From patchwork Tue Apr 9 16:11:01 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10891547 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AC8781708 for ; Tue, 9 Apr 2019 16:11:47 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8FA95286BF for ; Tue, 9 Apr 2019 16:11:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 83D57288C7; Tue, 9 Apr 2019 16:11:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D3EA6285A5 for ; Tue, 9 Apr 2019 16:11:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726612AbfDIQLp (ORCPT ); Tue, 9 Apr 2019 12:11:45 -0400 Received: from mail-wr1-f65.google.com ([209.85.221.65]:41403 "EHLO mail-wr1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726515AbfDIQLp (ORCPT ); Tue, 9 Apr 2019 12:11:45 -0400 Received: by mail-wr1-f65.google.com with SMTP id r4so21641915wrq.8 for ; Tue, 09 Apr 2019 09:11:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=XUl3DQ3ZGYAhJzG7PGKZHIvFALRhc78wPtePq4s0Ur8=; b=SzVIbzJjLhz95BK4PY+vkyd3yWqM8BW6oYFGiUR8o4Y6grJ2tCK66uE/gLH/w0WS9m yI3kdDycgO0IK/L4676MvKV+nV3X05/iYH8MHWwQyIURu7PVbv9dYcGNQs2DlIbaaJiF PYMLmLfVhoy8BHQwRVeDy6PzJHQEd7yYzceHXwMvf+5t0VWeNCN0XqLufJcX+6My3s95 zKfstCzqFsoiVU8bvZ7cfWKofW5v8MqxivvTjpW9ryplMejy1t1bP6gGPAOd11gT2Ylu dRPqSXTmgRiZ9h3YILCrCquduTEvgoBdrWRMs2CB0KB3VkpZRPUVfOEYY05HcJJnOVX0 Vc4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=XUl3DQ3ZGYAhJzG7PGKZHIvFALRhc78wPtePq4s0Ur8=; b=imQz/YdVsDDqYujHhnP/KWK2dmUCZo9sX8VfKusjwn3eXFjWvZlU6yCJxu0dSP515x eOE8UZb3eM/a2/9GZL8GzVqhMkLtiO23vY2o4Wcl1nbk6/lU5BL2JGtNpq56siFKpA77 Q1liKkST+bWUyIJyccI+6YA+YO/QaR9K3t9JhU/FrWI4uTVbb4ArCJOC4KlvSVNXzlTB 8oQ38R8eJrv38K6Gdh38FLYLmX8BIlWr37nbMj/AQwmLJKraBqbkdYy/BB2rp4uM7hp4 G3y4gEsxVmhHuTPY2zw3QCyvQUR8lrKLpS4x1slLzoiwbjGiXb/fLNZCKINhyAzPHxs/ pPow== X-Gm-Message-State: APjAAAX0omkgpxbVN1csK9irqUE2y9Og0T83TcUT/lk10+3TI7jkZrWM 2g4Do95UwRTxO4uDpAKl2sqojx48 X-Google-Smtp-Source: APXvYqzn3Et2GGT7G0HBdnjFjJ4/SjFrq8Pj+gcGlEaAY0fVs8G8V59lD42YOrWC7ZX0Yb8eVqHO3w== X-Received: by 2002:adf:ce8c:: with SMTP id r12mr4817543wrn.60.1554826302779; Tue, 09 Apr 2019 09:11:42 -0700 (PDT) Received: from ubuntu-N750HU.booking.pcln.com (access-114.38.rev.fr.colt.net. [213.41.38.114]) by smtp.gmail.com with ESMTPSA id i2sm4356012wmg.47.2019.04.09.09.11.42 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Apr 2019 09:11:42 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli , =?utf-8?q?SZEDER_G=C3=A1bor?= , Ramsay Jones Subject: [PATCH v5 01/16] t0410: remove pipes after git commands Date: Tue, 9 Apr 2019 18:11:01 +0200 Message-Id: <20190409161116.30256-2-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.750.g68c8ebb2ac In-Reply-To: <20190409161116.30256-1-chriscool@tuxfamily.org> References: <20190409161116.30256-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Let's not run a git command, especially one with "verify" in its name, upstream of a pipe, because the pipe will hide the git command's exit code. While at it, let's also avoid a useless `cat` command piping into `sed`. Helped-by: SZEDER Gábor Signed-off-by: Christian Couder --- t/t0410-partial-clone.sh | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh index 5bd892f2f7..3559313bd0 100755 --- a/t/t0410-partial-clone.sh +++ b/t/t0410-partial-clone.sh @@ -166,8 +166,9 @@ test_expect_success 'fetching of missing objects' ' # associated packfile contains the object ls repo/.git/objects/pack/pack-*.promisor >promisorlist && test_line_count = 1 promisorlist && - IDX=$(cat promisorlist | sed "s/promisor$/idx/") && - git verify-pack --verbose "$IDX" | grep "$HASH" + IDX=$(sed "s/promisor$/idx/" promisorlist) && + git verify-pack --verbose "$IDX" >out && + grep "$HASH" out ' test_expect_success 'fetching of missing objects works with ref-in-want enabled' ' @@ -514,8 +515,9 @@ test_expect_success 'fetching of missing objects from an HTTP server' ' # associated packfile contains the object ls repo/.git/objects/pack/pack-*.promisor >promisorlist && test_line_count = 1 promisorlist && - IDX=$(cat promisorlist | sed "s/promisor$/idx/") && - git verify-pack --verbose "$IDX" | grep "$HASH" + IDX=$(sed "s/promisor$/idx/" promisorlist) && + git verify-pack --verbose "$IDX" >out && + grep "$HASH" out ' test_done From patchwork Tue Apr 9 16:11:02 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10891551 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EEF1117E0 for ; Tue, 9 Apr 2019 16:11:49 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DAFCB285A5 for ; Tue, 9 Apr 2019 16:11:49 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CE8EA28892; Tue, 9 Apr 2019 16:11:49 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5DFD0285A5 for ; Tue, 9 Apr 2019 16:11:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726638AbfDIQLr (ORCPT ); Tue, 9 Apr 2019 12:11:47 -0400 Received: from mail-wm1-f65.google.com ([209.85.128.65]:55439 "EHLO mail-wm1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726523AbfDIQLq (ORCPT ); Tue, 9 Apr 2019 12:11:46 -0400 Received: by mail-wm1-f65.google.com with SMTP id o25so4223552wmf.5 for ; Tue, 09 Apr 2019 09:11:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=lEqc80MujqU7+JHs/Mji93LdoiHFAZ2tq0Zg1HbubbI=; b=gkK6X6xGMj0zvw7weqepR+hLzOxTzwyhv+ORaftdjJFkgH4i98vFF1S5INbGWEQyN3 BmtSJpmt/fbfcOIux523IDAXeypULGE+FmRcXEI9LlalpzYv+ZEGYCs2M/WnY6C/eM3j wPiIT8HeZ8qnv71Po5JRs4zcji19VCZstOXH8mLtUac5ZZGi1iq/NQlRoYWk3jyo/cAP MV/8A1cluxJ/xFY+XQXfUEe30OTzWBH9yScQgqpv+C3uM3A0Ua57c94mPyy5zcGOaWeR F7doejyAtsvT5fSXvEiuU7rHYxmJ2R66stlZbsyq1vJSmdXYxpglEzZY1qF4rxghgomp q2XQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=lEqc80MujqU7+JHs/Mji93LdoiHFAZ2tq0Zg1HbubbI=; b=MXRfbn0+30t7binEGC8ZEEy6qmp/GKyYPuonm93Dss4def4I2g3gpxEfDgFMSUExAX pZDi6JSWc7eCp372TDhhLCn6AoE8HN3Sk21gT63tH+tzXCIdAMLyL2yDw9CrxDXH+qVp RuipFH+x1TgUM5Xc8lBYYZ5X7JIfvrvTGwDn8Mz6ZVWW+6okLNEfw7mhAhGFfOEo1JwF xcYMdOAQheLfNU6C2lJVev5k1IMrvR5s7+oM/CZzTN8cWT5hMP33YSfMKlClw86IJ2vs TxD0zezYzKjb0znlla+51d8UBdGd0JdHEluL+ttVDkcDGSlhk7GQaVkBLy0eRrrW7UNH NgBQ== X-Gm-Message-State: APjAAAW0ezbjkU2E+XBN7E50joXI56yMofz4OkB61lXfbYY/+0Y3QpTH ioO8dczCr7+KZ6ks9/Yvr0/KmGkL X-Google-Smtp-Source: APXvYqwyZazLjARVmpPAbohUezUPtqZDdrbwKPoKRRPUySFyCjgkuTsiFMz/FymFrTkGhxnmo+zrwQ== X-Received: by 2002:a1c:f115:: with SMTP id p21mr21966969wmh.93.1554826303619; Tue, 09 Apr 2019 09:11:43 -0700 (PDT) Received: from ubuntu-N750HU.booking.pcln.com (access-114.38.rev.fr.colt.net. [213.41.38.114]) by smtp.gmail.com with ESMTPSA id i2sm4356012wmg.47.2019.04.09.09.11.42 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Apr 2019 09:11:42 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli , =?utf-8?q?SZEDER_G=C3=A1bor?= , Ramsay Jones , Christian Couder Subject: [PATCH v5 02/16] fetch-object: make functions return an error code Date: Tue, 9 Apr 2019 18:11:02 +0200 Message-Id: <20190409161116.30256-3-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.750.g68c8ebb2ac In-Reply-To: <20190409161116.30256-1-chriscool@tuxfamily.org> References: <20190409161116.30256-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Christian Couder The callers of the fetch_object() and fetch_objects() might be interested in knowing if these functions succeeded or not. Signed-off-by: Christian Couder Signed-off-by: Junio C Hamano --- fetch-object.c | 13 ++++++++----- fetch-object.h | 4 ++-- sha1-file.c | 4 ++-- 3 files changed, 12 insertions(+), 9 deletions(-) diff --git a/fetch-object.c b/fetch-object.c index 4266548800..eac4d448ef 100644 --- a/fetch-object.c +++ b/fetch-object.c @@ -5,11 +5,12 @@ #include "transport.h" #include "fetch-object.h" -static void fetch_refs(const char *remote_name, struct ref *ref) +static int fetch_refs(const char *remote_name, struct ref *ref) { struct remote *remote; struct transport *transport; int original_fetch_if_missing = fetch_if_missing; + int res; fetch_if_missing = 0; remote = remote_get(remote_name); @@ -19,12 +20,14 @@ static void fetch_refs(const char *remote_name, struct ref *ref) transport_set_option(transport, TRANS_OPT_FROM_PROMISOR, "1"); transport_set_option(transport, TRANS_OPT_NO_DEPENDENTS, "1"); - transport_fetch_refs(transport, ref); + res = transport_fetch_refs(transport, ref); fetch_if_missing = original_fetch_if_missing; + + return res; } -void fetch_objects(const char *remote_name, const struct object_id *oids, - int oid_nr) +int fetch_objects(const char *remote_name, const struct object_id *oids, + int oid_nr) { struct ref *ref = NULL; int i; @@ -36,5 +39,5 @@ void fetch_objects(const char *remote_name, const struct object_id *oids, new_ref->next = ref; ref = new_ref; } - fetch_refs(remote_name, ref); + return fetch_refs(remote_name, ref); } diff --git a/fetch-object.h b/fetch-object.h index d6444caa5a..7bcc7cadb0 100644 --- a/fetch-object.h +++ b/fetch-object.h @@ -3,7 +3,7 @@ struct object_id; -void fetch_objects(const char *remote_name, const struct object_id *oids, - int oid_nr); +int fetch_objects(const char *remote_name, const struct object_id *oids, + int oid_nr); #endif diff --git a/sha1-file.c b/sha1-file.c index ad02649124..83358737f3 100644 --- a/sha1-file.c +++ b/sha1-file.c @@ -1373,8 +1373,8 @@ int oid_object_info_extended(struct repository *r, const struct object_id *oid, !already_retried && r == the_repository && !(flags & OBJECT_INFO_FOR_PREFETCH)) { /* - * TODO Investigate having fetch_object() return - * TODO error/success and stopping the music here. + * TODO Investigate checking fetch_object() return + * TODO value and stopping on error here. * TODO Pass a repository struct through fetch_object, * such that arbitrary repositories work. */ From patchwork Tue Apr 9 16:11:03 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10891553 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 904FB17E0 for ; Tue, 9 Apr 2019 16:11:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7B95D285A5 for ; Tue, 9 Apr 2019 16:11:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6F20428892; Tue, 9 Apr 2019 16:11:51 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F34B3285A5 for ; Tue, 9 Apr 2019 16:11:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726653AbfDIQLt (ORCPT ); Tue, 9 Apr 2019 12:11:49 -0400 Received: from mail-wm1-f67.google.com ([209.85.128.67]:37233 "EHLO mail-wm1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726515AbfDIQLs (ORCPT ); Tue, 9 Apr 2019 12:11:48 -0400 Received: by mail-wm1-f67.google.com with SMTP id v14so4002820wmf.2 for ; Tue, 09 Apr 2019 09:11:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=CwuNyocFjBz/OD2XS+em3Lh8A8L82/w2MDgZ9CiAvL4=; b=fjiC/FkgN1SrpLYFNX65SIFfd21nm+zNDXWVvet94uJgCDqd/xdP9R7eGdwrme7KV5 6wVYXZk5nMkBtOMk2w3JBV0ykoSeCLOZpdyzj2VBH1kZO5tBwSoRXzp1pE8O2wrb/RAS RciJO9skytyphv7ZxKzulwLrTKG35cQN58hKffaKOkb45yQF2c9ZGjw+WGttUiQ0OSQY 5oPfnVlNU1v0u5R6q/ZjXui+6xlHnoBBaA5BWBSqKaPPTW+VRrLMdubJ5tbr8F9/YjGF CSSWfnzwdH1x/M1WlYRiuGyqd1FQ4fr0Y6KeOsJHQjSBblSCu2BJWSYqZeUHMTF2V7RO TtJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=CwuNyocFjBz/OD2XS+em3Lh8A8L82/w2MDgZ9CiAvL4=; b=I9dez/y63NP9fEuxrZTH4x+288aHpOX+5Xl+2lqQM8H+RnJNsryun8XclVNLxG1Oj8 jeJe3dtuX+ca7mgja2KnMXhn7BoGoG4Dsay6ShxZmNgw1T9brElGrDKC3h3yiSdEX+T7 aX/Ie41jUSoW3YAt4d9tNHOdDdIkP1dErVSbxysx8tjh8LZEPSraaNRlubDay3Wsm5Lp O/lh6nyj6HQjsPshGVnNdqxEhQFz0hM6LdnRvq01O3pnKLUhUXu0xhfJDjJhh9YWzKI4 sBXRq+kMLmndx6k7GUMnAn8yXmSi6p8f0EfkWp4+7Nnuz3M+Il/sF1P0S8Xe0zRR0yFQ pnBw== X-Gm-Message-State: APjAAAWSohxHu8QCjZ7wSEVX+5M84lEpZx72BOLNr4bPGCVUtOg2GGEF EjzuVKYI/Ol/9UVHx8XYBsG9fKHS X-Google-Smtp-Source: APXvYqzMWtpIv3mz8M+eM6t+ij+8k1c47zeDjWYiORjMjXxiRoe55GwKUARpU1nTNrah7Ld4mRimaw== X-Received: by 2002:a1c:486:: with SMTP id 128mr18549084wme.3.1554826305730; Tue, 09 Apr 2019 09:11:45 -0700 (PDT) Received: from ubuntu-N750HU.booking.pcln.com (access-114.38.rev.fr.colt.net. [213.41.38.114]) by smtp.gmail.com with ESMTPSA id i2sm4356012wmg.47.2019.04.09.09.11.43 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Apr 2019 09:11:43 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli , =?utf-8?q?SZEDER_G=C3=A1bor?= , Ramsay Jones , Christian Couder Subject: [PATCH v5 03/16] Add initial support for many promisor remotes Date: Tue, 9 Apr 2019 18:11:03 +0200 Message-Id: <20190409161116.30256-4-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.750.g68c8ebb2ac In-Reply-To: <20190409161116.30256-1-chriscool@tuxfamily.org> References: <20190409161116.30256-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Christian Couder The promisor-remote.{c,h} files will contain functions to manage many promisor remotes. We expect that there will not be a lot of promisor remotes, so it is ok to use a simple linked list to manage them. Helped-by: Jeff King Helped-by: SZEDER Gábor Signed-off-by: Christian Couder --- Makefile | 1 + promisor-remote.c | 92 +++++++++++++++++++++++++++++++++++++++++++++++ promisor-remote.h | 16 +++++++++ 3 files changed, 109 insertions(+) create mode 100644 promisor-remote.c create mode 100644 promisor-remote.h diff --git a/Makefile b/Makefile index 34fa9da39a..3523ae0517 100644 --- a/Makefile +++ b/Makefile @@ -954,6 +954,7 @@ LIB_OBJS += preload-index.o LIB_OBJS += pretty.o LIB_OBJS += prio-queue.o LIB_OBJS += progress.o +LIB_OBJS += promisor-remote.o LIB_OBJS += prompt.o LIB_OBJS += protocol.o LIB_OBJS += quote.o diff --git a/promisor-remote.c b/promisor-remote.c new file mode 100644 index 0000000000..c249b80e02 --- /dev/null +++ b/promisor-remote.c @@ -0,0 +1,92 @@ +#include "cache.h" +#include "promisor-remote.h" +#include "config.h" + +static struct promisor_remote *promisors; +static struct promisor_remote **promisors_tail = &promisors; + +static struct promisor_remote *promisor_remote_new(const char *remote_name) +{ + struct promisor_remote *r; + + if (*remote_name == '/') { + warning(_("promisor remote name cannot begin with '/': %s"), + remote_name); + return NULL; + } + + FLEX_ALLOC_STR(r, name, remote_name); + + *promisors_tail = r; + promisors_tail = &r->next; + + return r; +} + +static struct promisor_remote *promisor_remote_lookup(const char *remote_name, + struct promisor_remote **previous) +{ + struct promisor_remote *r, *p; + + for (p = NULL, r = promisors; r; p = r, r = r->next) + if (!strcmp(r->name, remote_name)) { + if (previous) + *previous = p; + return r; + } + + return NULL; +} + +static int promisor_remote_config(const char *var, const char *value, void *data) +{ + const char *name; + int namelen; + const char *subkey; + + if (parse_config_key(var, "remote", &name, &namelen, &subkey) < 0) + return 0; + + if (!strcmp(subkey, "promisor")) { + char *remote_name; + + if (!git_config_bool(var, value)) + return 0; + + remote_name = xmemdupz(name, namelen); + + if (!promisor_remote_lookup(remote_name, NULL)) + promisor_remote_new(remote_name); + + free(remote_name); + return 0; + } + + return 0; +} + +static void promisor_remote_init(void) +{ + static int initialized; + + if (initialized) + return; + initialized = 1; + + git_config(promisor_remote_config, NULL); +} + +struct promisor_remote *promisor_remote_find(const char *remote_name) +{ + promisor_remote_init(); + + if (!remote_name) + return promisors; + + return promisor_remote_lookup(remote_name, NULL); +} + +int has_promisor_remote(void) +{ + return !!promisor_remote_find(NULL); +} diff --git a/promisor-remote.h b/promisor-remote.h new file mode 100644 index 0000000000..01dcdf4dc7 --- /dev/null +++ b/promisor-remote.h @@ -0,0 +1,16 @@ +#ifndef PROMISOR_REMOTE_H +#define PROMISOR_REMOTE_H + +/* + * Promisor remote linked list + * Its information come from remote.XXX config entries. + */ +struct promisor_remote { + struct promisor_remote *next; + const char name[FLEX_ARRAY]; +}; + +extern struct promisor_remote *promisor_remote_find(const char *remote_name); +extern int has_promisor_remote(void); + +#endif /* PROMISOR_REMOTE_H */ From patchwork Tue Apr 9 16:11:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10891559 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3890517E0 for ; Tue, 9 Apr 2019 16:11:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 22EAE285A5 for ; Tue, 9 Apr 2019 16:11:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 16BE7286BF; Tue, 9 Apr 2019 16:11:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A26D9285A5 for ; Tue, 9 Apr 2019 16:11:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726672AbfDIQLx (ORCPT ); Tue, 9 Apr 2019 12:11:53 -0400 Received: from mail-wr1-f66.google.com ([209.85.221.66]:43428 "EHLO mail-wr1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726523AbfDIQLt (ORCPT ); Tue, 9 Apr 2019 12:11:49 -0400 Received: by mail-wr1-f66.google.com with SMTP id k17so21629455wrx.10 for ; Tue, 09 Apr 2019 09:11:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Q4yazDCGjft1r6zafJgy+9GM8LpQdf1guYns4hTQWHw=; b=fWkR48OKd9zvBZfghu9ch8AuQT0sfimWxr4tJpfpfelmY/5J59OW4lwAJ99yM1hSBQ 3iHaugG1ltJ6qadzFc87V0r/96+TRUsfTjDEd+xyUJbavSaMdmgl22rljIWjPXxqKahb KQGFr9kAtoVSb6HxrN5Ydc/CYg+heHAVG/bLc3ng9iapA4etaQaFkTj0nAuzr5snWs6y qxRE0oT0yc/PFfkOrMYBe+bb2SJlPWhbuRn0hMZfZB+3hucR47V15pOl+yv1geiPWNK9 bNOrqzFiN3zY7lPV+VKz9ZPQNNyZq0xEVrmOjKp1069lmMqqyHpw9Y7X3Ju12UVeFDV7 8xLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Q4yazDCGjft1r6zafJgy+9GM8LpQdf1guYns4hTQWHw=; b=bavS6PQUa7uPmezAv10qm/TdbNVYvWqWT6ZeX9zJHv2slNmXLmBHwMtQa1okHrhvUa QEml5Bu/iq85YJD7vaW82dWQe4YO+scz7ZyzH7ocL5SEo6IwFTDWJcAU1rX+SRMIZl4u OY9uUUN+zybiZjE9SNtsWGtlB9r7WmEM267/VXkenfX1VHkOZOwzeBLC5aDG91Z/i9h4 7o7fEth1oRRoDY6pKRX+s8h7MQyOmUqTl4sI9Y05p9y0Ly0PvzrLvyKbuCwtIm51aVcM C6attPXoUbX8E5mjG+LCvUbsrMaEB8OaBRIuCVuU0e/WvqmX4KSV8SjWeF/T+zayf+IY KWzw== X-Gm-Message-State: APjAAAWBShZW5BGFfeEWqTiWfiJP9eLJArIhZ7/RmkPat7A1bUR4elNI W01XKZZSjirkdO1y/KIJDk6u5WtJ X-Google-Smtp-Source: APXvYqy9LBIDHrDILAH4BYOhYIB5oZGJkJmQK9hZVJuvUw9dAcEJmcG1RCNYhQfk949ZiLIJyTkeqA== X-Received: by 2002:adf:9c91:: with SMTP id d17mr22526470wre.285.1554826307408; Tue, 09 Apr 2019 09:11:47 -0700 (PDT) Received: from ubuntu-N750HU.booking.pcln.com (access-114.38.rev.fr.colt.net. [213.41.38.114]) by smtp.gmail.com with ESMTPSA id i2sm4356012wmg.47.2019.04.09.09.11.45 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Apr 2019 09:11:45 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli , =?utf-8?q?SZEDER_G=C3=A1bor?= , Ramsay Jones , Christian Couder Subject: [PATCH v5 04/16] promisor-remote: implement promisor_remote_get_direct() Date: Tue, 9 Apr 2019 18:11:04 +0200 Message-Id: <20190409161116.30256-5-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.750.g68c8ebb2ac In-Reply-To: <20190409161116.30256-1-chriscool@tuxfamily.org> References: <20190409161116.30256-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Christian Couder This is implemented for now by calling fetch_objects(). It fetches from all the promisor remotes. Helped-by: Ramsay Jones Signed-off-by: Christian Couder --- promisor-remote.c | 66 +++++++++++++++++++++++++++++++++++++++++++++++ promisor-remote.h | 3 +++ 2 files changed, 69 insertions(+) diff --git a/promisor-remote.c b/promisor-remote.c index c249b80e02..289f1dd074 100644 --- a/promisor-remote.c +++ b/promisor-remote.c @@ -1,6 +1,8 @@ #include "cache.h" +#include "object-store.h" #include "promisor-remote.h" #include "config.h" +#include "fetch-object.h" static struct promisor_remote *promisors; static struct promisor_remote **promisors_tail = &promisors; @@ -90,3 +92,67 @@ int has_promisor_remote(void) { return !!promisor_remote_find(NULL); } + +static int remove_fetched_oids(struct object_id **oids, int oid_nr, int to_free) +{ + int i, missing_nr = 0; + int *missing = xcalloc(oid_nr, sizeof(*missing)); + struct object_id *old_oids = *oids; + struct object_id *new_oids; + int old_fetch_if_missing = fetch_if_missing; + + fetch_if_missing = 0; + + for (i = 0; i < oid_nr; i++) + if (oid_object_info_extended(the_repository, &old_oids[i], NULL, 0)) { + missing[i] = 1; + missing_nr++; + } + + fetch_if_missing = old_fetch_if_missing; + + if (missing_nr) { + int j = 0; + new_oids = xcalloc(missing_nr, sizeof(*new_oids)); + for (i = 0; i < oid_nr; i++) + if (missing[i]) + oidcpy(&new_oids[j++], &old_oids[i]); + *oids = new_oids; + if (to_free) + free(old_oids); + } + + free(missing); + + return missing_nr; +} + +int promisor_remote_get_direct(const struct object_id *oids, int oid_nr) +{ + struct promisor_remote *r; + struct object_id *missing_oids = (struct object_id *)oids; + int missing_nr = oid_nr; + int to_free = 0; + int res = -1; + + promisor_remote_init(); + + for (r = promisors; r; r = r->next) { + if (fetch_objects(r->name, missing_oids, missing_nr) < 0) { + if (missing_nr == 1) + continue; + missing_nr = remove_fetched_oids(&missing_oids, missing_nr, to_free); + if (missing_nr) { + to_free = 1; + continue; + } + } + res = 0; + break; + } + + if (to_free) + free(missing_oids); + + return res; +} diff --git a/promisor-remote.h b/promisor-remote.h index 01dcdf4dc7..92650cfd4c 100644 --- a/promisor-remote.h +++ b/promisor-remote.h @@ -1,6 +1,8 @@ #ifndef PROMISOR_REMOTE_H #define PROMISOR_REMOTE_H +struct object_id; + /* * Promisor remote linked list * Its information come from remote.XXX config entries. @@ -12,5 +14,6 @@ struct promisor_remote { extern struct promisor_remote *promisor_remote_find(const char *remote_name); extern int has_promisor_remote(void); +extern int promisor_remote_get_direct(const struct object_id *oids, int oid_nr); #endif /* PROMISOR_REMOTE_H */ From patchwork Tue Apr 9 16:11:05 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10891555 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C7A881708 for ; Tue, 9 Apr 2019 16:11:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B1CB0285A5 for ; Tue, 9 Apr 2019 16:11:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A5CD3286BF; Tue, 9 Apr 2019 16:11:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4FC03285A5 for ; Tue, 9 Apr 2019 16:11:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726677AbfDIQLx (ORCPT ); Tue, 9 Apr 2019 12:11:53 -0400 Received: from mail-wr1-f66.google.com ([209.85.221.66]:41421 "EHLO mail-wr1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726515AbfDIQLv (ORCPT ); Tue, 9 Apr 2019 12:11:51 -0400 Received: by mail-wr1-f66.google.com with SMTP id r4so21642325wrq.8 for ; Tue, 09 Apr 2019 09:11:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=N6vaBY91AoiJIMKSeCFmJ9mFQ4eiKK2CdaYB6w1obkQ=; b=cGr9X6WPzRh3WmhOMNRyPmHszSIMohABFBGb5ylADkwOwJQ9/EeGF2WQE1LStd0dk1 VBRam/ExVM3lcV4QB3arDADuQx/OY3nSaF3uCvzGhuC+7YbCpwlwtW3B0FmN7hdbRf77 Hvt5TUc7fB6anr5gvCQB6yB5ASUIsPAGUenKoSWU8Iy/qyJDrN6udV/fDB9dVZhymbJe 1mCi8cGUl7JZ9wvFlVtqM1IN3M1uuhkdP//ukgLKkRtUERusKqTjYdl8wfY8t6KzegIB iQH0Ny9pcDuniZjZrMLM40/nG5nS/VmULj5GTBKTf4xykuvWJFT4nQcX+SJG+oUwIWz3 QidA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=N6vaBY91AoiJIMKSeCFmJ9mFQ4eiKK2CdaYB6w1obkQ=; b=k1ACZ4sc8ZIRZg2wtgGEllcxaNN+OWF0/sDrnDMG2W+cXoXiB0N2dSIlMCLqcYhUQT 0bwJupI+f7YsHEz/R9NoH9UOEPYlj8NWllV7Zc6w09JKE7FQh9oegweBT6OgDtyqhIax Sqm9bthX38EjU2hbGUvRRbXFWSTdhnziMDX8DAYyF+iVB3H2QR2MYsYcSXICJtFuyIO6 XtXLBab6qX0q1XkATTBQZCqvh48gtyX7JUgE3rrSd3PnKXt74ExPbIDoUfVUCH+fQdEh phMjul0jfsTJXFkEsqyuyzID/0WsZNtf5X93LO6FW1gkjSxgc4a2/4qTPvRJwNaS/eWX H/qw== X-Gm-Message-State: APjAAAUaN+6/bnyuvFU/EQTXuAz5ZKdRIt1xvnzrsioWS+s1AUgBZ7h6 qXq2iSA3wEKp5GJclpj7kuq1gHWq X-Google-Smtp-Source: APXvYqxsEQUOerjaOMgBoLihm9XRg7sclRMBzttzbEpJVv9vGs5Avkkc55eicVUC5Aihcx5jWbUklQ== X-Received: by 2002:adf:c788:: with SMTP id l8mr23279612wrg.143.1554826309342; Tue, 09 Apr 2019 09:11:49 -0700 (PDT) Received: from ubuntu-N750HU.booking.pcln.com (access-114.38.rev.fr.colt.net. [213.41.38.114]) by smtp.gmail.com with ESMTPSA id i2sm4356012wmg.47.2019.04.09.09.11.47 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Apr 2019 09:11:47 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli , =?utf-8?q?SZEDER_G=C3=A1bor?= , Ramsay Jones , Christian Couder Subject: [PATCH v5 05/16] promisor-remote: add promisor_remote_reinit() Date: Tue, 9 Apr 2019 18:11:05 +0200 Message-Id: <20190409161116.30256-6-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.750.g68c8ebb2ac In-Reply-To: <20190409161116.30256-1-chriscool@tuxfamily.org> References: <20190409161116.30256-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Christian Couder We will need to reinitialize the promisor remote configuration as we will make some changes to it in a later commit. Signed-off-by: Christian Couder --- promisor-remote.c | 22 ++++++++++++++++++++-- promisor-remote.h | 1 + 2 files changed, 21 insertions(+), 2 deletions(-) diff --git a/promisor-remote.c b/promisor-remote.c index 289f1dd074..46271eb3e3 100644 --- a/promisor-remote.c +++ b/promisor-remote.c @@ -67,10 +67,10 @@ static int promisor_remote_config(const char *var, const char *value, void *data return 0; } +static int initialized; + static void promisor_remote_init(void) { - static int initialized; - if (initialized) return; initialized = 1; @@ -78,6 +78,24 @@ static void promisor_remote_init(void) git_config(promisor_remote_config, NULL); } +static void promisor_remote_clear(void) +{ + while (promisors) { + struct promisor_remote *r = promisors; + promisors = promisors->next; + free(r); + } + + promisors_tail = &promisors; +} + +void promisor_remote_reinit(void) +{ + initialized = 0; + promisor_remote_clear(); + promisor_remote_init(); +} + struct promisor_remote *promisor_remote_find(const char *remote_name) { promisor_remote_init(); diff --git a/promisor-remote.h b/promisor-remote.h index 92650cfd4c..ff69963907 100644 --- a/promisor-remote.h +++ b/promisor-remote.h @@ -12,6 +12,7 @@ struct promisor_remote { const char name[FLEX_ARRAY]; }; +extern void promisor_remote_reinit(void); extern struct promisor_remote *promisor_remote_find(const char *remote_name); extern int has_promisor_remote(void); extern int promisor_remote_get_direct(const struct object_id *oids, int oid_nr); From patchwork Tue Apr 9 16:11:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10891557 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9A2CF1708 for ; Tue, 9 Apr 2019 16:11:55 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 877E5285A5 for ; Tue, 9 Apr 2019 16:11:55 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7B0DA286BF; Tue, 9 Apr 2019 16:11:55 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1F2CE285A5 for ; Tue, 9 Apr 2019 16:11:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726685AbfDIQLy (ORCPT ); Tue, 9 Apr 2019 12:11:54 -0400 Received: from mail-wr1-f67.google.com ([209.85.221.67]:46897 "EHLO mail-wr1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726595AbfDIQLx (ORCPT ); Tue, 9 Apr 2019 12:11:53 -0400 Received: by mail-wr1-f67.google.com with SMTP id t17so21619028wrw.13 for ; Tue, 09 Apr 2019 09:11:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=L5QqTPdBPcJY8lzGpbv831cXvWC5SIOb/3P9ycLHAig=; b=bT2yUw2GtNLcbYjBiouLgYRua6Kk+HdCyMyNQIPenaIlRhTbyGT+8kXfJl3GGHmLPl iBBD3UyQRPiLmt9FcB2XvKYLA75QdasLXC+NCdQrg2b0WogXg4FhWBJUq5LOjt1PfOjR CUj6dzTIWjNdtWt07Kzff03MQvobEDemfA4GUPP/wGRUYkODZ11z3vXLSrwz7hriqHWM I+CWkkRSi/hcE3HOfBv5sJAAWRlS5ahXu+2NgAS1sfMloXLFp43Mmlo2j0PakTa/y/4F d+opoPq+tnVfyrWrEelNFibGOK116aBsjWAElNm/7pLk3UavvP6aa44dswy0ectnNFsh Db/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=L5QqTPdBPcJY8lzGpbv831cXvWC5SIOb/3P9ycLHAig=; b=I0x/5OED9OHPx9sWOVdjdMTfuKjAhIZDRCIhMy2i56SYQ7qIaWZWcnASkpE+ehTLo6 xMJqwW39orXp50rUxev2ytYfULSs4+S6DRlt9hdmZuo3JSV3i1RSe/1RNvUxP+DvR+JW v5xpP1NxhcQBdCA1E8UaGLVS4N43T08faVhFDA/e7wW974jsJXqi+MdH3bfY73f+Iq7J /az9x8MFLw/sOP4+ZT8A+AMvDK8+bimmgvEMFYFZZzeWxyGHY4ThO2YwAlnTrQrwjjfh fO0dnwCrsTVg21CzGkxmgiBKncoaXLEIZYgnsTkjRjPihSuyT+ZNuH5GaIqkpfaMTX6D Q0Lg== X-Gm-Message-State: APjAAAUBq4foERy/KHyw3nn3E/Mvb77UWoqjZNDcDQk07bKkI6+MVjBA Mp7fxpieRzD6vO/yMuu5BMDYylgI X-Google-Smtp-Source: APXvYqxiQ4hwdcXWHV9utx8HPO+Z7LmbQk39tbcRSBt+34V+Qgoj+lvGHBXeBhN3GnCNsbhTWUUgnw== X-Received: by 2002:adf:828f:: with SMTP id 15mr22858326wrc.17.1554826310959; Tue, 09 Apr 2019 09:11:50 -0700 (PDT) Received: from ubuntu-N750HU.booking.pcln.com (access-114.38.rev.fr.colt.net. [213.41.38.114]) by smtp.gmail.com with ESMTPSA id i2sm4356012wmg.47.2019.04.09.09.11.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Apr 2019 09:11:50 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli , =?utf-8?q?SZEDER_G=C3=A1bor?= , Ramsay Jones Subject: [PATCH v5 06/16] promisor-remote: use repository_format_partial_clone Date: Tue, 9 Apr 2019 18:11:06 +0200 Message-Id: <20190409161116.30256-7-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.750.g68c8ebb2ac In-Reply-To: <20190409161116.30256-1-chriscool@tuxfamily.org> References: <20190409161116.30256-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP A remote specified using the extensions.partialClone config option should be considered a promisor remote too. For simplicity and to make things predictable, this promisor remote should be either always the last one we try to get objects from, or the first one. So it should always be either at the end of the promisor remote list, or at its start. We decided to make it the last one we try, because it is likely that someone using many promisor remotes is doing so because the other promisor remotes are better for some reason (maybe they are closer or faster for some kind of objects) than the origin, and the origin is likely to be the remote specified by extensions.partialClone. This justification is not very strong, but one choice had to be made, and anyway the long term plan should be to make the order somehow fully configurable. Signed-off-by: Christian Couder --- promisor-remote.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/promisor-remote.c b/promisor-remote.c index 46271eb3e3..737689d044 100644 --- a/promisor-remote.c +++ b/promisor-remote.c @@ -40,6 +40,18 @@ static struct promisor_remote *promisor_remote_lookup(const char *remote_name, return NULL; } +static void promisor_remote_move_to_tail(struct promisor_remote *r, + struct promisor_remote *previous) +{ + if (previous) + previous->next = r->next; + else + promisors = r->next ? r->next : r; + r->next = NULL; + *promisors_tail = r; + promisors_tail = &r->next; +} + static int promisor_remote_config(const char *var, const char *value, void *data) { const char *name; @@ -76,6 +88,17 @@ static void promisor_remote_init(void) initialized = 1; git_config(promisor_remote_config, NULL); + + if (repository_format_partial_clone) { + struct promisor_remote *o, *previous; + + o = promisor_remote_lookup(repository_format_partial_clone, + &previous); + if (o) + promisor_remote_move_to_tail(o, previous); + else + promisor_remote_new(repository_format_partial_clone); + } } static void promisor_remote_clear(void) From patchwork Tue Apr 9 16:11:07 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10891565 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BBB7B1805 for ; Tue, 9 Apr 2019 16:11:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A5B67285A5 for ; Tue, 9 Apr 2019 16:11:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 99AF728892; Tue, 9 Apr 2019 16:11:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9752F285A5 for ; Tue, 9 Apr 2019 16:11:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726706AbfDIQL5 (ORCPT ); Tue, 9 Apr 2019 12:11:57 -0400 Received: from mail-wr1-f65.google.com ([209.85.221.65]:36356 "EHLO mail-wr1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726515AbfDIQLz (ORCPT ); Tue, 9 Apr 2019 12:11:55 -0400 Received: by mail-wr1-f65.google.com with SMTP id y13so21664513wrd.3 for ; Tue, 09 Apr 2019 09:11:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=6dY8p3PC3FPmzIiYwR+SfHDaGkLhxN1lcLnqyC+viH0=; b=ZlKdiFCNeKulp2U3zjGzhcstQW58bKT5U0QLzphNa5zHKLUj4Bmw3L/qbyt7HoEJrV RYbkAR9Wy5+/Y595OJwyN4t2oCScuDn2Fg78Bhb0M8853w+/quca96qFGKWCG8usA7dN lx+MLPgEFPAsoalypVsJgMYBUHYn8eyvdRO6dmMi1/OBkpqDPSs6JiwrUssrKRfMDd1v FMm9xmza+8psP+6hDN60Vvth/aUZ5mv4tcVTn0NfAZI7GNyXAUXe/1qmtXo+mCb3J/j8 pWz5Ch6XeJTOcoYTXaFS679Bgmw72rlBswiZH0ewApeGJCicJM4aRT9Lcn2Ofow2C2eb HCJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6dY8p3PC3FPmzIiYwR+SfHDaGkLhxN1lcLnqyC+viH0=; b=uM4S0oogtwkZbRi1Jn7E1ZCAxBOdoRWv68/iRINsm5gTpdV7ugklfhFIfY2/zUMG5N mywOFtTEOlKBachocUkY/mlFevpoOt8doUGRatqMfPL0i7iV7AmaXQo4HfwZBG59bF8P Ke8cC0r1vzRAe7vHWn4dBt2NGVIrp5qixYkwwmnR380VZ/SJrC9dBFhEWVAJKez4ldHk N8dhtpVsXsO4etIOns1SE672PgfFQJmq5HkD3Exl0PcmJNQr8Pp+YdyLaiIdqSl1oDxL nk4XNQaGWniYEk47+stTMqjih18BDze9QetfGTllgnLRNdg/RAcd3aBKXpEh1+0VIdsP 71mw== X-Gm-Message-State: APjAAAXTb+r5Ue0Zs88qRDNtBLQ+XD4nPYpSANWZ6kl5yoRSm28rWXf5 AEelhGYmlcWM7AjfoHPAO7WUpFOQ X-Google-Smtp-Source: APXvYqwW7NY+qsuGAKxQg8Y1+s5N8fIWxjVd470lfkwqVGIWYLByz+105Tb5H/p7KprMQaX/El+TpQ== X-Received: by 2002:a5d:68cd:: with SMTP id p13mr22731891wrw.22.1554826312657; Tue, 09 Apr 2019 09:11:52 -0700 (PDT) Received: from ubuntu-N750HU.booking.pcln.com (access-114.38.rev.fr.colt.net. [213.41.38.114]) by smtp.gmail.com with ESMTPSA id i2sm4356012wmg.47.2019.04.09.09.11.50 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Apr 2019 09:11:51 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli , =?utf-8?q?SZEDER_G=C3=A1bor?= , Ramsay Jones Subject: [PATCH v5 07/16] Use promisor_remote_get_direct() and has_promisor_remote() Date: Tue, 9 Apr 2019 18:11:07 +0200 Message-Id: <20190409161116.30256-8-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.750.g68c8ebb2ac In-Reply-To: <20190409161116.30256-1-chriscool@tuxfamily.org> References: <20190409161116.30256-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Instead of using the repository_format_partial_clone global and fetch_objects() directly, let's use has_promisor_remote() and promisor_remote_get_direct(). This way all the configured promisor remotes will be taken into account, not only the one specified by extensions.partialClone. Also when cloning or fetching using a partial clone filter, remote.origin.promisor will be set to "true" instead of setting extensions.partialClone to "origin". This makes it possible to use many promisor remote just by fetching from them. Signed-off-by: Christian Couder --- builtin/cat-file.c | 5 +++-- builtin/fetch.c | 11 ++++++----- builtin/gc.c | 3 ++- builtin/repack.c | 3 ++- cache-tree.c | 3 ++- connected.c | 3 ++- list-objects-filter-options.c | 28 +++++++++++++++------------- packfile.c | 3 ++- sha1-file.c | 15 ++++++++------- t/t5601-clone.sh | 2 +- t/t5616-partial-clone.sh | 2 +- unpack-trees.c | 7 +++---- 12 files changed, 47 insertions(+), 38 deletions(-) diff --git a/builtin/cat-file.c b/builtin/cat-file.c index 0f092382e1..85ae10bf0b 100644 --- a/builtin/cat-file.c +++ b/builtin/cat-file.c @@ -15,6 +15,7 @@ #include "sha1-array.h" #include "packfile.h" #include "object-store.h" +#include "promisor-remote.h" struct batch_options { int enabled; @@ -523,8 +524,8 @@ static int batch_objects(struct batch_options *opt) if (opt->all_objects) { struct object_cb_data cb; - if (repository_format_partial_clone) - warning("This repository has extensions.partialClone set. Some objects may not be loaded."); + if (has_promisor_remote()) + warning("This repository uses promisor remotes. Some objects may not be loaded."); cb.opt = opt; cb.expand = &data; diff --git a/builtin/fetch.c b/builtin/fetch.c index 4ba63d5ac6..f74bd78144 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -23,6 +23,7 @@ #include "packfile.h" #include "list-objects-filter-options.h" #include "commit-reach.h" +#include "promisor-remote.h" static const char * const builtin_fetch_usage[] = { N_("git fetch [] [ [...]]"), @@ -1460,7 +1461,7 @@ static inline void fetch_one_setup_partial(struct remote *remote) * If no prior partial clone/fetch and the current fetch DID NOT * request a partial-fetch, do a normal fetch. */ - if (!repository_format_partial_clone && !filter_options.choice) + if (!has_promisor_remote() && !filter_options.choice) return; /* @@ -1468,7 +1469,7 @@ static inline void fetch_one_setup_partial(struct remote *remote) * on this repo and remember the given filter-spec as the default * for subsequent fetches to this remote. */ - if (!repository_format_partial_clone && filter_options.choice) { + if (!has_promisor_remote() && filter_options.choice) { partial_clone_register(remote->name, &filter_options); return; } @@ -1477,7 +1478,7 @@ static inline void fetch_one_setup_partial(struct remote *remote) * We are currently limited to only ONE promisor remote and only * allow partial-fetches from the promisor remote. */ - if (strcmp(remote->name, repository_format_partial_clone)) { + if (!promisor_remote_find(remote->name)) { if (filter_options.choice) die(_("--filter can only be used with the remote " "configured in extensions.partialClone")); @@ -1611,7 +1612,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix) if (depth || deepen_since || deepen_not.nr) deepen = 1; - if (filter_options.choice && !repository_format_partial_clone) + if (filter_options.choice && !has_promisor_remote()) die("--filter can only be used when extensions.partialClone is set"); if (all) { @@ -1645,7 +1646,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix) } if (remote) { - if (filter_options.choice || repository_format_partial_clone) + if (filter_options.choice || has_promisor_remote()) fetch_one_setup_partial(remote); result = fetch_one(remote, argc, argv, prune_tags_ok); } else { diff --git a/builtin/gc.c b/builtin/gc.c index 8943bcc300..824a8832b5 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -27,6 +27,7 @@ #include "pack-objects.h" #include "blob.h" #include "tree.h" +#include "promisor-remote.h" #define FAILED_RUN "failed to run %s" @@ -661,7 +662,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix) argv_array_push(&prune, prune_expire); if (quiet) argv_array_push(&prune, "--no-progress"); - if (repository_format_partial_clone) + if (has_promisor_remote()) argv_array_push(&prune, "--exclude-promisor-objects"); if (run_command_v_opt(prune.argv, RUN_GIT_CMD)) diff --git a/builtin/repack.c b/builtin/repack.c index 3ea0583d02..57568ce9a8 100644 --- a/builtin/repack.c +++ b/builtin/repack.c @@ -11,6 +11,7 @@ #include "midx.h" #include "packfile.h" #include "object-store.h" +#include "promisor-remote.h" static int delta_base_offset = 1; static int pack_kept_objects = -1; @@ -359,7 +360,7 @@ int cmd_repack(int argc, const char **argv, const char *prefix) argv_array_push(&cmd.args, "--all"); argv_array_push(&cmd.args, "--reflog"); argv_array_push(&cmd.args, "--indexed-objects"); - if (repository_format_partial_clone) + if (has_promisor_remote()) argv_array_push(&cmd.args, "--exclude-promisor-objects"); if (write_bitmaps) argv_array_push(&cmd.args, "--write-bitmap-index"); diff --git a/cache-tree.c b/cache-tree.c index b13bfaf71e..64c285a746 100644 --- a/cache-tree.c +++ b/cache-tree.c @@ -5,6 +5,7 @@ #include "cache-tree.h" #include "object-store.h" #include "replace-object.h" +#include "promisor-remote.h" #ifndef DEBUG #define DEBUG 0 @@ -357,7 +358,7 @@ static int update_one(struct cache_tree *it, } ce_missing_ok = mode == S_IFGITLINK || missing_ok || - (repository_format_partial_clone && + (has_promisor_remote() && ce_skip_worktree(ce)); if (is_null_oid(oid) || (!ce_missing_ok && !has_object_file(oid))) { diff --git a/connected.c b/connected.c index 1bba888eff..0eaaedee6a 100644 --- a/connected.c +++ b/connected.c @@ -4,6 +4,7 @@ #include "connected.h" #include "transport.h" #include "packfile.h" +#include "promisor-remote.h" /* * If we feed all the commits we want to verify to this command @@ -56,7 +57,7 @@ int check_connected(oid_iterate_fn fn, void *cb_data, argv_array_push(&rev_list.args,"rev-list"); argv_array_push(&rev_list.args, "--objects"); argv_array_push(&rev_list.args, "--stdin"); - if (repository_format_partial_clone) + if (has_promisor_remote()) argv_array_push(&rev_list.args, "--exclude-promisor-objects"); if (!opt->is_deepening_fetch) { argv_array_push(&rev_list.args, "--not"); diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c index c0036f7378..f41a831fce 100644 --- a/list-objects-filter-options.c +++ b/list-objects-filter-options.c @@ -6,6 +6,7 @@ #include "list-objects.h" #include "list-objects-filter.h" #include "list-objects-filter-options.h" +#include "promisor-remote.h" /* * Parse value of the argument to the "filter" keyword. @@ -144,30 +145,31 @@ void partial_clone_register( const char *remote, const struct list_objects_filter_options *filter_options) { - /* - * Record the name of the partial clone remote in the - * config and in the global variable -- the latter is - * used throughout to indicate that partial clone is - * enabled and to expect missing objects. - */ - if (repository_format_partial_clone && - *repository_format_partial_clone && - strcmp(remote, repository_format_partial_clone)) - die(_("cannot change partial clone promisor remote")); + char *cfg_name; - git_config_set("core.repositoryformatversion", "1"); - git_config_set("extensions.partialclone", remote); + /* Check if it is already registered */ + if (!promisor_remote_find(remote)) { + git_config_set("core.repositoryformatversion", "1"); - repository_format_partial_clone = xstrdup(remote); + /* Add promisor config for the remote */ + cfg_name = xstrfmt("remote.%s.promisor", remote); + git_config_set(cfg_name, "true"); + free(cfg_name); + } /* * Record the initial filter-spec in the config as * the default for subsequent fetches from this remote. + * + * TODO: record it into remote..partialclonefilter */ core_partial_clone_filter_default = xstrdup(filter_options->filter_spec); git_config_set("core.partialclonefilter", core_partial_clone_filter_default); + + /* Make sure the config info are reset */ + promisor_remote_reinit(); } void partial_clone_get_default_filter_spec( diff --git a/packfile.c b/packfile.c index 498498b1e3..fa5a9bb136 100644 --- a/packfile.c +++ b/packfile.c @@ -16,6 +16,7 @@ #include "tree.h" #include "object-store.h" #include "midx.h" +#include "promisor-remote.h" char *odb_pack_name(struct strbuf *buf, const unsigned char *sha1, @@ -2147,7 +2148,7 @@ int is_promisor_object(const struct object_id *oid) static int promisor_objects_prepared; if (!promisor_objects_prepared) { - if (repository_format_partial_clone) { + if (has_promisor_remote()) { for_each_packed_object(add_promisor_object, &promisor_objects, FOR_EACH_OBJECT_PROMISOR_ONLY); diff --git a/sha1-file.c b/sha1-file.c index 83358737f3..e7e037b6ce 100644 --- a/sha1-file.c +++ b/sha1-file.c @@ -30,8 +30,8 @@ #include "mergesort.h" #include "quote.h" #include "packfile.h" -#include "fetch-object.h" #include "object-store.h" +#include "promisor-remote.h" /* The maximum size for an object header. */ #define MAX_HEADER_LEN 32 @@ -1369,16 +1369,17 @@ int oid_object_info_extended(struct repository *r, const struct object_id *oid, } /* Check if it is a missing object */ - if (fetch_if_missing && repository_format_partial_clone && + if (fetch_if_missing && has_promisor_remote() && !already_retried && r == the_repository && !(flags & OBJECT_INFO_FOR_PREFETCH)) { /* - * TODO Investigate checking fetch_object() return - * TODO value and stopping on error here. - * TODO Pass a repository struct through fetch_object, - * such that arbitrary repositories work. + * TODO Investigate checking promisor_remote_get_direct() + * TODO return value and stopping on error here. + * TODO Pass a repository struct through + * promisor_remote_get_direct(), such that arbitrary + * repositories work. */ - fetch_objects(repository_format_partial_clone, real, 1); + promisor_remote_get_direct(real, 1); already_retried = 1; continue; } diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh index 23854cab26..15720847b0 100755 --- a/t/t5601-clone.sh +++ b/t/t5601-clone.sh @@ -653,7 +653,7 @@ partial_clone () { git -C client fsck && # Ensure that unneeded blobs are not inadvertently fetched. - test_config -C client extensions.partialclone "not a remote" && + test_config -C client remote.origin.promisor "false" && test_must_fail git -C client cat-file -e "$HASH1" && # But this blob was fetched, because clone performs an initial checkout diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh index 9a8f9886b3..c9e5f14165 100755 --- a/t/t5616-partial-clone.sh +++ b/t/t5616-partial-clone.sh @@ -42,7 +42,7 @@ test_expect_success 'do partial clone 1' ' test_cmp expect_1.oids observed.oids && test "$(git -C pc1 config --local core.repositoryformatversion)" = "1" && - test "$(git -C pc1 config --local extensions.partialclone)" = "origin" && + test "$(git -C pc1 config --local remote.origin.promisor)" = "true" && test "$(git -C pc1 config --local core.partialclonefilter)" = "blob:none" ' diff --git a/unpack-trees.c b/unpack-trees.c index 22985aa4a3..d86cd922e0 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -16,7 +16,7 @@ #include "submodule-config.h" #include "fsmonitor.h" #include "object-store.h" -#include "fetch-object.h" +#include "promisor-remote.h" /* * Error messages expected by scripts out of plumbing commands such as @@ -398,7 +398,7 @@ static int check_updates(struct unpack_trees_options *o) load_gitmodules_file(index, &state); enable_delayed_checkout(&state); - if (repository_format_partial_clone && o->update && !o->dry_run) { + if (has_promisor_remote() && o->update && !o->dry_run) { /* * Prefetch the objects that are to be checked out in the loop * below. @@ -417,8 +417,7 @@ static int check_updates(struct unpack_trees_options *o) oid_array_append(&to_fetch, &ce->oid); } if (to_fetch.nr) - fetch_objects(repository_format_partial_clone, - to_fetch.oid, to_fetch.nr); + promisor_remote_get_direct(to_fetch.oid, to_fetch.nr); oid_array_clear(&to_fetch); } for (i = 0; i < index->cache_nr; i++) { From patchwork Tue Apr 9 16:11:08 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10891569 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9F7F817E0 for ; Tue, 9 Apr 2019 16:12:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8C1D4286BF for ; Tue, 9 Apr 2019 16:12:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7E674285A5; Tue, 9 Apr 2019 16:12:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 27CDC285A5 for ; Tue, 9 Apr 2019 16:12:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726697AbfDIQL5 (ORCPT ); Tue, 9 Apr 2019 12:11:57 -0400 Received: from mail-wr1-f65.google.com ([209.85.221.65]:40681 "EHLO mail-wr1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726595AbfDIQL4 (ORCPT ); Tue, 9 Apr 2019 12:11:56 -0400 Received: by mail-wr1-f65.google.com with SMTP id h4so21651934wre.7 for ; Tue, 09 Apr 2019 09:11:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=JaLcNLgH1FGvrESE5cWo/Yv1cI7dR1PzTsVxV3BN05A=; b=O0W287gaWtjeEWgGdg37MVgq19h/huaOh0UIGeW8mkvP2dg/iYqYH9nTyyDPXRWKB4 G/xad30WBfhtrTAQOKBROBcaa44G6PVJICoxmD/+RBRvMJRcNd9CaBnU2gcxt+e8otiT aYoglHxS60HBsbdK7sEa8+wo1KviOuxg00k0uGCFSSO0YAiIH1ZZxMtwOeihvtacS+Ak lulkq3lv5mi1WWxwabIx7F/4pae3Ba67CuAc1fytYWNvOY4sqpHt/Y1ZXqTrIN1/+V8p Egv1RzpWJPF3r7jSsQQKi52I5wKDhzeuM8Mza+Qn2ebCiyzYa/JjUARP5+UH/OCgq/3R M9Gg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JaLcNLgH1FGvrESE5cWo/Yv1cI7dR1PzTsVxV3BN05A=; b=LQ8IQLkSHqTI3dQkXzFJoortmNzX0K/7T6zO2dqPBhgc+qM1bFf5lFbWfBoeKeqRl2 FJvxVz4EBZ6VE7xEWB53TGYYvmSpHKRBtfpIC0faSXHEfCIYlsPIPKGdPEZJxfKUrIPe 1IWdBozP46SDpn0JMrgA1qHlbI0CdtO+PjCl74mKBgHPSwQYZawLlNsUJ02ctNumS2VA D3t5LiMAuwjx1sMDxdgRgNtUwvNr3Vwfd/KohvNBdQjNOlCfkaTNOWyd/S7KaO9K+N4x swLB6G6UQ/VaQxEpH2LEqpvC66wnRf5enhBq5yM/dDHaVbRPvwhgNmnEab6sty8A8iq/ 5IHA== X-Gm-Message-State: APjAAAUQSdEUaNhQyn5b05I+jDho6Buq0DzOpk38q5nxjdvcmoHdlNCn 2Xuq0w9pMZOnQWYHHOQi4kb8XqkA X-Google-Smtp-Source: APXvYqzDhU7z2xVtxuy9swj0BOrSXJ5t7kJC6KwdBVYh4ho2fBmcHgTv814KnTBg/OI7AhODU1LuBQ== X-Received: by 2002:adf:ea0b:: with SMTP id q11mr5369803wrm.233.1554826314195; Tue, 09 Apr 2019 09:11:54 -0700 (PDT) Received: from ubuntu-N750HU.booking.pcln.com (access-114.38.rev.fr.colt.net. [213.41.38.114]) by smtp.gmail.com with ESMTPSA id i2sm4356012wmg.47.2019.04.09.09.11.52 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Apr 2019 09:11:53 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli , =?utf-8?q?SZEDER_G=C3=A1bor?= , Ramsay Jones Subject: [PATCH v5 08/16] diff: use promisor-remote.h instead of fetch-object.h Date: Tue, 9 Apr 2019 18:11:08 +0200 Message-Id: <20190409161116.30256-9-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.750.g68c8ebb2ac In-Reply-To: <20190409161116.30256-1-chriscool@tuxfamily.org> References: <20190409161116.30256-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The repository_format_partial_clone global and fetch_objects() should not be used anymore when there can be more than one promisor remote. Instead let's use has_promisor_remote() and promisor_remote_get_direct() from "promisor-remote.h". This way all the configured promisor remotes will be taken into account, not only the one specified by extensions.partialClone. Signed-off-by: Christian Couder --- diff.c | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/diff.c b/diff.c index 4d3cf83a27..02864171fd 100644 --- a/diff.c +++ b/diff.c @@ -25,7 +25,7 @@ #include "packfile.h" #include "parse-options.h" #include "help.h" -#include "fetch-object.h" +#include "promisor-remote.h" #ifdef NO_FAST_WORKING_DIRECTORY #define FAST_WORKING_DIRECTORY 0 @@ -6490,8 +6490,7 @@ static void add_if_missing(struct repository *r, void diffcore_std(struct diff_options *options) { - if (options->repo == the_repository && - repository_format_partial_clone) { + if (options->repo == the_repository && has_promisor_remote()) { /* * Prefetch the diff pairs that are about to be flushed. */ @@ -6508,8 +6507,7 @@ void diffcore_std(struct diff_options *options) /* * NEEDSWORK: Consider deduplicating the OIDs sent. */ - fetch_objects(repository_format_partial_clone, - to_fetch.oid, to_fetch.nr); + promisor_remote_get_direct(to_fetch.oid, to_fetch.nr); oid_array_clear(&to_fetch); } From patchwork Tue Apr 9 16:11:09 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10891571 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6C9371708 for ; Tue, 9 Apr 2019 16:12:02 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 59217286BF for ; Tue, 9 Apr 2019 16:12:02 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4CE5B288D1; Tue, 9 Apr 2019 16:12:02 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A82A0286BF for ; Tue, 9 Apr 2019 16:12:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726712AbfDIQMA (ORCPT ); Tue, 9 Apr 2019 12:12:00 -0400 Received: from mail-wr1-f65.google.com ([209.85.221.65]:36362 "EHLO mail-wr1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726689AbfDIQL6 (ORCPT ); Tue, 9 Apr 2019 12:11:58 -0400 Received: by mail-wr1-f65.google.com with SMTP id y13so21664692wrd.3 for ; Tue, 09 Apr 2019 09:11:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=b7kJteAebVRyJIp9QFI9Vj/9kYAzKjTFmNSjo80SK4s=; b=l6gmz23nLdhmAn25y2zOT8RLh8UO/DfbEYR7Fjm7a0xgtm9TZm6c5AEPxQoR5+2Rvj PRjKpS1RgHWG8TJy86XbE9io42NWea1NuN3MI6SiBjbtp/e7HSkzuU4ZpQTO7Uiv13Fa cJaMNi+Agj5irUhX0r4ZiW0m5HTdTw4Q09vjaeYau0C7ON7AVij+puoV9wJ7JHG6VDsr y5z4C5dahm8I1gl9JMJFukiLzca4WJ4GPVngh0/bnQ4zL/CiUr9WT2Eju/+OTtJ6LAcu fJXl+krFqUSD/PofS7k5LFaC33F2hkyvIjGidquE/lsxbjShWmdci3EJPh8HBTIvRf93 bpqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=b7kJteAebVRyJIp9QFI9Vj/9kYAzKjTFmNSjo80SK4s=; b=XGG8IGKek2b9+NaitNzkLSj5Byg4vXSj4i+xb84qiIerL5+RhZdn3sC9Uxf9jwuYXW DCFEEdW+b2FGLKC8g9ayR7JUBehLzsuue2jvUvr8OS0l5JLlEugzVwuUL6xzCyLhSFr+ 8VXNWwCrh0sPsk13X0EEF17mXpWxproBkb14N908rt4FWQ7qTjypLanbqOTf51xtU9bD rNUvTR0blE69TBEtTQ3axJrcMKUTYzM7P1Ke/6BFa6GXToQq+2LptIusZwhPoBdhxJJw PlbkHT034ZGaOejeAJ1sJUUOhjkf5ZvlWFDYKPJoOTEx5MRVgCRSOv9dpeufXxjxoqSL 6qWA== X-Gm-Message-State: APjAAAXIfST43pd3s1LZ50R/yHGlI+DqRN+aZ7vnWGPTlZKEoDSaTBxu ZITgNewLTwz+fVlb2U6nKzu1CGAG X-Google-Smtp-Source: APXvYqygIFQ+ckuLyUGVVA9JbqPGNtP6T8/Sg0tQ+KswSfhIUWM3cDTkJ1QoGj7sFvp2T18X+qOT7A== X-Received: by 2002:adf:e510:: with SMTP id j16mr24485764wrm.259.1554826315645; Tue, 09 Apr 2019 09:11:55 -0700 (PDT) Received: from ubuntu-N750HU.booking.pcln.com (access-114.38.rev.fr.colt.net. [213.41.38.114]) by smtp.gmail.com with ESMTPSA id i2sm4356012wmg.47.2019.04.09.09.11.54 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Apr 2019 09:11:54 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli , =?utf-8?q?SZEDER_G=C3=A1bor?= , Ramsay Jones Subject: [PATCH v5 09/16] promisor-remote: parse remote.*.partialclonefilter Date: Tue, 9 Apr 2019 18:11:09 +0200 Message-Id: <20190409161116.30256-10-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.750.g68c8ebb2ac In-Reply-To: <20190409161116.30256-1-chriscool@tuxfamily.org> References: <20190409161116.30256-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This makes it possible to specify a different partial clone filter for each promisor remote. Signed-off-by: Christian Couder --- builtin/fetch.c | 2 +- list-objects-filter-options.c | 27 +++++++++++++++------------ list-objects-filter-options.h | 3 ++- promisor-remote.c | 15 +++++++++++++++ promisor-remote.h | 5 ++++- t/t0410-partial-clone.sh | 2 +- t/t5601-clone.sh | 1 + t/t5616-partial-clone.sh | 2 +- 8 files changed, 40 insertions(+), 17 deletions(-) diff --git a/builtin/fetch.c b/builtin/fetch.c index f74bd78144..13d8133130 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -1491,7 +1491,7 @@ static inline void fetch_one_setup_partial(struct remote *remote) * the config. */ if (!filter_options.choice) - partial_clone_get_default_filter_spec(&filter_options); + partial_clone_get_default_filter_spec(&filter_options, remote->name); return; } diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c index f41a831fce..02f48b7c40 100644 --- a/list-objects-filter-options.c +++ b/list-objects-filter-options.c @@ -30,6 +30,9 @@ static int gently_parse_list_objects_filter( { const char *v0; + if (!arg) + return 0; + if (filter_options->choice) { if (errbuf) { strbuf_addstr( @@ -146,6 +149,7 @@ void partial_clone_register( const struct list_objects_filter_options *filter_options) { char *cfg_name; + char *filter_name; /* Check if it is already registered */ if (!promisor_remote_find(remote)) { @@ -160,27 +164,26 @@ void partial_clone_register( /* * Record the initial filter-spec in the config as * the default for subsequent fetches from this remote. - * - * TODO: record it into remote..partialclonefilter */ - core_partial_clone_filter_default = - xstrdup(filter_options->filter_spec); - git_config_set("core.partialclonefilter", - core_partial_clone_filter_default); + filter_name = xstrfmt("remote.%s.partialclonefilter", remote); + git_config_set(filter_name, filter_options->filter_spec); + free(filter_name); /* Make sure the config info are reset */ promisor_remote_reinit(); } void partial_clone_get_default_filter_spec( - struct list_objects_filter_options *filter_options) + struct list_objects_filter_options *filter_options, + const char *remote) { + struct promisor_remote *promisor = promisor_remote_find(remote); + /* * Parse default value, but silently ignore it if it is invalid. */ - if (!core_partial_clone_filter_default) - return; - gently_parse_list_objects_filter(filter_options, - core_partial_clone_filter_default, - NULL); + if (promisor) + gently_parse_list_objects_filter(filter_options, + promisor->partial_clone_filter, + NULL); } diff --git a/list-objects-filter-options.h b/list-objects-filter-options.h index e3adc78ebf..70d27f44ef 100644 --- a/list-objects-filter-options.h +++ b/list-objects-filter-options.h @@ -89,6 +89,7 @@ void partial_clone_register( const char *remote, const struct list_objects_filter_options *filter_options); void partial_clone_get_default_filter_spec( - struct list_objects_filter_options *filter_options); + struct list_objects_filter_options *filter_options, + const char *remote); #endif /* LIST_OBJECTS_FILTER_OPTIONS_H */ diff --git a/promisor-remote.c b/promisor-remote.c index 737689d044..707a8005c5 100644 --- a/promisor-remote.c +++ b/promisor-remote.c @@ -75,6 +75,21 @@ static int promisor_remote_config(const char *var, const char *value, void *data free(remote_name); return 0; } + if (!strcmp(subkey, "partialclonefilter")) { + struct promisor_remote *r; + char *remote_name = xmemdupz(name, namelen); + + r = promisor_remote_lookup(remote_name, NULL); + if (!r) + r = promisor_remote_new(remote_name); + + free(remote_name); + + if (!r) + return 0; + + return git_config_string(&r->partial_clone_filter, var, value); + } return 0; } diff --git a/promisor-remote.h b/promisor-remote.h index ff69963907..562c7ad8a4 100644 --- a/promisor-remote.h +++ b/promisor-remote.h @@ -5,10 +5,13 @@ struct object_id; /* * Promisor remote linked list - * Its information come from remote.XXX config entries. + * + * Information in its fields come from remote.XXX config entries or + * from extensions.partialclone or core.partialclonefilter. */ struct promisor_remote { struct promisor_remote *next; + const char *partial_clone_filter; const char name[FLEX_ARRAY]; }; diff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh index 3559313bd0..3082eff2bf 100755 --- a/t/t0410-partial-clone.sh +++ b/t/t0410-partial-clone.sh @@ -26,7 +26,7 @@ promise_and_delete () { test_expect_success 'extensions.partialclone without filter' ' test_create_repo server && git clone --filter="blob:none" "file://$(pwd)/server" client && - git -C client config --unset core.partialclonefilter && + git -C client config --unset remote.origin.partialclonefilter && git -C client fetch origin ' diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh index 15720847b0..8842b029bd 100755 --- a/t/t5601-clone.sh +++ b/t/t5601-clone.sh @@ -654,6 +654,7 @@ partial_clone () { # Ensure that unneeded blobs are not inadvertently fetched. test_config -C client remote.origin.promisor "false" && + git -C client config --unset remote.origin.partialclonefilter && test_must_fail git -C client cat-file -e "$HASH1" && # But this blob was fetched, because clone performs an initial checkout diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh index c9e5f14165..107b247336 100755 --- a/t/t5616-partial-clone.sh +++ b/t/t5616-partial-clone.sh @@ -43,7 +43,7 @@ test_expect_success 'do partial clone 1' ' test_cmp expect_1.oids observed.oids && test "$(git -C pc1 config --local core.repositoryformatversion)" = "1" && test "$(git -C pc1 config --local remote.origin.promisor)" = "true" && - test "$(git -C pc1 config --local core.partialclonefilter)" = "blob:none" + test "$(git -C pc1 config --local remote.origin.partialclonefilter)" = "blob:none" ' # checkout master to force dynamic object fetch of blobs at HEAD. From patchwork Tue Apr 9 16:11:10 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10891573 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BE06717E0 for ; Tue, 9 Apr 2019 16:12:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AB9012862A for ; Tue, 9 Apr 2019 16:12:04 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A037A288C7; Tue, 9 Apr 2019 16:12:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BDB3D286BF for ; Tue, 9 Apr 2019 16:12:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726714AbfDIQMB (ORCPT ); Tue, 9 Apr 2019 12:12:01 -0400 Received: from mail-wr1-f65.google.com ([209.85.221.65]:35601 "EHLO mail-wr1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726515AbfDIQL7 (ORCPT ); Tue, 9 Apr 2019 12:11:59 -0400 Received: by mail-wr1-f65.google.com with SMTP id w1so21694731wrp.2 for ; Tue, 09 Apr 2019 09:11:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WGKD4t9/zQ0RVOKMppZZy5F8GoXk2Na8IU5uUJMWMnw=; b=QO8duTw2r4W7+GFJKu6ZrXZd1hTvO7pB1XXA/Y+efKDpRRuJIuU4qjmw8q5wyY+Tb7 70oMeDaNkDj8Q5oyLRy8zDv4Fxv7Uvcin/hhVxWRrWGtXIn9zhu20nlBqTscPiMMRPzC ALQOR06WSemjx+gEbcBh0tlRLFKs2AAiTwuglUuXXP8VSJt0cRQKzzrdjxMkD89SZYE2 gg4y+XkaN1Uxfu0TklAXk8pd9l5gckNwShKiVi4cUABMzy8dtAfqHqnfr5AgiUmlsYI2 Ek4ld9HN+fbdfjYq6U+UrObMlWj6GY45UAaf9XhEiszJskCLSvdHCnx/LVwtnw88Df+z y/mg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WGKD4t9/zQ0RVOKMppZZy5F8GoXk2Na8IU5uUJMWMnw=; b=CbpGNMbAzmW9r9ecn4F82qfJbDP8i06WM2zpYiFC4DFP4Wq3D0PW3G2a+80HNNKASS Vzco5ndjhp6OLQ8t/ahLY9peuoIZJd5cy980rDl7wkPSgypAp9sqg05XIKuFfW2Zi4/e T8M/dRS1J4vbBmtntG0d4kBqGnJGAEoXtM4II0xNT/KOx0D00DRUQV9tkRpPzmoWB5I1 I40Z1hn33zJ5/0W8IQ7+BwnnzNZSFe5hv9uRgbq/3hK1zekBiq9y7MYO886uAsbGY6dZ BVsgIXXqVJOZ8Oonjhr4u9bu1LOjv6FuurzWqjR1ErpVuWa9Ix4JOG5OEWFccuXlFxQk KUVg== X-Gm-Message-State: APjAAAUlhWzgm0rlGa6htCHceF32Rpi+SXf071wANbId0YhaGuSHDEAD qPnOTf3zbiPjoS/RAiWkjDaEOIVS X-Google-Smtp-Source: APXvYqyfsiE0quxq8oXxtFUgCMGdPJiC5CzHsB72Bd6h6wHj8nWupvstCorwbWXlUNssuflftjPixw== X-Received: by 2002:adf:fc0b:: with SMTP id i11mr5572635wrr.145.1554826316994; Tue, 09 Apr 2019 09:11:56 -0700 (PDT) Received: from ubuntu-N750HU.booking.pcln.com (access-114.38.rev.fr.colt.net. [213.41.38.114]) by smtp.gmail.com with ESMTPSA id i2sm4356012wmg.47.2019.04.09.09.11.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Apr 2019 09:11:55 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli , =?utf-8?q?SZEDER_G=C3=A1bor?= , Ramsay Jones Subject: [PATCH v5 10/16] builtin/fetch: remove unique promisor remote limitation Date: Tue, 9 Apr 2019 18:11:10 +0200 Message-Id: <20190409161116.30256-11-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.750.g68c8ebb2ac In-Reply-To: <20190409161116.30256-1-chriscool@tuxfamily.org> References: <20190409161116.30256-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP As the infrastructure for more than one promisor remote has been introduced in previous patches, we can remove code that forbids the registration of more than one promisor remote. Signed-off-by: Christian Couder --- builtin/fetch.c | 20 +++++--------------- 1 file changed, 5 insertions(+), 15 deletions(-) diff --git a/builtin/fetch.c b/builtin/fetch.c index 13d8133130..5657d054ec 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -1465,26 +1465,16 @@ static inline void fetch_one_setup_partial(struct remote *remote) return; /* - * If this is the FIRST partial-fetch request, we enable partial - * on this repo and remember the given filter-spec as the default - * for subsequent fetches to this remote. + * If this is a partial-fetch request, we enable partial on + * this repo if not already enabled and remember the given + * filter-spec as the default for subsequent fetches to this + * remote. */ - if (!has_promisor_remote() && filter_options.choice) { + if (filter_options.choice) { partial_clone_register(remote->name, &filter_options); return; } - /* - * We are currently limited to only ONE promisor remote and only - * allow partial-fetches from the promisor remote. - */ - if (!promisor_remote_find(remote->name)) { - if (filter_options.choice) - die(_("--filter can only be used with the remote " - "configured in extensions.partialClone")); - return; - } - /* * Do a partial-fetch from the promisor remote using either the * explicitly given filter-spec or inherit the filter-spec from From patchwork Tue Apr 9 16:11:11 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10891577 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CFDCB1708 for ; Tue, 9 Apr 2019 16:12:06 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BAC6E285A5 for ; Tue, 9 Apr 2019 16:12:06 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AE9C9286BF; Tue, 9 Apr 2019 16:12:06 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 429D92862A for ; Tue, 9 Apr 2019 16:12:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726730AbfDIQMD (ORCPT ); Tue, 9 Apr 2019 12:12:03 -0400 Received: from mail-wr1-f68.google.com ([209.85.221.68]:40691 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726628AbfDIQMB (ORCPT ); Tue, 9 Apr 2019 12:12:01 -0400 Received: by mail-wr1-f68.google.com with SMTP id h4so21652201wre.7 for ; Tue, 09 Apr 2019 09:11:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=YWcY89VksHxR3/rRZ++6PngRyphA2/XUIErvpM92YNQ=; b=lN6L3Il2WtzDFIa/ifYj8CZfmvaJGBd1OmajObzml+GdsZDgB6PJS/C2oiUTrcL+Mr fWcUn8ZC1UBsh1Xx7nKP8lK0DmV2+AdiOUnw8iIkL3TvRf0wTuZLDy4+HM2Ijvzyk9YM nFniudIb5eY+0Ve1Gusg8C1ljnTq3d56w27ARR5qU8Luc14aZwu7ydJQB8Rrm2ZwsuXy fl2Q5Q8wDhgg8ookDQIF9/3Ib6lzZTPKmxVfU2eiEYYoC6q5q2V+aY9N3GAuZtMvQby+ r4/pOzRrrG0zSasZMalsYDhMEwn2lw2LIBdk59/6BQGlYmKl85hXN9T20zXkTKX7fHuh Fq1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=YWcY89VksHxR3/rRZ++6PngRyphA2/XUIErvpM92YNQ=; b=Wk44aEbochRF4YLmMH4avVUjwmf5K4dphBHMeNJzV+lxC0rAo9Egf6IAvtFfcH7r2L 7Bq3T4ANFBZKurITlI0bzJLnOuBMde/8hq4U8FOTa/6GyYsXCp4SYHv8BIc09azr/UbW /yvOPaAufp2KJzrGL/TnR3FFEJGwCLMfNmHBa0VrCWkfxDOHQV+DYRMVd5Ui6ftqvPtd rKV7Q1kCAG27j7FLGmBsPYF3XiAxrJTt7rNFFOsBobfRI5Xkzfgy3mFBQ05bDbAhtF0A +jK4gjeG7Ng9mo6GxE+izPWA7eUBH/8SdQbQ4sqyCiVKgJP/RExIMLRZndnaP8ntsVsm xbnQ== X-Gm-Message-State: APjAAAVoe4sl0Oz7Y0lHhMRRwMv5/soshbzfZoIP/XExIBNjhUTHTM+O KMnjKdPcUipX0zgoV+wPkMhaUoKp X-Google-Smtp-Source: APXvYqwqGGUXANx+3ru3mupTU1eD0gd5ejV21N/iGhMhV1slcvIjRwE0S8gX2OKTgqzqN8bee0I/Gw== X-Received: by 2002:a05:6000:1250:: with SMTP id j16mr23520858wrx.135.1554826318530; Tue, 09 Apr 2019 09:11:58 -0700 (PDT) Received: from ubuntu-N750HU.booking.pcln.com (access-114.38.rev.fr.colt.net. [213.41.38.114]) by smtp.gmail.com with ESMTPSA id i2sm4356012wmg.47.2019.04.09.09.11.57 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Apr 2019 09:11:57 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli , =?utf-8?q?SZEDER_G=C3=A1bor?= , Ramsay Jones , Christian Couder Subject: [PATCH v5 11/16] t0410: test fetching from many promisor remotes Date: Tue, 9 Apr 2019 18:11:11 +0200 Message-Id: <20190409161116.30256-12-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.750.g68c8ebb2ac In-Reply-To: <20190409161116.30256-1-chriscool@tuxfamily.org> References: <20190409161116.30256-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Christian Couder This shows that it is now possible to fetch objects from more than one promisor remote, and that fetching from a new promisor remote can configure it as one. Helped-by: SZEDER Gábor Signed-off-by: Christian Couder --- t/t0410-partial-clone.sh | 49 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 48 insertions(+), 1 deletion(-) diff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh index 3082eff2bf..2498e72a34 100755 --- a/t/t0410-partial-clone.sh +++ b/t/t0410-partial-clone.sh @@ -183,8 +183,55 @@ test_expect_success 'fetching of missing objects works with ref-in-want enabled' grep "git< fetch=.*ref-in-want" trace ' +test_expect_success 'fetching of missing objects from another promisor remote' ' + git clone "file://$(pwd)/server" server2 && + test_commit -C server2 bar && + git -C server2 repack -a -d --write-bitmap-index && + HASH2=$(git -C server2 rev-parse bar) && + + git -C repo remote add server2 "file://$(pwd)/server2" && + git -C repo config remote.server2.promisor true && + git -C repo cat-file -p "$HASH2" && + + git -C repo fetch server2 && + rm -rf repo/.git/objects/* && + git -C repo cat-file -p "$HASH2" && + + # Ensure that the .promisor file is written, and check that its + # associated packfile contains the object + ls repo/.git/objects/pack/pack-*.promisor >promisorlist && + test_line_count = 1 promisorlist && + IDX=$(sed "s/promisor$/idx/" promisorlist) && + git verify-pack --verbose "$IDX" >out && + grep "$HASH2" out +' + +test_expect_success 'fetching of missing objects configures a promisor remote' ' + git clone "file://$(pwd)/server" server3 && + test_commit -C server3 baz && + git -C server3 repack -a -d --write-bitmap-index && + HASH3=$(git -C server3 rev-parse baz) && + git -C server3 config uploadpack.allowfilter 1 && + + rm repo/.git/objects/pack/pack-*.promisor && + + git -C repo remote add server3 "file://$(pwd)/server3" && + git -C repo fetch --filter="blob:none" server3 $HASH3 && + + test_cmp_config -C repo true remote.server3.promisor && + + # Ensure that the .promisor file is written, and check that its + # associated packfile contains the object + ls repo/.git/objects/pack/pack-*.promisor >promisorlist && + test_line_count = 1 promisorlist && + IDX=$(sed "s/promisor$/idx/" promisorlist) && + git verify-pack --verbose "$IDX" >out && + grep "$HASH3" out +' + test_expect_success 'fetching of missing blobs works' ' - rm -rf server repo && + rm -rf server server2 repo && + rm -rf server server3 repo && test_create_repo server && test_commit -C server foo && git -C server repack -a -d --write-bitmap-index && From patchwork Tue Apr 9 16:11:12 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10891575 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D51F717E0 for ; Tue, 9 Apr 2019 16:12:05 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C028A2862A for ; Tue, 9 Apr 2019 16:12:05 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B4074288C7; Tue, 9 Apr 2019 16:12:05 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EDA5D2862A for ; Tue, 9 Apr 2019 16:12:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726736AbfDIQME (ORCPT ); Tue, 9 Apr 2019 12:12:04 -0400 Received: from mail-wm1-f66.google.com ([209.85.128.66]:34918 "EHLO mail-wm1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726515AbfDIQMD (ORCPT ); Tue, 9 Apr 2019 12:12:03 -0400 Received: by mail-wm1-f66.google.com with SMTP id y197so4028715wmd.0 for ; Tue, 09 Apr 2019 09:12:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=/kZVztkpFeBPYPBLyVszoZaqZthE0KfXUhGm1E8KMZE=; b=tRQx7NkYn+/A32LuPE0iiFEeBrXmpPMIioPk8d2GnoBGY2dlX6yHa0oJei4b8pJJzm TXTRkbB9EwVXW0R5o6TqTbdt4EXsEDctR+slkn4g17vqwuM3B4JyIJNF9vuL1OuOT5gK INii0A0K/Zk6cDxdZgK2RqnQaSd/lVxlD4PJydVDKlbrp873hi8p6yGY7lDgVILL5ZQ1 eCswpAHq2/Wa0B+KtHIz2nVt+fFgrG1CLWfRLLsXTKuXJtQQg7XYTBdGEQi1BVa0D760 qXztOTsl5aZnyekl5crm6EoarC4lx2JC099oSnprgpoa+ncgYoB7mmQVcL7LujZbkrbR aK+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/kZVztkpFeBPYPBLyVszoZaqZthE0KfXUhGm1E8KMZE=; b=gwOeHbjshgM+Tpke8bm78rSH9NFgOZr5XwnfDRJnJcUaNEQGqDaLOOmtZwYScvcDMo 54EYoRkFPNCCKc55kdnjifeuXPDZVJERPttqJpajQgrphvjBopSxssb32RRHNQjcqtrq yPrz1Z7PAYkFzaOcXQhFK39v6YTWiIyPqLUUbJbxP2cdU9wRiV3lYSzwO0rUM7odGZ5r uW7UNm1jLrGEJZybPVQb+16wQm7BSJFVsRD5ihVAzXFBCTBnaicqw/itht5xbPzWCVtt gcHO5kPJLRgCI3Tg51+yiNGQYkYxeNuh45/eLuxuwm0wGgvymXYOdUGEOq+LuiThQ0cc GzVQ== X-Gm-Message-State: APjAAAVG/QyDfZssfOpoSAJcsLEV0muG0hvW869c6h6rx4IkABKijgQC 67kvTelg/BNT5lIB5ky6mV/Ld++e X-Google-Smtp-Source: APXvYqx+oR9JDxL3xYPZNvAux6ZgdkibGJIrtgfPEatl31jRyw5GX5EA7stCdPFJkXASxRbZpwti7g== X-Received: by 2002:a1c:c00b:: with SMTP id q11mr23246582wmf.38.1554826319901; Tue, 09 Apr 2019 09:11:59 -0700 (PDT) Received: from ubuntu-N750HU.booking.pcln.com (access-114.38.rev.fr.colt.net. [213.41.38.114]) by smtp.gmail.com with ESMTPSA id i2sm4356012wmg.47.2019.04.09.09.11.58 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Apr 2019 09:11:58 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli , =?utf-8?q?SZEDER_G=C3=A1bor?= , Ramsay Jones Subject: [PATCH v5 12/16] partial-clone: add multiple remotes in the doc Date: Tue, 9 Apr 2019 18:11:12 +0200 Message-Id: <20190409161116.30256-13-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.750.g68c8ebb2ac In-Reply-To: <20190409161116.30256-1-chriscool@tuxfamily.org> References: <20190409161116.30256-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP While at it, let's remove a reference to ODB effort as the ODB effort has been replaced by directly enhancing partial clone and promisor remote features. Signed-off-by: Christian Couder --- Documentation/technical/partial-clone.txt | 117 ++++++++++++++++------ 1 file changed, 84 insertions(+), 33 deletions(-) diff --git a/Documentation/technical/partial-clone.txt b/Documentation/technical/partial-clone.txt index 896c7b3878..210373e258 100644 --- a/Documentation/technical/partial-clone.txt +++ b/Documentation/technical/partial-clone.txt @@ -30,12 +30,20 @@ advance* during clone and fetch operations and thereby reduce download times and disk usage. Missing objects can later be "demand fetched" if/when needed. +A remote that can later provide the missing objects is called a +promisor remote, as it promises to send the objects when +requested. Initialy Git supported only one promisor remote, the origin +remote from which the user cloned and that was configured in the +"extensions.partialClone" config option. Later support for more than +one promisor remote has been implemented. + Use of partial clone requires that the user be online and the origin -remote be available for on-demand fetching of missing objects. This may -or may not be problematic for the user. For example, if the user can -stay within the pre-selected subset of the source tree, they may not -encounter any missing objects. Alternatively, the user could try to -pre-fetch various objects if they know that they are going offline. +remote or other promisor remotes be available for on-demand fetching +of missing objects. This may or may not be problematic for the user. +For example, if the user can stay within the pre-selected subset of +the source tree, they may not encounter any missing objects. +Alternatively, the user could try to pre-fetch various objects if they +know that they are going offline. Non-Goals @@ -100,18 +108,18 @@ or commits that reference missing trees. Handling Missing Objects ------------------------ -- An object may be missing due to a partial clone or fetch, or missing due - to repository corruption. To differentiate these cases, the local - repository specially indicates such filtered packfiles obtained from the - promisor remote as "promisor packfiles". +- An object may be missing due to a partial clone or fetch, or missing + due to repository corruption. To differentiate these cases, the + local repository specially indicates such filtered packfiles + obtained from promisor remotes as "promisor packfiles". + These promisor packfiles consist of a ".promisor" file with arbitrary contents (like the ".keep" files), in addition to their ".pack" and ".idx" files. - The local repository considers a "promisor object" to be an object that - it knows (to the best of its ability) that the promisor remote has promised - that it has, either because the local repository has that object in one of + it knows (to the best of its ability) that promisor remotes have promised + that they have, either because the local repository has that object in one of its promisor packfiles, or because another promisor object refers to it. + When Git encounters a missing object, Git can see if it is a promisor object @@ -123,12 +131,12 @@ expensive-to-modify list of missing objects.[a] - Since almost all Git code currently expects any referenced object to be present locally and because we do not want to force every command to do a dry-run first, a fallback mechanism is added to allow Git to attempt - to dynamically fetch missing objects from the promisor remote. + to dynamically fetch missing objects from promisor remotes. + When the normal object lookup fails to find an object, Git invokes -fetch-object to try to get the object from the server and then retry -the object lookup. This allows objects to be "faulted in" without -complicated prediction algorithms. +promisor_remote_get_direct() to try to get the object from a promisor +remote and then retry the object lookup. This allows objects to be +"faulted in" without complicated prediction algorithms. + For efficiency reasons, no check as to whether the missing object is actually a promisor object is performed. @@ -157,8 +165,7 @@ and prefetch those objects in bulk. + We are not happy with this global variable and would like to remove it, but that requires significant refactoring of the object code to pass an -additional flag. We hope that concurrent efforts to add an ODB API can -encompass this. +additional flag. Fetching Missing Objects @@ -182,21 +189,63 @@ has been updated to not use any object flags when the corresponding argument though they are not necessary. +Using many promisor remotes +--------------------------- + +Many promisor remotes can be configured and used. + +This allows for example a user to have multiple geographically-close +cache servers for fetching missing blobs while continuing to do +filtered `git-fetch` commands from the central server. + +When fetching objects, promisor remotes are tried one after the other +until all the objects have been fetched. + +Remotes that are considered "promisor" remotes are those specified by +the following configuration variables: + +- `extensions.partialClone = ` + +- `remote..promisor = true` + +- `remote..partialCloneFilter = ...` + +Only one promisor remote can be configured using the +`extensions.partialClone` config variable. This promisor remote will +be the last one tried when fetching objects. + +We decided to make it the last one we try, because it is likely that +someone using many promisor remotes is doing so because the other +promisor remotes are better for some reason (maybe they are closer or +faster for some kind of objects) than the origin, and the origin is +likely to be the remote specified by extensions.partialClone. + +This justification is not very strong, but one choice had to be made, +and anyway the long term plan should be to make the order somehow +fully configurable. + +For now though the other promisor remotes will be tried in the order +they appear in the config file. + Current Limitations ------------------- -- The remote used for a partial clone (or the first partial fetch - following a regular clone) is marked as the "promisor remote". +- It is not possible to specify the order in which the promisor + remotes are tried in other ways than the order in which they appear + in the config file. + -We are currently limited to a single promisor remote and only that -remote may be used for subsequent partial fetches. +It is also not possible to specify an order to be used when fetching +from one remote and a different order when fetching from another +remote. + +- It is not possible to push only specific objects to a promisor + remote. + -We accept this limitation because we believe initial users of this -feature will be using it on repositories with a strong single central -server. +It is not possible to push at the same time to multiple promisor +remote in a specific order. -- Dynamic object fetching will only ask the promisor remote for missing - objects. We assume that the promisor remote has a complete view of the +- Dynamic object fetching will only ask promisor remotes for missing + objects. We assume that promisor remotes have a complete view of the repository and can satisfy all such requests. - Repack essentially treats promisor and non-promisor packfiles as 2 @@ -218,15 +267,17 @@ server. Future Work ----------- -- Allow more than one promisor remote and define a strategy for fetching - missing objects from specific promisor remotes or of iterating over the - set of promisor remotes until a missing object is found. +- Improve the way to specify the order in which promisor remotes are + tried. + -A user might want to have multiple geographically-close cache servers -for fetching missing blobs while continuing to do filtered `git-fetch` -commands from the central server, for example. +For example this could allow to specify explicitly something like: +"When fetching from this remote, I want to use these promisor remotes +in this order, though, when pushing or fetching to that remote, I want +to use those promisor remotes in that order." + +- Allow pushing to promisor remotes. + -Or the user might want to work in a triangular work flow with multiple +The user might want to work in a triangular work flow with multiple promisor remotes that each have an incomplete view of the repository. - Allow repack to work on promisor packfiles (while keeping them distinct From patchwork Tue Apr 9 16:11:13 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10891579 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D1D0D17E0 for ; Tue, 9 Apr 2019 16:12:07 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BD6C9285A5 for ; Tue, 9 Apr 2019 16:12:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B1484286BF; Tue, 9 Apr 2019 16:12:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5DBBF285A5 for ; Tue, 9 Apr 2019 16:12:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726741AbfDIQMF (ORCPT ); Tue, 9 Apr 2019 12:12:05 -0400 Received: from mail-wm1-f65.google.com ([209.85.128.65]:55479 "EHLO mail-wm1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726716AbfDIQMD (ORCPT ); Tue, 9 Apr 2019 12:12:03 -0400 Received: by mail-wm1-f65.google.com with SMTP id o25so4224527wmf.5 for ; Tue, 09 Apr 2019 09:12:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Y0Do80d2PQBTqqe6EfxWfVFZp8FN+9ycPLcbMO84/AU=; b=mdXIHGJ/EYHEwq94dG/hjRwI1m27afXb144R/p9ttIOSvKFj0QxfOKCfOP8QlPGosm R2FHnzRAk6i505RZR/W5qYEDpn7brN5Z2Tilde5p6+hHu47zPAVMTkrE01UW+X3SfPMA BSx01ZjjrjThr5sBs+LuSZMWulWlTkqpF0zlnz6pWZeVP/Z4ys5jvhh6uai5rS1JlVxU xnsZzT1sHH3C7bnCQqbDQ+Mm3WJBxOgjC5OWR554gFFOLh9jykVDPqDItQDTcoHIj0li BYtsfve4WkkmmTlY06c6G7HQXFa6AQJB/nN2T0TAzmZDKuZr6JQe5Uungr9kvd0YrBno CIzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Y0Do80d2PQBTqqe6EfxWfVFZp8FN+9ycPLcbMO84/AU=; b=FCl32IJLpGjUPZOT+aQDmu+ZIW4cHwDn/216Q47fyLbCSZgIJ+m/r9owazHQNn5VwL 473zCMZC8atUwsnrdiSXLcL6mFueFQ2f16ZNI5YB6LWQfXXljf2Fh3EPeqKXe9CexnYu 1tWZDOKWHuNmYqwo4mPmvYTUClChSytM+q1x0mOUVrJvO4k9AWY4u7Zy8IwkAKkX2tRI fgT/wCKhORnW55tobW2fEJCg3khHsLn5uKvPuQ8Zx/znneUiXUxf7h+CkofIhR0oYqoS 30LF+9BmYtWl9iGOxgfzsUL9QZS11lLlssPmimFwPYCx6imCvxfOYnTdAy/bEXFKpfcv Hj2Q== X-Gm-Message-State: APjAAAXV1zbu3IwlbaUnHcNfpkcToVHRki59ytiLiz5c0Hkvb0zkUppy 61FzrdBwq8b0rNPHGcAQ44In9BiR X-Google-Smtp-Source: APXvYqxSy9QVbb35K4ycjKALO7qMkcqTQDJG47OKToWi9hUSOoGnmI0qBWjlMPhJOaeqG/tAJtJwvw== X-Received: by 2002:a05:600c:2309:: with SMTP id 9mr23080700wmo.52.1554826321497; Tue, 09 Apr 2019 09:12:01 -0700 (PDT) Received: from ubuntu-N750HU.booking.pcln.com (access-114.38.rev.fr.colt.net. [213.41.38.114]) by smtp.gmail.com with ESMTPSA id i2sm4356012wmg.47.2019.04.09.09.11.59 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Apr 2019 09:12:00 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli , =?utf-8?q?SZEDER_G=C3=A1bor?= , Ramsay Jones Subject: [PATCH v5 13/16] remote: add promisor and partial clone config to the doc Date: Tue, 9 Apr 2019 18:11:13 +0200 Message-Id: <20190409161116.30256-14-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.750.g68c8ebb2ac In-Reply-To: <20190409161116.30256-1-chriscool@tuxfamily.org> References: <20190409161116.30256-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Signed-off-by: Christian Couder --- Documentation/config/remote.txt | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/Documentation/config/remote.txt b/Documentation/config/remote.txt index 6c4cad83a2..a8e6437a90 100644 --- a/Documentation/config/remote.txt +++ b/Documentation/config/remote.txt @@ -76,3 +76,11 @@ remote..pruneTags:: + See also `remote..prune` and the PRUNING section of linkgit:git-fetch[1]. + +remote..promisor:: + When set to true, this remote will be used to fetch promisor + objects. + +remote..partialclonefilter:: + The filter that will be applied when fetching from this + promisor remote. From patchwork Tue Apr 9 16:11:14 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10891581 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D504117E0 for ; Tue, 9 Apr 2019 16:12:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C19F7285A5 for ; Tue, 9 Apr 2019 16:12:08 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B4254286BF; Tue, 9 Apr 2019 16:12:08 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 412A6285A5 for ; Tue, 9 Apr 2019 16:12:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726752AbfDIQMH (ORCPT ); Tue, 9 Apr 2019 12:12:07 -0400 Received: from mail-wm1-f66.google.com ([209.85.128.66]:53842 "EHLO mail-wm1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726515AbfDIQMG (ORCPT ); Tue, 9 Apr 2019 12:12:06 -0400 Received: by mail-wm1-f66.google.com with SMTP id q16so4258815wmj.3 for ; Tue, 09 Apr 2019 09:12:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ywh7pk8fJOay6WPvPOMpZ/tRvhES9EqCpsD1ysVDC1I=; b=OXsz5MGBTrfz19kRbQL4UnsMVpkUU6XOgRpFwZFbNhyhJqK75MeC1X53JsRb6W9Oh2 SAxL1B2eZKD54XYOEnX9o6gIqhMq9tGzjEzTMs3Lmu4gteQQJvWDfh3FcKJ3YRaDJJKN 5Ic7aXFFawB4KJRbZb+U+XScj4jJa8zhI69NEiVcDD/tzN8sVSzrdLYk37GkBKagqjoZ rLWCRNuvAPVcZx2e1B8UdoQwI8Q1WPBOf/oJuLKuM+BQQ2MzaVwA5557nqB/i2AC0qy+ AMicsG3bHqbKh3jBOBI/ccG5f3TcqwKfycys891rv4k3HdNyPzNh6e+bhI+eGNob1ns1 lu7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ywh7pk8fJOay6WPvPOMpZ/tRvhES9EqCpsD1ysVDC1I=; b=uFiZ4TOA0yCJBqpWmTZNVM7UGnK7u8S+ELcSxWrMJ6PedM3uHPrR1bhZOox7SuHtCd w2FGQc8bMTNcYNvI34t2nZG2lVF46CykZ5IG1S9o/n5x/ZEEKn6Wr2b53EeU1Ibso9h9 KJKkLEK+TxBrMPbUu3zpVFLkrb0PVouMWUrO4iH6NYLlteenie6zPrjCNr7qBs0CCcyF Xz/Rapu1b2Eg04nX+Cp/vOW1U7vKgPbHh05JdCxb3ZhYmXEMGC/RL8bVG7wJmt+0JBj8 RBDTMAgjL5H0OU3POeCbjz6gAKCeFKow4XWjZFiTxy5gouDpdZosKQPW4TpOc8MG1e4d kh4w== X-Gm-Message-State: APjAAAVKeZUQwMo618KzTEhnTCodyiuIr7avTTYEGVpUX+Kwt87BMzuS z3iMA9kxMpTwX6omjXfbkRTDaBys X-Google-Smtp-Source: APXvYqxiF3sVwl1q8DBgHDd7Aqos/3BdU1f5UUHGviiw2i/MDdnOnlZioW0YzAmDm4gwNu2hlxoIPA== X-Received: by 2002:a05:600c:28d:: with SMTP id 13mr21441287wmk.82.1554826323222; Tue, 09 Apr 2019 09:12:03 -0700 (PDT) Received: from ubuntu-N750HU.booking.pcln.com (access-114.38.rev.fr.colt.net. [213.41.38.114]) by smtp.gmail.com with ESMTPSA id i2sm4356012wmg.47.2019.04.09.09.12.01 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Apr 2019 09:12:02 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli , =?utf-8?q?SZEDER_G=C3=A1bor?= , Ramsay Jones Subject: [PATCH v5 14/16] Remove fetch-object.{c,h} in favor of promisor-remote.{c,h} Date: Tue, 9 Apr 2019 18:11:14 +0200 Message-Id: <20190409161116.30256-15-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.750.g68c8ebb2ac In-Reply-To: <20190409161116.30256-1-chriscool@tuxfamily.org> References: <20190409161116.30256-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP As fetch_objects() is now used only in promisor-remote.c and should't be used outside it, let's move it into promisor-remote.c, make it static there, and remove fetch-object.{c,h}. Signed-off-by: Christian Couder --- Makefile | 1 - fetch-object.c | 43 ------------------------------------------- fetch-object.h | 9 --------- promisor-remote.c | 40 +++++++++++++++++++++++++++++++++++++++- 4 files changed, 39 insertions(+), 54 deletions(-) delete mode 100644 fetch-object.c delete mode 100644 fetch-object.h diff --git a/Makefile b/Makefile index 3523ae0517..0996fb6646 100644 --- a/Makefile +++ b/Makefile @@ -890,7 +890,6 @@ LIB_OBJS += ewah/ewah_io.o LIB_OBJS += ewah/ewah_rlw.o LIB_OBJS += exec-cmd.o LIB_OBJS += fetch-negotiator.o -LIB_OBJS += fetch-object.o LIB_OBJS += fetch-pack.o LIB_OBJS += fsck.o LIB_OBJS += fsmonitor.o diff --git a/fetch-object.c b/fetch-object.c deleted file mode 100644 index eac4d448ef..0000000000 --- a/fetch-object.c +++ /dev/null @@ -1,43 +0,0 @@ -#include "cache.h" -#include "packfile.h" -#include "pkt-line.h" -#include "strbuf.h" -#include "transport.h" -#include "fetch-object.h" - -static int fetch_refs(const char *remote_name, struct ref *ref) -{ - struct remote *remote; - struct transport *transport; - int original_fetch_if_missing = fetch_if_missing; - int res; - - fetch_if_missing = 0; - remote = remote_get(remote_name); - if (!remote->url[0]) - die(_("Remote with no URL")); - transport = transport_get(remote, remote->url[0]); - - transport_set_option(transport, TRANS_OPT_FROM_PROMISOR, "1"); - transport_set_option(transport, TRANS_OPT_NO_DEPENDENTS, "1"); - res = transport_fetch_refs(transport, ref); - fetch_if_missing = original_fetch_if_missing; - - return res; -} - -int fetch_objects(const char *remote_name, const struct object_id *oids, - int oid_nr) -{ - struct ref *ref = NULL; - int i; - - for (i = 0; i < oid_nr; i++) { - struct ref *new_ref = alloc_ref(oid_to_hex(&oids[i])); - oidcpy(&new_ref->old_oid, &oids[i]); - new_ref->exact_oid = 1; - new_ref->next = ref; - ref = new_ref; - } - return fetch_refs(remote_name, ref); -} diff --git a/fetch-object.h b/fetch-object.h deleted file mode 100644 index 7bcc7cadb0..0000000000 --- a/fetch-object.h +++ /dev/null @@ -1,9 +0,0 @@ -#ifndef FETCH_OBJECT_H -#define FETCH_OBJECT_H - -struct object_id; - -int fetch_objects(const char *remote_name, const struct object_id *oids, - int oid_nr); - -#endif diff --git a/promisor-remote.c b/promisor-remote.c index 707a8005c5..066489b637 100644 --- a/promisor-remote.c +++ b/promisor-remote.c @@ -2,7 +2,45 @@ #include "object-store.h" #include "promisor-remote.h" #include "config.h" -#include "fetch-object.h" +#include "transport.h" + +static int fetch_refs(const char *remote_name, struct ref *ref) +{ + struct remote *remote; + struct transport *transport; + int original_fetch_if_missing = fetch_if_missing; + int res; + + fetch_if_missing = 0; + remote = remote_get(remote_name); + if (!remote->url[0]) + die(_("Remote with no URL")); + transport = transport_get(remote, remote->url[0]); + + transport_set_option(transport, TRANS_OPT_FROM_PROMISOR, "1"); + transport_set_option(transport, TRANS_OPT_NO_DEPENDENTS, "1"); + res = transport_fetch_refs(transport, ref); + fetch_if_missing = original_fetch_if_missing; + + return res; +} + +static int fetch_objects(const char *remote_name, + const struct object_id *oids, + int oid_nr) +{ + struct ref *ref = NULL; + int i; + + for (i = 0; i < oid_nr; i++) { + struct ref *new_ref = alloc_ref(oid_to_hex(&oids[i])); + oidcpy(&new_ref->old_oid, &oids[i]); + new_ref->exact_oid = 1; + new_ref->next = ref; + ref = new_ref; + } + return fetch_refs(remote_name, ref); +} static struct promisor_remote *promisors; static struct promisor_remote **promisors_tail = &promisors; From patchwork Tue Apr 9 16:11:15 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10891583 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6C27D1708 for ; Tue, 9 Apr 2019 16:12:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 57B3B285A5 for ; Tue, 9 Apr 2019 16:12:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 49FEC286BF; Tue, 9 Apr 2019 16:12:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D5793285A5 for ; Tue, 9 Apr 2019 16:12:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726756AbfDIQMJ (ORCPT ); Tue, 9 Apr 2019 12:12:09 -0400 Received: from mail-wr1-f66.google.com ([209.85.221.66]:37121 "EHLO mail-wr1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726742AbfDIQMH (ORCPT ); Tue, 9 Apr 2019 12:12:07 -0400 Received: by mail-wr1-f66.google.com with SMTP id w10so21680265wrm.4 for ; Tue, 09 Apr 2019 09:12:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=P6PjRo5DLBeEdKVtp29ZnCkB79MO7yqXoZpv9SpvVzQ=; b=kBoY2IL290DrlQ+mdQF1XNln5Bt1ZxHg/DKlLGdtEZSCmTxuFE3KcsoMjWW9+3CL0b Cy4nEhunLS1qzrLIWB0gwWTDxWwcQKvglHx6UAwY2oekNkAPSlkblfjSsjkMlHPepGCN i6Y1o+neyi9/rJqhatHcPGUQ9AwfCu7zJpdcQu1EKzFBAyowl/1XFb4xVXWY/rfJREGo mzq0BnP6nIFKl/iliDmsZQWlaFt+qIHjZfaO9IvL+K44cFJmUcQD5tE76oMxkKK54y42 r6zY1MEN0zZl1A0kkCqpdulwvWF67Dz2ewSdp997H5j6HifIlxg97AB9XJsngY5JxbRH 4Zhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=P6PjRo5DLBeEdKVtp29ZnCkB79MO7yqXoZpv9SpvVzQ=; b=fzO0XTzfbZ4zvjyMTw4b2HtZjL7bgOQbuRSUaYRhWARd5duMoNdmNK+FIBP/UMVzPX Gykccus03pv6qUauFRe2rQpCm93YaEMYND4WtsGWHUQn7j+WjQrTCWiyvtcWkrSuOVYj 71gmCrh5aivkNEUiRSM8DF1vLM+eo+T22wiNoNgJ/CWFaymW7ecQwBNEXrLZy2GjYt08 4LpzKlBladPRmi1Fz8Nfrv2deC6eBG/NoBOOpz9286i5LoDKhNPiu4nYGQNCqQVJp5Nt 0+qBJU/jRetU2vvmc68IIGNYOWnXXTYgarweEVM4LCt/Sba4ewiVgiz2sG0R5DvB8avd 3SPA== X-Gm-Message-State: APjAAAWEBwCmglHupIp9GlIIaWodgVzBW9+Ie8WjGqBUsjs15bc6qbFy Py6AQS2FxL7vQro7pqhnw0O80lLt X-Google-Smtp-Source: APXvYqyCczaVnMl2fiKIj6utsQjpYJwGmafvKm0r9PnOQrm9DXteDemC+BAg3XfNPwhBKUL64XWqOg== X-Received: by 2002:adf:9c91:: with SMTP id d17mr22527613wre.285.1554826325390; Tue, 09 Apr 2019 09:12:05 -0700 (PDT) Received: from ubuntu-N750HU.booking.pcln.com (access-114.38.rev.fr.colt.net. [213.41.38.114]) by smtp.gmail.com with ESMTPSA id i2sm4356012wmg.47.2019.04.09.09.12.03 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Apr 2019 09:12:03 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli , =?utf-8?q?SZEDER_G=C3=A1bor?= , Ramsay Jones Subject: [PATCH v5 15/16] Move repository_format_partial_clone to promisor-remote.c Date: Tue, 9 Apr 2019 18:11:15 +0200 Message-Id: <20190409161116.30256-16-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.750.g68c8ebb2ac In-Reply-To: <20190409161116.30256-1-chriscool@tuxfamily.org> References: <20190409161116.30256-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Now that we have has_promisor_remote() and can use many promisor remotes, let's hide repository_format_partial_clone as a static in promisor-remote.c to avoid it being use for anything other than managing backward compatibility. Signed-off-by: Christian Couder --- cache.h | 1 - environment.c | 1 - promisor-remote.c | 7 +++++++ promisor-remote.h | 6 ++++++ setup.c | 3 ++- 5 files changed, 15 insertions(+), 3 deletions(-) diff --git a/cache.h b/cache.h index e928fe9d3b..217e434d5d 100644 --- a/cache.h +++ b/cache.h @@ -960,7 +960,6 @@ extern int grafts_replace_parents; #define GIT_REPO_VERSION 0 #define GIT_REPO_VERSION_READ 1 extern int repository_format_precious_objects; -extern char *repository_format_partial_clone; extern const char *core_partial_clone_filter_default; extern int repository_format_worktree_config; diff --git a/environment.c b/environment.c index 89af47cb85..8855d2fc11 100644 --- a/environment.c +++ b/environment.c @@ -31,7 +31,6 @@ int warn_ambiguous_refs = 1; int warn_on_object_refname_ambiguity = 1; int ref_paranoia = -1; int repository_format_precious_objects; -char *repository_format_partial_clone; const char *core_partial_clone_filter_default; int repository_format_worktree_config; const char *git_commit_encoding; diff --git a/promisor-remote.c b/promisor-remote.c index 066489b637..371c78385f 100644 --- a/promisor-remote.c +++ b/promisor-remote.c @@ -4,6 +4,13 @@ #include "config.h" #include "transport.h" +static char *repository_format_partial_clone; + +void set_repository_format_partial_clone(char *partial_clone) +{ + repository_format_partial_clone = xstrdup_or_null(partial_clone); +} + static int fetch_refs(const char *remote_name, struct ref *ref) { struct remote *remote; diff --git a/promisor-remote.h b/promisor-remote.h index 562c7ad8a4..c3c07c2a23 100644 --- a/promisor-remote.h +++ b/promisor-remote.h @@ -20,4 +20,10 @@ extern struct promisor_remote *promisor_remote_find(const char *remote_name); extern int has_promisor_remote(void); extern int promisor_remote_get_direct(const struct object_id *oids, int oid_nr); +/* + * This should be used only once from setup.c to set the value we got + * from the extensions.partialclone config option. + */ +extern void set_repository_format_partial_clone(char *partial_clone); + #endif /* PROMISOR_REMOTE_H */ diff --git a/setup.c b/setup.c index d0c958c3b2..434628706b 100644 --- a/setup.c +++ b/setup.c @@ -4,6 +4,7 @@ #include "dir.h" #include "string-list.h" #include "chdir-notify.h" +#include "promisor-remote.h" static int inside_git_dir = -1; static int inside_work_tree = -1; @@ -477,7 +478,7 @@ static int check_repository_format_gently(const char *gitdir, struct repository_ } repository_format_precious_objects = candidate->precious_objects; - repository_format_partial_clone = xstrdup_or_null(candidate->partial_clone); + set_repository_format_partial_clone(candidate->partial_clone); repository_format_worktree_config = candidate->worktree_config; string_list_clear(&candidate->unknown_extensions, 0); From patchwork Tue Apr 9 16:11:16 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 10891587 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E48491708 for ; Tue, 9 Apr 2019 16:12:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D08BF2862A for ; Tue, 9 Apr 2019 16:12:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C4C61286BF; Tue, 9 Apr 2019 16:12:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1AF86286C4 for ; Tue, 9 Apr 2019 16:12:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726761AbfDIQMK (ORCPT ); Tue, 9 Apr 2019 12:12:10 -0400 Received: from mail-wr1-f68.google.com ([209.85.221.68]:45543 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726515AbfDIQMK (ORCPT ); Tue, 9 Apr 2019 12:12:10 -0400 Received: by mail-wr1-f68.google.com with SMTP id s15so21603966wra.12 for ; Tue, 09 Apr 2019 09:12:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ByOYcfo2Ie+3tPx+lyyUEesRW5ynkr17PCkvzC8Zauk=; b=UB609eS4ZlD2lC7aePL3hM/NHEwQ4/3vgBDbeqM/X7Jf0VqbtaSRKc7+4S/4xzC89i cmYVNm4uSSMDGlxTpLVeHWcU/zZdKBwxiMlGas5caET+eWcXuJwNWsMvHeuD9mECqpfb sI7F1hNd2CJu/i95aLJxW8eKDCVWjcm+/P7GoiLAs2yPQWNFE/mnu23zcFluf6ZpRPoJ MSGr0qJILZ4sNBwdQzqHUi92t7h+755//RmKOaViu2gLr5m9PrAhv/nhNB9Skee1x7mN sumpDAO1Eyg+YG6FHmLkoNGdUR0uOKj71m+AiGxt6fwjgqIRK46CucKIhr4r2dEfCE3b zo6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ByOYcfo2Ie+3tPx+lyyUEesRW5ynkr17PCkvzC8Zauk=; b=MzqeJm5UYB6llH+IO8hgxRZzCrT37kkLKShQfacSYl8ldeo4rU4BT1PSDFbfrsw1BT 6fvlNye/46dHKHWhg8uzI4NTJrz5qDb/Ou6rY2dkGCPfDZ3kbq2YEaJOhW8cbX+8DrUo mveQsTqmDqoxZIZvzrbkujR3f52OvSmdd4RQ2qwatvYA2ww2r7u1C6Bydooj4sW+TBeF PzbddWwKhRuv7NOf6a0I2dnP4OEX0s3GFw59qp/bH3WaBUpzRfCW29BXrDYIPhG0XPYq JTEB2ElOOuVjjnOiCKzoZv0SQdbhgu4zH7/PAXUXyv2/yVg/4Ok8sMzNHdUvKuqdxyDx NxjA== X-Gm-Message-State: APjAAAVs82/ddYxtxckdoJWH8L+9AIy5R2l3BdRHtcIM0ALJZCyy13+I 5EzKNUoEmIlXN+idZumnHwMOENSf X-Google-Smtp-Source: APXvYqwyXWzWSx5iNJzQ5wGAz9YjgvzjVCNfTRG5s9iUOEgTzJK0/8BiLE2ZVWIlGtrc0jLpg5LjwQ== X-Received: by 2002:adf:fc0b:: with SMTP id i11mr5573301wrr.145.1554826327183; Tue, 09 Apr 2019 09:12:07 -0700 (PDT) Received: from ubuntu-N750HU.booking.pcln.com (access-114.38.rev.fr.colt.net. [213.41.38.114]) by smtp.gmail.com with ESMTPSA id i2sm4356012wmg.47.2019.04.09.09.12.05 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Apr 2019 09:12:06 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Ben Peart , Jonathan Tan , Jonathan Nieder , Nguyen Thai Ngoc Duy , Mike Hommey , Lars Schneider , Eric Wong , Christian Couder , Jeff Hostetler , Eric Sunshine , Beat Bolli , =?utf-8?q?SZEDER_G=C3=A1bor?= , Ramsay Jones Subject: [PATCH v5 16/16] Move core_partial_clone_filter_default to promisor-remote.c Date: Tue, 9 Apr 2019 18:11:16 +0200 Message-Id: <20190409161116.30256-17-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.21.0.750.g68c8ebb2ac In-Reply-To: <20190409161116.30256-1-chriscool@tuxfamily.org> References: <20190409161116.30256-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Now that we can have a different default partial clone filter for each promisor remote, let's hide core_partial_clone_filter_default as a static in promisor-remote.c to avoid it being use for anything other than managing backward compatibility. Signed-off-by: Christian Couder --- cache.h | 1 - config.c | 5 ----- environment.c | 1 - promisor-remote.c | 5 +++++ 4 files changed, 5 insertions(+), 7 deletions(-) diff --git a/cache.h b/cache.h index 217e434d5d..7ed867b119 100644 --- a/cache.h +++ b/cache.h @@ -960,7 +960,6 @@ extern int grafts_replace_parents; #define GIT_REPO_VERSION 0 #define GIT_REPO_VERSION_READ 1 extern int repository_format_precious_objects; -extern const char *core_partial_clone_filter_default; extern int repository_format_worktree_config; /* diff --git a/config.c b/config.c index c2846df3f1..dd95b659b4 100644 --- a/config.c +++ b/config.c @@ -1344,11 +1344,6 @@ static int git_default_core_config(const char *var, const char *value, void *cb) return 0; } - if (!strcmp(var, "core.partialclonefilter")) { - return git_config_string(&core_partial_clone_filter_default, - var, value); - } - if (!strcmp(var, "core.usereplacerefs")) { read_replace_refs = git_config_bool(var, value); return 0; diff --git a/environment.c b/environment.c index 8855d2fc11..efa072680a 100644 --- a/environment.c +++ b/environment.c @@ -31,7 +31,6 @@ int warn_ambiguous_refs = 1; int warn_on_object_refname_ambiguity = 1; int ref_paranoia = -1; int repository_format_precious_objects; -const char *core_partial_clone_filter_default; int repository_format_worktree_config; const char *git_commit_encoding; const char *git_log_output_encoding; diff --git a/promisor-remote.c b/promisor-remote.c index 371c78385f..f77d0e3b50 100644 --- a/promisor-remote.c +++ b/promisor-remote.c @@ -5,6 +5,7 @@ #include "transport.h" static char *repository_format_partial_clone; +static const char *core_partial_clone_filter_default; void set_repository_format_partial_clone(char *partial_clone) { @@ -103,6 +104,10 @@ static int promisor_remote_config(const char *var, const char *value, void *data int namelen; const char *subkey; + if (!strcmp(var, "core.partialclonefilter")) + return git_config_string(&core_partial_clone_filter_default, + var, value); + if (parse_config_key(var, "remote", &name, &namelen, &subkey) < 0) return 0;