From patchwork Fri Mar 4 15:04:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robert Coup X-Patchwork-Id: 12769282 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DA41C433EF for ; Fri, 4 Mar 2022 15:04:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240041AbiCDPF2 (ORCPT ); Fri, 4 Mar 2022 10:05:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44436 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236011AbiCDPF0 (ORCPT ); Fri, 4 Mar 2022 10:05:26 -0500 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C4FF915B3CB for ; Fri, 4 Mar 2022 07:04:38 -0800 (PST) Received: by mail-wr1-x435.google.com with SMTP id e24so122317wrc.10 for ; Fri, 04 Mar 2022 07:04:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=TcVHonLBZGosq+7VKT77a72z0kemvaq/5GGDKB0cb0Q=; b=gGlbgoJtqPq4zhTuKjxRU2f59eoyWdT8LbDTs8NpdsM/FGKUa7zkMAm9RoXYyvzANO qX1cKS3KPfvXADp/5SaT/LzoD9KJVtBlEhH9gDlMC/LYDKK4wNhFYEKRIsB3ZvDW9wgd RcgECVvUygbuqLhdF4GwATDvlyssSkwR9X3kRcALB3DOSI0wdtLRbxxkvZjukrQ+6guU zuVF9ejScIxwtk78C4QLNu99DFHpmmhOD4hOrasYdtpEkuI51a7kNS6se2eyNag7MYzK 6OvBqFdlDOlXdvyrOK0OOETwes6OQgdAQZV7kMYynfzmqCw7cwFwuUuqcd33VBKrGDzB NdUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=TcVHonLBZGosq+7VKT77a72z0kemvaq/5GGDKB0cb0Q=; b=v6qbzi7EbawbOtxbK4EHQfHgWSfxXKVKZcjyW/xAHIeFWzu93YWTTUaPFos+hzqDki nBo+1FNNdUabVQu3LevAVq6dy3vVHTK4oqKsBa3TVuNoc6E31dPAcHIRrCQFVllSh8vU 0wSuelRceYgI/VJmG8WjUFzCjCXIv62mvyiv9pdsNM3r/rllNdgcmIX6XbTKs25hs6SB d/+Fwdyvkk4KbWtIraXHkOoubx+ntF54EmpMQDF7Kpn9RFLG8+O39OdVW3U4mRtf5qlO cLXJzrMwi1DbKwES6ctaub6JRtMf2Ljjt3HcNYtw3lrhaY4s7Rxdws2QS2A5W8fAC+Sk UIsg== X-Gm-Message-State: AOAM533P5ddSasVm9PHpi3byo2KLzRYWwhZj+Dn59v+HXa7yDkQVBO3d iSo/El8aiQ8iDJPUw6Zem0+UgcwUem8= X-Google-Smtp-Source: ABdhPJx7fF+CPJ+zHRoWrTjTuXNOvO91i551KfQOaNGrYcIM50CbDXbljfrNp4qPqndtRB8Ukr29rg== X-Received: by 2002:a05:6000:15c7:b0:1f0:b269:76ac with SMTP id y7-20020a05600015c700b001f0b26976acmr2133551wry.331.1646406277090; Fri, 04 Mar 2022 07:04:37 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f13-20020adff8cd000000b001f03439743fsm4869880wrq.75.2022.03.04.07.04.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Mar 2022 07:04:36 -0800 (PST) Message-Id: <96a75be3d8acf983db256206706c2e9651d4a2c2.1646406275.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 04 Mar 2022 15:04:28 +0000 Subject: [PATCH v3 1/7] fetch-negotiator: add specific noop initializer Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jonathan Tan , John Cai , Jeff Hostetler , Junio C Hamano , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Robert Coup , Robert Coup Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Robert Coup From: Robert Coup Add a specific initializer for the noop fetch negotiator. This is introduced to support allowing partial clones to skip commit negotiation when performing a "refetch". Signed-off-by: Robert Coup --- fetch-negotiator.c | 5 +++++ fetch-negotiator.h | 8 ++++++++ 2 files changed, 13 insertions(+) diff --git a/fetch-negotiator.c b/fetch-negotiator.c index 874797d767b..be383367f55 100644 --- a/fetch-negotiator.c +++ b/fetch-negotiator.c @@ -23,3 +23,8 @@ void fetch_negotiator_init(struct repository *r, return; } } + +void fetch_negotiator_init_noop(struct fetch_negotiator *negotiator) +{ + noop_negotiator_init(negotiator); +} diff --git a/fetch-negotiator.h b/fetch-negotiator.h index ea78868504b..e348905a1f0 100644 --- a/fetch-negotiator.h +++ b/fetch-negotiator.h @@ -53,7 +53,15 @@ struct fetch_negotiator { void *data; }; +/* + * Initialize a negotiator based on the repository settings. + */ void fetch_negotiator_init(struct repository *r, struct fetch_negotiator *negotiator); +/* + * Initialize a noop negotiator. + */ +void fetch_negotiator_init_noop(struct fetch_negotiator *negotiator); + #endif From patchwork Fri Mar 4 15:04:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robert Coup X-Patchwork-Id: 12769285 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14BE6C433F5 for ; Fri, 4 Mar 2022 15:04:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240048AbiCDPFc (ORCPT ); Fri, 4 Mar 2022 10:05:32 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44490 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240025AbiCDPF1 (ORCPT ); Fri, 4 Mar 2022 10:05:27 -0500 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B5A9F15B3EC for ; Fri, 4 Mar 2022 07:04:39 -0800 (PST) Received: by mail-wm1-x334.google.com with SMTP id i66so5171717wma.5 for ; Fri, 04 Mar 2022 07:04:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=QN4mRAiTxC5fu7XTz9VlYgbA6F2pTz4HHm2IG6o9d0g=; b=XpvmGWszfnh5foW9AJBIWK+YqFrV0h8xdC4WpS+QpYG2h4Ydd+JA/RCtBrkS7h6b8U AX2DapkDSbS2cnxggxCUicUKEl+z4JfZsArd0uA5hVGX1xKuroAOKN7lTlyymkAsYBIm ej3AlXwUfyt3bJWuqPzC6Lu1J1kt2PIONVqgNHGPEwUogEtIskZXyji3ZDEqQ8saW+mt PEj46Vw83xka1zGTWUXeRK4+V1rXwxClclEykVDyJpb7zbDWTlLGBKCV+wfbdFxdU8z+ fnL6JpCXsO4iQI/BwWrLuxtzx2QC2Tykm3cdPCF/7HWk/jzjkhXpETQ0hA8iGduuzRNT 3HGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=QN4mRAiTxC5fu7XTz9VlYgbA6F2pTz4HHm2IG6o9d0g=; b=NpNBnvPWCyhbegmTJuqrb5L5tpwX3N8je8zCcYGoSq0pIxlk268FHx5DGRXFLCCKpC H0xTJEzuodLhmvJbb2D1wFCi1qYuAuOEbFTIc3hRDFLgdSkdaof3YhUk/czvcAlm+dAa EnCrc0Vet2FKn63EGZYsQm7GoPApbn7ccvRVa641ndi+uDPVuaRWm08jmZ6P78clGlpy MpBFfCEpZs0JyNHNEB6zyI2O78+4rRdyt39YQwg0gGa5xpZc67g+e5/OLbyVsEFyydhV wzy4mIQmUXYgEWCJzchvjbmc/N+03vbbcQZrDmM7jqvYsthLPH5qbncFWRaW+vjjK1F2 6PxQ== X-Gm-Message-State: AOAM530MeAYVDVjhWJmHZlMKSUBynExsQZzX923Ny8WQwoWaT5cDIC9T tIp3WMWAfFTsNPs8NpXfrC0o1cTRp+o= X-Google-Smtp-Source: ABdhPJx7CKOG7bZsfGNeVWP1sfSTEivmQWpZo5IpRD5a8vhB6sKxfyTBId3HV41BzVtI0rARt37EGw== X-Received: by 2002:a05:600c:4f02:b0:382:c373:69e2 with SMTP id l2-20020a05600c4f0200b00382c37369e2mr8134844wmq.132.1646406277949; Fri, 04 Mar 2022 07:04:37 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o7-20020a5d6707000000b001f067c7b47fsm3708198wru.27.2022.03.04.07.04.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Mar 2022 07:04:37 -0800 (PST) Message-Id: <04ca6a07f859165cb06197dac862db251b342406.1646406275.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 04 Mar 2022 15:04:29 +0000 Subject: [PATCH v3 2/7] fetch-pack: add refetch Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jonathan Tan , John Cai , Jeff Hostetler , Junio C Hamano , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Robert Coup , Robert Coup Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Robert Coup From: Robert Coup Allow a "refetch" where the contents of the local object store are ignored and a full fetch is performed, not attempting to find or negotiate common commits with the remote. A key use case is to apply a new partial clone blob/tree filter and refetch all the associated matching content, which would otherwise not be transferred when the commit objects are already present locally. Signed-off-by: Robert Coup --- fetch-pack.c | 46 +++++++++++++++++++++++++++++----------------- fetch-pack.h | 1 + 2 files changed, 30 insertions(+), 17 deletions(-) diff --git a/fetch-pack.c b/fetch-pack.c index 87657907e78..4e1e88eea09 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -312,19 +312,21 @@ static int find_common(struct fetch_negotiator *negotiator, const char *remote_hex; struct object *o; - /* - * If that object is complete (i.e. it is an ancestor of a - * local ref), we tell them we have it but do not have to - * tell them about its ancestors, which they already know - * about. - * - * We use lookup_object here because we are only - * interested in the case we *know* the object is - * reachable and we have already scanned it. - */ - if (((o = lookup_object(the_repository, remote)) != NULL) && - (o->flags & COMPLETE)) { - continue; + if (!args->refetch) { + /* + * If that object is complete (i.e. it is an ancestor of a + * local ref), we tell them we have it but do not have to + * tell them about its ancestors, which they already know + * about. + * + * We use lookup_object here because we are only + * interested in the case we *know* the object is + * reachable and we have already scanned it. + */ + if (((o = lookup_object(the_repository, remote)) != NULL) && + (o->flags & COMPLETE)) { + continue; + } } remote_hex = oid_to_hex(remote); @@ -692,6 +694,9 @@ static void mark_complete_and_common_ref(struct fetch_negotiator *negotiator, int old_save_commit_buffer = save_commit_buffer; timestamp_t cutoff = 0; + if (args->refetch) + return; + save_commit_buffer = 0; trace2_region_enter("fetch-pack", "parse_remote_refs_and_find_cutoff", NULL); @@ -1028,7 +1033,11 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args, struct fetch_negotiator *negotiator; negotiator = &negotiator_alloc; - fetch_negotiator_init(r, negotiator); + if (args->refetch) { + fetch_negotiator_init_noop(negotiator); + } else { + fetch_negotiator_init(r, negotiator); + } sort_ref_list(&ref, ref_compare_name); QSORT(sought, nr_sought, cmp_ref_by_name); @@ -1121,7 +1130,7 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args, mark_complete_and_common_ref(negotiator, args, &ref); filter_refs(args, &ref, sought, nr_sought); - if (everything_local(args, &ref)) { + if (!args->refetch && everything_local(args, &ref)) { packet_flush(fd[1]); goto all_done; } @@ -1587,7 +1596,10 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, struct strvec index_pack_args = STRVEC_INIT; negotiator = &negotiator_alloc; - fetch_negotiator_init(r, negotiator); + if (args->refetch) + fetch_negotiator_init_noop(negotiator); + else + fetch_negotiator_init(r, negotiator); packet_reader_init(&reader, fd[0], NULL, 0, PACKET_READ_CHOMP_NEWLINE | @@ -1613,7 +1625,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, /* Filter 'ref' by 'sought' and those that aren't local */ mark_complete_and_common_ref(negotiator, args, &ref); filter_refs(args, &ref, sought, nr_sought); - if (everything_local(args, &ref)) + if (!args->refetch && everything_local(args, &ref)) state = FETCH_DONE; else state = FETCH_SEND_REQUEST; diff --git a/fetch-pack.h b/fetch-pack.h index 7f94a2a5831..8c7752fc821 100644 --- a/fetch-pack.h +++ b/fetch-pack.h @@ -42,6 +42,7 @@ struct fetch_pack_args { unsigned update_shallow:1; unsigned reject_shallow_remote:1; unsigned deepen:1; + unsigned refetch:1; /* * Indicate that the remote of this request is a promisor remote. The From patchwork Fri Mar 4 15:04:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robert Coup X-Patchwork-Id: 12769283 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DBCE0C433EF for ; Fri, 4 Mar 2022 15:04:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240046AbiCDPFa (ORCPT ); Fri, 4 Mar 2022 10:05:30 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44578 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240040AbiCDPF2 (ORCPT ); Fri, 4 Mar 2022 10:05:28 -0500 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 875E715B984 for ; Fri, 4 Mar 2022 07:04:40 -0800 (PST) Received: by mail-wm1-x32b.google.com with SMTP id p184-20020a1c29c1000000b0037f76d8b484so5254899wmp.5 for ; Fri, 04 Mar 2022 07:04:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=GQLIrddXetsyrX2ll69SBqBgf8QkoPOqnA7Y0Pv1zaE=; b=mTHRQsQwzAYK+gor9yV471uO+dvF6e4HYI42JdqJnVaZdzs0UVyC86ZSDUtUrDu2lQ pl8S2OQVUkoGSYGIKG7ojMHcb+yWr8fVES5IuuxF3hMIWvPid8x1odbabC9CVlBedYWQ QNjOKddVidvc+2rfBBsB5sqog5OCK8UHApCMRtIPDsDlKH/BNKwx/KmflwafPb2tu8bJ 2OcXGr72pye6gBHcy5koGWcWwCR0VX2xvd7QQl+oh5Xd9G/tV8VrVq7CpU3pW4Dl8Iun yX9tN8YNgRZ/uvsqECJLpi25KKdKS7ODDUnmp8+FsbqbHoj9+gee0aJf+GdEiha62vR3 1orQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=GQLIrddXetsyrX2ll69SBqBgf8QkoPOqnA7Y0Pv1zaE=; b=FW20Fj21lBm+Dbim6JS8AdUe1ocPxuT1M7+hkNq1zVX9b6zZCv4dzcC9rp5G9WODJo 3e1aM9Z2vQDYTFka46yHsDeAm6/L+BhTlNIs7C6R4CqG4B4jaNz/UGkmxeHCVEz5u9oJ 1/VbxrpOUIzyEiR5bkYIlP+SfEucTxDhNjmAhlRmJ3ycMOgtw9ITKWFO36KKRtvU91Z3 swWSm4NukQC7U86I5E7NrxNC5tsxXINUAH+ZaZdD6sWp9Rncv8+ksVo9+FH3ECLaptof hNvUSkOKU6Wab6kz40HAOjnp01x8FI4eRqNcjYz0xro13kr4N5XGKRJYBTtE+qSx6ipF CMYw== X-Gm-Message-State: AOAM532o/ubaSu0VmniYJtyyT6DH77WjkJZ/sqa4Fql4zbeZBxxL5MUI qcfH2ESt1qtHUdjZo7Vok0oolmeoJ9Q= X-Google-Smtp-Source: ABdhPJyKUAn5B2rXRlPDrZy6vx3gMM5EPtrn5L3M0BKrlFtyg3ncn8Hrb0KG4fYIodFZ8+LSVRKMmQ== X-Received: by 2002:a05:600c:1d08:b0:381:6eda:67d1 with SMTP id l8-20020a05600c1d0800b003816eda67d1mr8390734wms.88.1646406278857; Fri, 04 Mar 2022 07:04:38 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 10-20020adf808a000000b001edd413a952sm4758282wrl.95.2022.03.04.07.04.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Mar 2022 07:04:38 -0800 (PST) Message-Id: <879d30c4473ee31ad26b324cabf02eb37b3bd62a.1646406275.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 04 Mar 2022 15:04:30 +0000 Subject: [PATCH v3 3/7] builtin/fetch-pack: add --refetch option Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jonathan Tan , John Cai , Jeff Hostetler , Junio C Hamano , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Robert Coup , Robert Coup Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Robert Coup From: Robert Coup Add a refetch option to fetch-pack to force a full fetch. Use when applying a new partial clone filter to refetch all matching objects. Signed-off-by: Robert Coup --- Documentation/git-fetch-pack.txt | 4 ++++ builtin/fetch-pack.c | 4 ++++ remote-curl.c | 6 ++++++ 3 files changed, 14 insertions(+) diff --git a/Documentation/git-fetch-pack.txt b/Documentation/git-fetch-pack.txt index c9758847937..46747d5f429 100644 --- a/Documentation/git-fetch-pack.txt +++ b/Documentation/git-fetch-pack.txt @@ -101,6 +101,10 @@ be in a separate packet, and the list must end with a flush packet. current shallow boundary instead of from the tip of each remote branch history. +--refetch:: + Skips negotiating commits with the server in order to fetch all matching + objects. Use to reapply a new partial clone blob/tree filter. + --no-progress:: Do not show the progress. diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c index c2d96f4c89a..1f8aec97d47 100644 --- a/builtin/fetch-pack.c +++ b/builtin/fetch-pack.c @@ -153,6 +153,10 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix) args.from_promisor = 1; continue; } + if (!strcmp("--refetch", arg)) { + args.refetch = 1; + continue; + } if (skip_prefix(arg, ("--" CL_ARG__FILTER "="), &arg)) { parse_list_objects_filter(&args.filter_options, arg); continue; diff --git a/remote-curl.c b/remote-curl.c index 0dabef2dd7c..fc75600d4c6 100644 --- a/remote-curl.c +++ b/remote-curl.c @@ -43,6 +43,7 @@ struct options { /* see documentation of corresponding flag in fetch-pack.h */ from_promisor : 1, + refetch : 1, atomic : 1, object_format : 1, force_if_includes : 1; @@ -198,6 +199,9 @@ static int set_option(const char *name, const char *value) } else if (!strcmp(name, "from-promisor")) { options.from_promisor = 1; return 0; + } else if (!strcmp(name, "refetch")) { + options.refetch = 1; + return 0; } else if (!strcmp(name, "filter")) { options.filter = xstrdup(value); return 0; @@ -1182,6 +1186,8 @@ static int fetch_git(struct discovery *heads, strvec_push(&args, "--deepen-relative"); if (options.from_promisor) strvec_push(&args, "--from-promisor"); + if (options.refetch) + strvec_push(&args, "--refetch"); if (options.filter) strvec_pushf(&args, "--filter=%s", options.filter); strvec_push(&args, url.buf); From patchwork Fri Mar 4 15:04:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robert Coup X-Patchwork-Id: 12769286 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B732FC433EF for ; Fri, 4 Mar 2022 15:04:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240052AbiCDPFf (ORCPT ); Fri, 4 Mar 2022 10:05:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44640 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240023AbiCDPF3 (ORCPT ); Fri, 4 Mar 2022 10:05:29 -0500 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6716A15B99A for ; Fri, 4 Mar 2022 07:04:41 -0800 (PST) Received: by mail-wr1-x436.google.com with SMTP id p9so13087148wra.12 for ; Fri, 04 Mar 2022 07:04:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=tCVQy7Pb+9IIEdTsd+jypO1SgAlFt6YjxzWgUr20SHg=; b=Zb7cuDs/X//iK3D9ya5l8uM/XsgHhwxq+5KCgSC+TGoti3UbH21uFO55I2eXd7zGrA 8V4vehvkm6+VOUAeAQv5W6LICdgX/oFN5/j9h23NzcG3TAahPYPaSq6Qrvm0qchQmOBi Qdsa5j7KRqaAk/9rqZ/i6UaALMeLNBmiSixGABzY2S3qEy34HMRFVAllvN94wXWK4WfB G6GLofIinqijWHMY29XhDagEIkA5E7D11Jqr2d++nIBJZfnQxerMZ7fiRLVBZaH+xeLP ABMq2kUtJIgidKFm1JAY9jfFUCefmhssSobRih6NByoPH2ZaKXOeDKuVJDMtVHsNE9Cm uxyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=tCVQy7Pb+9IIEdTsd+jypO1SgAlFt6YjxzWgUr20SHg=; b=0+ny0zN4L7Ih8m9DFrGO8rrs/w7OUorjU87fIkgVou6j9BsR3dLZqGG+aHYh9u1MRR ar3/wTqVF/9/LVh9hd7oiRpJ9eRSETi/FIerqPNa1hF4IRTIUuebHh83R9cvXAQPR3Gi 2PWdH7bdCyqrZtrv3IAIb6OHDwo/oECi35jgtec9BeLhsfJ/OrZH3B2ML+9lFGaBScQG BwRPVc2wujhYXEzmv0c5U6ny6ZrP5P4rGUl8sgH6RbAQrCxyZaSctbrHHnKaI4Ym/Nfg pRcb0fQTUkkyEjE07NiwSulLM6+hRJ9eHl7kdIaHCGRKZr110C8jAo+2ZctdRhnJczDd Tycg== X-Gm-Message-State: AOAM532cB3j4cmqRqqzp3IylMzZ8mMm3QLL9N1KFEqcWsj60jkcDaZGS c9gUIJ3Iqb7OBECTCbcXTD9ZORN/2Js= X-Google-Smtp-Source: ABdhPJw/uvV4fmRmaOptCUKTsQeHj1lizLUVsinEmc1T+7C2iYw+FfAZg6lBk4uK4nbOc1hM8/GGuw== X-Received: by 2002:a05:6000:1c1c:b0:1ef:e989:260 with SMTP id ba28-20020a0560001c1c00b001efe9890260mr17079774wrb.668.1646406279701; Fri, 04 Mar 2022 07:04:39 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n15-20020a05600c4f8f00b003842f011bc5sm8621087wmq.2.2022.03.04.07.04.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Mar 2022 07:04:39 -0800 (PST) Message-Id: In-Reply-To: References: Date: Fri, 04 Mar 2022 15:04:31 +0000 Subject: [PATCH v3 4/7] fetch: add --refetch option Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jonathan Tan , John Cai , Jeff Hostetler , Junio C Hamano , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Robert Coup , Robert Coup Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Robert Coup From: Robert Coup Teach fetch and transports the --refetch option to force a full fetch without negotiating common commits with the remote. Use when applying a new partial clone filter to refetch all matching objects. Signed-off-by: Robert Coup --- Documentation/fetch-options.txt | 9 +++++++++ builtin/fetch.c | 15 ++++++++++++++- transport-helper.c | 3 +++ transport.c | 4 ++++ transport.h | 4 ++++ 5 files changed, 34 insertions(+), 1 deletion(-) diff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt index f9036831898..21a247abfa4 100644 --- a/Documentation/fetch-options.txt +++ b/Documentation/fetch-options.txt @@ -163,6 +163,15 @@ endif::git-pull[] behavior for a remote may be specified with the remote..tagOpt setting. See linkgit:git-config[1]. +ifndef::git-pull[] +--refetch:: + Instead of negotiating with the server to avoid transferring commits and + associated objects that are already present locally, this option fetches + all objects as a fresh clone would. Use this to reapply a partial clone + filter from configuration or using `--filter=` when the filter + definition has changed. +endif::git-pull[] + --refmap=:: When fetching refs listed on the command line, use the specified refspec (can be given more than once) to map the diff --git a/builtin/fetch.c b/builtin/fetch.c index 95832ba1dfd..f7bcf6fa64d 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -59,7 +59,7 @@ static int prune_tags = -1; /* unspecified */ static int all, append, dry_run, force, keep, multiple, update_head_ok; static int write_fetch_head = 1; -static int verbosity, deepen_relative, set_upstream; +static int verbosity, deepen_relative, set_upstream, refetch; static int progress = -1; static int enable_auto_gc = 1; static int tags = TAGS_DEFAULT, unshallow, update_shallow, deepen; @@ -190,6 +190,9 @@ static struct option builtin_fetch_options[] = { OPT_SET_INT_F(0, "unshallow", &unshallow, N_("convert to a complete repository"), 1, PARSE_OPT_NONEG), + OPT_SET_INT_F(0, "refetch", &refetch, + N_("re-fetch without negotiating common commits"), + 1, PARSE_OPT_NONEG), { OPTION_STRING, 0, "submodule-prefix", &submodule_prefix, N_("dir"), N_("prepend this to submodule path output"), PARSE_OPT_HIDDEN }, OPT_CALLBACK_F(0, "recurse-submodules-default", @@ -1296,6 +1299,14 @@ static int check_exist_and_connected(struct ref *ref_map) if (deepen) return -1; + /* + * Similarly, if we need to refetch, we always want to perform a full + * fetch ignoring existing objects. + */ + if (refetch) + return -1; + + /* * check_connected() allows objects to merely be promised, but * we need all direct targets to exist. @@ -1492,6 +1503,8 @@ static struct transport *prepare_transport(struct remote *remote, int deepen) set_option(transport, TRANS_OPT_DEEPEN_RELATIVE, "yes"); if (update_shallow) set_option(transport, TRANS_OPT_UPDATE_SHALLOW, "yes"); + if (refetch) + set_option(transport, TRANS_OPT_REFETCH, "yes"); if (filter_options.choice) { const char *spec = expand_list_objects_filter_spec(&filter_options); diff --git a/transport-helper.c b/transport-helper.c index a0297b0986c..b4dbbabb0c2 100644 --- a/transport-helper.c +++ b/transport-helper.c @@ -715,6 +715,9 @@ static int fetch_refs(struct transport *transport, if (data->transport_options.update_shallow) set_helper_option(transport, "update-shallow", "true"); + if (data->transport_options.refetch) + set_helper_option(transport, "refetch", "true"); + if (data->transport_options.filter_options.choice) { const char *spec = expand_list_objects_filter_spec( &data->transport_options.filter_options); diff --git a/transport.c b/transport.c index 253d6671b1f..e2817b7a715 100644 --- a/transport.c +++ b/transport.c @@ -243,6 +243,9 @@ static int set_git_option(struct git_transport_options *opts, list_objects_filter_die_if_populated(&opts->filter_options); parse_list_objects_filter(&opts->filter_options, value); return 0; + } else if (!strcmp(name, TRANS_OPT_REFETCH)) { + opts->refetch = !!value; + return 0; } else if (!strcmp(name, TRANS_OPT_REJECT_SHALLOW)) { opts->reject_shallow = !!value; return 0; @@ -377,6 +380,7 @@ static int fetch_refs_via_pack(struct transport *transport, args.update_shallow = data->options.update_shallow; args.from_promisor = data->options.from_promisor; args.filter_options = data->options.filter_options; + args.refetch = data->options.refetch; args.stateless_rpc = transport->stateless_rpc; args.server_options = transport->server_options; args.negotiation_tips = data->options.negotiation_tips; diff --git a/transport.h b/transport.h index a0bc6a1e9eb..12bc08fc339 100644 --- a/transport.h +++ b/transport.h @@ -16,6 +16,7 @@ struct git_transport_options { unsigned update_shallow : 1; unsigned reject_shallow : 1; unsigned deepen_relative : 1; + unsigned refetch : 1; /* see documentation of corresponding flag in fetch-pack.h */ unsigned from_promisor : 1; @@ -216,6 +217,9 @@ void transport_check_allowed(const char *type); /* Filter objects for partial clone and fetch */ #define TRANS_OPT_LIST_OBJECTS_FILTER "filter" +/* Refetch all objects without negotiating */ +#define TRANS_OPT_REFETCH "refetch" + /* Request atomic (all-or-nothing) updates when pushing */ #define TRANS_OPT_ATOMIC "atomic" From patchwork Fri Mar 4 15:04:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robert Coup X-Patchwork-Id: 12769287 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1561C433EF for ; Fri, 4 Mar 2022 15:05:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240063AbiCDPFw (ORCPT ); Fri, 4 Mar 2022 10:05:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44722 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240043AbiCDPFa (ORCPT ); Fri, 4 Mar 2022 10:05:30 -0500 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3FBED15B9B7 for ; Fri, 4 Mar 2022 07:04:42 -0800 (PST) Received: by mail-wr1-x42d.google.com with SMTP id bk29so13133961wrb.4 for ; Fri, 04 Mar 2022 07:04:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=d2lQjiwQo/d2+l3J+KrjN/UX3KHYu6mnvr6cgsfzWJw=; b=h4scN1PkQrUIo9FYCI8hiWNLgTnZtbx9w5yR9QM1t2AJl0SKAO8FksqZfPy25OZW1Z WZlYetulT/VyHPYCiCwRmMOEk+vEEO6mJXn4DNPSN+7kYMJsxBujeM2W12U2otkcgNNy /ib5ltrck4Ey//8s2q/IRs4brF20KG8pVt+imPCgkVaFk1OURsGbzHo/3ECoZJv1CFNO ORFU302L3ibzM/sE80MmMYVzQeOsKLRNoOhtFBLUTJpp8vlSEOxQTjIcxrhLkPj3AuMG LZf/XGDIOoDZRoey5C+tRmaHLSXTjLZ9AawOtXnRQmV/0+TGefIPxPc6JTRItxNW3mNE mjaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=d2lQjiwQo/d2+l3J+KrjN/UX3KHYu6mnvr6cgsfzWJw=; b=fW9YXDBd9It9Kce6GcqJdrK1v3ji7SlYalDr9d6MJAvS/s5bWxEWuLecaPh4grT7hF QIN4wY/F+o8QnYSKeJYECAYhAKWoL1AJff22vMsW/qH3hAW8BQ06KxfU4hFglwiYJMof Kkpax4tvwahR3s1fGqwaYmmy7A9agUUQxAGFFflKEIONaz69DiP2w3y6y/YIQin1Pob/ MrshksBQ3xOuojyJ8LENCIUwk/YDrCPojZRIx9TLv/GTerZgG0pe7SY1W7w49G3YtoPK 6lCWUX19Fdp75+S3LhG8CfpHOc9RDEarguOHiiiSPAAOgo4/GyvmQitxaZizMF7+c4fu j3bw== X-Gm-Message-State: AOAM533rs0ZY3mJ1odsR2RniH75v5U0/a644usP4u1ZXjghv6n1GgY8e JDvI8PDYK+saSYUrOqtNGpJ4QhPbUbU= X-Google-Smtp-Source: ABdhPJzrLrDE213y8NWvuxIHnNUYgSAUzkObRQ5vU1nX7zeoqF2phBNBK6C5YOWuQIWUBd3pMku4vQ== X-Received: by 2002:adf:eb4d:0:b0:1ed:c1f7:a951 with SMTP id u13-20020adfeb4d000000b001edc1f7a951mr29437881wrn.454.1646406280595; Fri, 04 Mar 2022 07:04:40 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id h36-20020a05600c49a400b00382aa0b1619sm4729902wmp.45.2022.03.04.07.04.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Mar 2022 07:04:40 -0800 (PST) Message-Id: <01f22e784a527f11c34703241bdb63b3d99f6fe8.1646406275.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 04 Mar 2022 15:04:32 +0000 Subject: [PATCH v3 5/7] t5615-partial-clone: add test for fetch --refetch Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jonathan Tan , John Cai , Jeff Hostetler , Junio C Hamano , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Robert Coup , Robert Coup Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Robert Coup From: Robert Coup Add a test for doing a refetch to apply a changed partial clone filter under protocol v0 and v2. Signed-off-by: Robert Coup --- t/t5616-partial-clone.sh | 52 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 51 insertions(+), 1 deletion(-) diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh index 34469b6ac10..87ebf4b0b1c 100755 --- a/t/t5616-partial-clone.sh +++ b/t/t5616-partial-clone.sh @@ -166,6 +166,56 @@ test_expect_success 'manual prefetch of missing objects' ' test_line_count = 0 observed.oids ' +# create new commits in "src" repo to establish a history on file.4.txt +# and push to "srv.bare". +test_expect_success 'push new commits to server for file.4.txt' ' + for x in a b c d e f + do + echo "Mod file.4.txt $x" >src/file.4.txt && + if list_contains "a,b" "$x"; then + printf "%10000s" X >>src/file.4.txt + fi && + if list_contains "c,d" "$x"; then + printf "%20000s" X >>src/file.4.txt + fi && + git -C src add file.4.txt && + git -C src commit -m "mod $x" || return 1 + done && + git -C src push -u srv main +' + +# Do partial fetch to fetch smaller files; then verify that without --refetch +# applying a new filter does not refetch missing large objects. Then use +# --refetch to apply the new filter on existing commits. Test it under both +# protocol v2 & v0. +test_expect_success 'apply a different filter using --refetch' ' + git -C pc1 fetch --filter=blob:limit=999 origin && + git -C pc1 rev-list --quiet --objects --missing=print \ + main..origin/main >observed && + test_line_count = 4 observed && + + git -C pc1 fetch --filter=blob:limit=19999 --refetch origin && + git -C pc1 rev-list --quiet --objects --missing=print \ + main..origin/main >observed && + test_line_count = 2 observed && + + git -c protocol.version=0 -C pc1 fetch --filter=blob:limit=29999 \ + --refetch origin && + git -C pc1 rev-list --quiet --objects --missing=print \ + main..origin/main >observed && + test_line_count = 0 observed +' + +test_expect_success 'fetch --refetch works with a shallow clone' ' + git clone --no-checkout --depth=1 --filter=blob:none "file://$(pwd)/srv.bare" pc1s && + git -C pc1s rev-list --objects --missing=print HEAD >observed && + test_line_count = 6 observed && + + GIT_TRACE=1 git -C pc1s fetch --filter=blob:limit=999 --refetch origin && + git -C pc1s rev-list --objects --missing=print HEAD >observed && + test_line_count = 6 observed +' + test_expect_success 'partial clone with transfer.fsckobjects=1 works with submodules' ' test_create_repo submodule && test_commit -C submodule mycommit && @@ -225,7 +275,7 @@ test_expect_success 'use fsck before and after manually fetching a missing subtr # Auto-fetch all remaining trees and blobs with --missing=error git -C dst rev-list --missing=error --objects main >fetched_objects && - test_line_count = 70 fetched_objects && + test_line_count = 88 fetched_objects && awk -f print_1.awk fetched_objects | xargs -n1 git -C dst cat-file -t >fetched_types && From patchwork Fri Mar 4 15:04:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Robert Coup X-Patchwork-Id: 12769288 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C9BBC433F5 for ; Fri, 4 Mar 2022 15:05:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240023AbiCDPFz (ORCPT ); Fri, 4 Mar 2022 10:05:55 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44834 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240049AbiCDPFc (ORCPT ); Fri, 4 Mar 2022 10:05:32 -0500 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 50BAB15C19B for ; Fri, 4 Mar 2022 07:04:43 -0800 (PST) Received: by mail-wr1-x42b.google.com with SMTP id e24so122730wrc.10 for ; Fri, 04 Mar 2022 07:04:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=eBFeyhJa1Ak/vERr0x00+QihR4ge9lNHluT1R3qiP3M=; b=qNlFcfrYymPfuThVo6d488pPDQXL4PRJAVu6zjj4UoZip+0s+oFHEZKf4qaGWeJb6x +zsXfwONJlW3yhn8EBTqOHCo3LzG+8Ir8iBfMeyS2qeNYjaqEQYVFahdvYB59QFuURnF sZTzOHwyrIO8g7I/g4GfbtB4mT2bqVocp7eutHzbY4DQmqfoMvNohZW4+H1JDKkYk/x2 AG3mMANX1eL0BMM8reJXFiXqxD69eTe9N9fbzWCSF6X8C3PHhuNTuhXHwgBUYXoiSGSG BCXhusabJ4wdC2+lTbnX3VZ7IsqlqJux8+/hu9Es08akNZrorMv6RuXHg6udC7U+6/Hd KuWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=eBFeyhJa1Ak/vERr0x00+QihR4ge9lNHluT1R3qiP3M=; b=1lNaiPxlRsj3dPCUWlskrmVJqqbVz4AJxs4PybVCQcHIvl74ktCnj8cL1KRUBiH5+K NQPbcMNRQ9k6fGu0+iQBv2zzRJJgfJvQqk6JoLFj1joZJm+CEauhIUTwMUvRmXfzuKYZ fNr19LJvyRdXfr4Og23s3WYZNjM9j3yzjUUuP+sFx90hep+5D0We8sqhwZyIrNVon3z8 wSEDAcQi4qPelpkeBvXMG1mgp8J7wpDNkkIAj7FVYS4pytK8Z8aDN66XxFJwl0B2FrOv x2y6gvrZ17ORilwuHj0B93Asfw41fERWWYGWlaMYVyt8BxYYN3i3lEb4wVmWjBxWz9tJ p87A== X-Gm-Message-State: AOAM532LKct3IAsSjrlVcqVkgUdbgOhH7kK2iTpnwn4TuHXpQv/x76+O EHskJBN4Ggx8t8izo/HCQeXzvHIaffw= X-Google-Smtp-Source: ABdhPJwszzGeHWS4hDzpnWGbmQw5Yypt9n+Vff2enguWEV9C/1+XrrKae7VaQfKgf6NNQ1q6iKtGKQ== X-Received: by 2002:a5d:6d4e:0:b0:1e3:3636:b104 with SMTP id k14-20020a5d6d4e000000b001e33636b104mr30854981wri.605.1646406281558; Fri, 04 Mar 2022 07:04:41 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id y4-20020adff144000000b001f022290737sm4668525wro.6.2022.03.04.07.04.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Mar 2022 07:04:41 -0800 (PST) Message-Id: <310466259878bcd32e09092b8de1db0b3e500c2a.1646406275.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 04 Mar 2022 15:04:33 +0000 Subject: [PATCH v3 6/7] fetch: after refetch, encourage auto gc repacking Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jonathan Tan , John Cai , Jeff Hostetler , Junio C Hamano , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Robert Coup , Robert Coup Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Robert Coup From: Robert Coup After invoking `fetch --refetch`, the object db will likely contain many duplicate objects. If auto-maintenance is enabled, invoke it with appropriate settings to encourage repacking/consolidation. * gc.autoPackLimit: unless this is set to 0 (disabled), override the value to 1 to force pack consolidation. * maintenance.incremental-repack.auto: unless this is set to 0, override the value to -1 to force incremental repacking. Signed-off-by: Robert Coup --- Documentation/fetch-options.txt | 3 ++- builtin/fetch.c | 19 ++++++++++++++++++- t/t5616-partial-clone.sh | 29 +++++++++++++++++++++++++++++ 3 files changed, 49 insertions(+), 2 deletions(-) diff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt index 21a247abfa4..49ae48dca32 100644 --- a/Documentation/fetch-options.txt +++ b/Documentation/fetch-options.txt @@ -169,7 +169,8 @@ ifndef::git-pull[] associated objects that are already present locally, this option fetches all objects as a fresh clone would. Use this to reapply a partial clone filter from configuration or using `--filter=` when the filter - definition has changed. + definition has changed. Automatic post-fetch maintenance will perform + object database pack consolidation to remove any duplicate objects. endif::git-pull[] --refmap=:: diff --git a/builtin/fetch.c b/builtin/fetch.c index f7bcf6fa64d..1557e8d57c5 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -2226,8 +2226,25 @@ int cmd_fetch(int argc, const char **argv, const char *prefix) NULL); } - if (enable_auto_gc) + if (enable_auto_gc) { + if (refetch) { + /* + * Hint auto-maintenance strongly to encourage repacking, + * but respect config settings disabling it. + */ + int opt_val; + if (git_config_get_int("gc.autopacklimit", &opt_val)) + opt_val = -1; + if (opt_val != 0) + git_config_push_parameter("gc.autoPackLimit=1"); + + if (git_config_get_int("maintenance.incremental-repack.auto", &opt_val)) + opt_val = -1; + if (opt_val != 0) + git_config_push_parameter("maintenance.incremental-repack.auto=-1"); + } run_auto_maintenance(verbosity < 0); + } cleanup: string_list_clear(&list, 0); diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh index 87ebf4b0b1c..4a3778d04a8 100755 --- a/t/t5616-partial-clone.sh +++ b/t/t5616-partial-clone.sh @@ -216,6 +216,35 @@ test_expect_success 'fetch --refetch works with a shallow clone' ' test_line_count = 6 observed ' +test_expect_success 'fetch --refetch triggers repacking' ' + GIT_TRACE2_CONFIG_PARAMS=gc.autoPackLimit,maintenance.incremental-repack.auto && + export GIT_TRACE2_CONFIG_PARAMS && + + GIT_TRACE2_EVENT="$PWD/trace1.event" \ + git -C pc1 fetch --refetch origin && + test_subcommand git maintenance run --auto --no-quiet X-Patchwork-Id: 12769289 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DB1F5C433EF for ; Fri, 4 Mar 2022 15:05:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240059AbiCDPF5 (ORCPT ); Fri, 4 Mar 2022 10:05:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44836 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240050AbiCDPFc (ORCPT ); Fri, 4 Mar 2022 10:05:32 -0500 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E442D15C1B6 for ; Fri, 4 Mar 2022 07:04:43 -0800 (PST) Received: by mail-wr1-x436.google.com with SMTP id j17so13184459wrc.0 for ; Fri, 04 Mar 2022 07:04:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=ICsNrHzXd4RZPq+Z7tpWCL81jwFbzmJwcvelPW681Yg=; b=VXVU3+YfDofTAqJM1HGITjFSLNvfjraa/xX3wFCNeBZSbvezjPQe0Z7A1IndiZoL35 MFTAEUNNns3s++O9W5nDNqQuD8TwlHdwKOEdOAL+ZJ1Etc0Mwxbwhwkwzjof126j0Sln oQ1YlpGxENWXDbQHXAuuqr7fVjPByPFlrlPAftdbJUQfdUjmzoLfvQZiDfek45xbXEmA hD+D88yWz4dLrdsRfRizCkOOCh/yclHB3XphDgAUrfkxdzMy6k36Cce47Dr1b4qDbYVM tC98ewdtadBVF6vXKViIzNe5LsjxnsVvM2K8Y0T5KQSZkTQD9zs4gAmDOV3MKvUW/Xx3 FQyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=ICsNrHzXd4RZPq+Z7tpWCL81jwFbzmJwcvelPW681Yg=; b=Y9884HFqpZPDgT8+mH06YhuO+qE4+MKIuLREN/65qTOSa4J9U1eJAFIO82jRPv+H2O QFtLJEQs2hH9VWOgkMg7+JjcQJbRFpMZ+XgxXK1dGBmX3JKZmldotHwqBqUdww/QyTNG I4QSwdjh+IGRL8ARA5iB1u6C/QfUh4qX2g+ylT97DJqrXg/CYKqDv5Nrl5KvEqQlzt3W g4YY9srDA1Hz5CUMk7wrxhIDrbAascUajt85mqsgxn29+XodmpffdRddD97RtS/6dipl 5QUmtu5fEdKEFrOWvIGf30DN3VU38qNcMusWznAmGVGVqahOLQZJYvo+M/tZA4kzjPvG argg== X-Gm-Message-State: AOAM530z55Z8VoD/Tu7ssfh7v7Bu3WvPcy2sgbpIRsR9c70iwY1KWz7G u6fc0Tc9Kum4TT6A+KiwgE2R3n9qXVA= X-Google-Smtp-Source: ABdhPJyTtPg9CDxdYqmHHZOzb7o02A1eY9kY8+xSd08cUXbAS0DoHVyXjXu7XjDK2JywVmZnbzlFeQ== X-Received: by 2002:adf:c792:0:b0:1f0:769:9ef3 with SMTP id l18-20020adfc792000000b001f007699ef3mr13124812wrg.336.1646406282393; Fri, 04 Mar 2022 07:04:42 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id z3-20020a1cf403000000b0037d1f4a2201sm5257147wma.21.2022.03.04.07.04.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 04 Mar 2022 07:04:41 -0800 (PST) Message-Id: In-Reply-To: References: Date: Fri, 04 Mar 2022 15:04:34 +0000 Subject: [PATCH v3 7/7] doc/partial-clone: mention --refetch fetch option Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jonathan Tan , John Cai , Jeff Hostetler , Junio C Hamano , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Robert Coup , Robert Coup Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Robert Coup From: Robert Coup Document it for partial clones as a means to apply a new filter. Signed-off-by: Robert Coup --- Documentation/technical/partial-clone.txt | 3 +++ 1 file changed, 3 insertions(+) diff --git a/Documentation/technical/partial-clone.txt b/Documentation/technical/partial-clone.txt index a0dd7c66f24..99f0eb30406 100644 --- a/Documentation/technical/partial-clone.txt +++ b/Documentation/technical/partial-clone.txt @@ -181,6 +181,9 @@ Fetching Missing Objects currently fetches all objects referred to by the requested objects, even though they are not necessary. +- Fetching with `--refetch` will request a complete new filtered packfile from + the remote, which can be used to change a filter without needing to + dynamically fetch missing objects. Using many promisor remotes ---------------------------