From patchwork Fri May 21 11:59:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12272945 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7E068C433B4 for ; Fri, 21 May 2021 11:59:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 451B06008E for ; Fri, 21 May 2021 11:59:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234806AbhEUMBP (ORCPT ); Fri, 21 May 2021 08:01:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35858 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230138AbhEUMBJ (ORCPT ); Fri, 21 May 2021 08:01:09 -0400 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A3F49C061763 for ; Fri, 21 May 2021 04:59:45 -0700 (PDT) Received: by mail-wm1-x32b.google.com with SMTP id z85-20020a1c7e580000b029017a76f3afbaso4476700wmc.2 for ; Fri, 21 May 2021 04:59:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=tnv8/RvaEXbX/IGlvdgT6jdK85+oA3TpgnT3UjIuqEE=; b=DJqtuulIFMaJcv0NaYeeZl/Bkav4zmMRilPcwbZQ87dkUgyUskvZ1K8vPo+brB+ZKa JXGd7jXu675i64jeuD6KIVGsh9eFLh6z/pwsTxv3mtPx0YbMVSMZnzZ2tnDagYycpPOJ pmtgWqe7UlWJC42S58x/cSqcNIpp7i+EzHNhyOj/psYsqyifLZtjxaI0ee194rgFVyhZ FpA1TN4miIqTMiKkKt/uhahnc/XBqXbMhw/8guIUrxRbvOgDUzhaiXgrEr1Z268kAysD Jf64AMfMh8lTtmLFrnVcWavOkIUq8qTDPJvpBqaLS3Ii+hqWojmYMXQ28lH+/RHq9sqJ 8HLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=tnv8/RvaEXbX/IGlvdgT6jdK85+oA3TpgnT3UjIuqEE=; b=Wfire/EACSOTVqgt2NYbZhBLnQkx9Bm/45+bbFfrb5RjdbOgkBt/Uf8aVShagyq0k9 x6VtmnAGjrhj25XIQTlsQKhdGZEvJg5fPqgMGE/BO/oin7rHBb4BN9/+5frrGRLlhiFY lGfDZtKL6RBW7khtZ1rZsFLmIAi7VLHgDlnuPb8JOg50GB+sz5NinhW6Dr5Mza2fEW2F kRaUBMUGhKPo9BbW23MOaqSK8JhuuP8IhhqgIhJs8vwKg7vM/ukU6ReoiyTSe9ZUkxI/ 5O1+qQ4FIT79CMEgFCo2TSeO0PZUv/qGgCEOUhWQt6U9sPWcVVos40eDHJAcGjZZFZ+1 2Eqw== X-Gm-Message-State: AOAM532hHrPvj+BdkkpNEjQ/imUB9ZuVsQIOAuZxoMb9GyMgKmwxti9o h83g7JUs9+prN5qm+u4E1ov7WA7VkHo= X-Google-Smtp-Source: ABdhPJz0G0WVQVtUu3BQA3mcFIeyJL7VZA5mvq56PsPPeYnkPkvF6zHYDQf+H1gPrFaK4WIrMzxoCg== X-Received: by 2002:a7b:c7ca:: with SMTP id z10mr8603617wmk.185.1621598384327; Fri, 21 May 2021 04:59:44 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l8sm1858651wry.55.2021.05.21.04.59.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 May 2021 04:59:44 -0700 (PDT) Message-Id: <5a2ed3d1d701b11c241bf3855db3bb35f5e93994.1621598382.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 21 May 2021 11:59:30 +0000 Subject: [PATCH v4 01/12] sparse-index: skip indexes with unmerged entries Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The sparse-index format is designed to be compatible with merge conflicts, even those outside the sparse-checkout definition. The reason is that when converting a full index to a sparse one, a cache entry with nonzero stage will not be collapsed into a sparse directory entry. However, this behavior was not tested, and a different behavior within convert_to_sparse() fails in this scenario. Specifically, cache_tree_update() will fail when unmerged entries exist. convert_to_sparse_rec() uses the cache-tree data to recursively walk the tree structure, but also to compute the OIDs used in the sparse-directory entries. Add an index scan to convert_to_sparse() that will detect if these merge conflict entries exist and skip the conversion before trying to update the cache-tree. This is marked as NEEDSWORK because this can be removed with a suitable update to cache_tree_update() or a similar method that can construct a cache-tree with invalid nodes, but still allow creating the nodes necessary for creating sparse directory entries. It is possible that in the future we will not need to make such an update, since if we do not expand a sparse-index into a full one, this conversion does not need to happen. Thus, this can be deferred until the merge machinery is made to integrate with the sparse-index. Signed-off-by: Derrick Stolee --- sparse-index.c | 18 ++++++++++++++++++ t/t1092-sparse-checkout-compatibility.sh | 22 ++++++++++++++++++++++ 2 files changed, 40 insertions(+) diff --git a/sparse-index.c b/sparse-index.c index 6f21397e2ee0..1b49898d0cb7 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -125,6 +125,17 @@ int set_sparse_index_config(struct repository *repo, int enable) return res; } +static int index_has_unmerged_entries(struct index_state *istate) +{ + int i; + for (i = 0; i < istate->cache_nr; i++) { + if (ce_stage(istate->cache[i])) + return 1; + } + + return 0; +} + int convert_to_sparse(struct index_state *istate) { int test_env; @@ -161,6 +172,13 @@ int convert_to_sparse(struct index_state *istate) return -1; } + /* + * NEEDSWORK: If we have unmerged entries, then stay full. + * Unmerged entries prevent the cache-tree extension from working. + */ + if (index_has_unmerged_entries(istate)) + return 0; + if (cache_tree_update(istate, 0)) { warning(_("unable to update cache-tree, staying full")); return -1; diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 12e6c453024f..4f2f09b53a32 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -352,6 +352,28 @@ test_expect_success 'merge with outside renames' ' done ' +# Sparse-index fails to convert the index in the +# final 'git cherry-pick' command. +test_expect_success 'cherry-pick with conflicts' ' + init_repos && + + write_script edit-conflict <<-\EOF && + echo $1 >conflict + EOF + + test_all_match git checkout -b to-cherry-pick && + run_on_all ../edit-conflict ABC && + test_all_match git add conflict && + test_all_match git commit -m "conflict to pick" && + + test_all_match git checkout -B base HEAD~1 && + run_on_all ../edit-conflict DEF && + test_all_match git add conflict && + test_all_match git commit -m "conflict in base" && + + test_all_match test_must_fail git cherry-pick to-cherry-pick +' + test_expect_success 'clean' ' init_repos && From patchwork Fri May 21 11:59:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12272951 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C4EF0C433ED for ; Fri, 21 May 2021 12:00:05 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A7EA8613DA for ; Fri, 21 May 2021 12:00:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233131AbhEUMB1 (ORCPT ); Fri, 21 May 2021 08:01:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35862 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233817AbhEUMBL (ORCPT ); Fri, 21 May 2021 08:01:11 -0400 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 289AFC0613CE for ; Fri, 21 May 2021 04:59:46 -0700 (PDT) Received: by mail-wr1-x42b.google.com with SMTP id n2so20897764wrm.0 for ; Fri, 21 May 2021 04:59:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=QTzuOgoXGKpn026RiRL8qt8jaloUIGodP1E75w97/mM=; b=maEYn8ggn8Hp5Pi8yXz1kVDITi+znkoOdSOu2if1IB7oAYApdssdonSJpQ+Qe1OakX cIE7axx5/TmSnmDpiuHg+zEjqX+pHW7f8qJ+J23yNUPEGQltEpOsQ0b2DPVwTSyv9NTf OadH9fyF2J0yYUTRfpu1aXY2FRP8b7XOfmVxnNkrx6vmc+oCKfO09GkezdqHaLqX+Vg5 TsXqIduSYjowCp7M8q2QqSK9U6Z4qL6FHLE6OAu1aUG0KAemCMR80ps8gq+WS80E86SC ju4nRHvElxC6In9wTxqwhEPcX1US3aZh3QKhL8TpPCcFbycRJ8Rnc7HD3rxeNUQGKuEZ CzJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=QTzuOgoXGKpn026RiRL8qt8jaloUIGodP1E75w97/mM=; b=soD1sZ7bZ4UNkQDF15yrLtnErdBogYkBKuJj00Cjqeo84DDljCsVD4HOSmnYW5Ox7n dP2bfWSKLVakf/DmKcAa3OfBlzooCfouA8TNlvitOQZFMSf8IviXPOVTEpnTO25z6xbh /3dLsutUxQWN3PD4Yj5OjZu7Vq5vNEP8KVFP3cLn2cgwLddU2Te7EUqbIuDWA1Ts8O/C 8gLydmZLTbBr5YQpHYLdNhJ4+pBLNQN6LTTqhkPGexLWfCf/AmKS63mr15x7k2ANVtf5 Fdn0x1nJDhRsenrTLIX/ud+ubcbvK1L9vxa9L6/ek0srCtvQJxbU9lcxJNegYV4YM1g9 X0qQ== X-Gm-Message-State: AOAM530K5SzhIi0w8SR8j8krjUKm2pxdP/i5jYrgHSJOYJVJTyExgFgX 2Rsmq6z5cKi9owqUvSSrKIamKhT5R7U= X-Google-Smtp-Source: ABdhPJxEvSHA0dlSud+jbn9j2mAsLLGgXP41G+rSkO9eSWcGZYtOn8CoSfF95W78mSI8rrDPp5VQog== X-Received: by 2002:a5d:524b:: with SMTP id k11mr9369627wrc.292.1621598384862; Fri, 21 May 2021 04:59:44 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id j18sm5410172wmq.27.2021.05.21.04.59.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 May 2021 04:59:44 -0700 (PDT) Message-Id: <8aa41e749471df3bd9d593b8f55db6506eafea12.1621598382.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 21 May 2021 11:59:31 +0000 Subject: [PATCH v4 02/12] sparse-index: include EXTENDED flag when expanding Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee When creating a full index from a sparse one, we create cache entries for every blob within a given sparse directory entry. These are correctly marked with the CE_SKIP_WORKTREE flag, but they must also be marked with the CE_EXTENDED flag to ensure that the skip-worktree bit is correctly written to disk in the case that the index is not converted back down to a sparse-index. Signed-off-by: Derrick Stolee --- sparse-index.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparse-index.c b/sparse-index.c index 1b49898d0cb7..b2b3fbd75050 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -222,7 +222,7 @@ static int add_path_to_index(const struct object_id *oid, strbuf_addstr(base, path); ce = make_cache_entry(istate, mode, oid, base->buf, 0, 0); - ce->ce_flags |= CE_SKIP_WORKTREE; + ce->ce_flags |= CE_SKIP_WORKTREE | CE_EXTENDED; set_index_entry(istate, istate->cache_nr++, ce); strbuf_setlen(base, len); From patchwork Fri May 21 11:59:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12272949 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 855E2C433B4 for ; Fri, 21 May 2021 12:00:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 616926008E for ; Fri, 21 May 2021 12:00:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235065AbhEUMBV (ORCPT ); Fri, 21 May 2021 08:01:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35870 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233754AbhEUMBL (ORCPT ); Fri, 21 May 2021 08:01:11 -0400 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BA358C0613ED for ; Fri, 21 May 2021 04:59:46 -0700 (PDT) Received: by mail-wm1-x32d.google.com with SMTP id l11-20020a05600c4f0bb029017a7cd488f5so4371407wmq.0 for ; Fri, 21 May 2021 04:59:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=JgmxC3ji1LdwWDcaPHjhS9lC0LNmteE5DJe06eKmBqM=; b=rSliofy1Ct9+U3VBWZiDOesr+WcsEObHZEW90vSfwExB3bAOY1h/iqMHcpy1u5cOoq /Ta89MOhl92/g55DgjV9Liu9TyGgOF/09Po+gVVWfNqUpYJggxuZNc2fX4BKCrbRz3VH X2gTSq3Uc+8vL05HxkE63nX+6r+/OL8gfPiu5XBluVR7vd/15+H7Lzcx12/d74wa/TFg cI7xr6mNoV45oHkawEl+lW8YSq3B8/aMhOjGTtDHTDKso880QbWPEl6YyVe5yDqZCgsu /T82PyPqBQuxW5YDf6/GQHVFVcZthvYHQzdrz2SrgXrMdKKNYCx60pwnYl3GDH8n3FBl 1rbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=JgmxC3ji1LdwWDcaPHjhS9lC0LNmteE5DJe06eKmBqM=; b=E/FbWkA3vGqh+/bWsnjf7OTm32Bz1HTuppQrtkI2K7CwNr7Be/Sz7dlagFkxW3RIbi FakumEtmi+RAAWU9Xw+oWgk9Jnm86qd+QcwmGcLoA5R6ZQFJC0QHoAWb1ZVzMXKMYXOf cKL92WWvxDIecv9EtVoa2/XEI2jzN6vPjk5ay6/emYRIKtZiKjt8UYvYySXhwolWx2eq GVU6WvxiNzts6KuGoryOl17V/KGx3OKPjeXuaww240bzKHzwDUHFLdZgqAiTlcwd7xMa HxJyEUItu1JJmoW60RPMcSwgCCinehYXPTCCRXMmOd+fq6QwQbC5vxCbHz/M6Io3P82w 7dwQ== X-Gm-Message-State: AOAM533WUOb3R7hTflKAscVS40uSiYo8E2XTfg8QQnb55Q/Qgk2hXyss HwjUkIpoQe9mVzFfgz2M0mouEzZi47c= X-Google-Smtp-Source: ABdhPJyN8cg+fpqOpqh/5lE+xcSJhbkHiA1Gv/x4i7mEAVOZ3XqPuV4EpC/iNekMP6TGbWjjC3d1ug== X-Received: by 2002:a05:600c:b58:: with SMTP id k24mr8688174wmr.155.1621598385461; Fri, 21 May 2021 04:59:45 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id s199sm12292947wme.43.2021.05.21.04.59.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 May 2021 04:59:45 -0700 (PDT) Message-Id: <70971b1f9261f2623ad6d6393be1a8e191003eb4.1621598382.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 21 May 2021 11:59:32 +0000 Subject: [PATCH v4 03/12] t1092: expand repository data shape Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee As more features integrate with the sparse-index feature, more and more special cases arise that require different data shapes within the tree structure of the repository in order to demonstrate those cases. Add several interesting special cases all at once instead of sprinkling them across several commits. The interesting cases being added here are: * Add sparse-directory entries on both sides of directories within the sparse-checkout definition. * Add directories outside the sparse-checkout definition who have only one entry and are the first entry of a directory with multiple entries. Later tests will take advantage of these shapes, but they also deepen the tests that already exist. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 4f2f09b53a32..98257695979a 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -17,7 +17,7 @@ test_expect_success 'setup' ' echo "after folder1" >g && echo "after x" >z && mkdir folder1 folder2 deep x && - mkdir deep/deeper1 deep/deeper2 && + mkdir deep/deeper1 deep/deeper2 deep/before deep/later && mkdir deep/deeper1/deepest && echo "after deeper1" >deep/e && echo "after deepest" >deep/deeper1/e && @@ -25,10 +25,16 @@ test_expect_success 'setup' ' cp a folder2 && cp a x && cp a deep && + cp a deep/before && cp a deep/deeper1 && cp a deep/deeper2 && + cp a deep/later && cp a deep/deeper1/deepest && cp -r deep/deeper1/deepest deep/deeper2 && + mkdir deep/deeper1/0 && + mkdir deep/deeper1/0/0 && + touch deep/deeper1/0/1 && + touch deep/deeper1/0/0/0 && git add . && git commit -m "initial commit" && git checkout -b base && From patchwork Fri May 21 11:59:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12272953 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6E21C433B4 for ; Fri, 21 May 2021 12:00:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C99006008E for ; Fri, 21 May 2021 12:00:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235094AbhEUMB3 (ORCPT ); Fri, 21 May 2021 08:01:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35872 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233858AbhEUMBM (ORCPT ); Fri, 21 May 2021 08:01:12 -0400 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 56EC7C06138A for ; Fri, 21 May 2021 04:59:47 -0700 (PDT) Received: by mail-wr1-x42f.google.com with SMTP id j14so19071960wrq.5 for ; Fri, 21 May 2021 04:59:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=pAyAAv6l4SLCdrEeaFHhQFp08BencxoPH4VLlINqObY=; b=O9Aey+dMZuvoPa/GLKjYM1UuiSUTX1c/Pv0gbc+dmJE00AUbvcT3A/Mu2SQib3/Xxu ZztK3zWz73DAUVY7qA9u6QcK1pXQ5GyGorR0fnB87OsszCXbv4xmyGQ68e+dH3TtcCKC pErH5Qt6bSdNDgpbYYDRGrfqfQpEdY/WBqGLkCS+5TDXHI7xNXBjFjGNX6SiToCgWHrC ufkBDU6UTLanGoSczQmB7B7QZ8UBTi2qPsYajCOEF413BuEKNUGsSI/oNvH33CQcaIb6 +/v1KiOfKD7+9b4VcRGmJPJgHvPILrPu5JFEoXm4VwlNd81qCmY9fZywtYHQIjteq8T5 sveg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=pAyAAv6l4SLCdrEeaFHhQFp08BencxoPH4VLlINqObY=; b=iElzx4IYqnfmQUI86nCm4RLnqSU7ANegbi3jyLyYnGopMwq+/aUxliSTABP1QgUw79 Ei0WEUjss+5BEgSUnvvX9opk/s/HN0mGrwW2JQvjILsAm6Amjk3zWPT+bsE4TCRnmKps bUTjciUgIR88rOMUxb6huVu1eIsT7V4KmoUKV5Hk4qWv0yQ3fBvBgogYVvANkLje5vzc /9e7Lkj0YtXoC457129ZLgUwlyhku0YUKlwBADxkvlcu2xQbUJIeoS9in0X93zQePcGg MaQ4gQYqCIxk8OIIoPNkKvxKMSRTYDpQpOVklxlvrtkYiAyWecb0iYwvMLGAu3AubTpR ufRA== X-Gm-Message-State: AOAM531ychIrGPlxldK9PbV5K+Y6I2PhgAv2cal/SV6Sv3GvI1RXc/FO RvFE7qBdgyqO8alOrB4xwdtUi4f3o3o= X-Google-Smtp-Source: ABdhPJyjd1rH9CnYVnR4bplmzMtsIq+v5qdWmpWNZZNoLRYLgvl3cDv8TNgFtQn9Zx4dS4nDYOlTKQ== X-Received: by 2002:a5d:4e8c:: with SMTP id e12mr9283367wru.94.1621598385999; Fri, 21 May 2021 04:59:45 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l18sm1867288wrt.97.2021.05.21.04.59.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 May 2021 04:59:45 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Fri, 21 May 2021 11:59:33 +0000 Subject: [PATCH v4 04/12] t1092: add tests for status/add and sparse files Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Before moving to update 'git status' and 'git add' to work with sparse indexes, add an explicit test that ensures the sparse-index works the same as a normal sparse-checkout when the worktree contains directories and files outside of the sparse cone. Specifically, 'folder1/a' is a file in our test repo, but 'folder1' is not in the sparse cone. When 'folder1/a' is modified, the file is not shown as modified and adding it will fail. This is new behavior as of a20f704 (add: warn when asked to update SKIP_WORKTREE entries, 2021-04-08). Before that change, these adds would be silently ignored. Untracked files are fine: adding new files both with 'git add .' and 'git add folder1/' works just as in a full checkout. This may not be entirely desirable, but we are not intending to change behavior at the moment, only document it. A future change could alter the behavior to be more sensible, and this test could be modified to satisfy the new expected behavior. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 38 ++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 98257695979a..fba98d5484ae 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -238,6 +238,44 @@ test_expect_success 'add, commit, checkout' ' test_all_match git checkout - ' +test_expect_success 'status/add: outside sparse cone' ' + init_repos && + + # adding a "missing" file outside the cone should fail + test_sparse_match test_must_fail git add folder1/a && + + # folder1 is at HEAD, but outside the sparse cone + run_on_sparse mkdir folder1 && + cp initial-repo/folder1/a sparse-checkout/folder1/a && + cp initial-repo/folder1/a sparse-index/folder1/a && + + test_sparse_match git status && + + write_script edit-contents <<-\EOF && + echo text >>$1 + EOF + run_on_sparse ../edit-contents folder1/a && + run_on_all ../edit-contents folder1/new && + + test_sparse_match git status --porcelain=v2 && + + # This "git add folder1/a" fails with a warning + # in the sparse repos, differing from the full + # repo. This is intentional. + test_sparse_match test_must_fail git add folder1/a && + test_sparse_match test_must_fail git add --refresh folder1/a && + test_all_match git status --porcelain=v2 && + + test_all_match git add . && + test_all_match git status --porcelain=v2 && + test_all_match git commit -m folder1/new && + + run_on_all ../edit-contents folder1/newer && + test_all_match git add folder1/ && + test_all_match git status --porcelain=v2 && + test_all_match git commit -m folder1/newer +' + test_expect_success 'checkout and reset --hard' ' init_repos && From patchwork Fri May 21 11:59:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12272955 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 097F6C433B4 for ; Fri, 21 May 2021 12:00:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E450B6008E for ; Fri, 21 May 2021 12:00:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235112AbhEUMBf (ORCPT ); Fri, 21 May 2021 08:01:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35878 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229988AbhEUMBN (ORCPT ); Fri, 21 May 2021 08:01:13 -0400 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 12995C06138B for ; Fri, 21 May 2021 04:59:47 -0700 (PDT) Received: by mail-wm1-x32e.google.com with SMTP id n17-20020a7bc5d10000b0290169edfadac9so7266750wmk.1 for ; Fri, 21 May 2021 04:59:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=wFfpok35DynSJzLw2iHIzdshoI7yibKc8zfxCLq0Mn8=; b=rUZgyKv1hTYqhO/kr37dPT82t4cqf7+tjhUP5h9R/MRV8loSdAjfliC2i+rc6ay0dR R5Oij6suEGsQ4gziHnHbfgjtPiHtcO3/FOXPkN/WtAvxzTqr+C/rCm8t80rHk60kK+Vx MjCIN5y1XwbO6MDb+P7EiYJcJWFlq8wnPuTCUYg8LwsAiWC+FW4mF66PeUqPcqRNkx8s Pu/OJf+jKtwtgTnns0vP3QHr4Itfgi94QxIZ2C8mxSOj6q/KvdWsn3vXrpmSG7dsMDzl /ywwugr8XhAVhMbnncDjIZffaKe01o3ejPWHsEIRMGGN6F8YphMvM6lPASboBbGSUwZb Thww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=wFfpok35DynSJzLw2iHIzdshoI7yibKc8zfxCLq0Mn8=; b=Z5pVbik4U7N64g+I3zLj68vahhfyOAX3kgWS9m1wsAVEZENXn9fKKZpLOzBQcJmSYi gNxfGekcRD8IngCUhwRsNKWbJrvS4u9uToOIFSUqUznnTdsr8r+1A0QuNBdT3Gew14At pgq+VURASsWruIFXPC5Xxm8bKu3jO1/03T6/KU2YWxi/J/Q57FHF0YTpAIQPfkvgBRtr oW3LAy4UhXOa6IqWe4r9YBuJbz2WZ6zwdAQ5cW2aSqm18UXjJPhrH0vtc5006YE7OBuH 3MKXTkgf2SKdcTffiXMh9RFO7kbmXVUjgo2j8WxlvO6cQErYsHBHA9CLlQZKWzOuarPN c3lw== X-Gm-Message-State: AOAM533gpIkmQ/D7mYVFR7C+4NNTc0HIzbwPRUn09jDN6qlN14zMYxtT glUKBTNT3okO+LpKuMe7gJolE2s6q1E= X-Google-Smtp-Source: ABdhPJxuYtCsDYXc20vCsg4iEUWEgWPNlXq2FI8OZg9qHbgdn6agkCXKS9NQe0I96bS+hocAucuBOw== X-Received: by 2002:a05:600c:4fd0:: with SMTP id o16mr8839071wmq.137.1621598386503; Fri, 21 May 2021 04:59:46 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id b10sm1962371wrt.24.2021.05.21.04.59.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 May 2021 04:59:46 -0700 (PDT) Message-Id: <07a45b661c4a6ecd63ede19c9e058a8879069742.1621598382.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 21 May 2021 11:59:34 +0000 Subject: [PATCH v4 05/12] unpack-trees: preserve cache_bottom Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The cache_bottom member of 'struct unpack_trees_options' is used to track the range of index entries corresponding to a node of the cache tree. While recursing with traverse_by_cache_tree(), this value is preserved on the call stack using a local and then restored as that method returns. The mark_ce_used() method normally modifies the cache_bottom member when it refers to the marked cache entry. However, sparse directory entries are stored as nodes in the cache-tree data structure as of 2de37c53 (cache-tree: integrate with sparse directory entries, 2021-03-30). Thus, the cache_bottom will be modified as the cache-tree walk advances. Do not update it as well within mark_ce_used(). Signed-off-by: Derrick Stolee --- unpack-trees.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/unpack-trees.c b/unpack-trees.c index dddf106d5bd4..1067db19c9d2 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -586,6 +586,13 @@ static void mark_ce_used(struct cache_entry *ce, struct unpack_trees_options *o) { ce->ce_flags |= CE_UNPACKED; + /* + * If this is a sparse directory, don't advance cache_bottom. + * That will be advanced later using the cache-tree data. + */ + if (S_ISSPARSEDIR(ce->ce_mode)) + return; + if (o->cache_bottom < o->src_index->cache_nr && o->src_index->cache[o->cache_bottom] == ce) { int bottom = o->cache_bottom; From patchwork Fri May 21 11:59:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12272957 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B9BB2C433ED for ; Fri, 21 May 2021 12:00:20 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9FCFB6008E for ; Fri, 21 May 2021 12:00:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235128AbhEUMBj (ORCPT ); Fri, 21 May 2021 08:01:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35876 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233890AbhEUMBN (ORCPT ); Fri, 21 May 2021 08:01:13 -0400 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A23C1C06138C for ; Fri, 21 May 2021 04:59:48 -0700 (PDT) Received: by mail-wr1-x435.google.com with SMTP id q5so20809289wrs.4 for ; Fri, 21 May 2021 04:59:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=8BiWBmJ/VC9KkVb4hI/TM5y8+2cX+/ZhlTUXVP76k9Q=; b=HHtefpG8ETuk49kqGS+ZArKLOZmby3wzMH0mEoODLvFfGoAhaQpOQLX/Z/2jMjMZXb meMLu5kQ3vvX0nIlU0YGYAsZIElX1u0WIDDF7WZ6OOTZZSr78UdkLSU3SaRtHrsmUCcg giXBBnzRbqhO5y66oC2rt7miBSXzfWGpXI+Qrst1dbNjqao3v1GYUKulOplGl99WnNW2 BUnj8MHZc64AA1ShZicZxTtlk60YshmjK13BbEi21Bfcx+gcAyhPFG0Udo+7rcglUOO/ e5ukt7sLR66w9vqd/eF2Vux/iCxQurWIjIiKfdsvU9BqjEBW8G8jxSPvaFCeod8Edvt4 FxyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=8BiWBmJ/VC9KkVb4hI/TM5y8+2cX+/ZhlTUXVP76k9Q=; b=ZPvnmwzcFwsOMCZ9fo23q+hnwNElm7jht9anHk21mnUXg0hByFE51hM6o5II19aMy6 KPj9+BR+sJkq8xn1ZzLgDr07+CNhIEJqVrQ1ApCxrZPsVOEPETXDvoivvB42ZuyzjG9W kya1xt9xbrypMOuMeXrJZjzhmDuqsXIr6YsqVac8ilh+S8v8sQ0pG3A7qbyQBnMyGXWN iGGuaHFOVycA5Xt3NdqbvNaucCVdP3onh8uMuiwrCoG5A7YW9eWezHAqy3ERg1ESlJBI RpZ2CjF1Q6LL/+12hL4DpSoj/aiqmVJBcjVcAJnH+v415w44ZFLlTDyT5la6bVlUymp3 VNOA== X-Gm-Message-State: AOAM532pPyzYHvN6ZUVgXKIkHSOtwcWWPWEpdMwC5j8gnJ/tMYzEFKE6 dErE2I7kTsPrhtOZV6JPMpPyIL8ag60= X-Google-Smtp-Source: ABdhPJzSx34vZyDyS8ukSca+j5xglx9/8XFoF9SX8g6fAJyRWB1M215hE5AQHL3b69ABJpkvx8AakQ== X-Received: by 2002:a5d:64c6:: with SMTP id f6mr9439338wri.18.1621598387101; Fri, 21 May 2021 04:59:47 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id e8sm1981161wrt.30.2021.05.21.04.59.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 May 2021 04:59:46 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Fri, 21 May 2021 11:59:35 +0000 Subject: [PATCH v4 06/12] unpack-trees: compare sparse directories correctly Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee As we further integrate the sparse-index into unpack-trees, we need to ensure that we compare sparse directory entries correctly with other entries. This affects searching for an exact path as well as sorting index entries. Sparse directory entries contain the trailing directory separator. This is important for the sorting, in particular. Thus, within do_compare_entry() we stop using S_IFREG in all cases, since sparse directories should use S_IFDIR to indicate that the comparison should treat the entry name as a dirctory. Within compare_entry(), it first calls do_compare_entry() to check the leading portion of the name. When the input path is a directory name, we could match exactly already. Thus, we should return 0 if we have an exact string match on a sparse directory entry. Signed-off-by: Derrick Stolee --- unpack-trees.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/unpack-trees.c b/unpack-trees.c index 1067db19c9d2..ef6a2b1c951c 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -969,6 +969,7 @@ static int do_compare_entry(const struct cache_entry *ce, int pathlen, ce_len; const char *ce_name; int cmp; + unsigned ce_mode; /* * If we have not precomputed the traverse path, it is quicker @@ -991,7 +992,8 @@ static int do_compare_entry(const struct cache_entry *ce, ce_len -= pathlen; ce_name = ce->name + pathlen; - return df_name_compare(ce_name, ce_len, S_IFREG, name, namelen, mode); + ce_mode = S_ISSPARSEDIR(ce->ce_mode) ? S_IFDIR : S_IFREG; + return df_name_compare(ce_name, ce_len, ce_mode, name, namelen, mode); } static int compare_entry(const struct cache_entry *ce, const struct traverse_info *info, const struct name_entry *n) @@ -1000,6 +1002,15 @@ static int compare_entry(const struct cache_entry *ce, const struct traverse_inf if (cmp) return cmp; + /* + * At this point, we know that we have a prefix match. If ce + * is a sparse directory, then allow an exact match. This only + * works when the input name is a directory, since ce->name + * ends in a directory separator. + */ + if (S_ISSPARSEDIR(ce->ce_mode)) + return 0; + /* * Even if the beginning compared identically, the ce should * compare as bigger than a directory leading up to it! From patchwork Fri May 21 11:59:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12272959 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 55779C433B4 for ; Fri, 21 May 2021 12:00:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 35A1C613D6 for ; Fri, 21 May 2021 12:00:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233890AbhEUMBo (ORCPT ); Fri, 21 May 2021 08:01:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35886 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234282AbhEUMBO (ORCPT ); Fri, 21 May 2021 08:01:14 -0400 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E879BC06138D for ; Fri, 21 May 2021 04:59:48 -0700 (PDT) Received: by mail-wr1-x42c.google.com with SMTP id x8so20804742wrq.9 for ; Fri, 21 May 2021 04:59:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Xlnwaf5kg5I86PpR29UBuzCeR2oJW7UhOX0TCCBSrvU=; b=OpYoM7rQJulmw+izVMlIR2/VeynrBzfU9SdCGFsvSBCunmILEvzfGO/jaWtN4eReGa qbg/vBf+SGjN+7errIZ//oy14abUzom2cEV1otpj0l47iqa39YipO9qoJJI2Rk08iHP0 6i4u5dytePQfmfg1GSqEgj0VrQ3Z2dBFRy3v1jGUtlo0EAnRF3sLZCMzc64bk0oYHIIj lCyJPWHstoUI3y2BmKd7Nuu5f6k9Mw54zzPtaXmN5UaPyKjQOHoZDh8LJeMkGqTh8dkj L7EC4UXQBDT2NhPORgdXyBtrceO4ckLFDsI8+aqWDAwk+xaox5USfmJN0CGidCnViPyr wzTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Xlnwaf5kg5I86PpR29UBuzCeR2oJW7UhOX0TCCBSrvU=; b=ulYWEIxw4ln0JDGYtainpSLaZSSqfkU4t8blKfz4H30qs+tErPM25DuV9+JUNEU5zE vkSmFfLRCvlqYTRWrbawj8Udos/38WSnmQcQM9HMn5wiou12G8EwKtUEkIcxlydZO4tB /uvrDcSHZCwmKCrOfn7+cITMfoD5qZdTSZDBbjmVpDPlfwydY4FuCpdymGVG80ghhovx Gw4eZvNISM6AhlsEvOyoDKXTTaiJYRFX4oGRW5CT0LYYayvgOw3Wc0n+ESKFK941NQLG ihVOx5mVSu/AP6rrTektgyjpXM6V1kNX+PmYBhv//D8MaSJscd2D+KeElXGbzkrVQXfo /tIQ== X-Gm-Message-State: AOAM532sbgI+fK+kPgqnrOqapGh+abFWV3k1d+mWXl8eiOwu/+ugewuj h4mjhyXoCjUVocBKBoM6vHkIEAzQ9ds= X-Google-Smtp-Source: ABdhPJzyJQCrYusKigGxlVWXuAqTpRplWdHrO+rMrvwGuRwIgX8FaThmO0Rves90uLxR33qno+T+IA== X-Received: by 2002:a05:6000:1189:: with SMTP id g9mr9115023wrx.385.1621598387631; Fri, 21 May 2021 04:59:47 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id x10sm1876109wrt.65.2021.05.21.04.59.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 May 2021 04:59:47 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Fri, 21 May 2021 11:59:36 +0000 Subject: [PATCH v4 07/12] unpack-trees: be careful around sparse directory entries Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The methods traverse_by_cache_tree() and unpack_nondirectories() have similar behavior in trying to demonstrate the difference between and index and a tree, with some differences about how they walk the index. Each of these is expecting every cache entry to correspond to a file path. We need to skip over the sparse directory entries in the case of a sparse-index. Those entries are discovered in the portion that looks for subtrees among the cache entries by scanning the paths for slashes. Skipping these sparse directory entries will have a measurable effect when we relax 'git status' to work with sparse-indexes: without this change these methods would call call_unpack_fn() which in turn calls oneway_diff() and then shows these sparse directory entries as added or modified files. Signed-off-by: Derrick Stolee --- unpack-trees.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/unpack-trees.c b/unpack-trees.c index ef6a2b1c951c..22634d98e72b 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -802,6 +802,9 @@ static int traverse_by_cache_tree(int pos, int nr_entries, int nr_names, src[0] = o->src_index->cache[pos + i]; + if (S_ISSPARSEDIR(src[0]->ce_mode)) + continue; + len = ce_namelen(src[0]); new_ce_len = cache_entry_size(len); @@ -1074,6 +1077,9 @@ static int unpack_nondirectories(int n, unsigned long mask, if (mask == dirmask && !src[0]) return 0; + if (src[0] && S_ISSPARSEDIR(src[0]->ce_mode)) + return 0; + /* * Ok, we've filled in up to any potential index entry in src[0], * now do the rest. From patchwork Fri May 21 11:59:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12272961 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DEB24C433ED for ; Fri, 21 May 2021 12:00:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C1838613D6 for ; Fri, 21 May 2021 12:00:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229571AbhEUMBs (ORCPT ); Fri, 21 May 2021 08:01:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35888 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234416AbhEUMBP (ORCPT ); Fri, 21 May 2021 08:01:15 -0400 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AE17DC061763 for ; Fri, 21 May 2021 04:59:49 -0700 (PDT) Received: by mail-wm1-x32e.google.com with SMTP id b7so10392017wmh.5 for ; Fri, 21 May 2021 04:59:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=IKPjTUqki+7odL+EABJRxmdzP4v6AWl9RXlIatOMDLM=; b=GWcqxVbAtIevRvM0kdQxvEuUHoHozSiDIaNxDoCq6NA2fnnbRe9ZJuHlsiOqRTSD4O pMvEMZ5Wnr8Ymo9qEzsRi/9xHoc7kStYRdyZuHTa+fAw68hd6GDO7SP3DDk/Yoqbni87 rD1y+14ik2PztLGKaHcHog4eLykzpVo8gONL6n9dvqQ47ktU2ScyBw10Z5cJZmAi+AzV 0pvma3AoB9Rt7N4StW9cqBfOixGpoIerj1E7YndP2vXTyuEbMl6Bra1WO7T9svTpFlzQ MmvJoaO+xfo85agIS93iR6P4AWZUIXxf22r8ZlWL+jYN3m0INKgqzHTDeSdznmVGuFqq Ww+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=IKPjTUqki+7odL+EABJRxmdzP4v6AWl9RXlIatOMDLM=; b=Bwct27KtStnGYEs2sdbLoo+L0PI9p5SHX8X5CHRyQtwHvSk8VRsYZ+LBGJ6QM0/JA4 1guujGYPBO8a/qN6aw7svtbDLjJvvbSGCK1NvgZMsVqJyteoNY83IrpU91N4iLDmdkUD BVjHS0KO+1bn7OAIWcjdLo3kLKILyI3B4QnWoM/UeLGoOke2DxGiMHqEyBLxU2dO6rEq C17A3bxC7Ea3P9FMc0p0m1UXNicew2u4WtEVlBOmaDLcdWb/UmuwqGBqqm41l8a+uYEC SG6/eoyAu2dUfCiaecEJZte+2/yaaU9hf+sb+9j2UEkRAcoGdtHgCULm5KAeRFMNcbt7 AIwQ== X-Gm-Message-State: AOAM531aR3J5dKtlh6YhH3kd+H+LUQ1O4/09zAczS9aHNcG0Qfjth1yp GbSQJrv/NgyeVnOVT8qUiU7JE4WWvFY= X-Google-Smtp-Source: ABdhPJwDuYtm7f9xiRJE92M4ar0JNOefHjBjuqkYLrt+8nbFUK7C2wCs83uN4U6KBY82ZumG/83iXg== X-Received: by 2002:a05:600c:410a:: with SMTP id j10mr8617376wmi.5.1621598388170; Fri, 21 May 2021 04:59:48 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id z12sm1924855wrv.68.2021.05.21.04.59.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 May 2021 04:59:47 -0700 (PDT) Message-Id: <2cc3a93d4434d546cf385fa2c902064748f1b2b6.1621598382.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 21 May 2021 11:59:37 +0000 Subject: [PATCH v4 08/12] dir.c: accept a directory as part of cone-mode patterns Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee When we have sparse directory entries in the index, we want to compare that directory against sparse-checkout patterns. Those pattern matching algorithms are built expecting a file path, not a directory path. This is especially important in the "cone mode" patterns which will match files that exist within the "parent directories" as well as the recursive directory matches. If path_matches_pattern_list() is given a directory, we can add a fake filename ("-") to the directory and get the same results as before, assuming we are in cone mode. Since sparse index requires cone mode patterns, this is an acceptable assumption. Signed-off-by: Derrick Stolee --- dir.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/dir.c b/dir.c index 166238e79f52..ab76ef286495 100644 --- a/dir.c +++ b/dir.c @@ -1378,6 +1378,17 @@ enum pattern_match_result path_matches_pattern_list( strbuf_addch(&parent_pathname, '/'); strbuf_add(&parent_pathname, pathname, pathlen); + /* + * Directory entries are matched if and only if a file + * contained immediately within them is matched. For the + * case of a directory entry, modify the path to create + * a fake filename within this directory, allowing us to + * use the file-base matching logic in an equivalent way. + */ + if (parent_pathname.len > 0 && + parent_pathname.buf[parent_pathname.len - 1] == '/') + strbuf_add(&parent_pathname, "-", 1); + if (hashmap_contains_path(&pl->recursive_hashmap, &parent_pathname)) { result = MATCHED_RECURSIVE; From patchwork Fri May 21 11:59:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12272963 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 04DF6C433B4 for ; Fri, 21 May 2021 12:00:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DDC7F6008E for ; Fri, 21 May 2021 12:00:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235046AbhEUMBv (ORCPT ); Fri, 21 May 2021 08:01:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35896 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234581AbhEUMBP (ORCPT ); Fri, 21 May 2021 08:01:15 -0400 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EBA5DC06138E for ; Fri, 21 May 2021 04:59:49 -0700 (PDT) Received: by mail-wr1-x42c.google.com with SMTP id q5so20809354wrs.4 for ; Fri, 21 May 2021 04:59:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=aieF5APTh0NLP7SmOzgU2W+ZxUYW0jJ5qZJaNE169FM=; b=rAVJ3tSQMA1G14w1rD+OGDpebbcPaeUKnc3CKqHosbqRMSBOgDkLtwmknhVbQObkhZ g4ixnMysWTkZXl7UAwYMlWdR3bEoosSDzhA7FihqEWMWxJdty+1gODF9OZECduFVbx+1 BbOUymeh9E809/1QxXZAXYhaU5fpzBgPqJzN3+M5tQmGoQkyz4d9an3PTGnnT8hs8EOq 0KkaXTR0HAOcfxf13WONREvW5SiCAjs4K6FlhlTjxVtlwJp21Xm++hSIxqlABocbPNxH F6yKaYlO3uqvkEyjADaex071GGfThJNteejlcX5QzOB2tLZ6fevN3oYI9c/3j58RVcT+ 5N6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=aieF5APTh0NLP7SmOzgU2W+ZxUYW0jJ5qZJaNE169FM=; b=rHf7WHjm7E60cJdVFvTK5h5cIga5QGOyXjUY8cnxt4C50l0P5ZpQVRwpAkER/8f/5G mCPxLIct7GOgQARikEeDqLXQUinZ7Iiw0BhKZkDucygZJD6rkOKbAqWcrj7+G8FsSpiV tDovLG2Oy99o+7BPyDGOMvz9vw/pXSWD7+hn7fW67edwhVF6IcfqfLvWMw4QsIvs9JcE RucOwWUntV+xAEhhHLBsLXBj4p5l/pMf8VB4BCzloM6i2ozSU5eS3xJP+4036VQ9K+75 FeL2DJL+eY/FrXqFR9WhgIY7yPuytplluAh8nPz9ke4EZdeXJnnP/mX98DsFvMBnZkf5 FeOw== X-Gm-Message-State: AOAM533sQOf1m/egJkPubd3REOymDRnmDH9Omta3ILLP/xaG1MSTxNBh mf8UcphU6lsJ1tILFwpslDFLbbXhCXk= X-Google-Smtp-Source: ABdhPJys7cHRBKRLn6G7S5D0eCuiTVCq9UoLl09+X0abkKZcp27jK+MkyNrnmcff5vCi6CYkagWm+w== X-Received: by 2002:adf:9cc1:: with SMTP id h1mr8967733wre.135.1621598388652; Fri, 21 May 2021 04:59:48 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id r14sm1818795wrx.74.2021.05.21.04.59.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 May 2021 04:59:48 -0700 (PDT) Message-Id: <5011feb1aa04885f6bb272305913257db5c304eb.1621598382.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 21 May 2021 11:59:38 +0000 Subject: [PATCH v4 09/12] status: skip sparse-checkout percentage with sparse-index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee 'git status' began reporting a percentage of populated paths when sparse-checkout is enabled in 051df3cf (wt-status: show sparse checkout status as well, 2020-07-18). This percentage is incorrect when the index has sparse directories. It would also be expensive to calculate as we would need to parse trees to count the total number of possible paths. Avoid the expensive computation by simplifying the output to only report that a sparse checkout exists, without the percentage. This change is the reason we use 'git status --porcelain=v2' in t1092-sparse-checkout-compatibility.sh. We don't want to ensure that this message is equal across both modes, but instead just the important information about staged, modified, and untracked files are compared. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 8 ++++++++ wt-status.c | 14 +++++++++++--- wt-status.h | 1 + 3 files changed, 20 insertions(+), 3 deletions(-) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index fba98d5484ae..34dae7fbcadd 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -202,6 +202,14 @@ test_expect_success 'status with options' ' test_all_match git status --porcelain=v2 -uno ' +test_expect_success 'status reports sparse-checkout' ' + init_repos && + git -C sparse-checkout status >full && + git -C sparse-index status >sparse && + test_i18ngrep "You are in a sparse checkout with " full && + test_i18ngrep "You are in a sparse checkout." sparse +' + test_expect_success 'add, commit, checkout' ' init_repos && diff --git a/wt-status.c b/wt-status.c index 0c8287a023e4..0425169c1895 100644 --- a/wt-status.c +++ b/wt-status.c @@ -1490,9 +1490,12 @@ static void show_sparse_checkout_in_use(struct wt_status *s, if (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_DISABLED) return; - status_printf_ln(s, color, - _("You are in a sparse checkout with %d%% of tracked files present."), - s->state.sparse_checkout_percentage); + if (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_SPARSE_INDEX) + status_printf_ln(s, color, _("You are in a sparse checkout.")); + else + status_printf_ln(s, color, + _("You are in a sparse checkout with %d%% of tracked files present."), + s->state.sparse_checkout_percentage); wt_longstatus_print_trailer(s); } @@ -1650,6 +1653,11 @@ static void wt_status_check_sparse_checkout(struct repository *r, return; } + if (r->index->sparse_index) { + state->sparse_checkout_percentage = SPARSE_CHECKOUT_SPARSE_INDEX; + return; + } + for (i = 0; i < r->index->cache_nr; i++) { struct cache_entry *ce = r->index->cache[i]; if (ce_skip_worktree(ce)) diff --git a/wt-status.h b/wt-status.h index 0d32799b28e1..ab9cc9d8f032 100644 --- a/wt-status.h +++ b/wt-status.h @@ -78,6 +78,7 @@ enum wt_status_format { }; #define SPARSE_CHECKOUT_DISABLED -1 +#define SPARSE_CHECKOUT_SPARSE_INDEX -2 struct wt_status_state { int merge_in_progress; From patchwork Fri May 21 11:59:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12272967 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DD9D7C43461 for ; Fri, 21 May 2021 12:00:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C1D7D6008E for ; Fri, 21 May 2021 12:00:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235143AbhEUMBw (ORCPT ); Fri, 21 May 2021 08:01:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35898 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234893AbhEUMBP (ORCPT ); Fri, 21 May 2021 08:01:15 -0400 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 855D2C06138F for ; Fri, 21 May 2021 04:59:50 -0700 (PDT) Received: by mail-wr1-x435.google.com with SMTP id v12so20810471wrq.6 for ; Fri, 21 May 2021 04:59:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=gu6GEJgKuJhO6OfWQXS55AAR+pJtihz9mG9qQegVuuA=; b=kLbV+7IDRHTgz83QewHX5eCVTBcAlS1b8QX523zN1zdlqiVsdoY+9iPRuh7vJTPoMz MDkWv0bMzRkKAN4Q75EhCDRk7vXpAO66rf1TUxJy/oynrlKkbuB7puMHL10qSY5FEdJQ yldQY3QaTvGoGviEuyto0iZuvJj+T99zxT1JFol3KUrWbt7YInut8I6GjJYSGOV1XLMZ boRgZnxKjCzDQZYMAtxcHKpi6IQHy1WQNSrluXavt+THShXGRPiLFKePCzIn+WvoXTWs Afle7han07LlJMl8y0/B7BOJA9gjI0s9KuHeQ3SDdTUi8PjQYNjuFtLwuXk/SgKZyikM CsTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=gu6GEJgKuJhO6OfWQXS55AAR+pJtihz9mG9qQegVuuA=; b=FzYQ5Bz3gwmydPijen0d1rK078YXhBmkkuZQ5qfkuyEDbkJ+V4WNCUWq07jfevR5Eq mM+SA3A0Q87EizT6kdFUGo4YJWiCOeIaLpwBHLslfWiB1V/oGP0cSQu1BETAz483z7zH W9OXusKS71c6nRTdRfW+JcSMWbqNDkfpVNfwM+wOfaEjN+WB6FCukZ2Lv3CoPE1yOd8r rZtqLJQmYFYzG3kgPR2on24LwVwLAYGOwIslD+8qYpX3wtX3Gv47GZhHf1YbnwWlcV0x meGfOjOIsVsbF0DKBnbkH9EEzTRwquMsp69dSuTcVHsDsfiNzjoGEeRTAwEMlSQle8BU nVcw== X-Gm-Message-State: AOAM531AfkJppFIETG7hZ0SRjpRA9/T1gV4Th0NVAxY7E61Yjvu9sXYx mHIrM67yMNxpuVeC0w3kohiv7RpWljc= X-Google-Smtp-Source: ABdhPJyUSwaBN0xRDIqT5PustfJDmw0yyQmcIOlqxDAsUWGN+qNa/xPO1LHUf8hsEeQTPd4Cyw3VPg== X-Received: by 2002:adf:a519:: with SMTP id i25mr9267417wrb.312.1621598389164; Fri, 21 May 2021 04:59:49 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id y6sm10839136wmy.23.2021.05.21.04.59.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 May 2021 04:59:48 -0700 (PDT) Message-Id: <9f2ce5301dc944d54f748390655dcc7ca3f0f9f9.1621598382.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 21 May 2021 11:59:39 +0000 Subject: [PATCH v4 10/12] status: use sparse-index throughout Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee By testing 'git -c core.fsmonitor= status -uno', we can check for the simplest index operations that can be made sparse-aware. The necessary implementation details are already integrated with sparse-checkout, so modify command_requires_full_index to be zero for cmd_status(). In refresh_index(), we loop through the index entries to refresh their stat() information. However, sparse directories have no stat() information to populate. Ignore these entries. This allows 'git status' to no longer expand a sparse index to a full one. This is further tested by dropping the "-uno" option and adding an untracked file into the worktree. The performance test p2000-sparse-checkout-operations.sh demonstrates these improvements: Test HEAD~1 HEAD ----------------------------------------------------------------------------- 2000.2: git status (full-index-v3) 0.31(0.30+0.05) 0.31(0.29+0.06) +0.0% 2000.3: git status (full-index-v4) 0.31(0.29+0.07) 0.34(0.30+0.08) +9.7% 2000.4: git status (sparse-index-v3) 2.35(2.28+0.10) 0.04(0.04+0.05) -98.3% 2000.5: git status (sparse-index-v4) 2.35(2.24+0.15) 0.05(0.04+0.06) -97.9% Note that since HEAD~1 was expanding the sparse index by parsing trees, it was artificially slower than the full index case. Thus, the 98% improvement is misleading, and instead we should celebrate the 0.34s to 0.05s improvement of 85%. This is more indicative of the peformance gains we are expecting by using a sparse index. Note: we are dropping the assignment of core.fsmonitor here. This is not necessary for the test script as we are not altering the config any other way. Correct integration with FS Monitor will be validated in later changes. Signed-off-by: Derrick Stolee --- builtin/commit.c | 3 +++ read-cache.c | 10 ++++++++-- t/t1092-sparse-checkout-compatibility.sh | 13 +++++++++---- 3 files changed, 20 insertions(+), 6 deletions(-) diff --git a/builtin/commit.c b/builtin/commit.c index cf0c36d1dcb2..e529da7beadd 100644 --- a/builtin/commit.c +++ b/builtin/commit.c @@ -1404,6 +1404,9 @@ int cmd_status(int argc, const char **argv, const char *prefix) if (argc == 2 && !strcmp(argv[1], "-h")) usage_with_options(builtin_status_usage, builtin_status_options); + prepare_repo_settings(the_repository); + the_repository->settings.command_requires_full_index = 0; + status_init_config(&s, git_status_config); argc = parse_options(argc, argv, prefix, builtin_status_options, diff --git a/read-cache.c b/read-cache.c index 29ffa9ac5db9..f80e26831b36 100644 --- a/read-cache.c +++ b/read-cache.c @@ -1578,8 +1578,7 @@ int refresh_index(struct index_state *istate, unsigned int flags, */ preload_index(istate, pathspec, 0); trace2_region_enter("index", "refresh", NULL); - /* TODO: audit for interaction with sparse-index. */ - ensure_full_index(istate); + for (i = 0; i < istate->cache_nr; i++) { struct cache_entry *ce, *new_entry; int cache_errno = 0; @@ -1594,6 +1593,13 @@ int refresh_index(struct index_state *istate, unsigned int flags, if (ignore_skip_worktree && ce_skip_worktree(ce)) continue; + /* + * If this entry is a sparse directory, then there isn't + * any stat() information to update. Ignore the entry. + */ + if (S_ISSPARSEDIR(ce->ce_mode)) + continue; + if (pathspec && !ce_path_match(istate, ce, pathspec, seen)) filtered = 1; diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 34dae7fbcadd..59faf7381093 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -479,12 +479,17 @@ test_expect_success 'sparse-index is expanded and converted back' ' GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ git -C sparse-index -c core.fsmonitor="" reset --hard && test_region index convert_to_sparse trace2.txt && - test_region index ensure_full_index trace2.txt && + test_region index ensure_full_index trace2.txt +' - rm trace2.txt && +test_expect_success 'sparse-index is not expanded' ' + init_repos && + + rm -f trace2.txt && + echo >>sparse-index/untracked.txt && GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ - git -C sparse-index -c core.fsmonitor="" status -uno && - test_region index ensure_full_index trace2.txt + git -C sparse-index status && + test_region ! index ensure_full_index trace2.txt ' test_done From patchwork Fri May 21 11:59:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12272965 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F1E23C43462 for ; Fri, 21 May 2021 12:00:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CE08A613DB for ; Fri, 21 May 2021 12:00:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235148AbhEUMBx (ORCPT ); Fri, 21 May 2021 08:01:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35902 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233238AbhEUMBR (ORCPT ); Fri, 21 May 2021 08:01:17 -0400 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 28706C061342 for ; Fri, 21 May 2021 04:59:51 -0700 (PDT) Received: by mail-wr1-x436.google.com with SMTP id i17so20791458wrq.11 for ; Fri, 21 May 2021 04:59:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=jWoJaiihOFErJJTdCshSCWDCsuIYLvVcwyV8zTPU6cQ=; b=ahpSKqFhwH+/YCU5veGzQFl8AphRvvD07KvI0RbgjQy4kJ9iMkV3E+3/HFpzzma7Jo DTvzN8cbAHrL+hFe5UMhuemsml7bw/7llLdH5zVVoeZsriQHqjZkxKNj8iTUsIKPtrTp a7dSeCFTcjhytiws1BlNlBwaZaM9uj/l1TysQ/q6iVi+xvSdqGUDLD4nYxYvCp9mRtf5 tLANLCbsnAOCJuk+fOKg1SMUjHsnEhwtjwwhFIHvgrg/DgCxRJ3fHzhekUnPEptgc6C/ 88HoMoC7g5ZnQpKlNoz0ei6GNmAD1V654Sx4uXvKERG8OL/OsK+1tIuqJQoi9sRUsrsI bd2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=jWoJaiihOFErJJTdCshSCWDCsuIYLvVcwyV8zTPU6cQ=; b=tT6pUExSiyJMaxGIJZdn4L4f0NOn7j0XSG3Fa8mdpWiYanh1UwdViDleACdd81MO1V pTaXzaUTZYejMEJUHkmj7YYEpWaGI1e4idGnkMTrn/ybN8qFBf1BvdAM5lDYSWe8CEsF 0WoegziaHusFuqBPnFHWJvkaSgHUlj2ET3Vg5npliTTcLSbJsPj3aLIqdYuLVvPRHhYO ZbkxMTwjELZpzkx3UTLt+BvDZIOV66kU1qStgfJOT4ehrh4Xy/FUuX6DW612+4Q9pbhU RIhrC+ZNV1HgXjnjAO4nqZcbNwIQAInlKllWjhxvJZU5NXhXfKzwzgXcWCcH4Ohvdj58 MQtA== X-Gm-Message-State: AOAM5304X2jflklLAy1El19317iUcOidvoQMMML96jByxz2HirT/yBMZ abVSqRMK0bX8JHOztW/gA87pFC5mfuA= X-Google-Smtp-Source: ABdhPJx5wKValg3Mg583+gkxriQxwWkkGr1Y8R9kFJd4m4vRJMg4DUMx1PVE8a/mgWsXeSoBI4/2qQ== X-Received: by 2002:adf:e987:: with SMTP id h7mr2062751wrm.347.1621598389737; Fri, 21 May 2021 04:59:49 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id y2sm14084974wmq.45.2021.05.21.04.59.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 May 2021 04:59:49 -0700 (PDT) Message-Id: <24417e095243979b13acde785a5bbd79145b050d.1621598382.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 21 May 2021 11:59:40 +0000 Subject: [PATCH v4 11/12] wt-status: expand added sparse directory entries Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee It is difficult, but possible, to get into a state where we intend to add a directory that is outside of the sparse-checkout definition. Add a test to t1092-sparse-checkout-compatibility.sh that demonstrates this using a combination of 'git reset --mixed' and 'git checkout --orphan'. This test failed before because the output of 'git status --porcelain=v2' would not match on the lines for folder1/: * The sparse-checkout repo (with a full index) would output each path name that is intended to be added. * The sparse-index repo would only output that "folder1/" is staged for addition. The status should report the full list of files to be added, and so this sparse-directory entry should be expanded to a full list when reaching it inside the wt_status_collect_changes_initial() method. Use read_tree_at() to assist. Somehow, this loop over the cache entries was not guarded by ensure_full_index() as intended. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 28 +++++++++++++ wt-status.c | 50 ++++++++++++++++++++++++ 2 files changed, 78 insertions(+) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 59faf7381093..cd3669d36b53 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -492,4 +492,32 @@ test_expect_success 'sparse-index is not expanded' ' test_region ! index ensure_full_index trace2.txt ' +test_expect_success 'reset mixed and checkout orphan' ' + init_repos && + + test_all_match git checkout rename-out-to-in && + test_all_match git reset --mixed HEAD~1 && + test_sparse_match test-tool read-cache --table --expand && + test_all_match git status --porcelain=v2 && + test_all_match git status --porcelain=v2 && + + # At this point, sparse-checkouts behave differently + # from the full-checkout. + test_sparse_match git checkout --orphan new-branch && + test_sparse_match test-tool read-cache --table --expand && + test_sparse_match git status --porcelain=v2 && + test_sparse_match git status --porcelain=v2 +' + +test_expect_success 'add everything with deep new file' ' + init_repos && + + run_on_sparse git sparse-checkout set deep/deeper1/deepest && + + run_on_all touch deep/deeper1/x && + test_all_match git add . && + test_all_match git status --porcelain=v2 && + test_all_match git status --porcelain=v2 +' + test_done diff --git a/wt-status.c b/wt-status.c index 0425169c1895..90db8bd659fa 100644 --- a/wt-status.c +++ b/wt-status.c @@ -654,6 +654,34 @@ static void wt_status_collect_changes_index(struct wt_status *s) run_diff_index(&rev, 1); } +static int add_file_to_list(const struct object_id *oid, + struct strbuf *base, const char *path, + unsigned int mode, void *context) +{ + struct string_list_item *it; + struct wt_status_change_data *d; + struct wt_status *s = context; + char *full_name; + + if (S_ISDIR(mode)) + return READ_TREE_RECURSIVE; + + full_name = xstrfmt("%s%s", base->buf, path); + it = string_list_insert(&s->change, full_name); + d = it->util; + if (!d) { + CALLOC_ARRAY(d, 1); + it->util = d; + } + + d->index_status = DIFF_STATUS_ADDED; + /* Leave {mode,oid}_head zero for adds. */ + d->mode_index = mode; + oidcpy(&d->oid_index, oid); + s->committable = 1; + return 0; +} + static void wt_status_collect_changes_initial(struct wt_status *s) { struct index_state *istate = s->repo->index; @@ -668,6 +696,28 @@ static void wt_status_collect_changes_initial(struct wt_status *s) continue; if (ce_intent_to_add(ce)) continue; + if (S_ISSPARSEDIR(ce->ce_mode)) { + /* + * This is a sparse directory entry, so we want to collect all + * of the added files within the tree. This requires recursively + * expanding the trees to find the elements that are new in this + * tree and marking them with DIFF_STATUS_ADDED. + */ + struct strbuf base = STRBUF_INIT; + struct pathspec ps; + struct tree *tree = lookup_tree(istate->repo, &ce->oid); + + memset(&ps, 0, sizeof(ps)); + ps.recursive = 1; + ps.has_wildcard = 1; + ps.max_depth = -1; + + strbuf_add(&base, ce->name, ce->ce_namelen); + read_tree_at(istate->repo, tree, &base, &ps, + add_file_to_list, s); + continue; + } + it = string_list_insert(&s->change, ce->name); d = it->util; if (!d) { From patchwork Fri May 21 11:59:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12272969 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6EB3CC433B4 for ; Fri, 21 May 2021 12:00:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 52D5B6008E for ; Fri, 21 May 2021 12:00:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235171AbhEUMB4 (ORCPT ); Fri, 21 May 2021 08:01:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35904 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234928AbhEUMBR (ORCPT ); Fri, 21 May 2021 08:01:17 -0400 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A179CC061343 for ; Fri, 21 May 2021 04:59:51 -0700 (PDT) Received: by mail-wm1-x334.google.com with SMTP id z85-20020a1c7e580000b029017a76f3afbaso4476870wmc.2 for ; Fri, 21 May 2021 04:59:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=HVK7x9ytblDj8iy4lmiMNtwNDpdpLC/uAkky1UzqoLw=; b=XOqzglHtzCj+pcrP8LKvmXpTRnix7OchY+qr2x9x4iAd0hTCvdwPtJ+kffk/hHeCM4 4esCUcZ8JqeE7Fy/oSMyZPHaRKCs2D+cht0FvpbK7tQLfumL+dUzk+KYWCvVU2hcRPUl 2Y20COv4T9zrdQNCsJDFoPbRw52V/5MqqUjTil3qg6PBNvdJ2Q4rMOCr4aWRZdJa83sm ieyl+WNVwWTLOCs2AK7juqldrlnxfX655s7/PiMjJji2RNxEi/t0zUn4Xw3nIp1xTIWi 43lNmgIdIlfvLkX+7t/8COfaQO4Yd8VP1dQTO1WAxS9K8P7Xu4V7iUcS0zS6ytiZJpyf CTFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=HVK7x9ytblDj8iy4lmiMNtwNDpdpLC/uAkky1UzqoLw=; b=cpVDncmpmqmffhAvUBF8VHP6tGzrPEgZEB3LmCDtUMyw5oQXhDScIY/uujBDxMyWyv Ebw11ZnSXlvRvV3KNhhplIn1vZA+KQWUx0l6Fa3Oo3ygD5vKmkOgUKnZg+Nb9yGP3Vke x8pgdsobNMrFF2XtFZKduAGuOFXFKh3xM4xN2llMRguSLHNqPC3xgp2KEZhB8e5PHQB8 qqbffLYY6M8k2eAsZNckBU2GSi7p4ckfhic6lHmAk+dIoZQT0RXu07rvpHMrE1mCDaa9 srvnrVXjunJr8UC9FYisQXxSx+kCoQXScQ+6yPbvqPV3/9hGF5jyp7Hry6aTmPxHYpJS WpTQ== X-Gm-Message-State: AOAM532ZUUn+AuI8KZEqCZyUtHnCdHa7WC4nWIE/dvL+TJa/gcmpSsWo 5YGEznQAdL/VFiR6BQVG3fjLp/e879Q= X-Google-Smtp-Source: ABdhPJxypoZG8GqvNI/26TnZOmKwUo8eKGGwIhBe1I/utl78JUMuA+zLJ3wK2giQuKqG2zNMpQ0Tqg== X-Received: by 2002:a7b:c1c6:: with SMTP id a6mr8556330wmj.189.1621598390284; Fri, 21 May 2021 04:59:50 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 61sm2032096wrm.52.2021.05.21.04.59.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 May 2021 04:59:50 -0700 (PDT) Message-Id: <584d4b559a91213287221b9731281431fc557627.1621598382.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 21 May 2021 11:59:41 +0000 Subject: [PATCH v4 12/12] fsmonitor: integrate with sparse index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee If we need to expand a sparse-index into a full one, then the FS Monitor bitmap is going to be incorrect. Ensure that we start fresh at such an event. While this is currently a performance drawback, the eventual hope of the sparse-index feature is that these expansions will be rare and hence we will be able to keep the FS Monitor data accurate across multiple Git commands. These tests are added to demonstrate that the behavior is the same across a full index and a sparse index, but also that file modifications to a tracked directory outside of the sparse cone will trigger ensure_full_index(). Signed-off-by: Derrick Stolee --- sparse-index.c | 7 ++++++ t/t7519-status-fsmonitor.sh | 48 +++++++++++++++++++++++++++++++++++++ 2 files changed, 55 insertions(+) diff --git a/sparse-index.c b/sparse-index.c index b2b3fbd75050..32ba0d17ef7c 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -195,6 +195,10 @@ int convert_to_sparse(struct index_state *istate) cache_tree_free(&istate->cache_tree); cache_tree_update(istate, 0); + istate->fsmonitor_has_run_once = 0; + FREE_AND_NULL(istate->fsmonitor_dirty); + FREE_AND_NULL(istate->fsmonitor_last_update); + istate->sparse_index = 1; trace2_region_leave("index", "convert_to_sparse", istate->repo); return 0; @@ -291,6 +295,9 @@ void ensure_full_index(struct index_state *istate) istate->cache = full->cache; istate->cache_nr = full->cache_nr; istate->cache_alloc = full->cache_alloc; + istate->fsmonitor_has_run_once = 0; + FREE_AND_NULL(istate->fsmonitor_dirty); + FREE_AND_NULL(istate->fsmonitor_last_update); strbuf_release(&base); free(full); diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh index 45d025f96010..f70fe961902e 100755 --- a/t/t7519-status-fsmonitor.sh +++ b/t/t7519-status-fsmonitor.sh @@ -73,6 +73,7 @@ test_expect_success 'setup' ' expect* actual* marker* + trace2* EOF ' @@ -383,4 +384,51 @@ test_expect_success 'status succeeds after staging/unstaging' ' ) ' +# Usage: +# check_sparse_index_behavior [!] +# If "!" is supplied, then we verify that we do not call ensure_full_index +# during a call to 'git status'. Otherwise, we verify that we _do_ call it. +check_sparse_index_behavior () { + git status --porcelain=v2 >expect && + git sparse-checkout init --cone --sparse-index && + git sparse-checkout set dir1 dir2 && + GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ + git status --porcelain=v2 >actual && + test_region $1 index ensure_full_index trace2.txt && + test_cmp expect actual && + rm trace2.txt && + git sparse-checkout disable +} + +test_expect_success 'status succeeds with sparse index' ' + git reset --hard && + + test_config core.fsmonitor "$TEST_DIRECTORY/t7519/fsmonitor-all" && + check_sparse_index_behavior ! && + + write_script .git/hooks/fsmonitor-test<<-\EOF && + printf "last_update_token\0" + EOF + git config core.fsmonitor .git/hooks/fsmonitor-test && + check_sparse_index_behavior ! && + + write_script .git/hooks/fsmonitor-test<<-\EOF && + printf "last_update_token\0" + printf "dir1/modified\0" + EOF + check_sparse_index_behavior ! && + + cp -r dir1 dir1a && + git add dir1a && + git commit -m "add dir1a" && + + # This one modifies outside the sparse-checkout definition + # and hence we expect to expand the sparse-index. + write_script .git/hooks/fsmonitor-test<<-\EOF && + printf "last_update_token\0" + printf "dir1a/modified\0" + EOF + check_sparse_index_behavior +' + test_done