From patchwork Fri May 14 18:31:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12258583 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 48B28C43461 for ; Fri, 14 May 2021 18:31:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2ECB661440 for ; Fri, 14 May 2021 18:31:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233700AbhENScc (ORCPT ); Fri, 14 May 2021 14:32:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50248 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233573AbhENSc3 (ORCPT ); Fri, 14 May 2021 14:32:29 -0400 Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0756BC06174A for ; Fri, 14 May 2021 11:31:17 -0700 (PDT) Received: by mail-wm1-x331.google.com with SMTP id j3-20020a05600c4843b02901484662c4ebso1884090wmo.0 for ; Fri, 14 May 2021 11:31:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=tnv8/RvaEXbX/IGlvdgT6jdK85+oA3TpgnT3UjIuqEE=; b=IrfaZiskLcw2xXbrucGPffQmiT7+NOfN+27cCzby6mzT+B9qRumuwNkFQyCehBV8TB kxPkWG0FMoeRe6VfbISJPZWl9iHNTQ/qyKa9th+FAfcSRPk2uR0C7z4pxPuiCitxVgCe kRvPv68XeNYcT2LxiqUTViveMrTNp7UhDYFvJcoqLnmX3YD6ztckaoTThLdkDNue/rTB BwITaBpcYAnKA53LVWgy4l8s0U5CnNTfbDbsBKhHwhc1pS1RC6J7GZ/zLAkqalJcw7cH rrmfsAWkjVpkE93xoEhx6p+00AUWxdsUfy0/23yF9FVcAxEKCEvubZgdmmxNEItTuGud ljEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=tnv8/RvaEXbX/IGlvdgT6jdK85+oA3TpgnT3UjIuqEE=; b=JeqheM1qHyaeQBKkmjglQizn2hfvakfV/r3+cwIjawl3BHHRe6LwVMJObXYQ6t/j6U Wxc2CU2fI9+PoWjczoVDb98yPWbFt9HhhYuE1jCX//IDyTmPq4z10YGi8G4tW8JJFG8M OZeaiOacHUIlEjRr9a1Ef1+UanzOSQCJ55AKtyciujbd2kV5wk3iSBQftTHn5+KG9mRJ cnB05Mln9uT6tQr/pwXL7zbmsGUis6Ns6mSdkBUKmJQZSidxfusssgt8LNiYBxZBw8sH ri4lbMyXwB8tZhHlAiNFav+oMLh126rVOxz0flf81/5eVxukOpE3Ot/CLp3zOL1KratS BIDw== X-Gm-Message-State: AOAM5339FEopMKEWFMVmPQexP78Hg1XZiFkNDOZsr+mltGXOav11eQMw mBUiZZN2BZRm3dese/kUoBhAxnielv0= X-Google-Smtp-Source: ABdhPJw7Rd+3uEW93+sgzqzfpK9O9M1Y66Ra+Vrg2h6f5OJ+izuH0YbKN/M1pn8Yq7crwZhBElk6fw== X-Received: by 2002:a1c:4e0b:: with SMTP id g11mr10815043wmh.3.1621017074788; Fri, 14 May 2021 11:31:14 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id t18sm5790339wmq.19.2021.05.14.11.31.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 May 2021 11:31:14 -0700 (PDT) Message-Id: <5a2ed3d1d701b11c241bf3855db3bb35f5e93994.1621017072.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 14 May 2021 18:31:00 +0000 Subject: [PATCH v3 01/12] sparse-index: skip indexes with unmerged entries Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The sparse-index format is designed to be compatible with merge conflicts, even those outside the sparse-checkout definition. The reason is that when converting a full index to a sparse one, a cache entry with nonzero stage will not be collapsed into a sparse directory entry. However, this behavior was not tested, and a different behavior within convert_to_sparse() fails in this scenario. Specifically, cache_tree_update() will fail when unmerged entries exist. convert_to_sparse_rec() uses the cache-tree data to recursively walk the tree structure, but also to compute the OIDs used in the sparse-directory entries. Add an index scan to convert_to_sparse() that will detect if these merge conflict entries exist and skip the conversion before trying to update the cache-tree. This is marked as NEEDSWORK because this can be removed with a suitable update to cache_tree_update() or a similar method that can construct a cache-tree with invalid nodes, but still allow creating the nodes necessary for creating sparse directory entries. It is possible that in the future we will not need to make such an update, since if we do not expand a sparse-index into a full one, this conversion does not need to happen. Thus, this can be deferred until the merge machinery is made to integrate with the sparse-index. Signed-off-by: Derrick Stolee --- sparse-index.c | 18 ++++++++++++++++++ t/t1092-sparse-checkout-compatibility.sh | 22 ++++++++++++++++++++++ 2 files changed, 40 insertions(+) diff --git a/sparse-index.c b/sparse-index.c index 6f21397e2ee0..1b49898d0cb7 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -125,6 +125,17 @@ int set_sparse_index_config(struct repository *repo, int enable) return res; } +static int index_has_unmerged_entries(struct index_state *istate) +{ + int i; + for (i = 0; i < istate->cache_nr; i++) { + if (ce_stage(istate->cache[i])) + return 1; + } + + return 0; +} + int convert_to_sparse(struct index_state *istate) { int test_env; @@ -161,6 +172,13 @@ int convert_to_sparse(struct index_state *istate) return -1; } + /* + * NEEDSWORK: If we have unmerged entries, then stay full. + * Unmerged entries prevent the cache-tree extension from working. + */ + if (index_has_unmerged_entries(istate)) + return 0; + if (cache_tree_update(istate, 0)) { warning(_("unable to update cache-tree, staying full")); return -1; diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 12e6c453024f..4f2f09b53a32 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -352,6 +352,28 @@ test_expect_success 'merge with outside renames' ' done ' +# Sparse-index fails to convert the index in the +# final 'git cherry-pick' command. +test_expect_success 'cherry-pick with conflicts' ' + init_repos && + + write_script edit-conflict <<-\EOF && + echo $1 >conflict + EOF + + test_all_match git checkout -b to-cherry-pick && + run_on_all ../edit-conflict ABC && + test_all_match git add conflict && + test_all_match git commit -m "conflict to pick" && + + test_all_match git checkout -B base HEAD~1 && + run_on_all ../edit-conflict DEF && + test_all_match git add conflict && + test_all_match git commit -m "conflict in base" && + + test_all_match test_must_fail git cherry-pick to-cherry-pick +' + test_expect_success 'clean' ' init_repos && From patchwork Fri May 14 18:31:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12258581 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 656C3C433ED for ; Fri, 14 May 2021 18:31:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 42BE96144C for ; Fri, 14 May 2021 18:31:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233680AbhENScb (ORCPT ); Fri, 14 May 2021 14:32:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50246 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233563AbhENSc2 (ORCPT ); Fri, 14 May 2021 14:32:28 -0400 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C6D74C061574 for ; Fri, 14 May 2021 11:31:16 -0700 (PDT) Received: by mail-wr1-x42b.google.com with SMTP id m9so182615wrx.3 for ; Fri, 14 May 2021 11:31:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=QTzuOgoXGKpn026RiRL8qt8jaloUIGodP1E75w97/mM=; b=iVhb01muPrs8Po4zHYEovwlFGbDUCkRZr+qjURvmQEnOVYWCP0+MyAMYwPulBkCbpl 0VVFw54JpWptMq0tP9/peEzFajQssHDrb/tZxDjbQm+Z/GOfBYd4eyTwZZKBr8pWRb4J hn5F3r1dll0KflHtl8pi8VWyGuGo0rKDPEN6DkuOEifpRvlzcLKdEYjd9fRbB9lya7YP yind4eAg9SAO4RbJCf0QP5/wJG91DxWRupJSB4sSS4t2YlkndP4EdBb6//CiCHKRuZ6U l+zJnkUbfmLsBo63/8upVKJe9Dk0RdfPcLwDNiZkzF3nIwp1z2x7ufVf6FvVbsdLceIo nHgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=QTzuOgoXGKpn026RiRL8qt8jaloUIGodP1E75w97/mM=; b=gQXbOzdYh1TQaTJTVmQdA426sB5kLt0+n/hmLEeeaXD+gTO76hlXpuh517jwI/6LiS ftDaX37HVW6q8NYVI+WCcnhEr8H7A3YVgH/LOCLB00djoG7JT1TCYLrXfXwGyYMj8GYt QFtqhCN/5+KHvCQF25L2Bt7ImV5ZUxpgPatmUJJ3D2xgJ1d7k786hFi0Y1iGiOeKloRX +kbzrHS1Lt/TnQN/2qChaMGjdTZf30h+FMsq/ZZ+p91sxWhYk/xxgAy7ZvVef8TtbhTH cEl08n6h4DrJRZeb28coT1jpwxL18yDxr5pq2che/bCqMYRxkE5gPc6jp4n9pEFkPoI9 MTIQ== X-Gm-Message-State: AOAM5318s+iai5bWef/GMhd2MCvC0GdfW+EouwzJ50q+TCJVXoJ7ZZ07 E9sjTPROThOAZO10JyCWHYk6T/zq3pI= X-Google-Smtp-Source: ABdhPJwwgCL8ePyfSm5mx5WaL1MYyqdDp2lAJAeYdoi79DpOByZIzLdvYmgZseBKdSP63JXXefBf+A== X-Received: by 2002:adf:ed47:: with SMTP id u7mr13710481wro.254.1621017075650; Fri, 14 May 2021 11:31:15 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id a15sm8439123wrx.9.2021.05.14.11.31.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 May 2021 11:31:15 -0700 (PDT) Message-Id: <8aa41e749471df3bd9d593b8f55db6506eafea12.1621017072.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 14 May 2021 18:31:01 +0000 Subject: [PATCH v3 02/12] sparse-index: include EXTENDED flag when expanding Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee When creating a full index from a sparse one, we create cache entries for every blob within a given sparse directory entry. These are correctly marked with the CE_SKIP_WORKTREE flag, but they must also be marked with the CE_EXTENDED flag to ensure that the skip-worktree bit is correctly written to disk in the case that the index is not converted back down to a sparse-index. Signed-off-by: Derrick Stolee --- sparse-index.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparse-index.c b/sparse-index.c index 1b49898d0cb7..b2b3fbd75050 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -222,7 +222,7 @@ static int add_path_to_index(const struct object_id *oid, strbuf_addstr(base, path); ce = make_cache_entry(istate, mode, oid, base->buf, 0, 0); - ce->ce_flags |= CE_SKIP_WORKTREE; + ce->ce_flags |= CE_SKIP_WORKTREE | CE_EXTENDED; set_index_entry(istate, istate->cache_nr++, ce); strbuf_setlen(base, len); From patchwork Fri May 14 18:31:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12258585 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C907C433ED for ; Fri, 14 May 2021 18:31:23 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7EEB461442 for ; Fri, 14 May 2021 18:31:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233706AbhENScd (ORCPT ); Fri, 14 May 2021 14:32:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50250 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233597AbhENSc3 (ORCPT ); Fri, 14 May 2021 14:32:29 -0400 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9ECBFC061756 for ; Fri, 14 May 2021 11:31:17 -0700 (PDT) Received: by mail-wr1-x430.google.com with SMTP id a4so186727wrr.2 for ; Fri, 14 May 2021 11:31:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=JgmxC3ji1LdwWDcaPHjhS9lC0LNmteE5DJe06eKmBqM=; b=Ifs7ofUwR7nF/LTeYWHUsJ8AdRLR03DTa/suMJ7VcuucRdYDbCGjuiPKhdbZKFPOV0 Ivoj7mRKFyxKWj5nCfY2LdR2a2sPl855yBIjFW4xNzrblRfuOeLOCaii9fyfWXU16VTo aPg8B28USGIqdfMqSVemfRjS7nIjWVwn+jvP3h0XMh+pAIjqSqw55NVDLLJ95ZgOzKZr LqqUtcUhFGJDAWljuj82VbDyJOdIrdCUGdvHg5REqZPSUnUhfqmVP+mF3Tb8GlRVGsqF zmkzEN/H0p3FU5n0jEZYxD55vT+tjtazYk/BtbL/10AM1x0NQf0KsbodSbNWRUnC7LIj XyAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=JgmxC3ji1LdwWDcaPHjhS9lC0LNmteE5DJe06eKmBqM=; b=Yp+TJaOQoxxfHm2fw4m0FkeR2frfoL7vG3p9u2vpfm6BbbNXR7d/tAiF6GUdyP6LEK iteimQrurkw4mjM6eWrXimxT1w93dQjTgbpnaGhV10Bsx9A5sDf55lZ4N5hfTeL2hi4S Y3QG4+ikoZlXUoZYZ0LhOq9zNHR+jqvSR1Wa42NrDJGokJHdg2bzJUc3ATrKu5u45XcU RHtGBBDlQQ2QPWu5/PTzyc1rKbLq1ECtDd3/8E8iT52cLRS1JcObGtGGpvoNhPS2x8Tx 3IGkeTWCeW56L1YFxtVn1zmKG7TOTb8/Ef+9ZVGvLlknMM+Xd1imSjacZwNgC/YC8UnA BqIQ== X-Gm-Message-State: AOAM531nK+qZVXse87z/AR0ZyaSREAAgMbiX6EqgKCU3uVnhFIPiPfbc JyXUC2XWdHT0naUk7vdM9z0D9geDF6Q= X-Google-Smtp-Source: ABdhPJwFWWiEJEuvZIo3v4XiUQD9lbcthCSxA8d28Eqxe0HGMfmw6Cdm3iUAdVDVJWRaUZxc60L7nA== X-Received: by 2002:a5d:670c:: with SMTP id o12mr8419029wru.223.1621017076401; Fri, 14 May 2021 11:31:16 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l14sm7624767wrv.94.2021.05.14.11.31.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 May 2021 11:31:15 -0700 (PDT) Message-Id: <70971b1f9261f2623ad6d6393be1a8e191003eb4.1621017072.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 14 May 2021 18:31:02 +0000 Subject: [PATCH v3 03/12] t1092: expand repository data shape Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee As more features integrate with the sparse-index feature, more and more special cases arise that require different data shapes within the tree structure of the repository in order to demonstrate those cases. Add several interesting special cases all at once instead of sprinkling them across several commits. The interesting cases being added here are: * Add sparse-directory entries on both sides of directories within the sparse-checkout definition. * Add directories outside the sparse-checkout definition who have only one entry and are the first entry of a directory with multiple entries. Later tests will take advantage of these shapes, but they also deepen the tests that already exist. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 4f2f09b53a32..98257695979a 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -17,7 +17,7 @@ test_expect_success 'setup' ' echo "after folder1" >g && echo "after x" >z && mkdir folder1 folder2 deep x && - mkdir deep/deeper1 deep/deeper2 && + mkdir deep/deeper1 deep/deeper2 deep/before deep/later && mkdir deep/deeper1/deepest && echo "after deeper1" >deep/e && echo "after deepest" >deep/deeper1/e && @@ -25,10 +25,16 @@ test_expect_success 'setup' ' cp a folder2 && cp a x && cp a deep && + cp a deep/before && cp a deep/deeper1 && cp a deep/deeper2 && + cp a deep/later && cp a deep/deeper1/deepest && cp -r deep/deeper1/deepest deep/deeper2 && + mkdir deep/deeper1/0 && + mkdir deep/deeper1/0/0 && + touch deep/deeper1/0/1 && + touch deep/deeper1/0/0/0 && git add . && git commit -m "initial commit" && git checkout -b base && From patchwork Fri May 14 18:31:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12258587 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DEB23C433B4 for ; Fri, 14 May 2021 18:31:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C397861444 for ; Fri, 14 May 2021 18:31:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233749AbhENScf (ORCPT ); Fri, 14 May 2021 14:32:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50258 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233616AbhENSca (ORCPT ); Fri, 14 May 2021 14:32:30 -0400 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 93F6EC06175F for ; Fri, 14 May 2021 11:31:18 -0700 (PDT) Received: by mail-wm1-x330.google.com with SMTP id o6-20020a05600c4fc6b029015ec06d5269so235948wmq.0 for ; Fri, 14 May 2021 11:31:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=pAyAAv6l4SLCdrEeaFHhQFp08BencxoPH4VLlINqObY=; b=Dn0MYHe13HsK0uSFKBIBOt7KkWU6fMGjJaZ0QpHBGIx/VPMBdOIyOxuXVElkYGwdeD DfzqxiVSLyNCk0bb002lcTCvbEcj2KUVpWLqjxt2/ocNBdNz6ITn/Uio1cyXNrPsF5T+ 0m/JgndP2OTsdr4eUshDWJL3ZFHD3P0+CjIH5AqFhYb5cfl7W+CUtNWFAPZN88foJxCg d7hk87jjNU9GxkAlkRdcWn6XY9UNQtQShc5oN+yv0W1efjAwuCTk5hQ79bfQACcs0ZpU UWhfUhEZudOG7zxzEWm299d5PatfTNmg4TWIT4j50QvG1uMomXl74NZw0uYo50VbR2jS BYTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=pAyAAv6l4SLCdrEeaFHhQFp08BencxoPH4VLlINqObY=; b=JO6lK6EOio0XGSAFpvsAgc2AUnnCE6v7kC79/rjcFgXyW4OOy7rsBl3abNQGx17+a9 5Pw17TjtOiSnlObPWSUApSnuVhzbrl5M1OKHABIQ29tuVRjg8GGkj1F51fM3aPb8DAFZ qdWcQncN7kF8zI1Qrql9koTr/meFMOiL+Azz111AzVxg+DCbrYsuNxyYT4wrDoZGKbcb qqkdSYJ/DIZvjjwzSquzYmBkimzMhlC9WBmbJSv+iSnY5BLG5qznctxWkQ732gKR4d+R nHdNbcWgLhDCNbj6HfsgyCwqqhAAdaz68EV2I6hV0hNAa8cGGDiaNMLnhK5bW1Gqtljk CX9w== X-Gm-Message-State: AOAM531zOAlQ1TxVDOZ3G4Hzi6pH3pL5r0xuwOrp+vvNkz19p6Ucdi2S uJYrgMSqZRWA5cPvvsevOuM71XXuF30= X-Google-Smtp-Source: ABdhPJy5p2ZCykh5Tqe72WY4LHhv9mg/hoNqPy2w1J7s1XO3DzFxseg/CcjRXMbkZIDFty2utQQlbg== X-Received: by 2002:a05:600c:4fd3:: with SMTP id o19mr10488350wmq.17.1621017077351; Fri, 14 May 2021 11:31:17 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 61sm8008268wrm.52.2021.05.14.11.31.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 May 2021 11:31:16 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Fri, 14 May 2021 18:31:03 +0000 Subject: [PATCH v3 04/12] t1092: add tests for status/add and sparse files Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Before moving to update 'git status' and 'git add' to work with sparse indexes, add an explicit test that ensures the sparse-index works the same as a normal sparse-checkout when the worktree contains directories and files outside of the sparse cone. Specifically, 'folder1/a' is a file in our test repo, but 'folder1' is not in the sparse cone. When 'folder1/a' is modified, the file is not shown as modified and adding it will fail. This is new behavior as of a20f704 (add: warn when asked to update SKIP_WORKTREE entries, 2021-04-08). Before that change, these adds would be silently ignored. Untracked files are fine: adding new files both with 'git add .' and 'git add folder1/' works just as in a full checkout. This may not be entirely desirable, but we are not intending to change behavior at the moment, only document it. A future change could alter the behavior to be more sensible, and this test could be modified to satisfy the new expected behavior. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 38 ++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 98257695979a..fba98d5484ae 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -238,6 +238,44 @@ test_expect_success 'add, commit, checkout' ' test_all_match git checkout - ' +test_expect_success 'status/add: outside sparse cone' ' + init_repos && + + # adding a "missing" file outside the cone should fail + test_sparse_match test_must_fail git add folder1/a && + + # folder1 is at HEAD, but outside the sparse cone + run_on_sparse mkdir folder1 && + cp initial-repo/folder1/a sparse-checkout/folder1/a && + cp initial-repo/folder1/a sparse-index/folder1/a && + + test_sparse_match git status && + + write_script edit-contents <<-\EOF && + echo text >>$1 + EOF + run_on_sparse ../edit-contents folder1/a && + run_on_all ../edit-contents folder1/new && + + test_sparse_match git status --porcelain=v2 && + + # This "git add folder1/a" fails with a warning + # in the sparse repos, differing from the full + # repo. This is intentional. + test_sparse_match test_must_fail git add folder1/a && + test_sparse_match test_must_fail git add --refresh folder1/a && + test_all_match git status --porcelain=v2 && + + test_all_match git add . && + test_all_match git status --porcelain=v2 && + test_all_match git commit -m folder1/new && + + run_on_all ../edit-contents folder1/newer && + test_all_match git add folder1/ && + test_all_match git status --porcelain=v2 && + test_all_match git commit -m folder1/newer +' + test_expect_success 'checkout and reset --hard' ' init_repos && From patchwork Fri May 14 18:31:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12258595 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19DF7C433ED for ; Fri, 14 May 2021 18:31:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id F1BFA61444 for ; Fri, 14 May 2021 18:31:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233813AbhENSci (ORCPT ); Fri, 14 May 2021 14:32:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50264 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233661AbhENScb (ORCPT ); Fri, 14 May 2021 14:32:31 -0400 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6ECC8C06174A for ; Fri, 14 May 2021 11:31:19 -0700 (PDT) Received: by mail-wm1-x332.google.com with SMTP id b19-20020a05600c06d3b029014258a636e8so214751wmn.2 for ; Fri, 14 May 2021 11:31:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=wFfpok35DynSJzLw2iHIzdshoI7yibKc8zfxCLq0Mn8=; b=iFlK0hB5qcmq25w1ecdyMbTFXPTdD3be4NsiXaAJ5jsj9tfSnkP4tXkVuXwtASaCYs IC8jC/1fHnDg4HZxa9Tn4rnziHQLYhq9ys2zjKpN1zOlnuIZOEs4L0XIfTViQR8gi65v av9E69PrrUSJgnXzhCmRfg6WSKLHL1YTyghNrfcyZCLzY31Az/IGfbzgAD7AwE6ohtos UKMD+ZhBAx0SaQ8mFligUMCTkC0XoHM7tYnAWqn+7K5NSfUy4itCs+3REAgucONVIZWH 0rV6Izboc8C179FC6skqWO2zPMArJP0DWD+PqBCDr9ffP4LzXq5eBEtD3TgyqNWrZqJ4 /R8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=wFfpok35DynSJzLw2iHIzdshoI7yibKc8zfxCLq0Mn8=; b=le44p7g+l0tLnN84EDjERngS+m/zpCyovTmFSLDJQdm9qcGNPvjcYCM5Z56f3SAXkR z1WngUblRZF5q2EME6CnWqGya+Mrr5ZRlm82laZD1k/d0Yo/z1oZFk6c7nDlhGDqL3Ad NkP2fM9mfZxFl1tFAStaKE3qivjsvSRFJmta3I1IY4w1Z6hwtLqQCymOhmchAUUTKz4J PY+ZWPnEv8A96WzlQwm3kdQshyqfmkYOkpEmLQj8o6rsYUabjC8oA9yExthJjqnqlpMs LcjeQiM4PiFxvnV+gip7VlhDSuM1MqVioSN6Qln0Q00e4gdU2JXVYapBwx1XSBnkgQYk XdTA== X-Gm-Message-State: AOAM531bfBCC0XKUuiecgeQqKpY56kVcFslUaOZQXrPVk9CcTARNHKiX il+gqab5I/G0psv0/UjGxfDP+JsbLTk= X-Google-Smtp-Source: ABdhPJzSFOweu4X0DRneEM3cpPmTJC0Ti31ex93IWax0cxBYnMlDGl1/T7oKp/0VLNZ7DF2eGG0DLA== X-Received: by 2002:a05:600c:4f4d:: with SMTP id m13mr51509375wmq.4.1621017078195; Fri, 14 May 2021 11:31:18 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id h9sm5877183wmb.35.2021.05.14.11.31.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 May 2021 11:31:17 -0700 (PDT) Message-Id: <07a45b661c4a6ecd63ede19c9e058a8879069742.1621017072.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 14 May 2021 18:31:04 +0000 Subject: [PATCH v3 05/12] unpack-trees: preserve cache_bottom Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The cache_bottom member of 'struct unpack_trees_options' is used to track the range of index entries corresponding to a node of the cache tree. While recursing with traverse_by_cache_tree(), this value is preserved on the call stack using a local and then restored as that method returns. The mark_ce_used() method normally modifies the cache_bottom member when it refers to the marked cache entry. However, sparse directory entries are stored as nodes in the cache-tree data structure as of 2de37c53 (cache-tree: integrate with sparse directory entries, 2021-03-30). Thus, the cache_bottom will be modified as the cache-tree walk advances. Do not update it as well within mark_ce_used(). Signed-off-by: Derrick Stolee --- unpack-trees.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/unpack-trees.c b/unpack-trees.c index dddf106d5bd4..1067db19c9d2 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -586,6 +586,13 @@ static void mark_ce_used(struct cache_entry *ce, struct unpack_trees_options *o) { ce->ce_flags |= CE_UNPACKED; + /* + * If this is a sparse directory, don't advance cache_bottom. + * That will be advanced later using the cache-tree data. + */ + if (S_ISSPARSEDIR(ce->ce_mode)) + return; + if (o->cache_bottom < o->src_index->cache_nr && o->src_index->cache[o->cache_bottom] == ce) { int bottom = o->cache_bottom; From patchwork Fri May 14 18:31:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12258599 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32125C43460 for ; Fri, 14 May 2021 18:31:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 19345610FC for ; Fri, 14 May 2021 18:31:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235227AbhENScn (ORCPT ); Fri, 14 May 2021 14:32:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50260 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233718AbhENSce (ORCPT ); Fri, 14 May 2021 14:32:34 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3A29BC061756 for ; Fri, 14 May 2021 11:31:20 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id x8so152034wrq.9 for ; Fri, 14 May 2021 11:31:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=8BiWBmJ/VC9KkVb4hI/TM5y8+2cX+/ZhlTUXVP76k9Q=; b=irV5qHlWmd+EKrzcuilaJibVR0/LQO0LOT8OMyIDyqNXzOI/X7BKDqCBYijLal2kjU 5OHG1jXyaU1Ph3X7YpwoRm0rQM5gwZ/r+pSI/egFRakr64ioXresb7XxI38rJ2fr67Vq sEkHJxlOLS8OwRsNEiHUfjw2ZnaIwx4td2cDqq+DH4LDwaTGzf2WMgbdYXftMwo7y0uJ PmycvuE/8WogU8PT+0GI2obB6LoCOsFILjCJCpSfgL1G2sGCc2M2xN3JJbn0jiAJN+Ky MfDHiz3pe1I7iXzk4XRlozYnw5pSNcGiRkfn4GBA+uWsrppBZtXWNtC7iD/rKHLXfsIg X3nQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=8BiWBmJ/VC9KkVb4hI/TM5y8+2cX+/ZhlTUXVP76k9Q=; b=SZknkx5dLbIoUYPlL+7p9WlTA6VfRMRc2Gl+f69QUf00fOvRm/DW+KEOpsUFvFrn9D CF48nRQB6UjbtP6mpv5XLOf4J8hmOdLMAJk3qJOwRP9J3XTssxaSvfN93xg4ERyDzyTR svAlJL7lpTV2Az8qlYRUO35q/F7SJBipt/4nGZE/+TKlPcwiWvR99musXv+fyVKNFML2 m2vz3olUmb/owj+aL53PKRvNPxXLrbZUUbkkboPZKFyqS+dIq1T65ttXAZbiXOaqeT8Q Nym1FzSS5WgFAegbLNBAuhCeYO5U9UPxLca2BxudPz0/KP4c7CyGJ6reZOTebN4VSipI ZBQA== X-Gm-Message-State: AOAM530S3vmPisOdOIe0VYiCO64uqlZBqjjO9ASfrLZxUJa9hCluZrmp TRyd1dkwjQ/ejZmtHx1+rUG6KW3MwZo= X-Google-Smtp-Source: ABdhPJwSMTvJFMD0L3E07OIKChk9DyWrmyIE/hSAc8Tbt60+nKc4g7rKjnwTommhKwoQJsta79q1zA== X-Received: by 2002:a05:6000:18a4:: with SMTP id b4mr58849930wri.86.1621017079051; Fri, 14 May 2021 11:31:19 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id b12sm7354246wro.28.2021.05.14.11.31.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 May 2021 11:31:18 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Fri, 14 May 2021 18:31:05 +0000 Subject: [PATCH v3 06/12] unpack-trees: compare sparse directories correctly Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee As we further integrate the sparse-index into unpack-trees, we need to ensure that we compare sparse directory entries correctly with other entries. This affects searching for an exact path as well as sorting index entries. Sparse directory entries contain the trailing directory separator. This is important for the sorting, in particular. Thus, within do_compare_entry() we stop using S_IFREG in all cases, since sparse directories should use S_IFDIR to indicate that the comparison should treat the entry name as a dirctory. Within compare_entry(), it first calls do_compare_entry() to check the leading portion of the name. When the input path is a directory name, we could match exactly already. Thus, we should return 0 if we have an exact string match on a sparse directory entry. Signed-off-by: Derrick Stolee --- unpack-trees.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/unpack-trees.c b/unpack-trees.c index 1067db19c9d2..ef6a2b1c951c 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -969,6 +969,7 @@ static int do_compare_entry(const struct cache_entry *ce, int pathlen, ce_len; const char *ce_name; int cmp; + unsigned ce_mode; /* * If we have not precomputed the traverse path, it is quicker @@ -991,7 +992,8 @@ static int do_compare_entry(const struct cache_entry *ce, ce_len -= pathlen; ce_name = ce->name + pathlen; - return df_name_compare(ce_name, ce_len, S_IFREG, name, namelen, mode); + ce_mode = S_ISSPARSEDIR(ce->ce_mode) ? S_IFDIR : S_IFREG; + return df_name_compare(ce_name, ce_len, ce_mode, name, namelen, mode); } static int compare_entry(const struct cache_entry *ce, const struct traverse_info *info, const struct name_entry *n) @@ -1000,6 +1002,15 @@ static int compare_entry(const struct cache_entry *ce, const struct traverse_inf if (cmp) return cmp; + /* + * At this point, we know that we have a prefix match. If ce + * is a sparse directory, then allow an exact match. This only + * works when the input name is a directory, since ce->name + * ends in a directory separator. + */ + if (S_ISSPARSEDIR(ce->ce_mode)) + return 0; + /* * Even if the beginning compared identically, the ce should * compare as bigger than a directory leading up to it! From patchwork Fri May 14 18:31:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12258597 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C61FBC433B4 for ; Fri, 14 May 2021 18:31:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AAD4161440 for ; Fri, 14 May 2021 18:31:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233722AbhENScl (ORCPT ); Fri, 14 May 2021 14:32:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50276 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229986AbhENScd (ORCPT ); Fri, 14 May 2021 14:32:33 -0400 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F139DC06175F for ; Fri, 14 May 2021 11:31:20 -0700 (PDT) Received: by mail-wm1-x332.google.com with SMTP id b11-20020a7bc24b0000b0290148da0694ffso1868027wmj.2 for ; Fri, 14 May 2021 11:31:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=7yWmoxa4P2IhgSPJe7dSIEDaLEpg2XrJKHHPNNjTM+s=; b=gF6+fkUuFl/3oPStvz44QlDWpG6FBg/3UDbBtooZsqd/1tk5X0LJk0esWJuC2xGoju U6579ZlW/kCdSR8YV9QHqSyVagDh3iHC0TgH/MgmgpwjQyY4E0X2tsjippw49I7oR+Ir Fe0A+yhR5IQ/ux7BZHBoBmpGVG3AywbGZyqsWqzFg1o+TZUgJuS+5HsJwSb9Yv0PsOci PzsyzAJHBzigI/QS5s1pUCYPEdM++aCd/fNkByL+FiVAPqdBgyh1TYTgXoZEy/LvDp0T PC9d69n09673OTwr75ClxLhSw/DbvRSBNgkJo+W+8+7fvGXbYcHID98+6dii5cKh+AFi zkBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=7yWmoxa4P2IhgSPJe7dSIEDaLEpg2XrJKHHPNNjTM+s=; b=IS6FuEr03pDSewnQSA4RYYQ6VQ2WGb3cGCy64JgFGDPKO16bOGeHAFHpL0yM63SvUS /rs/LwJIjpzkX63okeq6vedpRncdL6GkktxsE8S+r0f6365EKeUIP9CNeWM+/l90Q24W BxRqQqPquciQVGcv7xQ7RSLlYUNAxU2NTq9POhDNjIweSrdjm3sHFJrzDADth6aJULGU KNEpnk8JD0+wgn6xKUQOpUbmakcEYWlI+hjZKeRHhnYXtQ/GPU/ybh5WDA3D8WMJFwsB rVDQ1Lv/8foRFiOi4TamnIpRiOj98AgLCvMDQtqdMMava6W81vexOP2Qo/IG6t60zdSY XS8Q== X-Gm-Message-State: AOAM532KUyuPhCBUNKT6F/XZUDjOdHiYsODA3QEqcnKAxqbp8AD3SC8I j8xK4DaCb3H8O1odOSFhU710VGTK/o8= X-Google-Smtp-Source: ABdhPJxUu/Rb+IW1JeaxB0Ui6fMlFtPSxzw//uNniNdfGTIktrv6Q120NRzhTbPQkKsazFKcSXNQxw== X-Received: by 2002:a7b:cc15:: with SMTP id f21mr10961177wmh.86.1621017079758; Fri, 14 May 2021 11:31:19 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id u126sm106111wmb.9.2021.05.14.11.31.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 May 2021 11:31:19 -0700 (PDT) Message-Id: <598375d3531fabe8582cb6d93838df09e1bd06c3.1621017072.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 14 May 2021 18:31:06 +0000 Subject: [PATCH v3 07/12] unpack-trees: stop recursing into sparse directories Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee When walking trees using traverse_trees_recursive() and unpack_callback(), we must not attempt to walk into a sparse directory entry. There are no index entries within that directory to compare to the tree object at that position, so skip over the entries of that tree. This code is used in many places, so the only way to test it is to start removing the command_requres_full_index option from one builtin at a time and carefully test that its use of unpack_trees() behaves correctly with a sparse-index. Such tests will be added by later changes. Signed-off-by: Derrick Stolee --- diff-lib.c | 6 ++++++ unpack-trees.c | 7 +++++-- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/diff-lib.c b/diff-lib.c index b73cc1859a49..d5e7e01132ee 100644 --- a/diff-lib.c +++ b/diff-lib.c @@ -322,6 +322,9 @@ static void show_new_file(struct rev_info *revs, unsigned int mode; unsigned dirty_submodule = 0; + if (S_ISSPARSEDIR(new_file->ce_mode)) + return; + /* * New file in the index: it might actually be different in * the working tree. @@ -343,6 +346,9 @@ static int show_modified(struct rev_info *revs, const struct object_id *oid; unsigned dirty_submodule = 0; + if (S_ISSPARSEDIR(new_entry->ce_mode)) + return 0; + if (get_stat_data(new_entry, &oid, &mode, cached, match_missing, &dirty_submodule, &revs->diffopt) < 0) { if (report_missing) diff --git a/unpack-trees.c b/unpack-trees.c index ef6a2b1c951c..703b0bdc9dfd 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -1261,6 +1261,7 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str struct cache_entry *src[MAX_UNPACK_TREES + 1] = { NULL, }; struct unpack_trees_options *o = info->data; const struct name_entry *p = names; + unsigned unpack_tree = 1; /* Find first entry with a real name (we could use "mask" too) */ while (!p->mode) @@ -1307,7 +1308,8 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str } } - if (unpack_nondirectories(n, mask, dirmask, src, names, info) < 0) + if (unpack_tree && + unpack_nondirectories(n, mask, dirmask, src, names, info) < 0) return -1; if (o->merge && src[0]) { @@ -1337,7 +1339,8 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str } } - if (traverse_trees_recursive(n, dirmask, mask & ~dirmask, + if (unpack_tree && + traverse_trees_recursive(n, dirmask, mask & ~dirmask, names, info) < 0) return -1; return mask; From patchwork Fri May 14 18:31:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12258609 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E26DBC433ED for ; Fri, 14 May 2021 18:31:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C8DAF61440 for ; Fri, 14 May 2021 18:31:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233819AbhENScv (ORCPT ); Fri, 14 May 2021 14:32:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50278 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233573AbhENSci (ORCPT ); Fri, 14 May 2021 14:32:38 -0400 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EE72AC061760 for ; Fri, 14 May 2021 11:31:21 -0700 (PDT) Received: by mail-wm1-x32d.google.com with SMTP id f75-20020a1c1f4e0000b0290171001e7329so218752wmf.1 for ; Fri, 14 May 2021 11:31:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=IKPjTUqki+7odL+EABJRxmdzP4v6AWl9RXlIatOMDLM=; b=pSB21RPeDgNv9QiLYrSqwohqTkokd+n7t+kdchckQBtqYMLNUtlf0ABeaY4pXAO+qN wzChLu+mSOlFrdcoY6JYqG1m327M8KPcixEg91cYZ7niujCIVSwhp9UM4UOEaHylH9KC 2pPV6/eAn5H001CwtY1aEZSq6a6K2G+aMaSFRJfhWhYgW+r7YrxpyO8zMLRtYmr9aP1e v9Fv00QvDTnLVItNyBzngNVvDLN7VvGkOynh7FB8U6Czzbqab0FY/UWE4jok7cpeVchN usoIE0AYs3JrMrA9u69q3cUTjZAVOboIZCajzglFMa2WPIit2sjVME3UCr7rJ+KDYzYo aMkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=IKPjTUqki+7odL+EABJRxmdzP4v6AWl9RXlIatOMDLM=; b=DD8KwHi/ovtYbWbXoCzaHcLyYzdix9lYGxDq0Qy89IvblHs+vGd6eDczz3UoIwaKWt ld3oMBTYWA+TQJ8GVw/g4gxupujp5V1ZB9/ksE0rQeWr7gvJ8HhoGgf9D3cpGGIxAXNQ ifsNczaoVZLjPWU8A1XcKkV9+z3B9ENk8QTcYeOg2wHBD67hrX8i6tMsjFX8Y0FM+2sC mjniQ5w0QdXBlVXM9haTurt/+ObCPnz4uWseJIIj3pLREciWpbFloUvs0qiyTnsHQbv8 vSPRrlK0oMSDGZ10Zxotl1KcxQAFXw7AwLDART8+QTnQxWoi9cvvdSEEoXxnIMRwsM8z Q3Og== X-Gm-Message-State: AOAM533iut/LWJuTPCHZrT99a9DQOVh94URG4HtfsHrgd/8CTJRzT0Jq 4AGcSIWyn+Zf0pjfjp8LedTLsw8G9ZY= X-Google-Smtp-Source: ABdhPJzpcvAFORdLbmQSvbRGNEAYgu5u32zQVIldfBvjYaV2DKlD122j1HhpHF1Qal02q+3Nt7tY3A== X-Received: by 2002:a1c:7711:: with SMTP id t17mr51809198wmi.6.1621017080758; Fri, 14 May 2021 11:31:20 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id b8sm7532729wrx.15.2021.05.14.11.31.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 May 2021 11:31:20 -0700 (PDT) Message-Id: <47da2b317237c4ce8c95ce847e1c65c2d82fcfb7.1621017072.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 14 May 2021 18:31:07 +0000 Subject: [PATCH v3 08/12] dir.c: accept a directory as part of cone-mode patterns Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee When we have sparse directory entries in the index, we want to compare that directory against sparse-checkout patterns. Those pattern matching algorithms are built expecting a file path, not a directory path. This is especially important in the "cone mode" patterns which will match files that exist within the "parent directories" as well as the recursive directory matches. If path_matches_pattern_list() is given a directory, we can add a fake filename ("-") to the directory and get the same results as before, assuming we are in cone mode. Since sparse index requires cone mode patterns, this is an acceptable assumption. Signed-off-by: Derrick Stolee --- dir.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/dir.c b/dir.c index 166238e79f52..ab76ef286495 100644 --- a/dir.c +++ b/dir.c @@ -1378,6 +1378,17 @@ enum pattern_match_result path_matches_pattern_list( strbuf_addch(&parent_pathname, '/'); strbuf_add(&parent_pathname, pathname, pathlen); + /* + * Directory entries are matched if and only if a file + * contained immediately within them is matched. For the + * case of a directory entry, modify the path to create + * a fake filename within this directory, allowing us to + * use the file-base matching logic in an equivalent way. + */ + if (parent_pathname.len > 0 && + parent_pathname.buf[parent_pathname.len - 1] == '/') + strbuf_add(&parent_pathname, "-", 1); + if (hashmap_contains_path(&pl->recursive_hashmap, &parent_pathname)) { result = MATCHED_RECURSIVE; From patchwork Fri May 14 18:31:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12258601 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C764C433ED for ; Fri, 14 May 2021 18:31:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1131D613EB for ; Fri, 14 May 2021 18:31:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235332AbhENScp (ORCPT ); Fri, 14 May 2021 14:32:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50286 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233719AbhENSce (ORCPT ); Fri, 14 May 2021 14:32:34 -0400 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D0658C0613ED for ; Fri, 14 May 2021 11:31:22 -0700 (PDT) Received: by mail-wr1-x42f.google.com with SMTP id i17so142903wrq.11 for ; Fri, 14 May 2021 11:31:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=aieF5APTh0NLP7SmOzgU2W+ZxUYW0jJ5qZJaNE169FM=; b=k2sdDVtPNJTvSDz2IiiXIcGazfEI/AA7B28nxGLAWHtyHPYFk9beudZ6W8GDsPH2gk yzDDcF7SZlNITTYd3jGgG9MOhjhu/K5AjkJ4/B6MXNWcKuems1DDB6Hbs7kdX62FPmiD 1gKWya1yVMY7bRcOfBNq6I4aTKm6GxgNmb3PHxKm0DgxShTXLdWlmtNdVsrjhOCKXD0n TZDftlwgr0wsQel52LPdhAvn4o4DaMb8mY1swNli1O5FyLzUjTqkJY6qjS4tqMiBp+S/ nE+dMhsiiX1yF8z2yHK2Pai/jNa9E1VIRk0BnpMh451g1cy7G2q4uOXNf9mnWfMW2Mf5 9+Zw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=aieF5APTh0NLP7SmOzgU2W+ZxUYW0jJ5qZJaNE169FM=; b=jdRNt6WEVXSchlnf4Mv3mtqNmRxvWS8Y1nem1VeRU9cxdtJ1HX5qLCwAgRuy5qSgGH qWz0pjCsnSx1L4MiGUVdJK26d9c9sRgIOD+C9Nz6l+sqqnID3ER6JZG6gY/exPNqCPB/ lWGpBlYmDePXVuaKoDlDLNdgXZ314nUa7KKutLYXg2PAHFl9T1QuT3Qz8RxM4tZ8p/Pg SUyxHP2AurmUPu688vA/mLb2QNxz/yN3OYWsuiGz/Kx8llnMGxIeVKTcISbx7s/M+5Su cNCB8JJPUcBiLr/9YOmsGcrg44BjpV4GE8Ht7lmRFGpKeYPStO3TUEGO49e597oW8o+4 2hGg== X-Gm-Message-State: AOAM531ZZxBWQfh4aGtwVjRJQvOrIRv2MVElkecywffjCKfbCGhvlwU6 5yjRUiw9gv8OsH8eDtdAFL+tj2lZ2lY= X-Google-Smtp-Source: ABdhPJy9cWkGbHEVLwyJkY2LU5m14WmP95ih27W1zMV3+AT5YrxAF4bvRjxBeFgSIgd4NGbdc0QNZg== X-Received: by 2002:a5d:5301:: with SMTP id e1mr17738450wrv.36.1621017081673; Fri, 14 May 2021 11:31:21 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f11sm12266639wmq.41.2021.05.14.11.31.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 May 2021 11:31:21 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Fri, 14 May 2021 18:31:08 +0000 Subject: [PATCH v3 09/12] status: skip sparse-checkout percentage with sparse-index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee 'git status' began reporting a percentage of populated paths when sparse-checkout is enabled in 051df3cf (wt-status: show sparse checkout status as well, 2020-07-18). This percentage is incorrect when the index has sparse directories. It would also be expensive to calculate as we would need to parse trees to count the total number of possible paths. Avoid the expensive computation by simplifying the output to only report that a sparse checkout exists, without the percentage. This change is the reason we use 'git status --porcelain=v2' in t1092-sparse-checkout-compatibility.sh. We don't want to ensure that this message is equal across both modes, but instead just the important information about staged, modified, and untracked files are compared. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 8 ++++++++ wt-status.c | 14 +++++++++++--- wt-status.h | 1 + 3 files changed, 20 insertions(+), 3 deletions(-) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index fba98d5484ae..34dae7fbcadd 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -202,6 +202,14 @@ test_expect_success 'status with options' ' test_all_match git status --porcelain=v2 -uno ' +test_expect_success 'status reports sparse-checkout' ' + init_repos && + git -C sparse-checkout status >full && + git -C sparse-index status >sparse && + test_i18ngrep "You are in a sparse checkout with " full && + test_i18ngrep "You are in a sparse checkout." sparse +' + test_expect_success 'add, commit, checkout' ' init_repos && diff --git a/wt-status.c b/wt-status.c index 0c8287a023e4..0425169c1895 100644 --- a/wt-status.c +++ b/wt-status.c @@ -1490,9 +1490,12 @@ static void show_sparse_checkout_in_use(struct wt_status *s, if (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_DISABLED) return; - status_printf_ln(s, color, - _("You are in a sparse checkout with %d%% of tracked files present."), - s->state.sparse_checkout_percentage); + if (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_SPARSE_INDEX) + status_printf_ln(s, color, _("You are in a sparse checkout.")); + else + status_printf_ln(s, color, + _("You are in a sparse checkout with %d%% of tracked files present."), + s->state.sparse_checkout_percentage); wt_longstatus_print_trailer(s); } @@ -1650,6 +1653,11 @@ static void wt_status_check_sparse_checkout(struct repository *r, return; } + if (r->index->sparse_index) { + state->sparse_checkout_percentage = SPARSE_CHECKOUT_SPARSE_INDEX; + return; + } + for (i = 0; i < r->index->cache_nr; i++) { struct cache_entry *ce = r->index->cache[i]; if (ce_skip_worktree(ce)) diff --git a/wt-status.h b/wt-status.h index 0d32799b28e1..ab9cc9d8f032 100644 --- a/wt-status.h +++ b/wt-status.h @@ -78,6 +78,7 @@ enum wt_status_format { }; #define SPARSE_CHECKOUT_DISABLED -1 +#define SPARSE_CHECKOUT_SPARSE_INDEX -2 struct wt_status_state { int merge_in_progress; From patchwork Fri May 14 18:31:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12258603 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AF67EC433B4 for ; Fri, 14 May 2021 18:31:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 92285613EB for ; Fri, 14 May 2021 18:31:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233807AbhENScr (ORCPT ); Fri, 14 May 2021 14:32:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50262 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233795AbhENSch (ORCPT ); Fri, 14 May 2021 14:32:37 -0400 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BF176C06138B for ; Fri, 14 May 2021 11:31:23 -0700 (PDT) Received: by mail-wr1-x42f.google.com with SMTP id r12so192417wrp.1 for ; Fri, 14 May 2021 11:31:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=gu6GEJgKuJhO6OfWQXS55AAR+pJtihz9mG9qQegVuuA=; b=AnHNx2IRWamOduVnOTyuFXlCCbL+wLdma6oxVRwNyUnH7iDqxolm/tCFUiCM+cor/P wwkmKY8CwMHpU8mY1TmSf46EDdL7eZKFxXidxs2W/sxLexNWd7qH9bBMFCJMazn6MGlT DWsoYmFnrsCsk59hOeX2XsunIrZKvQCek8Y0JcYmqEzzOYXycLD1xkO/yPwi4+ZYh9PM IVzEvuDhz7dwTpnkmFuUE5V86DBuP35dHGIpscApf822GoyeuDrHiNysBFfZrPCG8bMY o6+8fS/eq1Xow19jvDzBfr5/ToCVRUf9swgMBhJQj2FkLdMh9ieiHpcdZ05IHYAOowyX oIbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=gu6GEJgKuJhO6OfWQXS55AAR+pJtihz9mG9qQegVuuA=; b=NDgRHyiC0i7TGjs1QRybUouERz/dwm26XjYVanW3kb+RrN8ar4HSyEWxQ0gYmTm3Sh EBVks+lumA1vk73nev3uGi6ytn7NTN1E0gofce0jbIpH711W+KkTg0qUQadVGDw0MFG6 5GBM+V8pJ/jnTn4s8Sf8YU0ZxICWCpdSHIfYTy8cXyWphQ4C9yl0XWCpeFy+ynfrDvc6 CkDzWsVRoR5h+EHcQLhF4MNIptEiEmiTHWKQVuEk9zKr21/dhYRMRTMiQSI51R1sndBk 0wJzFOiwTsOUqlojacpO4aYjZdxA+C3JReOzrwUF3QHW+HKZ0BPQKfnf3Um7fUv646ts lYiA== X-Gm-Message-State: AOAM532lfTAG38CkrpsKrJ4mnjdXoojq0IUdOFuu6aw53BIRyrIOjRu5 ujxfiYGCQG+8OqalOEWBvKcmhjzhZxI= X-Google-Smtp-Source: ABdhPJzI3oblZPG2BS+LJ1SVVWVN242tIeFMb9sB+YbHUuS+vve6Dz0N04kyuY62UapJ/XPssuD8qQ== X-Received: by 2002:adf:f981:: with SMTP id f1mr21961781wrr.381.1621017082519; Fri, 14 May 2021 11:31:22 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id r5sm1080637wrw.96.2021.05.14.11.31.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 May 2021 11:31:21 -0700 (PDT) Message-Id: <5b1ae369a7cd2700288a6719c496c149b4240d82.1621017072.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 14 May 2021 18:31:09 +0000 Subject: [PATCH v3 10/12] status: use sparse-index throughout Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee By testing 'git -c core.fsmonitor= status -uno', we can check for the simplest index operations that can be made sparse-aware. The necessary implementation details are already integrated with sparse-checkout, so modify command_requires_full_index to be zero for cmd_status(). In refresh_index(), we loop through the index entries to refresh their stat() information. However, sparse directories have no stat() information to populate. Ignore these entries. This allows 'git status' to no longer expand a sparse index to a full one. This is further tested by dropping the "-uno" option and adding an untracked file into the worktree. The performance test p2000-sparse-checkout-operations.sh demonstrates these improvements: Test HEAD~1 HEAD ----------------------------------------------------------------------------- 2000.2: git status (full-index-v3) 0.31(0.30+0.05) 0.31(0.29+0.06) +0.0% 2000.3: git status (full-index-v4) 0.31(0.29+0.07) 0.34(0.30+0.08) +9.7% 2000.4: git status (sparse-index-v3) 2.35(2.28+0.10) 0.04(0.04+0.05) -98.3% 2000.5: git status (sparse-index-v4) 2.35(2.24+0.15) 0.05(0.04+0.06) -97.9% Note that since HEAD~1 was expanding the sparse index by parsing trees, it was artificially slower than the full index case. Thus, the 98% improvement is misleading, and instead we should celebrate the 0.34s to 0.05s improvement of 85%. This is more indicative of the peformance gains we are expecting by using a sparse index. Note: we are dropping the assignment of core.fsmonitor here. This is not necessary for the test script as we are not altering the config any other way. Correct integration with FS Monitor will be validated in later changes. Signed-off-by: Derrick Stolee --- builtin/commit.c | 3 +++ read-cache.c | 10 ++++++++-- t/t1092-sparse-checkout-compatibility.sh | 13 +++++++++---- 3 files changed, 20 insertions(+), 6 deletions(-) diff --git a/builtin/commit.c b/builtin/commit.c index cf0c36d1dcb2..e529da7beadd 100644 --- a/builtin/commit.c +++ b/builtin/commit.c @@ -1404,6 +1404,9 @@ int cmd_status(int argc, const char **argv, const char *prefix) if (argc == 2 && !strcmp(argv[1], "-h")) usage_with_options(builtin_status_usage, builtin_status_options); + prepare_repo_settings(the_repository); + the_repository->settings.command_requires_full_index = 0; + status_init_config(&s, git_status_config); argc = parse_options(argc, argv, prefix, builtin_status_options, diff --git a/read-cache.c b/read-cache.c index 29ffa9ac5db9..f80e26831b36 100644 --- a/read-cache.c +++ b/read-cache.c @@ -1578,8 +1578,7 @@ int refresh_index(struct index_state *istate, unsigned int flags, */ preload_index(istate, pathspec, 0); trace2_region_enter("index", "refresh", NULL); - /* TODO: audit for interaction with sparse-index. */ - ensure_full_index(istate); + for (i = 0; i < istate->cache_nr; i++) { struct cache_entry *ce, *new_entry; int cache_errno = 0; @@ -1594,6 +1593,13 @@ int refresh_index(struct index_state *istate, unsigned int flags, if (ignore_skip_worktree && ce_skip_worktree(ce)) continue; + /* + * If this entry is a sparse directory, then there isn't + * any stat() information to update. Ignore the entry. + */ + if (S_ISSPARSEDIR(ce->ce_mode)) + continue; + if (pathspec && !ce_path_match(istate, ce, pathspec, seen)) filtered = 1; diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 34dae7fbcadd..59faf7381093 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -479,12 +479,17 @@ test_expect_success 'sparse-index is expanded and converted back' ' GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ git -C sparse-index -c core.fsmonitor="" reset --hard && test_region index convert_to_sparse trace2.txt && - test_region index ensure_full_index trace2.txt && + test_region index ensure_full_index trace2.txt +' - rm trace2.txt && +test_expect_success 'sparse-index is not expanded' ' + init_repos && + + rm -f trace2.txt && + echo >>sparse-index/untracked.txt && GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ - git -C sparse-index -c core.fsmonitor="" status -uno && - test_region index ensure_full_index trace2.txt + git -C sparse-index status && + test_region ! index ensure_full_index trace2.txt ' test_done From patchwork Fri May 14 18:31:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12258605 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF87AC43461 for ; Fri, 14 May 2021 18:31:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B5648613EB for ; Fri, 14 May 2021 18:31:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235340AbhENScs (ORCPT ); Fri, 14 May 2021 14:32:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50300 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233808AbhENSch (ORCPT ); Fri, 14 May 2021 14:32:37 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 91DBFC06138E for ; Fri, 14 May 2021 11:31:24 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id x8so152174wrq.9 for ; Fri, 14 May 2021 11:31:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=jWoJaiihOFErJJTdCshSCWDCsuIYLvVcwyV8zTPU6cQ=; b=Fa+HDuoypsoLHPg44OH2Jq18lfl+J6qXC9ukFH2QPYOsJu4WBP6hRiUYH76XQJf4nR GAyqtJnVfRlLrslwp8nUJTgJ/Vglf/bP/MvAAzzKuyAm2Z+qmu9gMrsKTAvrec/0Ndav gzI6h0C5kyfpx6wY8NbiTI9rBJMOM0VaI+ZUQLHaoLCyLu+S1Y4OgpLzyRqzf/1qXQzB 7TbKz4Dn6y5CExS37MFKuOEibZ6qwvS9NZdBM+qblqlUvtT89r9CvkWmklR0UdAm/YRR /q7dom5G5hSqIJGZYpkjoigVnH34FNpx+0OiSdYfItJm0oj7HorN1VGjRVWThHngaoLB zSVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=jWoJaiihOFErJJTdCshSCWDCsuIYLvVcwyV8zTPU6cQ=; b=ejxrBJZqp9BD1p11wvgl663MfvtkpJDcp+xxOg7xU0GraW7e0DmrVJ+jFhqbZZtdjZ 7MezBK9w+Kgd2S6pOWXcFT57DdKb7YvkPdUlDMKFs44zgTx9gMLN5xY+MIvYvYIFkq1q eO/qe2A5lAEWhHkCQIig37JacLhE4GBu0fLIFl2uaBmxjM5HzBDcuWXCpQWIZzkllv1Y t0dNldTLFmKuP10I3+JsHS1uWd+QL3AzuF3XVq+8OkLCHdPjf1pukF3z9TKo0RULw17S BL8IwSCQ+VU7ScSWc6nxMO1rrKXfzTnPaSXm6heTPgq9GKhHDeTdhTL996ngPDsJjHXt KKLw== X-Gm-Message-State: AOAM532cOagflENaNXgkMkGVSs+pmOeoIxKPwwIf0kDSyYHY/sS2zw2X jO1n3oXuLNROkIzu9ESKC9YQj2qCcjQ= X-Google-Smtp-Source: ABdhPJxUdzwIO6FgT9ULz/IT7X0aIu4KkiHuN/FgrBOTKj5HQ3knub6cFvnTaZo3kOo8/XAmTVKF4Q== X-Received: by 2002:a05:6000:186a:: with SMTP id d10mr61355685wri.41.1621017083323; Fri, 14 May 2021 11:31:23 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id s7sm12009490wmh.35.2021.05.14.11.31.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 May 2021 11:31:22 -0700 (PDT) Message-Id: <3b42783d4a86473420480b2789d61d8103e6e7d4.1621017072.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 14 May 2021 18:31:10 +0000 Subject: [PATCH v3 11/12] wt-status: expand added sparse directory entries Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee It is difficult, but possible, to get into a state where we intend to add a directory that is outside of the sparse-checkout definition. Add a test to t1092-sparse-checkout-compatibility.sh that demonstrates this using a combination of 'git reset --mixed' and 'git checkout --orphan'. This test failed before because the output of 'git status --porcelain=v2' would not match on the lines for folder1/: * The sparse-checkout repo (with a full index) would output each path name that is intended to be added. * The sparse-index repo would only output that "folder1/" is staged for addition. The status should report the full list of files to be added, and so this sparse-directory entry should be expanded to a full list when reaching it inside the wt_status_collect_changes_initial() method. Use read_tree_at() to assist. Somehow, this loop over the cache entries was not guarded by ensure_full_index() as intended. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 28 +++++++++++++ wt-status.c | 50 ++++++++++++++++++++++++ 2 files changed, 78 insertions(+) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 59faf7381093..cd3669d36b53 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -492,4 +492,32 @@ test_expect_success 'sparse-index is not expanded' ' test_region ! index ensure_full_index trace2.txt ' +test_expect_success 'reset mixed and checkout orphan' ' + init_repos && + + test_all_match git checkout rename-out-to-in && + test_all_match git reset --mixed HEAD~1 && + test_sparse_match test-tool read-cache --table --expand && + test_all_match git status --porcelain=v2 && + test_all_match git status --porcelain=v2 && + + # At this point, sparse-checkouts behave differently + # from the full-checkout. + test_sparse_match git checkout --orphan new-branch && + test_sparse_match test-tool read-cache --table --expand && + test_sparse_match git status --porcelain=v2 && + test_sparse_match git status --porcelain=v2 +' + +test_expect_success 'add everything with deep new file' ' + init_repos && + + run_on_sparse git sparse-checkout set deep/deeper1/deepest && + + run_on_all touch deep/deeper1/x && + test_all_match git add . && + test_all_match git status --porcelain=v2 && + test_all_match git status --porcelain=v2 +' + test_done diff --git a/wt-status.c b/wt-status.c index 0425169c1895..90db8bd659fa 100644 --- a/wt-status.c +++ b/wt-status.c @@ -654,6 +654,34 @@ static void wt_status_collect_changes_index(struct wt_status *s) run_diff_index(&rev, 1); } +static int add_file_to_list(const struct object_id *oid, + struct strbuf *base, const char *path, + unsigned int mode, void *context) +{ + struct string_list_item *it; + struct wt_status_change_data *d; + struct wt_status *s = context; + char *full_name; + + if (S_ISDIR(mode)) + return READ_TREE_RECURSIVE; + + full_name = xstrfmt("%s%s", base->buf, path); + it = string_list_insert(&s->change, full_name); + d = it->util; + if (!d) { + CALLOC_ARRAY(d, 1); + it->util = d; + } + + d->index_status = DIFF_STATUS_ADDED; + /* Leave {mode,oid}_head zero for adds. */ + d->mode_index = mode; + oidcpy(&d->oid_index, oid); + s->committable = 1; + return 0; +} + static void wt_status_collect_changes_initial(struct wt_status *s) { struct index_state *istate = s->repo->index; @@ -668,6 +696,28 @@ static void wt_status_collect_changes_initial(struct wt_status *s) continue; if (ce_intent_to_add(ce)) continue; + if (S_ISSPARSEDIR(ce->ce_mode)) { + /* + * This is a sparse directory entry, so we want to collect all + * of the added files within the tree. This requires recursively + * expanding the trees to find the elements that are new in this + * tree and marking them with DIFF_STATUS_ADDED. + */ + struct strbuf base = STRBUF_INIT; + struct pathspec ps; + struct tree *tree = lookup_tree(istate->repo, &ce->oid); + + memset(&ps, 0, sizeof(ps)); + ps.recursive = 1; + ps.has_wildcard = 1; + ps.max_depth = -1; + + strbuf_add(&base, ce->name, ce->ce_namelen); + read_tree_at(istate->repo, tree, &base, &ps, + add_file_to_list, s); + continue; + } + it = string_list_insert(&s->change, ce->name); d = it->util; if (!d) { From patchwork Fri May 14 18:31:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12258607 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E8E1C433B4 for ; Fri, 14 May 2021 18:31:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 77D8C61166 for ; Fri, 14 May 2021 18:31:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233686AbhENScv (ORCPT ); Fri, 14 May 2021 14:32:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50258 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233640AbhENSch (ORCPT ); Fri, 14 May 2021 14:32:37 -0400 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6F74DC06138F for ; Fri, 14 May 2021 11:31:25 -0700 (PDT) Received: by mail-wr1-x436.google.com with SMTP id h4so140948wrt.12 for ; Fri, 14 May 2021 11:31:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=HVK7x9ytblDj8iy4lmiMNtwNDpdpLC/uAkky1UzqoLw=; b=hQ1lrAIRnMZo58X9mawtlOtMDVP1vsRXgcc6X814AUjB686fzIA37E1yfAP8M0DvOJ ZpJMyoU8M+1x6sRGKaRQdLNljXyxExxmMS6+lmbnsiYDCMqELk/liLgA6/JhTET/blSp f08XKky9kjbkoNUJJaUGY9BxwvUMSvZOCohJBFTlWEA8o3d0zKlhna4GkbCW5FqsRo5i 8Yyfg+MK4KjJUR1lia8HDYEoKMW5g1HBKEFSGQShbwSbBLSeV4i8hoeVBT0OYDlFCrDA WNi3wwCKfcmyO8nF5lPTcDUREH3lFeGBaH9LiMh6cY/EO/F2scl1/V7TD05YWWOraqL3 0tdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=HVK7x9ytblDj8iy4lmiMNtwNDpdpLC/uAkky1UzqoLw=; b=j9DXAzGNBXfMO1QpoOhwOAxbcJvpE4HeDZS+WfXtXotGD9Tlo/31umRdu3iR70vNp5 Ot5BvK11VgmHMX37RddEAqR9D0xZdIMst5GuGOuP4ZcDTlTkYlAD+kHyCBOfUd8DAuhZ htPH7zfwP31RB9QEL6Ote/kLdfwWP1OoU5840Geyo2ukGAMeQyFYIyu9r9bwsrdb+5Av k/qc4hnR3YwY0NF9YDXaughJ5Usdn46WtoqIZI30YxiwevYo8M5njwv49oz3sqgYBxhy TdjJgbS3xzbV/HI9w1uu07RuKZNcT+9lrxiHkmIrAtwGVpxyv7MD1K79SR8nKI72vKAu ITLA== X-Gm-Message-State: AOAM533vGdPQ6+IJoHGMVn+D9qIRsElpiC2XA4CQU0KXTzHbDvhVU++l MILGeBzqr1aCpFQZ87SFuKqfy32C3t0= X-Google-Smtp-Source: ABdhPJx7WWwFgulPceYHRmIEoaVQpcX1PgoWyuDNDL0j5SV7BvIYkA+7bvw8EgE6iUXdcfqsuTOesA== X-Received: by 2002:a5d:410e:: with SMTP id l14mr15014563wrp.230.1621017084253; Fri, 14 May 2021 11:31:24 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id g10sm710454wrq.12.2021.05.14.11.31.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 14 May 2021 11:31:23 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Fri, 14 May 2021 18:31:11 +0000 Subject: [PATCH v3 12/12] fsmonitor: integrate with sparse index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee If we need to expand a sparse-index into a full one, then the FS Monitor bitmap is going to be incorrect. Ensure that we start fresh at such an event. While this is currently a performance drawback, the eventual hope of the sparse-index feature is that these expansions will be rare and hence we will be able to keep the FS Monitor data accurate across multiple Git commands. These tests are added to demonstrate that the behavior is the same across a full index and a sparse index, but also that file modifications to a tracked directory outside of the sparse cone will trigger ensure_full_index(). Signed-off-by: Derrick Stolee --- sparse-index.c | 7 ++++++ t/t7519-status-fsmonitor.sh | 48 +++++++++++++++++++++++++++++++++++++ 2 files changed, 55 insertions(+) diff --git a/sparse-index.c b/sparse-index.c index b2b3fbd75050..32ba0d17ef7c 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -195,6 +195,10 @@ int convert_to_sparse(struct index_state *istate) cache_tree_free(&istate->cache_tree); cache_tree_update(istate, 0); + istate->fsmonitor_has_run_once = 0; + FREE_AND_NULL(istate->fsmonitor_dirty); + FREE_AND_NULL(istate->fsmonitor_last_update); + istate->sparse_index = 1; trace2_region_leave("index", "convert_to_sparse", istate->repo); return 0; @@ -291,6 +295,9 @@ void ensure_full_index(struct index_state *istate) istate->cache = full->cache; istate->cache_nr = full->cache_nr; istate->cache_alloc = full->cache_alloc; + istate->fsmonitor_has_run_once = 0; + FREE_AND_NULL(istate->fsmonitor_dirty); + FREE_AND_NULL(istate->fsmonitor_last_update); strbuf_release(&base); free(full); diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh index 45d025f96010..f70fe961902e 100755 --- a/t/t7519-status-fsmonitor.sh +++ b/t/t7519-status-fsmonitor.sh @@ -73,6 +73,7 @@ test_expect_success 'setup' ' expect* actual* marker* + trace2* EOF ' @@ -383,4 +384,51 @@ test_expect_success 'status succeeds after staging/unstaging' ' ) ' +# Usage: +# check_sparse_index_behavior [!] +# If "!" is supplied, then we verify that we do not call ensure_full_index +# during a call to 'git status'. Otherwise, we verify that we _do_ call it. +check_sparse_index_behavior () { + git status --porcelain=v2 >expect && + git sparse-checkout init --cone --sparse-index && + git sparse-checkout set dir1 dir2 && + GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ + git status --porcelain=v2 >actual && + test_region $1 index ensure_full_index trace2.txt && + test_cmp expect actual && + rm trace2.txt && + git sparse-checkout disable +} + +test_expect_success 'status succeeds with sparse index' ' + git reset --hard && + + test_config core.fsmonitor "$TEST_DIRECTORY/t7519/fsmonitor-all" && + check_sparse_index_behavior ! && + + write_script .git/hooks/fsmonitor-test<<-\EOF && + printf "last_update_token\0" + EOF + git config core.fsmonitor .git/hooks/fsmonitor-test && + check_sparse_index_behavior ! && + + write_script .git/hooks/fsmonitor-test<<-\EOF && + printf "last_update_token\0" + printf "dir1/modified\0" + EOF + check_sparse_index_behavior ! && + + cp -r dir1 dir1a && + git add dir1a && + git commit -m "add dir1a" && + + # This one modifies outside the sparse-checkout definition + # and hence we expect to expand the sparse-index. + write_script .git/hooks/fsmonitor-test<<-\EOF && + printf "last_update_token\0" + printf "dir1a/modified\0" + EOF + check_sparse_index_behavior +' + test_done