From patchwork Tue Oct 5 13:20:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Willford X-Patchwork-Id: 12536651 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2FEE3C433FE for ; Tue, 5 Oct 2021 13:21:05 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1746E613D5 for ; Tue, 5 Oct 2021 13:21:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234982AbhJENWy (ORCPT ); Tue, 5 Oct 2021 09:22:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57018 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234950AbhJENWu (ORCPT ); Tue, 5 Oct 2021 09:22:50 -0400 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3CF6FC061760 for ; Tue, 5 Oct 2021 06:21:00 -0700 (PDT) Received: by mail-wm1-x333.google.com with SMTP id g193-20020a1c20ca000000b0030d55f1d984so3021731wmg.3 for ; Tue, 05 Oct 2021 06:21:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=cYtH0ucxg2svi35ld1sfbei8Bkq0DfTY/p92ZD7PBbs=; b=CV7cKd+8n9VqjjO8qS5MJFeTQLqW/6sWQt7KUrQX+c8OXZLkG4e0DK5ibYE93vs3OR GJgOdMfTZpvhWrbq4NdOM4S90LRyy/rzOs7r282H0ohCYsKy6Z07+N6r0nHXi/rY7xIH f9/GbQppUUI9j9Hj8bvEAb7lieshj5a6TijT+qC2+DYEFoSZeSjR/SIkDnZ/5Vkx4lCv 4x0rv02uVV9Z/+/V7mt7EUlUR85OJilWnnCsueuX0Z3JzAABggyAUU6ByvehvF9XVMPJ iWzn6O0aCGtGN5d4KSmhktWLOEWMsbQ8QmpfgGv/n33Y+v3G5VCyFRHyxYNtMnW5fXtz Ng1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=cYtH0ucxg2svi35ld1sfbei8Bkq0DfTY/p92ZD7PBbs=; b=ZlP7RwMIIISX8H7iZm7FnTutyR8MtiQNefwTg4XhJ1cfiCKLZf0h5Kg/MDVXyDZhuB kkM/RbzpyI7NKfPUCUD2VtiGKGVayZMeC3IcwlnDw1UiATWUW5XRR9+HaUoCvnXKx77Q 39Mq9jc69YNKmBOVknor3Q2z4YmGbr06Etv6/ANOsk2zgExgFevifsJcChp6ZC+ly9kE 3enNGfrE+6OjSiEh2M4+8jZwc25MWil+8jnx/OX5NxGeQkuWa1V7RXGaC4RF4EZJ0AhF RVq9xvD3R6T/O30X/UME2Vd38AYRlDOUhiH+9L0/j12EX1vAJYQjln75elAW7nCb7Nns dvFA== X-Gm-Message-State: AOAM532wY16h+fUulg5p7vOWLpjliTpQGErq3dEaUbHIgfVDDkhQC7dH AWCKeR081B1sNzbJEmCf76Ey4kgeV1g= X-Google-Smtp-Source: ABdhPJxGsA6QoCtj47lNtLqrd5udUD+FciZ4TZl6uglUOW+g/5oljlID8TbI9E5McCrRU7GHp3cN8w== X-Received: by 2002:a1c:f405:: with SMTP id z5mr3448629wma.72.1633440058874; Tue, 05 Oct 2021 06:20:58 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f1sm19499998wri.43.2021.10.05.06.20.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Oct 2021 06:20:58 -0700 (PDT) Message-Id: <22c69bc60308fef13acd7c3aab4e11e175c89440.1633440057.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 05 Oct 2021 13:20:51 +0000 Subject: [PATCH v2 1/7] reset: behave correctly with sparse-checkout Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: stolee@gmail.com, gitster@pobox.com, newren@gmail.com, Taylor Blau , Bagas Sanjaya , Victoria Dye , Kevin Willford Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Kevin Willford From: Kevin Willford When using the sparse checkout feature, 'git reset' will add entries to the index that will have the skip-worktree bit off but will leave the working directory empty. File data is lost because the index version of the files has been changed but there is nothing that is in the working directory. This will cause the next 'git status' call to show either deleted for files modified or deleting or nothing for files added. The added files should be shown as untracked and modified files should be shown as modified. To fix this when the reset is running if there is not a file in the working directory and if it will be missing with the new index entry or was not missing in the previous version, we create the previous index version of the file in the working directory so that status will report correctly and the files will be availble for the user to deal with. This fixes a documented failure from t1092 that was created in 19a0acc (t1092: test interesting sparse-checkout scenarios, 2021-01-23). Signed-off-by: Kevin Willford Signed-off-by: Derrick Stolee Signed-off-by: Victoria Dye --- builtin/reset.c | 24 ++++++++-- t/t1092-sparse-checkout-compatibility.sh | 4 +- t/t7114-reset-sparse-checkout.sh | 61 ++++++++++++++++++++++++ 3 files changed, 83 insertions(+), 6 deletions(-) create mode 100755 t/t7114-reset-sparse-checkout.sh diff --git a/builtin/reset.c b/builtin/reset.c index 51c9e2f43ff..3b75d3b2f20 100644 --- a/builtin/reset.c +++ b/builtin/reset.c @@ -25,6 +25,8 @@ #include "cache-tree.h" #include "submodule.h" #include "submodule-config.h" +#include "dir.h" +#include "entry.h" #define REFRESH_INDEX_DELAY_WARNING_IN_MS (2 * 1000) @@ -130,11 +132,27 @@ static void update_index_from_diff(struct diff_queue_struct *q, int intent_to_add = *(int *)data; for (i = 0; i < q->nr; i++) { + int pos; struct diff_filespec *one = q->queue[i]->one; - int is_missing = !(one->mode && !is_null_oid(&one->oid)); + struct diff_filespec *two = q->queue[i]->two; + int is_in_reset_tree = one->mode && !is_null_oid(&one->oid); struct cache_entry *ce; - if (is_missing && !intent_to_add) { + /* + * If the file being reset has `skip-worktree` enabled, we need + * to check it out to prevent the file from being hard reset. + */ + pos = cache_name_pos(two->path, strlen(two->path)); + if (pos >= 0 && ce_skip_worktree(active_cache[pos])) { + struct checkout state = CHECKOUT_INIT; + state.force = 1; + state.refresh_cache = 1; + state.istate = &the_index; + + checkout_entry(active_cache[pos], &state, NULL, NULL); + } + + if (!is_in_reset_tree && !intent_to_add) { remove_file_from_cache(one->path); continue; } @@ -144,7 +162,7 @@ static void update_index_from_diff(struct diff_queue_struct *q, if (!ce) die(_("make_cache_entry failed for path '%s'"), one->path); - if (is_missing) { + if (!is_in_reset_tree) { ce->ce_flags |= CE_INTENT_TO_ADD; set_object_name_for_intent_to_add_entry(ce); } diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 886e78715fe..c5977152661 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -459,9 +459,7 @@ test_expect_failure 'blame with pathspec outside sparse definition' ' test_all_match git blame deep/deeper2/deepest/a ' -# NEEDSWORK: a sparse-checkout behaves differently from a full checkout -# in this scenario, but it shouldn't. -test_expect_failure 'checkout and reset (mixed)' ' +test_expect_success 'checkout and reset (mixed)' ' init_repos && test_all_match git checkout -b reset-test update-deep && diff --git a/t/t7114-reset-sparse-checkout.sh b/t/t7114-reset-sparse-checkout.sh new file mode 100755 index 00000000000..a8029707fb1 --- /dev/null +++ b/t/t7114-reset-sparse-checkout.sh @@ -0,0 +1,61 @@ +#!/bin/sh + +test_description='reset when using a sparse-checkout' + +. ./test-lib.sh + +test_expect_success 'setup' ' + test_tick && + echo "checkout file" >c && + echo "modify file" >m && + echo "delete file" >d && + git add . && + git commit -m "initial commit" && + echo "added file" >a && + echo "modification of a file" >m && + git rm d && + git add . && + git commit -m "second commit" && + git checkout -b endCommit +' + +test_expect_success 'reset when there is a sparse-checkout' ' + echo "/c" >.git/info/sparse-checkout && + test_config core.sparsecheckout true && + git checkout -B resetBranch && + test_path_is_missing m && + test_path_is_missing a && + test_path_is_missing d && + git reset HEAD~1 && + echo "checkout file" >expect && + test_cmp expect c && + echo "added file" >expect && + test_cmp expect a && + echo "modification of a file" >expect && + test_cmp expect m && + test_path_is_missing d +' + +test_expect_success 'reset after deleting file without skip-worktree bit' ' + git checkout -f endCommit && + git clean -xdf && + cat >.git/info/sparse-checkout <<-\EOF && + /c + /m + EOF + test_config core.sparsecheckout true && + git checkout -B resetAfterDelete && + test_path_is_file m && + test_path_is_missing a && + test_path_is_missing d && + rm -f m && + git reset HEAD~1 && + echo "checkout file" >expect && + test_cmp expect c && + echo "added file" >expect && + test_cmp expect a && + test_path_is_missing m && + test_path_is_missing d +' + +test_done From patchwork Tue Oct 5 13:20:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victoria Dye X-Patchwork-Id: 12536655 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 02CADC43219 for ; Tue, 5 Oct 2021 13:21:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D9DCB61503 for ; Tue, 5 Oct 2021 13:21:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234993AbhJENW6 (ORCPT ); Tue, 5 Oct 2021 09:22:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57026 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234958AbhJENWw (ORCPT ); Tue, 5 Oct 2021 09:22:52 -0400 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BECEEC061762 for ; Tue, 5 Oct 2021 06:21:00 -0700 (PDT) Received: by mail-wm1-x334.google.com with SMTP id m42so5834190wms.2 for ; Tue, 05 Oct 2021 06:21:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=0diNHvAg+DnAH3GtSgMzhFnRO7Oz642uheHT03o4VjU=; b=I+/3diw1YA+sRfq8YCL47ZnMvd2YghiL+GYCZtZi4fsD0DBjMBLulQsBtSDP/a2jia KoQ0ikn+U6zNTYh1TH5QEEGoYKC2vXBooIG7bxO6Y4HMiNDZKyuQiuDugsCE5wzSUSsp QDGztkIicLt+5QTKq7nNhCDntyXVQgDLBYtWSVtTaX/U7hIU8o3lyjoe5PyqppLnrdfl CvL9hlHvtFQPvR6XdJ/R9fSgxroYA+N4ZDbGBXl8TlEzcFZaRTbxNV74o14N1k0q9J4O 5GQvxQZRlSN8S7EO0Myo7GsH7x1Yeoyrggj0rK35yGoff7dqx2rkdCNDkLLfqLSlRKLJ 4mTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=0diNHvAg+DnAH3GtSgMzhFnRO7Oz642uheHT03o4VjU=; b=AAoOJPfF/RwhFrNktpyb4YhI8DoluMvHmHAS/srwmWcDNEPqaBQri2vExxQ6L1o9ZT vtP8gIdOZ9by/aTQCCwCnguO8Vr+/bPU1Nc9UjyUHMo0ePlnCeS3HEEkqtlpJVADhV4I E1TLwJPKrwvdRAVLj54Cg2gktTP+IIT3Oe5ZnWGK3ZJW2e+BkYIN/jZwLifUf5GXp2Gc aVMWUlyN+TgPZ6PDmKWOfllCfzHePcjk3Jqv3yL8C9WnuuJt2/ufrehDIpUU0FcgeKH8 WjnYmgM95A8fiyKZq41mKyQJceWOAO/PzlNdiP/hzU0cjfubAoS2aMAT4RqyLbt3U0Pp TAYg== X-Gm-Message-State: AOAM531eQifq2KI4erA8lOGGvr3PxuFefmb7ZK/GIX3BURWS/26BK5ju 4MJbMH9tGjLRturM71HnzvmGtjEQlNI= X-Google-Smtp-Source: ABdhPJzag+yZzFz32hFnvRWN/2yqSdxUc+Eyb3wsPlLl3UOUtBLRKUT+d+EzXkqjNGNWMBE0HpAkDw== X-Received: by 2002:a7b:cd8f:: with SMTP id y15mr3459617wmj.135.1633440059421; Tue, 05 Oct 2021 06:20:59 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id y13sm17680003wrm.48.2021.10.05.06.20.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Oct 2021 06:20:59 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 05 Oct 2021 13:20:52 +0000 Subject: [PATCH v2 2/7] update-index: add --force-full-index option for expand/collapse test Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: stolee@gmail.com, gitster@pobox.com, newren@gmail.com, Taylor Blau , Bagas Sanjaya , Victoria Dye , Victoria Dye Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Victoria Dye From: Victoria Dye Add a new `--force-full-index` option to `git update-index`, which skips explicitly setting `command_requires_full_index`. This lets `git update-index --force-full-index` run as a command without sparse index compatibility implemented, even after it receives sparse index compatibility updates. By using `git update-index --force-full-index` in the `t1092` test `sparse-index is expanded and converted back`, commands can continue to integrate with the sparse index without the need to keep modifying the command used in the test. Signed-off-by: Victoria Dye --- Documentation/git-update-index.txt | 5 +++++ builtin/update-index.c | 11 +++++++++++ t/t1092-sparse-checkout-compatibility.sh | 2 +- 3 files changed, 17 insertions(+), 1 deletion(-) diff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt index 2853f168d97..06255e321a3 100644 --- a/Documentation/git-update-index.txt +++ b/Documentation/git-update-index.txt @@ -24,6 +24,7 @@ SYNOPSIS [--[no-]fsmonitor] [--really-refresh] [--unresolve] [--again | -g] [--info-only] [--index-info] + [--force-full-index] [-z] [--stdin] [--index-version ] [--verbose] [--] [...] @@ -170,6 +171,10 @@ time. Version 4 is relatively young (first released in 1.8.0 in October 2012). Other Git implementations such as JGit and libgit2 may not support it yet. +--force-full-index:: + Force the command to operate on a full index, expanding a sparse + index if necessary. + -z:: Only meaningful with `--stdin` or `--index-info`; paths are separated with NUL character instead of LF. diff --git a/builtin/update-index.c b/builtin/update-index.c index 187203e8bb5..32ada3ead77 100644 --- a/builtin/update-index.c +++ b/builtin/update-index.c @@ -964,6 +964,7 @@ int cmd_update_index(int argc, const char **argv, const char *prefix) int split_index = -1; int force_write = 0; int fsmonitor = -1; + int use_default_full_index = 0; struct lock_file lock_file = LOCK_INIT; struct parse_opt_ctx_t ctx; strbuf_getline_fn getline_fn; @@ -1069,6 +1070,8 @@ int cmd_update_index(int argc, const char **argv, const char *prefix) {OPTION_SET_INT, 0, "no-fsmonitor-valid", &mark_fsmonitor_only, NULL, N_("clear fsmonitor valid bit"), PARSE_OPT_NOARG | PARSE_OPT_NONEG, NULL, UNMARK_FLAG}, + OPT_SET_INT(0, "force-full-index", &use_default_full_index, + N_("run with full index explicitly required"), 1), OPT_END() }; @@ -1082,6 +1085,14 @@ int cmd_update_index(int argc, const char **argv, const char *prefix) if (newfd < 0) lock_error = errno; + /* + * If --force-full-index is set, the command should skip manually + * setting `command_requires_full_index`. + */ + prepare_repo_settings(r); + if (!use_default_full_index) + r->settings.command_requires_full_index = 1; + entries = read_cache(); if (entries < 0) die("cache corrupted"); diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index c5977152661..b3c0d3b98ee 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -642,7 +642,7 @@ test_expect_success 'sparse-index is expanded and converted back' ' init_repos && GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ - git -C sparse-index -c core.fsmonitor="" reset --hard && + git -C sparse-index -c core.fsmonitor="" update-index --force-full-index && test_region index convert_to_sparse trace2.txt && test_region index ensure_full_index trace2.txt ' From patchwork Tue Oct 5 13:20:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victoria Dye X-Patchwork-Id: 12536657 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2F297C433F5 for ; Tue, 5 Oct 2021 13:21:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1622661503 for ; Tue, 5 Oct 2021 13:21:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234990AbhJENWz (ORCPT ); Tue, 5 Oct 2021 09:22:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57028 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234959AbhJENWw (ORCPT ); Tue, 5 Oct 2021 09:22:52 -0400 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5CFE9C061764 for ; Tue, 5 Oct 2021 06:21:01 -0700 (PDT) Received: by mail-wm1-x332.google.com with SMTP id p21so6636555wmq.1 for ; Tue, 05 Oct 2021 06:21:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=MMsbOv/bbgTRd4gzYHz/ThjOh6qlXknsOXHwR0Dnvbk=; b=bT2BwGhwMXaIZbBqw08LpTOEbEFVWnpPS/Lfm8Yje2ejR4zTI//eL3RhFRM6d5WHcZ SqZtZ2l6copUZHX28uPblfbgzrmiyTuvaPVcv/gpi3MZRVCapBoUkDt+u5E6hHFLKBvf wea6twoYZVPQvCsAVjeZ0gForwrGX/FeaB0w5xyUVvyNE4gq0SeNHapiKx0qiyaptBZf BL3JMSMAB36ymhosw3/rzx8x/efz/uDZg8LW8jKEFK0fq7wd5i73oBzSHejrnL7AANg4 w2EYW40FuJtfoRkGNukWcjU6N52iYN1RrIP4qONgz7dyh+0DqB9hGZY2ju8PnLRpXTQE HiMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=MMsbOv/bbgTRd4gzYHz/ThjOh6qlXknsOXHwR0Dnvbk=; b=LzxrwVlwuOYfu8nuNGsnmAu+Vi4dOnfcHVTwYr/MxF1+/m/aAz+n+ahI4Jsgp2t2cc a1zzCy2DcNPvclfwA6DVWXCDdoAg6tj6AOvtWfcAKm1EZvXGkaVc3tvAmv3oAcmvHw43 DRrxraDCmHBw5YTne4ogIpUrWQ9V3N2jCXcMkNcx9eI7MelxslJdsq3Szi+eGcL/2fUC /ZFlbyTkunM9XxDBmK4tkFWa9lNtiD4RFZKla9WxAVByA86LK/cePbruKF+Nm1sCsoLb 3PcR2/1NIawct61y5D3MpJqvJheeVZwj/QdnTb0BbA6AR4iv43i3C/iUkaowcrOuarRP xkBA== X-Gm-Message-State: AOAM530paWq00nhIdtab2mG/I4bEApuXoBkI7vfN6AlUpJWhbab5D4fW RuBTh72hqWh16PBDIDpp2rA4Z6A9uVI= X-Google-Smtp-Source: ABdhPJxMG7uD288wEPcpdv5kLKmHrgbbpdH8xMTOaNE0LBIq9VxC3qX8DPDhD67R3NIV8spr94t0/Q== X-Received: by 2002:a05:600c:21d6:: with SMTP id x22mr3491660wmj.121.1633440060025; Tue, 05 Oct 2021 06:21:00 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id p18sm6447815wrt.96.2021.10.05.06.20.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Oct 2021 06:20:59 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 05 Oct 2021 13:20:53 +0000 Subject: [PATCH v2 3/7] reset: expand test coverage for sparse checkouts Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: stolee@gmail.com, gitster@pobox.com, newren@gmail.com, Taylor Blau , Bagas Sanjaya , Victoria Dye , Victoria Dye Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Victoria Dye From: Victoria Dye Add new tests for `--merge` and `--keep` modes, as well as mixed reset with pathspecs both inside and outside of the sparse checkout definition. New performance test cases exercise various execution paths for `reset`. Co-authored-by: Derrick Stolee Signed-off-by: Derrick Stolee Signed-off-by: Victoria Dye --- t/perf/p2000-sparse-operations.sh | 3 + t/t1092-sparse-checkout-compatibility.sh | 107 +++++++++++++++++++++++ 2 files changed, 110 insertions(+) diff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh index 597626276fb..bfd332120c8 100755 --- a/t/perf/p2000-sparse-operations.sh +++ b/t/perf/p2000-sparse-operations.sh @@ -110,5 +110,8 @@ test_perf_on_all git add -A test_perf_on_all git add . test_perf_on_all git commit -a -m A test_perf_on_all git checkout -f - +test_perf_on_all git reset +test_perf_on_all git reset --hard +test_perf_on_all git reset -- does-not-exist test_done diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index b3c0d3b98ee..f0723a6ac97 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -479,6 +479,113 @@ test_expect_success 'checkout and reset (mixed) [sparse]' ' test_sparse_match git reset update-folder2 ' +# NEEDSWORK: with mixed reset, files with differences between HEAD and +# will be added to the work tree even if outside the sparse checkout +# definition, and even if the file is modified to a state of having no local +# changes. The file is "re-ignored" if a hard reset is executed. We may want to +# change this behavior in the future and enforce that files are not written +# outside of the sparse checkout definition. +test_expect_success 'checkout and mixed reset file tracking [sparse]' ' + init_repos && + + test_all_match git checkout -b reset-test update-deep && + test_all_match git reset update-folder1 && + test_all_match git reset update-deep && + + # At this point, there are no changes in the working tree. However, + # folder1/a now exists locally (even though it is outside of the sparse + # paths). + run_on_sparse test_path_exists folder1 && + + run_on_all rm folder1/a && + test_all_match git status --porcelain=v2 && + + test_all_match git reset --hard update-deep && + run_on_sparse test_path_is_missing folder1 && + test_path_exists full-checkout/folder1 +' + +test_expect_success 'checkout and reset (merge)' ' + init_repos && + + write_script edit-contents <<-\EOF && + echo text >>$1 + EOF + + test_all_match git checkout -b reset-test update-deep && + run_on_all ../edit-contents a && + test_all_match git reset --merge deepest && + test_all_match git status --porcelain=v2 && + + test_all_match git reset --hard update-deep && + run_on_all ../edit-contents deep/a && + test_all_match test_must_fail git reset --merge deepest +' + +test_expect_success 'checkout and reset (keep)' ' + init_repos && + + write_script edit-contents <<-\EOF && + echo text >>$1 + EOF + + test_all_match git checkout -b reset-test update-deep && + run_on_all ../edit-contents a && + test_all_match git reset --keep deepest && + test_all_match git status --porcelain=v2 && + + test_all_match git reset --hard update-deep && + run_on_all ../edit-contents deep/a && + test_all_match test_must_fail git reset --keep deepest +' + +test_expect_success 'reset with pathspecs inside sparse definition' ' + init_repos && + + write_script edit-contents <<-\EOF && + echo text >>$1 + EOF + + test_all_match git checkout -b reset-test update-deep && + run_on_all ../edit-contents deep/a && + + test_all_match git reset base -- deep/a && + test_all_match git status --porcelain=v2 && + + test_all_match git reset base -- nonexistent-file && + test_all_match git status --porcelain=v2 && + + test_all_match git reset deepest -- deep && + test_all_match git status --porcelain=v2 +' + +test_expect_success 'reset with sparse directory pathspec outside definition' ' + init_repos && + + test_all_match git checkout -b reset-test update-deep && + test_all_match git reset --hard update-folder1 && + test_all_match git reset base -- folder1 && + test_all_match git status --porcelain=v2 +' + +test_expect_success 'reset with pathspec match in sparse directory' ' + init_repos && + + test_all_match git checkout -b reset-test update-deep && + test_all_match git reset --hard update-folder1 && + test_all_match git reset base -- folder1/a && + test_all_match git status --porcelain=v2 +' + +test_expect_success 'reset with wildcard pathspec' ' + init_repos && + + test_all_match git checkout -b reset-test update-deep && + test_all_match git reset --hard update-folder1 && + test_all_match git reset base -- \*/a && + test_all_match git status --porcelain=v2 +' + test_expect_success 'merge, cherry-pick, and rebase' ' init_repos && From patchwork Tue Oct 5 13:20:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victoria Dye X-Patchwork-Id: 12536653 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 04DBFC4332F for ; Tue, 5 Oct 2021 13:21:08 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DD59761503 for ; Tue, 5 Oct 2021 13:21:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235001AbhJENW5 (ORCPT ); Tue, 5 Oct 2021 09:22:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57030 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234979AbhJENWx (ORCPT ); Tue, 5 Oct 2021 09:22:53 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 16129C061765 for ; Tue, 5 Oct 2021 06:21:02 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id s15so9131286wrv.11 for ; Tue, 05 Oct 2021 06:21:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=QGx9YF/7G8kGWdo5Kg9eFed6hLSyObhsMNY0mgtnrKk=; b=UDfMhIBdLeVY4EHIedpBT270bflypzHA/v6QCNVUwipFnQliXqeYgk0p4yO9Ifol6O azGlEgty6fAWGusXM1/sw64UvpM0KbQueBIYIVhRzBMu48sewF7WPsWCYUNRTcK7alrr sw9AOiOL9aAvawoPOO9TzKQozfnI3x6NDMGU/GXyVmy1pSkCXg1vAAqoyRziXcmn/Gvw FyyHATGG67AqdvJHNRvPc3T4i39SkRatcZqNMcuSJJ2UwDo6M3mn9jT900E2LfPKnE8y uw4iU37ZKb84yXhZJGsR3x/m9ud0ycXfEXWV0il8w+bWGP8Rdtaoxt2US/1/QOnpbcwN Qlxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=QGx9YF/7G8kGWdo5Kg9eFed6hLSyObhsMNY0mgtnrKk=; b=BgklmM5jvVGYzqNue6mji03DdqWSF1PC+oJQ/HkXMzJpTmxbxUDuWVBowXhonnE6IK DQT2Bdqan59Fpk6swtD+bniROb7AMoRYDqqSl3z3+/tLnRgBF+jpJkr3JBAg0cyMT5Ec Q5ugKhTGvVwMnRyfE98+61m/ElH05FR0myTDDzRdPU0a9KOyGJLhqwiYeoh9VC409gY5 pGksf0B8hKDckBijgcvTeKPPI663rUMFwLEZJLJAMXRZTl2SwMCzTmjHkjtjALxoUCfU 6ueGbhq7u5AxtOqfFN04dOVKxEIeqGeqn6yAN3YI4CsCHR6uYIQoZlk3cGTv7B2sKxiV HrRQ== X-Gm-Message-State: AOAM532ZR/Ctr9TpHk+H5970Ea/vYomIl8EkehA/BRomXXEJ8I5648eq PUgkYJMHoQ2NattntAMyL1VwbCz2CCg= X-Google-Smtp-Source: ABdhPJzHo0tXVhQkgFoolXRlnOKEaYnUl743ERBp/bnjENV1S8rSmy9bE8ti1I8P8J35Ezbr/1/mJQ== X-Received: by 2002:adf:b1c7:: with SMTP id r7mr9562772wra.392.1633440060683; Tue, 05 Oct 2021 06:21:00 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id h18sm17185547wrs.75.2021.10.05.06.21.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Oct 2021 06:21:00 -0700 (PDT) Message-Id: <49813c8d9ed94fd56f30eb204d346eb5a30473ca.1633440057.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 05 Oct 2021 13:20:54 +0000 Subject: [PATCH v2 4/7] reset: integrate with sparse index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: stolee@gmail.com, gitster@pobox.com, newren@gmail.com, Taylor Blau , Bagas Sanjaya , Victoria Dye , Victoria Dye Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Victoria Dye From: Victoria Dye `reset --soft` does not modify the index, so no compatibility changes are needed for it to function without expanding the index. For all other reset modes (`--mixed`, `--hard`, `--keep`, `--merge`), the full index is explicitly expanded with `ensure_full_index` to maintain current behavior. Additionally, the `read_cache()` check verifying an uncorrupted index is moved after argument parsing and preparing the repo settings. The index is not used by the preceding argument handling, but `read_cache()` does need to be run after enabling sparse index for the command and before resetting. Signed-off-by: Victoria Dye --- builtin/reset.c | 10 +++++++--- cache-tree.c | 1 + 2 files changed, 8 insertions(+), 3 deletions(-) diff --git a/builtin/reset.c b/builtin/reset.c index 3b75d3b2f20..e1f2a2bb2c4 100644 --- a/builtin/reset.c +++ b/builtin/reset.c @@ -184,6 +184,7 @@ static int read_from_tree(const struct pathspec *pathspec, opt.flags.override_submodule_config = 1; opt.repo = the_repository; + ensure_full_index(&the_index); if (do_diff_cache(tree_oid, &opt)) return 1; diffcore_std(&opt); @@ -261,9 +262,6 @@ static void parse_args(struct pathspec *pathspec, } *rev_ret = rev; - if (read_cache() < 0) - die(_("index file corrupt")); - parse_pathspec(pathspec, 0, PATHSPEC_PREFER_FULL | (patch_mode ? PATHSPEC_PREFIX_ORIGIN : 0), @@ -409,6 +407,12 @@ int cmd_reset(int argc, const char **argv, const char *prefix) if (intent_to_add && reset_type != MIXED) die(_("-N can only be used with --mixed")); + prepare_repo_settings(the_repository); + the_repository->settings.command_requires_full_index = 0; + + if (read_cache() < 0) + die(_("index file corrupt")); + /* Soft reset does not touch the index file nor the working tree * at all, but requires them in a good order. Other resets reset * the index file to the tree object we are switching to. */ diff --git a/cache-tree.c b/cache-tree.c index 90919f9e345..9be19c85b66 100644 --- a/cache-tree.c +++ b/cache-tree.c @@ -776,6 +776,7 @@ void prime_cache_tree(struct repository *r, cache_tree_free(&istate->cache_tree); istate->cache_tree = cache_tree(); + ensure_full_index(istate); prime_cache_tree_rec(r, istate->cache_tree, tree); istate->cache_changed |= CACHE_TREE_CHANGED; trace2_region_leave("cache-tree", "prime_cache_tree", the_repository); From patchwork Tue Oct 5 13:20:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victoria Dye X-Patchwork-Id: 12536659 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49305C43217 for ; Tue, 5 Oct 2021 13:21:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 353B36138F for ; Tue, 5 Oct 2021 13:21:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234958AbhJENW7 (ORCPT ); Tue, 5 Oct 2021 09:22:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57040 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233910AbhJENWx (ORCPT ); Tue, 5 Oct 2021 09:22:53 -0400 Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C67D9C061766 for ; Tue, 5 Oct 2021 06:21:02 -0700 (PDT) Received: by mail-wm1-x336.google.com with SMTP id l18-20020a05600c4f1200b002f8cf606262so3041972wmq.1 for ; Tue, 05 Oct 2021 06:21:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=E8peHUGeSN0CW7vLeX8jUKgpN5LLCbqWtr/BN26N/aU=; b=bWFiZLWCk8KlM9TxPoMxsGpeTpCICKBUflc9B3/kWOgKiFvEuhgsG1iD8ez+7stGTu SRhgJXIR+SRd2Dr0ZU976DfKhGmw3LxJlP3wgOSvmIVWxpfftvF5Ses/9zjztMLNLGG9 kdeWlD9ENdJpWlgXJ0ldN5+IxUNUWsNm0stjYO/AUvMyUsMR59QJQK2fkYTe/R+uaQ20 dEz1fPQ3BhPtF4Dov2G04Hz7CPEZN3jH+o5gJ0SM5ia1DRxpwKCSV71mnHXMYSMcD05i CN6M+165ZxJkLPPF0DNqwAl0iNkP+LI9SVyHL8jgDIk1kjd3wc8azIu/+FqKS3jly+af +GCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=E8peHUGeSN0CW7vLeX8jUKgpN5LLCbqWtr/BN26N/aU=; b=y69MLpYk0Xa7OCjxiR2RrDCmVOZRKUOpKELqwSREry4OFHiRbrKEmwt/2wYfo64CQr UW8Y601zB09x6RxynC4bXj2DmBUpEpSGqN7k+RuKfckoJ9thXd2RQzpgwgSFtGdJB41D yxOApD2Auc+W0BgoAH4hwduS1om6+YJ3GLiPnVDkP8Jpx146cyPuZXU45kxhi3XGdBC2 K5eDIEzmoYptNrCxgOSmZmPWrg4y0u+oFch8+ojkm3GsU1e/3qnGfWrPttr94J9yocjC YYbfmOdt6bf2n/Tld51dk9nnXIL+F9ALU2Rv6b5BEJmYDTF5YNmNjxHuqHXynmJkuBKl Vlsg== X-Gm-Message-State: AOAM531qUPEHknqwXzC8iq64pDwFZqDFAYeIV5+CM3OYqQg76Jl0eHYN fs2n9/OCD1b//Lt6C7pcCk0RsJI7wfI= X-Google-Smtp-Source: ABdhPJzi8UKM3X9AFYaqX7wHlYY5XGDFsN704kl7rbWwMYa4v9P3MPGU0fTHOVgaKponvfggmQ37tg== X-Received: by 2002:a1c:a941:: with SMTP id s62mr3396791wme.55.1633440061335; Tue, 05 Oct 2021 06:21:01 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id w7sm5166217wrm.54.2021.10.05.06.21.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Oct 2021 06:21:01 -0700 (PDT) Message-Id: <78cd85d8dcc790251ce8235e649902cf6adf091a.1633440057.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 05 Oct 2021 13:20:55 +0000 Subject: [PATCH v2 5/7] reset: make sparse-aware (except --mixed) Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: stolee@gmail.com, gitster@pobox.com, newren@gmail.com, Taylor Blau , Bagas Sanjaya , Victoria Dye , Victoria Dye Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Victoria Dye From: Victoria Dye In order to accurately reconstruct the cache tree in `prime_cache_tree_rec`, the function must determine whether the currently-processing directory in the tree is sparse or not. If it is not sparse, the tree is parsed and subtree recursively constructed. If it is sparse, no subtrees are added to the tree and the entry count is set to 1 (representing the sparse directory itself). Signed-off-by: Victoria Dye --- cache-tree.c | 44 +++++++++++++++++++++--- cache.h | 10 ++++++ read-cache.c | 22 ++++++++---- t/t1092-sparse-checkout-compatibility.sh | 15 ++++++-- 4 files changed, 78 insertions(+), 13 deletions(-) diff --git a/cache-tree.c b/cache-tree.c index 9be19c85b66..9021669d682 100644 --- a/cache-tree.c +++ b/cache-tree.c @@ -740,15 +740,29 @@ out: return ret; } +static void prime_cache_tree_sparse_dir(struct repository *r, + struct cache_tree *it, + struct tree *tree, + struct strbuf *tree_path) +{ + + oidcpy(&it->oid, &tree->object.oid); + it->entry_count = 1; + return; +} + static void prime_cache_tree_rec(struct repository *r, struct cache_tree *it, - struct tree *tree) + struct tree *tree, + struct strbuf *tree_path) { + struct strbuf subtree_path = STRBUF_INIT; struct tree_desc desc; struct name_entry entry; int cnt; oidcpy(&it->oid, &tree->object.oid); + init_tree_desc(&desc, tree->buffer, tree->size); cnt = 0; while (tree_entry(&desc, &entry)) { @@ -757,27 +771,49 @@ static void prime_cache_tree_rec(struct repository *r, else { struct cache_tree_sub *sub; struct tree *subtree = lookup_tree(r, &entry.oid); + if (!subtree->object.parsed) parse_tree(subtree); sub = cache_tree_sub(it, entry.path); sub->cache_tree = cache_tree(); - prime_cache_tree_rec(r, sub->cache_tree, subtree); + strbuf_reset(&subtree_path); + strbuf_grow(&subtree_path, tree_path->len + entry.pathlen + 1); + strbuf_addbuf(&subtree_path, tree_path); + strbuf_add(&subtree_path, entry.path, entry.pathlen); + strbuf_addch(&subtree_path, '/'); + + /* + * If a sparse index is in use, the directory being processed may be + * sparse. To confirm that, we can check whether an entry with that + * exact name exists in the index. If it does, the created subtree + * should be sparse. Otherwise, cache tree expansion should continue + * as normal. + */ + if (r->index->sparse_index && + index_entry_exists(r->index, subtree_path.buf, subtree_path.len)) + prime_cache_tree_sparse_dir(r, sub->cache_tree, subtree, &subtree_path); + else + prime_cache_tree_rec(r, sub->cache_tree, subtree, &subtree_path); cnt += sub->cache_tree->entry_count; } } it->entry_count = cnt; + + strbuf_release(&subtree_path); } void prime_cache_tree(struct repository *r, struct index_state *istate, struct tree *tree) { + struct strbuf tree_path = STRBUF_INIT; + trace2_region_enter("cache-tree", "prime_cache_tree", the_repository); cache_tree_free(&istate->cache_tree); istate->cache_tree = cache_tree(); - ensure_full_index(istate); - prime_cache_tree_rec(r, istate->cache_tree, tree); + prime_cache_tree_rec(r, istate->cache_tree, tree, &tree_path); + strbuf_release(&tree_path); istate->cache_changed |= CACHE_TREE_CHANGED; trace2_region_leave("cache-tree", "prime_cache_tree", the_repository); } diff --git a/cache.h b/cache.h index f6295f3b048..1d3e4665562 100644 --- a/cache.h +++ b/cache.h @@ -816,6 +816,16 @@ struct cache_entry *index_file_exists(struct index_state *istate, const char *na */ int index_name_pos(struct index_state *, const char *name, int namelen); +/* + * Determines whether an entry with the given name exists within the + * given index. The return value is 1 if an exact match is found, otherwise + * it is 0. Note that, unlike index_name_pos, this function does not expand + * the index if it is sparse. If an item exists within the full index but it + * is contained within a sparse directory (and not in the sparse index), 0 is + * returned. + */ +int index_entry_exists(struct index_state *, const char *name, int namelen); + /* * Some functions return the negative complement of an insert position when a * precise match was not found but a position was found where the entry would diff --git a/read-cache.c b/read-cache.c index f5d4385c408..ea1166895f8 100644 --- a/read-cache.c +++ b/read-cache.c @@ -551,7 +551,10 @@ int cache_name_stage_compare(const char *name1, int len1, int stage1, const char return 0; } -static int index_name_stage_pos(struct index_state *istate, const char *name, int namelen, int stage) +static int index_name_stage_pos(struct index_state *istate, + const char *name, int namelen, + int stage, + int search_sparse) { int first, last; @@ -570,7 +573,7 @@ static int index_name_stage_pos(struct index_state *istate, const char *name, in first = next+1; } - if (istate->sparse_index && + if (search_sparse && istate->sparse_index && first > 0) { /* Note: first <= istate->cache_nr */ struct cache_entry *ce = istate->cache[first - 1]; @@ -586,7 +589,7 @@ static int index_name_stage_pos(struct index_state *istate, const char *name, in ce_namelen(ce) < namelen && !strncmp(name, ce->name, ce_namelen(ce))) { ensure_full_index(istate); - return index_name_stage_pos(istate, name, namelen, stage); + return index_name_stage_pos(istate, name, namelen, stage, search_sparse); } } @@ -595,7 +598,12 @@ static int index_name_stage_pos(struct index_state *istate, const char *name, in int index_name_pos(struct index_state *istate, const char *name, int namelen) { - return index_name_stage_pos(istate, name, namelen, 0); + return index_name_stage_pos(istate, name, namelen, 0, 1); +} + +int index_entry_exists(struct index_state *istate, const char *name, int namelen) +{ + return index_name_stage_pos(istate, name, namelen, 0, 0) >= 0; } int remove_index_entry_at(struct index_state *istate, int pos) @@ -1222,7 +1230,7 @@ static int has_dir_name(struct index_state *istate, */ } - pos = index_name_stage_pos(istate, name, len, stage); + pos = index_name_stage_pos(istate, name, len, stage, 1); if (pos >= 0) { /* * Found one, but not so fast. This could @@ -1322,7 +1330,7 @@ static int add_index_entry_with_check(struct index_state *istate, struct cache_e strcmp(ce->name, istate->cache[istate->cache_nr - 1]->name) > 0) pos = index_pos_to_insert_pos(istate->cache_nr); else - pos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce)); + pos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce), 1); /* existing match? Just replace it. */ if (pos >= 0) { @@ -1357,7 +1365,7 @@ static int add_index_entry_with_check(struct index_state *istate, struct cache_e if (!ok_to_replace) return error(_("'%s' appears as both a file and as a directory"), ce->name); - pos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce)); + pos = index_name_stage_pos(istate, ce->name, ce_namelen(ce), ce_stage(ce), 1); pos = -pos-1; } return pos + 1; diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index f0723a6ac97..e301ef5633a 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -786,9 +786,9 @@ test_expect_success 'sparse-index is not expanded' ' ensure_not_expanded checkout - && ensure_not_expanded switch rename-out-to-out && ensure_not_expanded switch - && - git -C sparse-index reset --hard && + ensure_not_expanded reset --hard && ensure_not_expanded checkout rename-out-to-out -- deep/deeper1 && - git -C sparse-index reset --hard && + ensure_not_expanded reset --hard && ensure_not_expanded restore -s rename-out-to-out -- deep/deeper1 && echo >>sparse-index/README.md && @@ -798,6 +798,17 @@ test_expect_success 'sparse-index is not expanded' ' echo >>sparse-index/untracked.txt && ensure_not_expanded add . && + for ref in update-deep update-folder1 update-folder2 update-deep + do + echo >>sparse-index/README.md && + ensure_not_expanded reset --hard $ref || return 1 + done && + + ensure_not_expanded reset --hard update-deep && + ensure_not_expanded reset --keep base && + ensure_not_expanded reset --merge update-deep && + ensure_not_expanded reset --hard && + ensure_not_expanded checkout -f update-deep && test_config -C sparse-index pull.twohead ort && ( From patchwork Tue Oct 5 13:20:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victoria Dye X-Patchwork-Id: 12536661 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99C5DC433EF for ; Tue, 5 Oct 2021 13:21:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 81F0761090 for ; Tue, 5 Oct 2021 13:21:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235016AbhJENXA (ORCPT ); Tue, 5 Oct 2021 09:23:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57028 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234964AbhJENWx (ORCPT ); Tue, 5 Oct 2021 09:22:53 -0400 Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 490C0C061749 for ; Tue, 5 Oct 2021 06:21:03 -0700 (PDT) Received: by mail-wm1-x336.google.com with SMTP id a11-20020a7bc1cb000000b0030d6aae48b5so2873679wmj.4 for ; Tue, 05 Oct 2021 06:21:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=eqLVmiatgqvUMXys9C2h3HzvqIRcQW4fhYbW7/JKGVE=; b=mu6dwm9zYmDsFZ4K8P0H8ZbQRP6mtjIChGKvRTJ0d1jhDVEeDVpPai5PXqLH60tuhK ZD2OXF2ausuD4kYbf77uxP4FXwtDxGopDtTL6/IUZHwIYWBt+p8FpdIX8jIGRVzak9Qd cCB3hNzOK8WQODg6zA9JEOHkEoKpWBT5hQw3iZMxkcPfy48filpuDQ9g8tdR7A6h1aGt snJwVrIcvu/SZTjKuQ3OhxwFruNbwcoJr7aqGi2m9loA+F/G7eHOcrgLn/YeGRiXiJFd ZFVxhdEIGXGG6zN9BreksWN1cq1kJSxS2WKM8sjgdyKqsRlgC1CjeP8ADCk2RHHlPegD JiIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=eqLVmiatgqvUMXys9C2h3HzvqIRcQW4fhYbW7/JKGVE=; b=LYIZJh+0qztlyIrGBa+VanA1eJXLYyxMGFmbnXSHJ7v4cRXVWJxxBmHMqQM6R4JRMV FL03efGFJXTS+9AxriHcTfQ4ui9/NbrNYAN6ySYNT1FktFi8lBvVFkbEq3ThIZKfYF9i jqYs6SdF9Yw82k8/dKsLeGl3BSngBjt7fbwtK+exsUAVu8KoZ1cf1H6q2m9iM3NaOksT wXpG1KrljGCqDgFLH4XpoLubUaipr4AG/MNSLfhZXTcUupSRfAzgEpK4Q1ORVtVBxo1d 8CVX/iseyTts6JiYtlXeKcK9EcnGY86V2aaYYsPwmZTZ9MXxseGrpt4uNvWhlCnFV2ig M3YA== X-Gm-Message-State: AOAM530jKEgTd6z9FHkn0Ytwd7aNKYWP7tMdv1snRB9WuJX+/tRzQGBI 981aalhsSqcW3z57/l+pL/DYxDgZmT4= X-Google-Smtp-Source: ABdhPJx167pMQcEXwAgEnGCzwfc0xzlCCMkMwRiQxcOPAzFPbfBFjRA3am433/PjycRAofyY0MkGTg== X-Received: by 2002:a05:600c:a49:: with SMTP id c9mr3336100wmq.159.1633440061951; Tue, 05 Oct 2021 06:21:01 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id e15sm1267767wrr.38.2021.10.05.06.21.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Oct 2021 06:21:01 -0700 (PDT) Message-Id: <5eaae0825af4cee84040b784c32190bb1de2f919.1633440057.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 05 Oct 2021 13:20:56 +0000 Subject: [PATCH v2 6/7] reset: make --mixed sparse-aware Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: stolee@gmail.com, gitster@pobox.com, newren@gmail.com, Taylor Blau , Bagas Sanjaya , Victoria Dye , Victoria Dye Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Victoria Dye From: Victoria Dye Sparse directory entries are "diffed" as trees in `diff_cache` (used internally by `reset --mixed`), following a code path separate from individual file handling. The use of `diff_tree_oid` there requires setting explicit `change` and `add_remove` functions to process the internal contents of a sparse directory. Additionally, the `recursive` diff option handles cases in which `reset --mixed` must diff/merge files that are nested multiple levels deep in a sparse directory. Signed-off-by: Victoria Dye --- builtin/reset.c | 30 +++++++++++++++++++++++- t/t1092-sparse-checkout-compatibility.sh | 13 +++++++++- 2 files changed, 41 insertions(+), 2 deletions(-) diff --git a/builtin/reset.c b/builtin/reset.c index e1f2a2bb2c4..ceb9b122897 100644 --- a/builtin/reset.c +++ b/builtin/reset.c @@ -175,6 +175,8 @@ static int read_from_tree(const struct pathspec *pathspec, int intent_to_add) { struct diff_options opt; + unsigned int i; + char *skip_worktree_seen = NULL; memset(&opt, 0, sizeof(opt)); copy_pathspec(&opt.pathspec, pathspec); @@ -182,9 +184,35 @@ static int read_from_tree(const struct pathspec *pathspec, opt.format_callback = update_index_from_diff; opt.format_callback_data = &intent_to_add; opt.flags.override_submodule_config = 1; + opt.flags.recursive = 1; opt.repo = the_repository; + opt.change = diff_change; + opt.add_remove = diff_addremove; + + /* + * When pathspec is given for resetting a cone-mode sparse checkout, it may + * identify entries that are nested in sparse directories, in which case the + * index should be expanded. For the sake of efficiency, this check is + * overly-cautious: anything with a wildcard or a magic prefix requires + * expansion, as well as literal paths that aren't in the sparse checkout + * definition AND don't match any directory in the index. + */ + if (pathspec->nr && the_index.sparse_index) { + if (pathspec->magic || pathspec->has_wildcard) { + ensure_full_index(&the_index); + } else { + for (i = 0; i < pathspec->nr; i++) { + if (!path_in_cone_mode_sparse_checkout(pathspec->items[i].original, &the_index) && + !matches_skip_worktree(pathspec, i, &skip_worktree_seen)) { + ensure_full_index(&the_index); + break; + } + } + } + } + + free(skip_worktree_seen); - ensure_full_index(&the_index); if (do_diff_cache(tree_oid, &opt)) return 1; diffcore_std(&opt); diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index e301ef5633a..4afcbc2d673 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -804,11 +804,22 @@ test_expect_success 'sparse-index is not expanded' ' ensure_not_expanded reset --hard $ref || return 1 done && + ensure_not_expanded reset --mixed base && ensure_not_expanded reset --hard update-deep && ensure_not_expanded reset --keep base && ensure_not_expanded reset --merge update-deep && - ensure_not_expanded reset --hard && + ensure_not_expanded reset base -- deep/a && + ensure_not_expanded reset base -- nonexistent-file && + ensure_not_expanded reset deepest -- deep && + + # Although folder1 is outside the sparse definition, it exists as a + # directory entry in the index, so it will be reset without needing to + # expand the full index. + ensure_not_expanded reset --hard update-folder1 && + ensure_not_expanded reset base -- folder1 && + + ensure_not_expanded reset --hard update-deep && ensure_not_expanded checkout -f update-deep && test_config -C sparse-index pull.twohead ort && ( From patchwork Tue Oct 5 13:20:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victoria Dye X-Patchwork-Id: 12536663 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7B93C433F5 for ; Tue, 5 Oct 2021 13:21:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D4D0161090 for ; Tue, 5 Oct 2021 13:21:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235030AbhJENXD (ORCPT ); Tue, 5 Oct 2021 09:23:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57044 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234983AbhJENWz (ORCPT ); Tue, 5 Oct 2021 09:22:55 -0400 Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DD3C8C061749 for ; Tue, 5 Oct 2021 06:21:03 -0700 (PDT) Received: by mail-wm1-x336.google.com with SMTP id p21so6636661wmq.1 for ; Tue, 05 Oct 2021 06:21:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=2HRlxeP3tz/uNY5+unoDO9/v7e3CHEEqIbEgAsKvEsY=; b=OWNRDww1IBqNcEJLodYknz4ejxx44n1r1OgP+zY6fJzpWoAeQecEWkGSw/oI4Y1UMG 036qqtQ9eTqs3KoC5hADJ3qWOmDuzIqpiYrYmkJeVLLt7WXjXkLC2e7r3CNYe3TIn7Sg OfHSgA1pNf2cwY5UkiHZIDiRYby51e4umLLA3a+AD9WYJqSTn3GO3qHihbaKTG71lTFL KNzuq5rWg6A09Z4fp6slfvCSjdWXqB0BdQEuPQCc9lJL7BTBtGdujtsPe62katm2ACPo jXV9UdAHlo/Q6vfKisWpKxFN1y5RMoNCIyTuq6LgEnsjK5CPR0twZsKxjp2jS0vr4mGH kUbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=2HRlxeP3tz/uNY5+unoDO9/v7e3CHEEqIbEgAsKvEsY=; b=gQnuzJTGJS/MyQkZwy2g4eRxrK5QZHPk/1y9yz9eVhk5DSOz7wo4iQRNDRBi7/A3mp Xq7bP2RZCu05gJ8j39279bsmD4A7KKoSqh6hPVP834mED1RE0umT2OuoVzeo8MMAnhGP lYgT81dQteq8h7d+B2KEtI52Gnng18YvFNiemaebsaiPIxB20/KquqIRNvuYVL+Mokyo Vks8Utm7Ei0Ys9xHbDFZTFFwC8SFr230MjfXZs4lRTZZLhjRXcZLjwfS7l1WB/jY2s65 47c1eVEtVIXmHihvCDYoTIImMoFrlDCE7m3gR0lcXxluNv9MKMcElWGvzP+LJpDlr6Oo mU3w== X-Gm-Message-State: AOAM533YhrSddF1bOyMuMEIxq5m0/1bFMoObxW8bnI/04TO8AWphtbfn CpJUOyMtJkzedCoPRvCjwf1qdQH9q4Q= X-Google-Smtp-Source: ABdhPJwbryv3r8lybsPmtRrJpCr3X4p6qQ0nN9F39ui7VySIw3LjDyXMzCvIVo7gvblSFZnUYJfJmA== X-Received: by 2002:a1c:7f11:: with SMTP id a17mr3398257wmd.166.1633440062535; Tue, 05 Oct 2021 06:21:02 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id t12sm14638556wrw.12.2021.10.05.06.21.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Oct 2021 06:21:02 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 05 Oct 2021 13:20:57 +0000 Subject: [PATCH v2 7/7] unpack-trees: improve performance of next_cache_entry Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: stolee@gmail.com, gitster@pobox.com, newren@gmail.com, Taylor Blau , Bagas Sanjaya , Victoria Dye , Victoria Dye Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Victoria Dye From: Victoria Dye To find the first non-unpacked cache entry, `next_cache_entry` iterates through index, starting at `cache_bottom`. The performance of this in full indexes is helped by `cache_bottom` advancing with each invocation of `mark_ce_used` (called by `unpack_index_entry`). However, the presence of sparse directories can prevent the `cache_bottom` from advancing in a sparse index case, effectively forcing `next_cache_entry` to search from the beginning of the index each time it is called. The `cache_bottom` must be preserved for the sparse index (see 17a1bb570b (unpack-trees: preserve cache_bottom, 2021-07-14)). Therefore, to retain the benefit `cache_bottom` provides in non-sparse index cases, a separate `hint` position indicates the first position `next_cache_entry` should search, updated each execution with a new position. The performance of `git reset -- does-not-exist` (testing the "worst case" in which all entries in the index are unpacked with `next_cache_entry`) is significantly improved for the sparse index case: Test before after ------------------------------------------------------ (full-v3) 0.79(0.38+0.30) 0.91(0.43+0.34) +15.2% (full-v4) 0.80(0.38+0.29) 0.85(0.40+0.35) +6.2% (sparse-v3) 0.76(0.43+0.69) 0.44(0.08+0.67) -42.1% (sparse-v4) 0.71(0.40+0.65) 0.41(0.09+0.65) -42.3% Signed-off-by: Victoria Dye --- unpack-trees.c | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/unpack-trees.c b/unpack-trees.c index 8ea0a542da8..b94733de6be 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -645,17 +645,24 @@ static void mark_ce_used_same_name(struct cache_entry *ce, } } -static struct cache_entry *next_cache_entry(struct unpack_trees_options *o) +static struct cache_entry *next_cache_entry(struct unpack_trees_options *o, int *hint) { const struct index_state *index = o->src_index; int pos = o->cache_bottom; + if (*hint > pos) + pos = *hint; + while (pos < index->cache_nr) { struct cache_entry *ce = index->cache[pos]; - if (!(ce->ce_flags & CE_UNPACKED)) + if (!(ce->ce_flags & CE_UNPACKED)) { + *hint = pos + 1; return ce; + } pos++; } + + *hint = pos; return NULL; } @@ -1365,12 +1372,13 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str /* Are we supposed to look at the index too? */ if (o->merge) { + int hint = -1; while (1) { int cmp; struct cache_entry *ce; if (o->diff_index_cached) - ce = next_cache_entry(o); + ce = next_cache_entry(o, &hint); else ce = find_cache_entry(info, p); @@ -1690,7 +1698,7 @@ static int verify_absent(const struct cache_entry *, int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options *o) { struct repository *repo = the_repository; - int i, ret; + int i, hint, ret; static struct cache_entry *dfc; struct pattern_list pl; int free_pattern_list = 0; @@ -1763,13 +1771,15 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options info.pathspec = o->pathspec; if (o->prefix) { + hint = -1; + /* * Unpack existing index entries that sort before the * prefix the tree is spliced into. Note that o->merge * is always true in this case. */ while (1) { - struct cache_entry *ce = next_cache_entry(o); + struct cache_entry *ce = next_cache_entry(o, &hint); if (!ce) break; if (ce_in_traverse_path(ce, &info)) @@ -1790,8 +1800,9 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options /* Any left-over entries in the index? */ if (o->merge) { + hint = -1; while (1) { - struct cache_entry *ce = next_cache_entry(o); + struct cache_entry *ce = next_cache_entry(o, &hint); if (!ce) break; if (unpack_index_entry(ce, o) < 0)