From patchwork Wed Mar 22 16:18:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shuqi Liang X-Patchwork-Id: 13184272 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5884C6FD1C for ; Wed, 22 Mar 2023 16:18:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229556AbjCVQSv (ORCPT ); Wed, 22 Mar 2023 12:18:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41840 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229997AbjCVQSp (ORCPT ); Wed, 22 Mar 2023 12:18:45 -0400 Received: from mail-il1-x131.google.com (mail-il1-x131.google.com [IPv6:2607:f8b0:4864:20::131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1CC344229 for ; Wed, 22 Mar 2023 09:18:37 -0700 (PDT) Received: by mail-il1-x131.google.com with SMTP id s1so4210008ild.6 for ; Wed, 22 Mar 2023 09:18:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1679501916; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=shIfNMYTcTevhJNijnWjuk99N5vM7RXgRpkLvBQsGwc=; b=G5wa2inEPbeCOjcFcQwATE+4XzeUaJORI2SZkk+8C8n5OsVhVv/fPa4ZztUVsuIgX1 c2WyO4iUgb3swboS4x3pUEWXRzXiJDoPqINvyv3qxM5dji50pRmxbV7+iqQRMFEHcWhV EMiS/oxxv2jBdAcH+VlA8/J1kl79McU1Hl++Nh9ZTwsSq8lHNxShffJryjYbmSIQ1AIN DhQxyqqd9YtMNmajleLIaef4y9lfd+5vlZ29XDPCskdv2VdmIr1JBseiWZAKcCB83yhz JSJ49Xxz+22pTaX/LaUF6RPC4rKa4rHy7zQrR5Xm/WGDdFkPNhUeCnx7Puvmvy7MNKMH FOZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679501916; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=shIfNMYTcTevhJNijnWjuk99N5vM7RXgRpkLvBQsGwc=; b=j6nqJboV0xbQfc3oJ7n66tq2bd20fCipalNvTKuR8+IQZo3lAiy7oMS6C8jz6lkV3U 05nrO1i6n+Nx0/fk62D6iTC/FVCtVh15il06gwTnB6lnxcrN0dqgI9uK8/jNL4OKcUKZ nDX1FqEr6rVRwi3xKvbmFM8PAF6h7dfWsZINusCcHipbTEoVUCSrKXFBIUICPTUIhpOI 2scwQqxsrGaI5CsWjV9RtZZzyvKvz2nNDihq34TD/YxG8PfNPSedAMBycG2oxlosUvlh PJj7o5X2UbjTtRb70MpeboxtqV8Jxp7Kt22Y2nhGkcysEuWkaJxQ9ajuaQikA1D4IV7M IQyA== X-Gm-Message-State: AO0yUKUdLNP6ztj8ZTiDBUKljue+cqZEolk27ul82mJO+PUV3jzPGqMe Q5eXEUh0fou3Pl4JO1PJkfZpE+vcrWU= X-Google-Smtp-Source: AK7set+laG8sNnw/P+fCfHct6rxnhU8uZnXhlNYTWhc08ROSRZGvUkDBc1LLHmKDHa2eDLJvoJMHtQ== X-Received: by 2002:a92:730a:0:b0:319:23c7:5553 with SMTP id o10-20020a92730a000000b0031923c75553mr4474096ilc.32.1679501916093; Wed, 22 Mar 2023 09:18:36 -0700 (PDT) Received: from localhost.localdomain (bras-base-london142cw-grc-20-69-158-191-243.dsl.bell.ca. [69.158.191.243]) by smtp.googlemail.com with ESMTPSA id q18-20020a056e02107200b0032304e1814bsm4460129ilj.40.2023.03.22.09.18.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Mar 2023 09:18:35 -0700 (PDT) From: Shuqi Liang To: git@vger.kernel.org Cc: Shuqi Liang , vdye@github.com, gitster@pobox.com, derrickstolee@github.com Subject: [PATCH v7 1/2] t1092: add tests for `git diff-files` Date: Wed, 22 Mar 2023 12:18:19 -0400 Message-Id: <20230322161820.3609-2-cheskaqiqi@gmail.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230322161820.3609-1-cheskaqiqi@gmail.com> References: <20230320205241.105476-1-cheskaqiqi@gmail.com> <20230322161820.3609-1-cheskaqiqi@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Before integrating the 'git diff-files' builtin with the sparse index feature, add tests to t1092-sparse-checkout-compatibility.sh to ensure it currently works with sparse-checkout and will still work with sparse index after that integration. When adding tests against a sparse-checkout definition, we test two modes: all changes are within the sparse-checkout cone and some changes are outside the sparse-checkout cone. In order to have staged changes outside of the sparse-checkout cone, make a directory called 'folder1' and copy `a` into 'folder1/a'. 'folder1/a' is identical to `a` in the base commit. These make 'folder1/a' in the index, while leaving it outside of the sparse-checkout definition. Test 'folder1/a'being present on-disk without modifications, then change content inside 'folder1/a' in order to test 'folder1/a' being present on-disk with modifications. Signed-off-by: Shuqi Liang --- t/t1092-sparse-checkout-compatibility.sh | 46 ++++++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 801919009e..d23041e27a 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -2055,4 +2055,50 @@ test_expect_success 'grep sparse directory within submodules' ' test_cmp actual expect ' +test_expect_success 'diff-files with pathspec inside sparse definition' ' + init_repos && + + write_script edit-contents <<-\EOF && + echo text >>"$1" + EOF + + run_on_all ../edit-contents deep/a && + + test_all_match git diff-files && + + test_all_match git diff-files deep/a && + + # test wildcard + test_all_match git diff-files deep/* +' + +test_expect_success 'diff-files with pathspec outside sparse definition' ' + init_repos && + + test_sparse_match test_must_fail git diff-files folder2/a && + + write_script edit-contents <<-\EOF && + echo text >>"$1" + EOF + + # Add file to the index but outside of cone for sparse-checkout cases. + # Add file to the index without sparse-checkout cases to ensure all have + # same output. + run_on_all mkdir -p folder1 && + run_on_all cp a folder1/a && + + # file present on-disk without modifications + # use `--stat` to ignore file creation time differences in + # unrefreshed index + test_all_match git diff-files --stat && + test_all_match git diff-files --stat folder1/a && + test_all_match git diff-files --stat "folder*/a" && + + # file present on-disk with modifications + run_on_all ../edit-contents folder1/a && + test_all_match git diff-files && + test_all_match git diff-files folder1/a && + test_all_match git diff-files "folder*/a" +' + test_done From patchwork Wed Mar 22 16:18:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shuqi Liang X-Patchwork-Id: 13184273 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22EFEC6FD1F for ; Wed, 22 Mar 2023 16:18:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229838AbjCVQSw (ORCPT ); Wed, 22 Mar 2023 12:18:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41844 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229786AbjCVQSq (ORCPT ); Wed, 22 Mar 2023 12:18:46 -0400 Received: from mail-il1-x130.google.com (mail-il1-x130.google.com [IPv6:2607:f8b0:4864:20::130]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8268649C7 for ; Wed, 22 Mar 2023 09:18:39 -0700 (PDT) Received: by mail-il1-x130.google.com with SMTP id l15so8616405ilv.4 for ; Wed, 22 Mar 2023 09:18:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1679501918; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Doa9IcuIP5f+/qKJeIDd8s3hsl9gxWEWivHxKOMjcuI=; b=QYqvqkRNuCfrjC7uy4yTmaGCPVIJ5xlqflUfvXHpa15ky1ch75g/U7a3Z3Hx/HweZC 2DJmEp/Le9eUuRoA/Tw4XqShXx5Z+jrkmnks8Q3dQVKEfgdV/kGJbUzi842bYgkikc84 8bHuEhaDYUTJVfoY1FR4TQYhFM7ux8HYkzKGDOwexrLTykpm2ILXH+HyWfzE41BB/11l +IeQomzKGzR+HebjEcsuHoIHX7Bj4H/Fl4cyCkLHjzWsMmJxmtlKcaN+QvuonoovXbUq 8Sk4qpkJfK6Fsqovl2MzudPkKuPVwrEVHKOCtO35KYE0dSWQ+dj5pc84lOAIGZ9y1crV ujmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1679501918; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Doa9IcuIP5f+/qKJeIDd8s3hsl9gxWEWivHxKOMjcuI=; b=YC9P4FqhE5ODY2WpfaqIeFvvRlOI9AsLClbr4Iduupy6D3ElIOJC8BGWa5KANsZqxc cA9YROjkawa0bBhfDlkhz7meLCiFHFhouZtZzKK9av9AUFUPuK233tw0Fx0EB46ee3uj DaqDXopWg74GkhucR7QCsy1AoNC5GG3xIuRh873W453eFf1HtScnmyyAh4QKTczMi6lY wsrwiyVe/YXTsOOgqmYIgjU4D09rOQn5R6oUrPyVJPZ2BFS9U3niGN1KdCNYgdOwYM0U EkOs1d7FNCe+JR5/qsnoC0i/kNygbSURuJ5Zwc94DI5WfUY3XEgkmblHFAJB/0jzwtf0 c+EQ== X-Gm-Message-State: AO0yUKVl+zq/Zv3T5gNI0mkCt+rZhq6RejfFd5lTHQZjrndZwJUHRwx3 WzbLecQCHuLyQ+di1uAcbdLytrNFbao= X-Google-Smtp-Source: AK7set8lRLSNu1DyVwjVpS23tGCeSFglIJ0gpXmQzJ7hiFgLAucLLsZ19ggtsi/2jSKrE1JMDmfkUQ== X-Received: by 2002:a92:c641:0:b0:325:a6c5:70fa with SMTP id 1-20020a92c641000000b00325a6c570famr4561277ill.26.1679501918400; Wed, 22 Mar 2023 09:18:38 -0700 (PDT) Received: from localhost.localdomain (bras-base-london142cw-grc-20-69-158-191-243.dsl.bell.ca. [69.158.191.243]) by smtp.googlemail.com with ESMTPSA id q18-20020a056e02107200b0032304e1814bsm4460129ilj.40.2023.03.22.09.18.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Mar 2023 09:18:38 -0700 (PDT) From: Shuqi Liang To: git@vger.kernel.org Cc: Shuqi Liang , vdye@github.com, gitster@pobox.com, derrickstolee@github.com Subject: [PATCH v7 2/2] diff-files: integrate with sparse index Date: Wed, 22 Mar 2023 12:18:20 -0400 Message-Id: <20230322161820.3609-3-cheskaqiqi@gmail.com> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230322161820.3609-1-cheskaqiqi@gmail.com> References: <20230320205241.105476-1-cheskaqiqi@gmail.com> <20230322161820.3609-1-cheskaqiqi@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Originally, diff-files a pathspec that is out-of-cone in a sparse-index environment, Git dies with "pathspec '' did not match any files", mainly because it does not expand the index so nothing is matched. Expand the index when the needs an expanded index, i.e. the contains wildcard that may need a full-index or the is simply outside of sparse-checkout definition. Remove full index requirement for `git diff-files`.Create an 'ensure_expanded' to handle silent failures. Add test to ensure the index only expanded when necessary in `git diff-files`. The `p2000` tests demonstrate a ~96% execution time reduction for 'git diff-files' and a ~97% execution time reduction for 'git diff-files' for a file using a sparse index: Test before after ----------------------------------------------------------------- 2000.78: git diff-files (full-v3) 0.09 0.08 -11.1% 2000.79: git diff-files (full-v4) 0.09 0.09 +0.0% 2000.80: git diff-files (sparse-v3) 0.52 0.02 -96.2% 2000.81: git diff-files (sparse-v4) 0.51 0.02 -96.1% 2000.82: git diff-files f2/f4/a (full-v3) 0.06 0.07 +16.7% 2000.83: git diff-files f2/f4/a (full-v4) 0.08 0.08 +0.0% 2000.84: git diff-files f2/f4/a (sparse-v3) 0.46 0.01 -97.8% 2000.85: git diff-files f2/f4/a (sparse-v4) 0.51 0.02 -96.1% Signed-off-by: Shuqi Liang --- builtin/diff-files.c | 8 ++++ t/perf/p2000-sparse-operations.sh | 2 + t/t1092-sparse-checkout-compatibility.sh | 52 ++++++++++++++++++++++++ 3 files changed, 62 insertions(+) diff --git a/builtin/diff-files.c b/builtin/diff-files.c index dc991f753b..db90592090 100644 --- a/builtin/diff-files.c +++ b/builtin/diff-files.c @@ -27,6 +27,10 @@ int cmd_diff_files(int argc, const char **argv, const char *prefix) usage(diff_files_usage); git_config(git_diff_basic_config, NULL); /* no "diff" UI options */ + + prepare_repo_settings(the_repository); + the_repository->settings.command_requires_full_index = 0; + repo_init_revisions(the_repository, &rev, prefix); rev.abbrev = 0; @@ -80,6 +84,10 @@ int cmd_diff_files(int argc, const char **argv, const char *prefix) result = -1; goto cleanup; } + + if (pathspec_needs_expanded_index(the_repository->index, &rev.diffopt.pathspec)) + ensure_full_index(the_repository->index); + result = run_diff_files(&rev, options); result = diff_result_code(&rev.diffopt, result); cleanup: diff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh index 3242cfe91a..82751f2ca3 100755 --- a/t/perf/p2000-sparse-operations.sh +++ b/t/perf/p2000-sparse-operations.sh @@ -125,5 +125,7 @@ test_perf_on_all git checkout-index -f --all test_perf_on_all git update-index --add --remove $SPARSE_CONE/a test_perf_on_all "git rm -f $SPARSE_CONE/a && git checkout HEAD -- $SPARSE_CONE/a" test_perf_on_all git grep --cached --sparse bogus -- "f2/f1/f1/*" +test_perf_on_all git diff-files +test_perf_on_all git diff-files $SPARSE_CONE/a test_done diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index d23041e27a..152f3f752e 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -1401,6 +1401,30 @@ ensure_not_expanded () { test_region ! index ensure_full_index trace2.txt } +ensure_expanded () { + rm -f trace2.txt && + if test -z "$WITHOUT_UNTRACKED_TXT" + then + echo >>sparse-index/untracked.txt + fi && + + if test "$1" = "!" + then + shift && + test_must_fail env \ + GIT_TRACE2_EVENT="$(pwd)/trace2.txt" \ + git -C sparse-index "$@" \ + >sparse-index-out \ + 2>sparse-index-error || return 1 + else + GIT_TRACE2_EVENT="$(pwd)/trace2.txt" \ + git -C sparse-index "$@" \ + >sparse-index-out \ + 2>sparse-index-error || return 1 + fi && + test_region index ensure_full_index trace2.txt +} + test_expect_success 'sparse-index is not expanded' ' init_repos && @@ -2101,4 +2125,32 @@ test_expect_success 'diff-files with pathspec outside sparse definition' ' test_all_match git diff-files "folder*/a" ' +test_expect_success 'diff-files pathspec expands index when necessary' ' + init_repos && + + write_script edit-contents <<-\EOF && + echo text >>"$1" + EOF + + run_on_all ../edit-contents deep/a && + + # pathspec that should expand index + ensure_expanded diff-files "*/a" && + ensure_expanded diff-files "**a" +' + +test_expect_success 'sparse index is not expanded: diff-files' ' + init_repos && + + write_script edit-contents <<-\EOF && + echo text >>"$1" + EOF + + run_on_all ../edit-contents deep/a && + + ensure_not_expanded diff-files && + ensure_not_expanded diff-files deep/a && + ensure_not_expanded diff-files deep/* +' + test_done