From patchwork Thu Jul 29 14:52:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12408925 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 69519C4320A for ; Thu, 29 Jul 2021 14:52:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 43EF560EBC for ; Thu, 29 Jul 2021 14:52:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237656AbhG2OwS (ORCPT ); Thu, 29 Jul 2021 10:52:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38846 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236712AbhG2OwP (ORCPT ); Thu, 29 Jul 2021 10:52:15 -0400 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 116EDC0613C1 for ; Thu, 29 Jul 2021 07:52:11 -0700 (PDT) Received: by mail-wm1-x330.google.com with SMTP id o5-20020a1c4d050000b02901fc3a62af78so7026356wmh.3 for ; Thu, 29 Jul 2021 07:52:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=m3Z0ant7md1f9wJf0Q9VKxSUWOMpn0V3rnRB1ujIDL0=; b=tKfoOlPtSQLVc3GHej+ZTFayP6L8qVb5qWMtdKjBraxmSL1EdIoik43FrKsmIKtBAT XLDLRsZGZJYI2K3k9+ZC6fw81rzASUYGwjhkITU3J0rg78E5IwBl7JR8GXoJwSUZuSpn gGpp2IobtAKOWKdLz5OatAvMT/9f++hX84cGDOcfnZCqVlztz5zV+RWZpPe/tzjyDE7r 0H6oCY7FqJPOxcZrAHrQCyJCZSZMr/Ogj67yR/JuEpqDyLLhNm6RQTGef5SBeBJ2wWVK CZu32/t4NKwbZQXp96uQ/ZboDEimNMnXmMPMtMPsMrvuuXspMx9hFauPXE9ymiFDU4O9 VASw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=m3Z0ant7md1f9wJf0Q9VKxSUWOMpn0V3rnRB1ujIDL0=; b=oK6b0gU2U6DYWTMjg2DRhI+IfMCiY/ZrfsjG8Cy3Lt/Bn5gIt4OXlE1uJ2R8LPW94q WG7860EQPOveXLZNTM6ZEyZitwcn1Wf+C3dAGMU9ABT0PCAdLxeUn+XLx1KSHtcHEGlI 3K4/gW9HTVm/+t+M4FOtVl44hl1HSM5j+xsDwTg5epwFsoFO6GA4oNuff6zAv6jTrw2L 8iewrlqL9DCxnMhN7dcYTJxCSujpUGbquOMUiSFZ+uwl6PFm6RWrx9hm8NEITi9jPGPG yyblJpAVl9kHzub9UcnWuJL6xJvFkeQ/8WGGwU6ckdRQw4w1RXYlAmQLmyQbS+or7AGX z6SA== X-Gm-Message-State: AOAM5335SfIdDp5bVSq9Hy7T5UXhxQJ4ZzS0kGjeXAJ4B7nRb0SzepKy av1J/p8O/gPqWM+CssT5OtQ4JRbqVeo= X-Google-Smtp-Source: ABdhPJwRTAhh41XW176IEbMnoMM4FSRVBqtV7dg5+GFpmngFBYJmVY6cwq7zDAbI3KQgJftuCjaFBQ== X-Received: by 2002:a05:600c:206:: with SMTP id 6mr15153354wmi.137.1627570329701; Thu, 29 Jul 2021 07:52:09 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id b6sm9645925wmj.34.2021.07.29.07.52.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 29 Jul 2021 07:52:09 -0700 (PDT) Message-Id: <5e96df4df582744a89b10ce55887a2c2aacc7e70.1627570327.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 29 Jul 2021 14:52:03 +0000 Subject: [PATCH v3 1/5] t1092: test merge conflicts outside cone Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, matheus.bernardino@usp.br, stolee@gmail.com, Eric Sunshine , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Conflicts can occur outside of the sparse-checkout definition, and in that case users might try to resolve the conflicts in several ways. Document a few of these ways in a test. Make it clear that this behavior is not necessarily the optimal flow, since users can become confused when Git deletes these files from the worktree in later commands. Reviewed-by: Elijah Newren Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 43 ++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 91e30d6ec22..4c3bcb34999 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -114,6 +114,16 @@ test_expect_success 'setup' ' git add . && git commit -m "file to dir" && + for side in left right + do + git checkout -b merge-$side base && + echo $side >>deep/deeper2/a && + echo $side >>folder1/a && + echo $side >>folder2/a && + git add . && + git commit -m "$side" || return 1 + done && + git checkout -b deepest base && echo "updated deepest" >deep/deeper1/deepest/a && git commit -a -m "update deepest" && @@ -482,6 +492,39 @@ test_expect_success 'merge' ' test_all_match git rev-parse HEAD^{tree} ' +# NEEDSWORK: This test is documenting current behavior, but that +# behavior can be confusing to users so there is desire to change it. +# Right now, users might be using this flow to work through conflicts, +# so any solution should present advice to users who try this sequence +# of commands to follow whatever new method we create. +test_expect_success 'merge with conflict outside cone' ' + init_repos && + + test_all_match git checkout -b merge-tip merge-left && + test_all_match git status --porcelain=v2 && + test_all_match test_must_fail git merge -m merge merge-right && + test_all_match git status --porcelain=v2 && + + # Resolve the conflict in different ways: + # 1. Revert to the base + test_all_match git checkout base -- deep/deeper2/a && + test_all_match git status --porcelain=v2 && + + # 2. Add the file with conflict markers + test_all_match git add folder1/a && + test_all_match git status --porcelain=v2 && + + # 3. Rename the file to another sparse filename and + # accept conflict markers as resolved content. + run_on_all mv folder2/a folder2/z && + test_all_match git add folder2 && + test_all_match git status --porcelain=v2 && + + test_all_match git merge --continue && + test_all_match git status --porcelain=v2 && + test_all_match git rev-parse HEAD^{tree} +' + test_expect_success 'merge with outside renames' ' init_repos && From patchwork Thu Jul 29 14:52:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12408923 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41398C4338F for ; Thu, 29 Jul 2021 14:52:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 21CB960EBC for ; Thu, 29 Jul 2021 14:52:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237640AbhG2OwR (ORCPT ); Thu, 29 Jul 2021 10:52:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38854 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229934AbhG2OwP (ORCPT ); Thu, 29 Jul 2021 10:52:15 -0400 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B9C33C0613CF for ; Thu, 29 Jul 2021 07:52:11 -0700 (PDT) Received: by mail-wr1-x42d.google.com with SMTP id l18so7307685wrv.5 for ; Thu, 29 Jul 2021 07:52:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=UgvhPu+guleWuAFXyyEnpZpN9uiWr9d7+4IVmwrw76s=; b=UvhIwpTGWbGkMmh55pNXrDLwFQBXSmMS0/RxUUuPg7dfMO6MukwCmbKDSPuHSBFF/x ZtIPXleMYzt1V7xh8S2raVJ04FsgZoReVxkM6erUb+x7U85xfAtSBjNDaUpp53jZyuj7 rVgNRV2XhNH0YmBET2gZ41j/NdkThSD1ZXml8c8X0iHKS28gp+Fb4jGjhrvQO3QQxaPZ hm5T8FDyT83V2G7wztGGbsUrqNOhfMEpD23RAKUo6kKeGC33+weFjPeAi+BuwVihYI3l bBhAO3MC5w+j3rkAflzarBpJxR+DLFa0pUk03WdwFzEHVVBq9dLj2X9Dh/n6U6Xpu7Z4 2+vw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=UgvhPu+guleWuAFXyyEnpZpN9uiWr9d7+4IVmwrw76s=; b=Frt1W6ysQ7erzzCUW9Ku2PAeRK7Wsf7PIpYaEhjHy4OLgsnO+iIqYRviyHS7+bqZ6z uFSuyAGg8iyqIo5+pJrlYo4ygnwEL4hYG/zMumwvTTQhm4UDdahwDqPSnE5gBxli42rV BMpx21YIUkV65UDnNJM0HOHe5o1aZhdrPGhZqLCqp3GfoXRfEtsWSWH6QXECxZPM2NXu WgpVBaTAxVgYym6wjvP1adNz5/rHTWZAidglwwJCGyDix2mGpCiBmkqwboMHfoP0O3uH DQA1yE/HwPROqb7CdLb0Vk86rRoFSRucNQZGXReEtll3VAsu48Q+uEZbeIsmUyMh+q+4 X+VQ== X-Gm-Message-State: AOAM531I1CQuepezzHZzxIrP9+s6xb737aM61mJXOUb6dE9iGBXmuRi9 fRmDGeZ0DnMD07Y3BpCAQwvUzDeTJk4= X-Google-Smtp-Source: ABdhPJwUyAMzEk+ZkhU5j6+fA0kinPSvCDgPcivKH42zoHlfYtdFrLrA3n9ZqkXfQGCmzrVxXtNtOg== X-Received: by 2002:adf:eec9:: with SMTP id a9mr5167414wrp.226.1627570330385; Thu, 29 Jul 2021 07:52:10 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id t1sm3601447wrm.42.2021.07.29.07.52.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 29 Jul 2021 07:52:10 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Thu, 29 Jul 2021 14:52:04 +0000 Subject: [PATCH v3 2/5] add: allow operating on a sparse-only index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, matheus.bernardino@usp.br, stolee@gmail.com, Eric Sunshine , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Disable command_requires_full_index for 'git add'. This does not require any additional removals of ensure_full_index(). The main reason is that 'git add' discovers changes based on the pathspec and the worktree itself. These are then inserted into the index directly, and calls to index_name_pos() or index_file_exists() already call expand_to_path() at the appropriate time to support a sparse-index. Add a test to check that 'git add -A' and 'git add ' does not expand the index at all, as long as is not within a sparse directory. This does not help the global 'git add .' case. We can measure the improvement using p2000-sparse-operations.sh with these results: Test HEAD~1 HEAD ------------------------------------------------------------------------------ 2000.6: git add -A (full-index-v3) 0.35(0.30+0.05) 0.37(0.29+0.06) +5.7% 2000.7: git add -A (full-index-v4) 0.31(0.26+0.06) 0.33(0.27+0.06) +6.5% 2000.8: git add -A (sparse-index-v3) 0.57(0.53+0.07) 0.05(0.04+0.08) -91.2% 2000.9: git add -A (sparse-index-v4) 0.58(0.55+0.06) 0.05(0.05+0.06) -91.4% While the 91% improvement seems impressive, it's important to recognize that previously we had significant overhead for expanding the sparse-index. Comparing to the full index case, 'git add -A' goes from 0.37s to 0.05s, which is "only" an 86% improvement. This modification to 'git add' creates some behavior change depending on the use of a sparse index. We modify a test in t1092 to demonstrate these changes which will be remedied in future changes. Reviewed-by: Elijah Newren Signed-off-by: Derrick Stolee --- builtin/add.c | 3 +++ t/t1092-sparse-checkout-compatibility.sh | 25 +++++++++++++++++------- 2 files changed, 21 insertions(+), 7 deletions(-) diff --git a/builtin/add.c b/builtin/add.c index b773b5a4993..c76e6ddd359 100644 --- a/builtin/add.c +++ b/builtin/add.c @@ -528,6 +528,9 @@ int cmd_add(int argc, const char **argv, const char *prefix) add_new_files = !take_worktree_changes && !refresh_only && !add_renormalize; require_pathspec = !(take_worktree_changes || (0 < addremove_explicit)); + prepare_repo_settings(the_repository); + the_repository->settings.command_requires_full_index = 0; + hold_locked_index(&lock_file, LOCK_DIE_ON_ERROR); /* diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 4c3bcb34999..77343cb6d95 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -340,21 +340,27 @@ test_expect_success 'status/add: outside sparse cone' ' test_sparse_match git status --porcelain=v2 && - # This "git add folder1/a" fails with a warning - # in the sparse repos, differing from the full - # repo. This is intentional. + # Adding the path outside of the sparse-checkout cone should fail. test_sparse_match test_must_fail git add folder1/a && - test_sparse_match test_must_fail git add --refresh folder1/a && - test_all_match git status --porcelain=v2 && + + test_must_fail git -C sparse-checkout add --refresh folder1/a 2>sparse-checkout-err && + test_must_fail git -C sparse-index add --refresh folder1/a 2>sparse-index-err && + # NEEDSWORK: A sparse index changes the error message. + ! test_cmp sparse-checkout-err sparse-index-err && + + # NEEDSWORK: Adding a newly-tracked file outside the cone succeeds + test_sparse_match git add folder1/new && test_all_match git add . && test_all_match git status --porcelain=v2 && test_all_match git commit -m folder1/new && + test_all_match git rev-parse HEAD^{tree} && run_on_all ../edit-contents folder1/newer && test_all_match git add folder1/ && test_all_match git status --porcelain=v2 && - test_all_match git commit -m folder1/newer + test_all_match git commit -m folder1/newer && + test_all_match git rev-parse HEAD^{tree} ' test_expect_success 'checkout and reset --hard' ' @@ -641,7 +647,12 @@ test_expect_success 'sparse-index is not expanded' ' git -C sparse-index reset --hard && ensure_not_expanded checkout rename-out-to-out -- deep/deeper1 && git -C sparse-index reset --hard && - ensure_not_expanded restore -s rename-out-to-out -- deep/deeper1 + ensure_not_expanded restore -s rename-out-to-out -- deep/deeper1 && + + echo >>sparse-index/README.md && + ensure_not_expanded add -A && + echo >>sparse-index/extra.txt && + ensure_not_expanded add extra.txt ' # NEEDSWORK: a sparse-checkout behaves differently from a full checkout From patchwork Thu Jul 29 14:52:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12408929 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A8D0C4338F for ; Thu, 29 Jul 2021 14:52:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 597B260EBC for ; Thu, 29 Jul 2021 14:52:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237783AbhG2OwU (ORCPT ); Thu, 29 Jul 2021 10:52:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38858 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237056AbhG2OwQ (ORCPT ); Thu, 29 Jul 2021 10:52:16 -0400 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 80139C0613D5 for ; Thu, 29 Jul 2021 07:52:12 -0700 (PDT) Received: by mail-wm1-x334.google.com with SMTP id l34-20020a05600c1d22b02902573c214807so1584279wms.2 for ; Thu, 29 Jul 2021 07:52:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=JZWrFzVBIfX/W7/g/PdIA7sxwEGWtK+913aI4fIL6IE=; b=ep+fqvl0tlWQDZgAqlVs6tke0ohRIHLDTw4lGT1xskW1dI0CMv1HBnFt4KnehDbxyZ XqgMJjxhbuQioORDdG6Pe4QKWCEstjALXXNJyvrcZSQjY1YcfVhVTGmHpZCvz0vQMRCZ tD0wDIyCytW2PMubnAbyiq9kixIo1Aa0ZNf5k18YrTZrUK19Oga6Nq1c7bSdkMzNiYg2 OreJhakzdjfj5k+xTSWoAXQGm6SzPsG/Ej/hK8pa6ifOSfMTlNui1klW5+xUKim3S9CQ PLqnayk1EGaClpi6mL8LuInDNnFOQtUH/VSROJpoMHvamwHfSDP1KRshgQWnBs9xST01 J2WQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=JZWrFzVBIfX/W7/g/PdIA7sxwEGWtK+913aI4fIL6IE=; b=ggGNzuOLPBe3E0H6fYNurF9Xsp34KWDBre/q9j2VJardXetxEKVxaD4KHcRieWFYIP XycpkVMdlOPJh3fAjUpXGumH3luFPO2ugDx1zQbOyGLol32gFHAae07IaSCvO79mRr6J A7B5AKYUY040CCpiivzyWl3DZHYdxxppe5TmG4xqzTFZMWqSYTBRokbOA7B8brWz+NFm fb6KppwJ2DTY3IzGfD8QEnIwA6DnzRj0cuL2rf5MOSCKtSRK333R/6pxT7CiMvtVpFph 6hq+HYChCW2o8PDc/1nyVbK+fevM/HF+BD0JutIIRN9MKwqIaXMy3+toXw6ACjb7EqOx siYg== X-Gm-Message-State: AOAM532v+tlLPeYNoUKHwds9AxdKFmDBlwLE5xcX4lrH499Rp8I0OTeT 9qQdhlW9EqhLDazt3xSAaTFkt0L14Mo= X-Google-Smtp-Source: ABdhPJzbwEmbc9V9Xsd4jLdyuve6UbVXXMN6mEZZpU65mdkQ3VFNdSPh5h4LHtn/NiNEQQL1XRTTWA== X-Received: by 2002:a1c:7419:: with SMTP id p25mr14463362wmc.160.1627570331149; Thu, 29 Jul 2021 07:52:11 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id j6sm3949428wmq.29.2021.07.29.07.52.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 29 Jul 2021 07:52:10 -0700 (PDT) Message-Id: <9fc4313c88959cb6eab43b2e34750b4ed889771a.1627570327.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 29 Jul 2021 14:52:05 +0000 Subject: [PATCH v3 3/5] pathspec: stop calling ensure_full_index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, matheus.bernardino@usp.br, stolee@gmail.com, Eric Sunshine , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The add_pathspec_matches_against_index() focuses on matching a pathspec to file entries in the index. This already works correctly for its only use: checking if untracked files exist in the index. The compatibility checks in t1092 already test that 'git add ' works for a directory outside of the sparse cone. That provides coverage for removing this guard. This finalizes our ability to run 'git add .' without expanding a sparse index to a full one. This is evidenced by an update to t1092 and by these performance numbers for p2000-sparse-operations.sh: Test HEAD~1 HEAD -------------------------------------------------------------------------------- 2000.10: git add . (full-index-v3) 0.37(0.28+0.07) 0.36(0.27+0.06) -2.7% 2000.11: git add . (full-index-v4) 0.33(0.26+0.06) 0.32(0.28+0.05) -3.0% 2000.12: git add . (sparse-index-v3) 0.57(0.53+0.07) 0.06(0.06+0.07) -89.5% 2000.13: git add . (sparse-index-v4) 0.57(0.53+0.07) 0.05(0.03+0.09) -91.2% While the ~90% improvement is shown by the test results, it is worth noting that expanding the sparse index was adding overhead in previous commits. Comparing to the full index case, we see the performance go from 0.33s to 0.05s, an 85% improvement. Reviewed-by: Elijah Newren Signed-off-by: Derrick Stolee --- pathspec.c | 2 -- t/t1092-sparse-checkout-compatibility.sh | 7 +++---- 2 files changed, 3 insertions(+), 6 deletions(-) diff --git a/pathspec.c b/pathspec.c index 08f8d3eedc3..44306fdaca2 100644 --- a/pathspec.c +++ b/pathspec.c @@ -37,8 +37,6 @@ void add_pathspec_matches_against_index(const struct pathspec *pathspec, num_unmatched++; if (!num_unmatched) return; - /* TODO: audit for interaction with sparse-index. */ - ensure_full_index(istate); for (i = 0; i < istate->cache_nr; i++) { const struct cache_entry *ce = istate->cache[i]; if (sw_action == PS_IGNORE_SKIP_WORKTREE && ce_skip_worktree(ce)) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 77343cb6d95..dee20db5bb1 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -322,9 +322,6 @@ test_expect_success 'commit including unstaged changes' ' test_expect_success 'status/add: outside sparse cone' ' init_repos && - # adding a "missing" file outside the cone should fail - test_sparse_match test_must_fail git add folder1/a && - # folder1 is at HEAD, but outside the sparse cone run_on_sparse mkdir folder1 && cp initial-repo/folder1/a sparse-checkout/folder1/a && @@ -652,7 +649,9 @@ test_expect_success 'sparse-index is not expanded' ' echo >>sparse-index/README.md && ensure_not_expanded add -A && echo >>sparse-index/extra.txt && - ensure_not_expanded add extra.txt + ensure_not_expanded add extra.txt && + echo >>sparse-index/untracked.txt && + ensure_not_expanded add . ' # NEEDSWORK: a sparse-checkout behaves differently from a full checkout From patchwork Thu Jul 29 14:52:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12408927 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88BBFC432BE for ; Thu, 29 Jul 2021 14:52:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6BB4060EBB for ; Thu, 29 Jul 2021 14:52:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237669AbhG2OwT (ORCPT ); Thu, 29 Jul 2021 10:52:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38866 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237404AbhG2OwQ (ORCPT ); Thu, 29 Jul 2021 10:52:16 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1D161C061765 for ; Thu, 29 Jul 2021 07:52:13 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id q3so7356117wrx.0 for ; Thu, 29 Jul 2021 07:52:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=KkF6IMbAMZ/rUGP/m1B+r+kVHZeLn8AMYuVyXWvvMuQ=; b=BZmim6mFMv5Tm6IAj5awT/TjDmrOmyirXZ4ccEIg5mi3pDnJ+qzjlarws/3FpfRNVi 2M5kQdscPjRVf9Z+JnIG/uRD5dMxWnhIVK/qC3lpfP/IJ+9OSXhnx8aBX8si5tufKDY1 SKOxORXVHDdLn+O0Gt0IhqeqaZzvSAQbMPdm4HvMzt6P9mvnJS7QgFnxI3thWeTCdXll mA90A+t0DT4ZIm9/zpEyKlQYlQaml6dL9fYItiYuztWIIl68rhiHs+xlvk7wgQHPiIYB 7yPq9RCK5LHv7NdC9zgt/jDfYOUrOxwm3qoBFDEgA3GVC+NTla0Z2DpZuCru139Nkavy w99A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=KkF6IMbAMZ/rUGP/m1B+r+kVHZeLn8AMYuVyXWvvMuQ=; b=GaT/VwmQG6lggsDfandF7rmEXSbB7zD8zdo5rFOU4hPLQ9L2rLF7X6o5uBCUzmQrJk iAWYqe3r/3XxTLK4lhDARRWouQBbhv4Jbu6J5HG5+OFs958Wo7ofQ4OuGea8rsVO7wZH NZz+wOtAfdPcQFsqkn4DYQOGd126kV+byX5j3T2fED03f/yBNwjVCxpvYbyyyFY2l5Dl oLUsbsTcndPpUMEiGObmhZX78dfGLcc9t/bH0NUT9hEN0GTizOsWhxPnuYMg62rCdLqG rwKDP5NidnuhF9MBxldeek4tSjn/slH9Ca4ZtUoMqGdj5S3yJTKO7RtOf/w0mhtiK/h6 FUFQ== X-Gm-Message-State: AOAM5303g3axM2hBlD1BU3uiHQTtC55JdxcPY6gE5rePXwQwllubir6Y /MsLy4Z3ogiWc/WMc+3FKDpE1ssNS54= X-Google-Smtp-Source: ABdhPJwZ9e8zKQL2ivBHas7eORZiGSAOakkgHd82pbaltnDKM5Mb/zybvW8xeDQF5GQhjFP7+6QtFg== X-Received: by 2002:adf:f852:: with SMTP id d18mr1201877wrq.201.1627570331770; Thu, 29 Jul 2021 07:52:11 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f15sm3570476wrp.12.2021.07.29.07.52.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 29 Jul 2021 07:52:11 -0700 (PDT) Message-Id: <0ec03ab021da8b8a2278af3e14960c5cf4689646.1627570327.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 29 Jul 2021 14:52:06 +0000 Subject: [PATCH v3 4/5] add: ignore outside the sparse-checkout in refresh() Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, matheus.bernardino@usp.br, stolee@gmail.com, Eric Sunshine , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Since b243012 (refresh_index(): add flag to ignore SKIP_WORKTREE entries, 2021-04-08), 'git add --refresh ' will output a warning message when the path is outside the sparse-checkout definition. The implementation of this warning happened in parallel with the sparse-index work to add ensure_full_index() calls throughout the codebase. Update this loop to have the proper logic that checks to see if the pathspec is outside the sparse-checkout definition. This avoids the need to expand the sparse directory entry and determine if the path is tracked, untracked, or ignored. We simply avoid updating the stat() information because there isn't even an entry that matches the path! Reviewed-by: Elijah Newren Signed-off-by: Derrick Stolee --- builtin/add.c | 10 +++++++++- t/t1092-sparse-checkout-compatibility.sh | 6 +----- 2 files changed, 10 insertions(+), 6 deletions(-) diff --git a/builtin/add.c b/builtin/add.c index c76e6ddd359..d512ece655b 100644 --- a/builtin/add.c +++ b/builtin/add.c @@ -192,13 +192,21 @@ static int refresh(int verbose, const struct pathspec *pathspec) struct string_list only_match_skip_worktree = STRING_LIST_INIT_NODUP; int flags = REFRESH_IGNORE_SKIP_WORKTREE | (verbose ? REFRESH_IN_PORCELAIN : REFRESH_QUIET); + struct pattern_list pl = { 0 }; + int sparse_checkout_enabled = !get_sparse_checkout_patterns(&pl); seen = xcalloc(pathspec->nr, 1); refresh_index(&the_index, flags, pathspec, seen, _("Unstaged changes after refreshing the index:")); for (i = 0; i < pathspec->nr; i++) { if (!seen[i]) { - if (matches_skip_worktree(pathspec, i, &skip_worktree_seen)) { + const char *path = pathspec->items[i].original; + int dtype = DT_REG; + + if (matches_skip_worktree(pathspec, i, &skip_worktree_seen) || + (sparse_checkout_enabled && + !path_matches_pattern_list(path, strlen(path), NULL, + &dtype, &pl, &the_index))) { string_list_append(&only_match_skip_worktree, pathspec->items[i].original); } else { diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index dee20db5bb1..ddc86bb4152 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -339,11 +339,7 @@ test_expect_success 'status/add: outside sparse cone' ' # Adding the path outside of the sparse-checkout cone should fail. test_sparse_match test_must_fail git add folder1/a && - - test_must_fail git -C sparse-checkout add --refresh folder1/a 2>sparse-checkout-err && - test_must_fail git -C sparse-index add --refresh folder1/a 2>sparse-index-err && - # NEEDSWORK: A sparse index changes the error message. - ! test_cmp sparse-checkout-err sparse-index-err && + test_sparse_match test_must_fail git add --refresh folder1/a && # NEEDSWORK: Adding a newly-tracked file outside the cone succeeds test_sparse_match git add folder1/new && From patchwork Thu Jul 29 14:52:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12408931 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8503C432BE for ; Thu, 29 Jul 2021 14:52:19 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C85FE60EBC for ; Thu, 29 Jul 2021 14:52:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237833AbhG2OwV (ORCPT ); Thu, 29 Jul 2021 10:52:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38866 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237035AbhG2OwQ (ORCPT ); Thu, 29 Jul 2021 10:52:16 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A7375C061765 for ; Thu, 29 Jul 2021 07:52:13 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id h14so7266694wrx.10 for ; Thu, 29 Jul 2021 07:52:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=4FmGwAHuC6p1ZUMm2KxbrnzsZI0XqDa0wUNb99/XewA=; b=WVEsj8aUoy15Krw/7KOxSbas6dxGhQwrOaVWyKY8ckDzPglquu7RwSbCjQFs7ZVr6+ KaANqQguJn8P2fA00Dawt5k9U+PiLBLmW0GbqtnYSZqtaBErjrQo4/NDdGNjGZh4b3pb HcG8mz9iMmACAZbK9Zr23P/Z2rk2DHgvl6IM63poijMwpg2mgwdmASdqFgI1e9eVssqt D29d4v0p3gSyjTRSkCiwjtoXm2fj0Qa5vbnNcn/G2GLEEzW4+dZnzvyNOEhEE6ZvDL7C tkQd8qmN+idJ8eKnAxjc0YvT7khLpUye4oTlslF0cxA8hPtLc1DIG+QGaLo9VFYVfx40 jJNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=4FmGwAHuC6p1ZUMm2KxbrnzsZI0XqDa0wUNb99/XewA=; b=CSpSGLE4lIcXo8xuNPzYzFJ7UFveFn8KbXOSY9jY0cnb4zuQLPmnMV5v/y717z2xDR nTTTIcp1RZ+/eB+e9Qm4ND+R86Sg+ijqCRXTl1KmaoiZil6a9adjKW82TNVUQSUvBU2X Qk+paV0AywlXcTTi35MweN/70OuIoOjc8KmkIat0UDMkaczvjXIb0qcfKtpAnGdVlVwq BczKXZN3AUbGIfprZjezH8JigXFvNB8dZNxpMHYqEzjQA+Vayk3WeWZ+FIM+1qoqE4nG AFaTN2lWC0z8EQy7huxo8bmGapv0hI87Xxk3D5mQBh/x+0xGf8/6uwnqcyngah5QQTrE ZIGA== X-Gm-Message-State: AOAM531gwlu8nQcA6OBRtNm9MAcMMmo0Tiaimlt3XIqYai3KAgMQuuUF 40pd2A1mRDcbWoQzi7FTpURP0ujFSqI= X-Google-Smtp-Source: ABdhPJy0JFdhZqfR+UqVWxZRt/wMMyoSMKNvO37Wmg2QspcVB6XjaZw1UFAvYfOFgAna7241NqGarw== X-Received: by 2002:a5d:4751:: with SMTP id o17mr5303438wrs.252.1627570332364; Thu, 29 Jul 2021 07:52:12 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id p4sm3835557wrt.23.2021.07.29.07.52.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 29 Jul 2021 07:52:12 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Thu, 29 Jul 2021 14:52:07 +0000 Subject: [PATCH v3 5/5] add: remove ensure_full_index() with --renormalize Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, matheus.bernardino@usp.br, stolee@gmail.com, Eric Sunshine , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The --renormalize option updates the EOL conversions for the tracked files. However, the loop already ignores files marked with the SKIP_WORKTREE bit, so it will continue to do so with a sparse index because the sparse directory entries also have this bit set. Reviewed-by: Elijah Newren Signed-off-by: Derrick Stolee --- builtin/add.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/builtin/add.c b/builtin/add.c index d512ece655b..c49e179abc3 100644 --- a/builtin/add.c +++ b/builtin/add.c @@ -144,8 +144,6 @@ static int renormalize_tracked_files(const struct pathspec *pathspec, int flags) { int i, retval = 0; - /* TODO: audit for interaction with sparse-index. */ - ensure_full_index(&the_index); for (i = 0; i < active_nr; i++) { struct cache_entry *ce = active_cache[i];