From patchwork Wed Sep 8 01:42:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12479689 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A04EC433EF for ; Wed, 8 Sep 2021 01:42:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D1EC46113C for ; Wed, 8 Sep 2021 01:42:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346667AbhIHBnp (ORCPT ); Tue, 7 Sep 2021 21:43:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55578 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346441AbhIHBno (ORCPT ); Tue, 7 Sep 2021 21:43:44 -0400 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3D951C061575 for ; Tue, 7 Sep 2021 18:42:37 -0700 (PDT) Received: by mail-wr1-x42c.google.com with SMTP id z4so736143wrr.6 for ; Tue, 07 Sep 2021 18:42:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=2790nSllAGE+Jryr1ZNUmiwSlcYNs/tn/3C9+QWeo8U=; b=BuGNiODQrhalsQel8u5hELGxdYBrGQIzVdiqQyMdbfVzXh2mcX8ODMJLKfXsGIai4P 9XQo7oPLKWYpiynsZQFJBIs1B/PbxY2cbKsZ/NSsYmTNDHxz20IUheLvL7FRMSTDeU8M GvrxmrsfMTU+11/5fK71F1aIu6Awr1f7xkRPtbzAvuEap/AyHK52gYn/GV3O3Yaek8uG xLxwum41msqzoT/eEKNeq++WWEtB6nAG7PgJdyA9oteevqwMkHfusOtQIexE/Vm6yCBy fPN49icDt6B/v3iqYa4eslCKKgU1/5zaDfJraUl2dG4Kn5oZb2yHC6oWwvUFvgD/2Izy +Q0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=2790nSllAGE+Jryr1ZNUmiwSlcYNs/tn/3C9+QWeo8U=; b=W6+MRTYKAmXZFDY8AVCHMtIFFTeRkx4h3dKGULdtGveEoaP9xwWGd2teRCU2Juc1DB cp+8x/OgdvrSjABorhcFUNmsqUV9UBOBZQIvHjehrI/6Z8Qx9iyaIxcccAeR+Jj/TtCj w4ZjRg7mdHhIMp4RECbfWwgzWFpo5cPQm7cWmc1WspA65X0X9dqhE3pFmrsOD70hTFl9 fSqHh+lrgaQjw1RlXbXRkDUkkGbb2ofYOwbtQ8aM3isS50ERAK/CjBNRPxMwe7p7wDvj +X10KOH5Jw4aVz9SI6rbv/HepKfuPyNj5qzvMBRtdIPQaECNcXkG7ULcI2QDtk7SYdUQ KMUw== X-Gm-Message-State: AOAM531BMaI2rpIu556Y/9YhImi1lknbi2E0nah+Zu25ibWVmxubNQEs ihQQ2EQ5WPPDug1lALOMZdSt9vVQpDA= X-Google-Smtp-Source: ABdhPJyYMNSC3Iw5pZJBcEY0kHBhQ8F6gFz28BkL/lPVaQEMhmkNOxPE4ClfRGhK3rhGr45qI4cSEg== X-Received: by 2002:a5d:6cca:: with SMTP id c10mr1093141wrc.224.1631065355841; Tue, 07 Sep 2021 18:42:35 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l15sm555977wrv.84.2021.09.07.18.42.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Sep 2021 18:42:35 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 08 Sep 2021 01:42:25 +0000 Subject: [PATCH v5 1/9] t7519: rewrite sparse index test Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, matheus.bernardino@usp.br, stolee@gmail.com, Johannes Schindelin , Eric Sunshine , =?utf-8?b?UmVuw6k=?= Scharfe , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The sparse index is tested with the FS Monitor hook and extension since f8fe49e (fsmonitor: integrate with sparse index, 2021-07-14). This test was very fragile because it shared an index across sparse and non-sparse behavior. Since that expansion and contraction could cause the index to lose its FS Monitor bitmap and token, behavior is fragile to changes in 'git sparse-checkout set'. Rewrite the test to use two clones of the original repo: full and sparse. This allows us to also keep the test files (actual, expect, trace2.txt) out of the repos we are testing with 'git status'. Signed-off-by: Derrick Stolee --- t/t7519-status-fsmonitor.sh | 38 ++++++++++++++++++++----------------- 1 file changed, 21 insertions(+), 17 deletions(-) diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh index deea88d4431..f1463197b99 100755 --- a/t/t7519-status-fsmonitor.sh +++ b/t/t7519-status-fsmonitor.sh @@ -389,43 +389,47 @@ test_expect_success 'status succeeds after staging/unstaging' ' # If "!" is supplied, then we verify that we do not call ensure_full_index # during a call to 'git status'. Otherwise, we verify that we _do_ call it. check_sparse_index_behavior () { - git status --porcelain=v2 >expect && - git sparse-checkout init --cone --sparse-index && - git sparse-checkout set dir1 dir2 && + git -C full status --porcelain=v2 >expect && GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ - git status --porcelain=v2 >actual && + git -C sparse status --porcelain=v2 >actual && test_region $1 index ensure_full_index trace2.txt && test_region fsm_hook query trace2.txt && test_cmp expect actual && - rm trace2.txt && - git sparse-checkout disable + rm trace2.txt } test_expect_success 'status succeeds with sparse index' ' - git reset --hard && + git clone . full && + git clone --sparse . sparse && + git -C sparse sparse-checkout init --cone --sparse-index && + git -C sparse sparse-checkout set dir1 dir2 && - test_config core.fsmonitor "$TEST_DIRECTORY/t7519/fsmonitor-all" && - check_sparse_index_behavior ! && - - write_script .git/hooks/fsmonitor-test<<-\EOF && + write_script .git/hooks/fsmonitor-test <<-\EOF && printf "last_update_token\0" EOF - git config core.fsmonitor .git/hooks/fsmonitor-test && + git -C full config core.fsmonitor ../.git/hooks/fsmonitor-test && + git -C sparse config core.fsmonitor ../.git/hooks/fsmonitor-test && check_sparse_index_behavior ! && - write_script .git/hooks/fsmonitor-test<<-\EOF && + write_script .git/hooks/fsmonitor-test <<-\EOF && printf "last_update_token\0" printf "dir1/modified\0" EOF check_sparse_index_behavior ! && - cp -r dir1 dir1a && - git add dir1a && - git commit -m "add dir1a" && + git -C sparse sparse-checkout add dir1a && + + for repo in full sparse + do + cp -r $repo/dir1 $repo/dir1a && + git -C $repo add dir1a && + git -C $repo commit -m "add dir1a" || return 1 + done && + git -C sparse sparse-checkout set dir1 dir2 && # This one modifies outside the sparse-checkout definition # and hence we expect to expand the sparse-index. - write_script .git/hooks/fsmonitor-test<<-\EOF && + write_script .git/hooks/fsmonitor-test <<-\EOF && printf "last_update_token\0" printf "dir1a/modified\0" EOF From patchwork Wed Sep 8 01:42:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12479691 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C016EC433F5 for ; Wed, 8 Sep 2021 01:42:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A272A61139 for ; Wed, 8 Sep 2021 01:42:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346778AbhIHBnt (ORCPT ); Tue, 7 Sep 2021 21:43:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55582 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346638AbhIHBnp (ORCPT ); Tue, 7 Sep 2021 21:43:45 -0400 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D53E6C061575 for ; Tue, 7 Sep 2021 18:42:37 -0700 (PDT) Received: by mail-wr1-x42c.google.com with SMTP id x6so692613wrv.13 for ; Tue, 07 Sep 2021 18:42:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=NRMabsLggRiXlRQv0kIMAy+bZxcG89mSHktSlIoVFk4=; b=XUHXOJstN4jJINHPfUF74W92rpoAE9et4X0GxfSzWDE5NYvYLl0qLDpBWx5b8ECOUa bEC+MhAxPtYis7/0QbCWwxJbllV3YF8RB028q7niyDsr4hKdxR5mhBJiVDtkDkDAytsS 9eqQc7PpO/92wqnzCsGwI94meTlw7JarrSv5TD1O/n4ob3RcWJj3Rj2DeO5Vkqt/ILgF AzT1feCUH+hdtyGlGCnC22r3+QZDaMFNPvW8zptYnXgjF8EqNCDPUArmSwZgOFIQRltd mX7XzAl7nxEqZYp+g7YO2tOX2M445NaQ3Jgck8LnVq7xrmnR9FyyNaefBh2QnD+56KVn Yviw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=NRMabsLggRiXlRQv0kIMAy+bZxcG89mSHktSlIoVFk4=; b=QlmPQKN4557F/Hd5K9kVnZcCH5bbmFNY+vd7cLhSN+epR9T3BKMSeCAS5lj1ZWFQI3 s8LPBPubeeNfEx1fJ0xngQ10+H6RAikiyoSE/VJ/D3H3K7ZZSQ9qsBG0GSfTfDDaWpH9 Y5q2i+3BuL5jOyKG+Uzed1vgaCeKAxihQfviIxRjl0JoIlBBpR4fH68nVVGQpNaKygCs 8DKD2lOOU5wVpOVYrVVlbbGVuK2tOpAUH4ZhU9LGc3tu/xjUVCKuzajGF6ojVmsD6hN/ HvusHKAeYFwsTuDaYUbG17HB6rv+qFSTlrK2IV+yY63zG/USzEm1oaytDjW6OzihrByT jPSg== X-Gm-Message-State: AOAM530NoZZLHhgz6jUKPMptavZHp44zFYn+IYXR9DTsLn0PCOrPuoVW hgOCIhIr2RfHbQQdxI2pE3MeLLu3qpc= X-Google-Smtp-Source: ABdhPJy3qjSaO9d+TJZ7+SPj6bxOFzL4W3Sl76o3LAkGJZWxzGD7x62PFtwZbwNBdB3uKD6Al1uu8g== X-Received: by 2002:a05:6000:92:: with SMTP id m18mr1081361wrx.293.1631065356526; Tue, 07 Sep 2021 18:42:36 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id s12sm583855wru.41.2021.09.07.18.42.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Sep 2021 18:42:36 -0700 (PDT) Message-Id: <8660877ba7a06971bb5c443bbc1e07825e950a60.1631065353.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 08 Sep 2021 01:42:26 +0000 Subject: [PATCH v5 2/9] sparse-index: silently return when not using cone-mode patterns Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, matheus.bernardino@usp.br, stolee@gmail.com, Johannes Schindelin , Eric Sunshine , =?utf-8?b?UmVuw6k=?= Scharfe , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee While the sparse-index is only enabled when core.sparseCheckoutCone is also enabled, it is possible for the user to modify the sparse-checkout file manually in a way that does not match cone-mode patterns. In this case, we should refuse to convert an index into a sparse index, since the sparse_checkout_patterns will not be initialized with recursive and parent path hashsets. Also silently return if there are no cache entries, which is a simple case: there are no paths to make sparse! Signed-off-by: Derrick Stolee --- sparse-index.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/sparse-index.c b/sparse-index.c index c6b4feec413..cd6e0d5f408 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -130,7 +130,7 @@ static int index_has_unmerged_entries(struct index_state *istate) int convert_to_sparse(struct index_state *istate) { int test_env; - if (istate->split_index || istate->sparse_index || + if (istate->split_index || istate->sparse_index || !istate->cache_nr || !core_apply_sparse_checkout || !core_sparse_checkout_cone) return 0; @@ -158,10 +158,16 @@ int convert_to_sparse(struct index_state *istate) return 0; } - if (!istate->sparse_checkout_patterns->use_cone_patterns) { - warning(_("attempting to use sparse-index without cone mode")); - return -1; - } + /* + * We need cone-mode patterns to use sparse-index. If a user edits + * their sparse-checkout file manually, then we can detect during + * parsing that they are not actually using cone-mode patterns and + * hence we need to abort this conversion _without error_. Warnings + * already exist in the pattern parsing to inform the user of their + * bad patterns. + */ + if (!istate->sparse_checkout_patterns->use_cone_patterns) + return 0; /* * NEEDSWORK: If we have unmerged entries, then stay full. From patchwork Wed Sep 8 01:42:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12479693 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 571A7C433EF for ; Wed, 8 Sep 2021 01:42:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3ADDB6113C for ; Wed, 8 Sep 2021 01:42:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346685AbhIHBnu (ORCPT ); Tue, 7 Sep 2021 21:43:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55588 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346223AbhIHBnp (ORCPT ); Tue, 7 Sep 2021 21:43:45 -0400 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6F158C061757 for ; Tue, 7 Sep 2021 18:42:38 -0700 (PDT) Received: by mail-wm1-x32e.google.com with SMTP id k5-20020a05600c1c8500b002f76c42214bso307207wms.3 for ; Tue, 07 Sep 2021 18:42:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:mime-version :content-transfer-encoding:fcc:to:cc; bh=JRS95RVbeDzgbDO/0MMlbbEuXlEVV1AwpA9tdsh+v3Y=; b=W+fFeFiwA+DSYtTr8G0KQZEPCfxWez4fUsNScuLFpMUNw4d7JWo0tzF/FoLoZb0PJC TZV8lO/vZzkcrV5ihcUPdyJwMznzWF9UfHODdKpNvV4E/7y0DHVVhpkD64FTaIQtC7gR Hwmt4iLSjFsM6BtDHh6m/BuWZM9bhXtol7DQpCcg8HfB80RT8miRicDA4ciLkQTiJtyt F1d9WeWdtffoXM4qZztTABBlHfro6+2zXrek9gmXDyAfNDiT+lv8030dXcW3CRfAM1zH EaeHEauf93SMxapC7SEdqOmKO4tcRqrM3F9xEqwcf7oN7b0sLE7qr9mKNzUSbCXoGFlQ JjHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:mime-version:content-transfer-encoding:fcc:to:cc; bh=JRS95RVbeDzgbDO/0MMlbbEuXlEVV1AwpA9tdsh+v3Y=; b=XJ2oFRs81BcvtQZk5FIuGDbYVtWVetLiT004i66uM/YNDjpqrTxsNR9+VQgJ9DvVXS GDpwKQTw2jVt5Bt/yG962WGE2avXLAN2IRTjK0QvI7b9rLfF8eIvV8jrrfiqOUuUUNQs Uc1MspSuNAzb3KvZwp5bfT2VWJWkARyJi3aVm19X7pEOZeh240a5TzQT68+8orx3tM4u 9eenteDoqUW/bs2jxX6JXirErXIzhsz9ev8LjKQMdsImrqO3ePrkp5gxjEb0ESs0lFy+ EuW0FqWTxOFC5UPDLzj1yQLjvddi8YBxzDyI8HBcovPZYg6X/auwyEf+wPgxl17MoNn8 1l/g== X-Gm-Message-State: AOAM5304L1q3A8ELN/x+NHqKECdR8FF00xwHuP7lWTwt04Wqy4zypvZM KiSnJP4cfioexu0JbWgbTdHx82KzaGo= X-Google-Smtp-Source: ABdhPJxLLelxYZcKNBjHtTdcEc9+3wJ9/lbkdOAq7mBMVIUfOQe4lMalyXQXNsKAK8NGLHlf03ebyg== X-Received: by 2002:a05:600c:295:: with SMTP id 21mr936710wmk.37.1631065357068; Tue, 07 Sep 2021 18:42:37 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id p9sm597685wrg.14.2021.09.07.18.42.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Sep 2021 18:42:36 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 08 Sep 2021 01:42:27 +0000 Subject: [PATCH v5 3/9] unpack-trees: fix nested sparse-dir search MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, matheus.bernardino@usp.br, stolee@gmail.com, Johannes Schindelin , Eric Sunshine , =?utf-8?b?UmVuw6k=?= Scharfe , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The iterated search in find_cache_entry() was recently modified to include a loop that searches backwards for a sparse directory entry that matches the given traverse_info and name_entry. However, the string comparison failed to actually concatenate those two strings, so this failed to find a sparse directory when it was not a top-level directory. This caused some errors in rare cases where a 'git checkout' spanned a diff that modified files within the sparse directory entry, but we could not correctly find the entry. Helped-by: Johannes Schindelin Helped-by: René Scharfe Signed-off-by: Derrick Stolee --- unpack-trees.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/unpack-trees.c b/unpack-trees.c index 5786645f315..b78d7ee6c09 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -1255,7 +1255,7 @@ static int sparse_dir_matches_path(const struct cache_entry *ce, static struct cache_entry *find_cache_entry(struct traverse_info *info, const struct name_entry *p) { - struct cache_entry *ce; + const char *path; int pos = find_cache_pos(info, p->path, p->pathlen); struct unpack_trees_options *o = info->data; @@ -1281,9 +1281,11 @@ static struct cache_entry *find_cache_entry(struct traverse_info *info, * paths (e.g. "subdir-"). */ while (pos >= 0) { - ce = o->src_index->cache[pos]; + struct cache_entry *ce = o->src_index->cache[pos]; - if (strncmp(ce->name, p->path, p->pathlen)) + if (!skip_prefix(ce->name, info->traverse_path, &path) || + strncmp(path, p->path, p->pathlen) || + path[p->pathlen] != '/') return NULL; if (S_ISSPARSEDIR(ce->ce_mode) && From patchwork Wed Sep 8 01:42:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12479697 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 42F31C433F5 for ; Wed, 8 Sep 2021 01:42:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 27F426113C for ; Wed, 8 Sep 2021 01:42:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346726AbhIHBnv (ORCPT ); Tue, 7 Sep 2021 21:43:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55590 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235180AbhIHBnq (ORCPT ); Tue, 7 Sep 2021 21:43:46 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F081EC06175F for ; Tue, 7 Sep 2021 18:42:38 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id y132so485129wmc.1 for ; Tue, 07 Sep 2021 18:42:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=0n87QV9Z/mw0KmAZmpsOxn8gQVw5CMem+cn2Z07qKf8=; b=CEMopTY0UaBuwxqRFcbdqPo1XYCY3NSU80ZX1vZkhVDrHaPf8WU4UKqKl2PGw7ZE/A rxvuIbo/aOPxoSaId1Ei0iC4mObl13eXxCcrnDfphacwtTCMV0IV7u7gYvBgjvC19b9j M2OVflHaAeRa+/w240HzX5wi+U3AHyRn6mxENUusPViIMHIMpOhC4UuaX2+M534EPirj Y9XA8duiNzutQ6hresTQbYigWowb0UXxH/oPpIIhzraRe3kyJNb0e61saaB0tPl8wGFS Ka1owaHXKBl8Ob8RQKtDR9A2lOkTUoc44wo2336iPVzJ0pRyBu1630anOPa01f7MCmTp oTyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=0n87QV9Z/mw0KmAZmpsOxn8gQVw5CMem+cn2Z07qKf8=; b=W2XHTOvgVeEoMgQihuU00+ztjGQFWOYrBXX83utLAyn/orfeLoiJ6TEAfXlVBiX63P OPIulBioHuCnE+dXqp17QJT8PTm8MimqopRqIoOJqf1NAX8TEtdnj5lJlgCKbkJLJd7r ZbdCAWMYlW/f6NdAJi47R8+5tJb3556jYc4tQIqadDqURXvtUDtHA9KLsU8jmGjjoX5H BFArcGboFZfmFKrbJh5WfaYdWkeZOV0VheDWr7/RP4xv6TBRK+cHhdIEvQP5t41RMjyW 7LmZ8Hsuzbj2yvue0Y+hcZka1OGSxa188fzOhVi+GmyNtK5z5V+5bPdQ8H9vGjwyYfqV wLjA== X-Gm-Message-State: AOAM531Yae1gOOkjeDovXP7IbBgj/mdQrlXf5AaxDczsCTN8wcaYYHK2 +18ti2fnTVCdK9J8UOrS5wA+PDXRAc4= X-Google-Smtp-Source: ABdhPJyTocP4gntY/Q4V5EaiZqGaMXRs50blo13CrTfUGeA6j37tZxse/t3+JUsfGxp/lDfUEqIPtw== X-Received: by 2002:a05:600c:2f90:: with SMTP id t16mr978844wmn.136.1631065357652; Tue, 07 Sep 2021 18:42:37 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o2sm597200wrh.13.2021.09.07.18.42.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Sep 2021 18:42:37 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 08 Sep 2021 01:42:28 +0000 Subject: [PATCH v5 4/9] sparse-index: silently return when cache tree fails Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, matheus.bernardino@usp.br, stolee@gmail.com, Johannes Schindelin , Eric Sunshine , =?utf-8?b?UmVuw6k=?= Scharfe , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee If cache_tree_update() returns a non-zero value, then it could not create the cache tree. This is likely due to a path having a merge conflict. Since we are already returning early, let's return silently to avoid making it seem like we failed to write the index at all. If we remove our dependence on the cache tree within convert_to_sparse(), then we could still recover from this scenario and have a sparse index. Signed-off-by: Derrick Stolee --- sparse-index.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/sparse-index.c b/sparse-index.c index cd6e0d5f408..d9b07695953 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -178,10 +178,12 @@ int convert_to_sparse(struct index_state *istate) /* Clear and recompute the cache-tree */ cache_tree_free(&istate->cache_tree); - if (cache_tree_update(istate, 0)) { - warning(_("unable to update cache-tree, staying full")); - return -1; - } + /* + * Silently return if there is a problem with the cache tree update, + * which might just be due to a conflict state in some entry. + */ + if (cache_tree_update(istate, 0)) + return 0; remove_fsmonitor(istate); From patchwork Wed Sep 8 01:42:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12479695 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC0DAC4332F for ; Wed, 8 Sep 2021 01:42:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C6EE760555 for ; Wed, 8 Sep 2021 01:42:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346689AbhIHBnu (ORCPT ); Tue, 7 Sep 2021 21:43:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55594 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346441AbhIHBnr (ORCPT ); Tue, 7 Sep 2021 21:43:47 -0400 Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 785F1C0613C1 for ; Tue, 7 Sep 2021 18:42:39 -0700 (PDT) Received: by mail-wm1-x331.google.com with SMTP id j17-20020a05600c1c1100b002e754875260so290494wms.4 for ; Tue, 07 Sep 2021 18:42:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=MbU88O7tq8/LF2BlKKzEtM7WyKkge6STN6O1kXeP2X4=; b=QAoA2hozMExu4pnkALgP+1Q9R7VMM7hhgc30I4guL2KlB6DVZxidffNwjED3A+I8Ig ii3SJaeh1tOpTySb6EE0fILuAD7aXI1tLi8WFmAnkHGjUeyxlda6y+BrYcVKBtuFRH1B A2jqo/47h3fDP2Q1E1vEPgThX0sV5Qu4eF4xARzg3pCEHwKahqetQ5m8QNFEui8FkFTT YJD/6531xHEcULkxgQeueqZIhbgCR5isXxTG03u1PpnvXq64FkWzAKHe78dVXgC+eDkl fnUjqCTlKSH8frX3t7BezgkX4fVgxCwmvhWkcK97qFUi4sWgtK4QnVITS1woAZ09w/6U yn8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=MbU88O7tq8/LF2BlKKzEtM7WyKkge6STN6O1kXeP2X4=; b=BFEDeiEmz8rehtUeY0fxnokUC8tGBOZgEohfLlOmhuKca5i8HSE1Gny9A9IFEeEdXJ aoAkdZO4dIxkFcWEc2//HDL/PQWRrSTGlVSbyO7pymXYPhN9OJg4NVQvy8BF7TBWLG6+ lMOY/Gaqj1yh0LNvSbnE9pKnTHcm6lGh120zE+nn5+QOkCZEK0SeHfIhDy0xoWWpbOQt d9qlzyOfIzGXgsD4BAQ85fmBYSDTd9X+7Ra32JdcqkJnkB++h7bz3zX/BgmbDSEjAzaY WdkgGmTXbHSJjaLpLs7HbSIDYxLrTQ8/UPVQrgrZ/ajNe2W02HOpbJnDzhqeYFM5MXUE 3+yA== X-Gm-Message-State: AOAM533sKZAo5xw39+l67vnRn+9S4/w6UrvlAqEiGi9OpVvEc3i4dcqI WqKy4WREfD/YiIR9eYI+t9Lx/3xy7ug= X-Google-Smtp-Source: ABdhPJyMkZeJchJV3sTtCNkKpqCHSSDmkAEJ+1uldM33GXuXHnd/CBwj1wpIhP7BCHEfs/nx5fmTIw== X-Received: by 2002:a05:600c:4fc1:: with SMTP id o1mr1021741wmq.4.1631065358185; Tue, 07 Sep 2021 18:42:38 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f7sm624782wmh.20.2021.09.07.18.42.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Sep 2021 18:42:37 -0700 (PDT) Message-Id: <371716124247cf89069c1344e77c08853a1f76fb.1631065353.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 08 Sep 2021 01:42:29 +0000 Subject: [PATCH v5 5/9] sparse-index: use WRITE_TREE_MISSING_OK Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, matheus.bernardino@usp.br, stolee@gmail.com, Johannes Schindelin , Eric Sunshine , =?utf-8?b?UmVuw6k=?= Scharfe , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee When updating the cache tree in convert_to_sparse(), the WRITE_TREE_MISSING_OK flag indicates that trees might be computed that do not already exist within the object database. This happens in cases such as 'git add' creating new trees that it wants to store in anticipation of a following 'git commit'. If this flag is not specified, then it might trigger a promisor fetch or a failure due to the object not existing locally. Use WRITE_TREE_MISSING_OK during convert_to_sparse() to avoid these possible reasons for the cache_tree_update() to fail. Signed-off-by: Derrick Stolee --- sparse-index.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/sparse-index.c b/sparse-index.c index d9b07695953..880c5f72338 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -181,8 +181,11 @@ int convert_to_sparse(struct index_state *istate) /* * Silently return if there is a problem with the cache tree update, * which might just be due to a conflict state in some entry. + * + * This might create new tree objects, so be sure to use + * WRITE_TREE_MISSING_OK. */ - if (cache_tree_update(istate, 0)) + if (cache_tree_update(istate, WRITE_TREE_MISSING_OK)) return 0; remove_fsmonitor(istate); From patchwork Wed Sep 8 01:42:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12479699 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E9DF1C433EF for ; Wed, 8 Sep 2021 01:42:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C007B60555 for ; Wed, 8 Sep 2021 01:42:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346829AbhIHBnx (ORCPT ); Tue, 7 Sep 2021 21:43:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55596 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346668AbhIHBnr (ORCPT ); Tue, 7 Sep 2021 21:43:47 -0400 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0D071C0613CF for ; Tue, 7 Sep 2021 18:42:40 -0700 (PDT) Received: by mail-wm1-x332.google.com with SMTP id y132so485144wmc.1 for ; Tue, 07 Sep 2021 18:42:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=xgVFZpCgEwxSR63IhFeNQZTeJ2cnt9VEz2UJ67AnEPM=; b=LBg7lWldvLCo/HWZKcBeHbiKxwwUPWzXF4dUdFUOC+0FbCr1eQfB/8A2wzSt6ISYab 8bPubTsANccEmhDQD3s+IixkZleoJyF23MuV1DTwy5hYUTdmHQS3jMlvwupmuzm5/O60 UKw0zv6JBOB2mVYr6Q64yMZgH99BpEKHlz3Nopi4I5vNZqRF93k1LnxXbp3BfPHRfuwm UfsiJiHCWM+lpGwN+9fKrd3ZCXrKQQ0GWCWEeaBdWckvhY8FMzO74qh7DMGn/zT8D4Dh P9tM7RJBrpHfpMyaG70anYz5uZ9QzmOzCkZhVccXf4KW6qEEbX7uyjuy2qrk4zuA4O5G FrnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=xgVFZpCgEwxSR63IhFeNQZTeJ2cnt9VEz2UJ67AnEPM=; b=nR8K8ac6SX4yTAuEJrhgJXxq0XJs3hLJzPs73lPc5I8aw06YT9rB95nQzgSgGS+9ov WtGNj32ig7hdtvPZsZ9HZ0rruLIlJC/ZDEQo10FykEmfxMhx32oWNuAKYiYXe3d91oy5 fV+RMhifAIRGTR7zOTYo3S+GqoQG7wz27jBowivnmgF2/eYVi3X7cBhBtiuFxxQXYqGX Q61n7Y5xUtzw5BPsSl3vr9ebWCqo5NFzBsw3umSv+m3EWbSwiGB9Re23PaDGWcF9u5lW 5PUqemkFSYI6ht1fjVFb3iEqs9Cj9cDfrSxPEWSa3txwlEYNyHXPjwFyftlJOQ56Q6Z3 gsRA== X-Gm-Message-State: AOAM532XRqY5uwEGxf9jAtuuQ4vofO9/k2mzC+ejmL+5BY18PtZkFNpv 82ym9XKm5567cnp3x/+f7/4UknMuZX8= X-Google-Smtp-Source: ABdhPJxw/rVqs3zzQC/bTHFiWTPRLwskPgeW5boVbDZKMUgiO2RiVDDAZ8i9SxpOqggES1vUMnZP5Q== X-Received: by 2002:a7b:c408:: with SMTP id k8mr921925wmi.184.1631065358704; Tue, 07 Sep 2021 18:42:38 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n20sm541594wms.15.2021.09.07.18.42.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Sep 2021 18:42:38 -0700 (PDT) Message-Id: <98b4cae2297cc93abe83a573360859c16806a6fd.1631065353.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 08 Sep 2021 01:42:30 +0000 Subject: [PATCH v5 6/9] sparse-checkout: create helper methods Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, matheus.bernardino@usp.br, stolee@gmail.com, Johannes Schindelin , Eric Sunshine , =?utf-8?b?UmVuw6k=?= Scharfe , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee As we integrate the sparse index into more builtins, we occasionally need to check the sparse-checkout patterns to see if a path is within the sparse-checkout cone. Create some helper methods that help initialize the patterns and check for pattern matching to make this easier. The existing callers of commands like get_sparse_checkout_patterns() use a custom 'struct pattern_list' that is not necessarily the one in the 'struct index_state', so there are not many previous uses that could adopt these helpers. There are just two in builtin/add.c and sparse-index.c that can use path_in_sparse_checkout(). We add a path_in_cone_mode_sparse_checkout() as well that will only return false if the path is outside of the sparse-checkout definition _and_ the sparse-checkout patterns are in cone mode. Signed-off-by: Derrick Stolee --- builtin/add.c | 7 +------ dir.c | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++ dir.h | 8 ++++++++ sparse-index.c | 14 +++----------- 4 files changed, 64 insertions(+), 17 deletions(-) diff --git a/builtin/add.c b/builtin/add.c index 17528e8f922..88a6c0c69fb 100644 --- a/builtin/add.c +++ b/builtin/add.c @@ -190,8 +190,6 @@ static int refresh(int verbose, const struct pathspec *pathspec) struct string_list only_match_skip_worktree = STRING_LIST_INIT_NODUP; int flags = REFRESH_IGNORE_SKIP_WORKTREE | (verbose ? REFRESH_IN_PORCELAIN : REFRESH_QUIET); - struct pattern_list pl = { 0 }; - int sparse_checkout_enabled = !get_sparse_checkout_patterns(&pl); seen = xcalloc(pathspec->nr, 1); refresh_index(&the_index, flags, pathspec, seen, @@ -199,12 +197,9 @@ static int refresh(int verbose, const struct pathspec *pathspec) for (i = 0; i < pathspec->nr; i++) { if (!seen[i]) { const char *path = pathspec->items[i].original; - int dtype = DT_REG; if (matches_skip_worktree(pathspec, i, &skip_worktree_seen) || - (sparse_checkout_enabled && - !path_matches_pattern_list(path, strlen(path), NULL, - &dtype, &pl, &the_index))) { + !path_in_sparse_checkout(path, &the_index)) { string_list_append(&only_match_skip_worktree, pathspec->items[i].original); } else { diff --git a/dir.c b/dir.c index 03c4d212672..86afa2eae00 100644 --- a/dir.c +++ b/dir.c @@ -1439,6 +1439,58 @@ done: return result; } +int init_sparse_checkout_patterns(struct index_state *istate) +{ + if (!core_apply_sparse_checkout) + return 1; + if (istate->sparse_checkout_patterns) + return 0; + + CALLOC_ARRAY(istate->sparse_checkout_patterns, 1); + + if (get_sparse_checkout_patterns(istate->sparse_checkout_patterns) < 0) { + FREE_AND_NULL(istate->sparse_checkout_patterns); + return -1; + } + + return 0; +} + +static int path_in_sparse_checkout_1(const char *path, + struct index_state *istate, + int require_cone_mode) +{ + const char *base; + int dtype = DT_REG; + + /* + * We default to accepting a path if there are no patterns or + * they are of the wrong type. + */ + if (init_sparse_checkout_patterns(istate) || + (require_cone_mode && + !istate->sparse_checkout_patterns->use_cone_patterns)) + return 1; + + base = strrchr(path, '/'); + return path_matches_pattern_list(path, strlen(path), base ? base + 1 : path, + &dtype, + istate->sparse_checkout_patterns, + istate) > 0; +} + +int path_in_sparse_checkout(const char *path, + struct index_state *istate) +{ + return path_in_sparse_checkout_1(path, istate, 0); +} + +int path_in_cone_mode_sparse_checkout(const char *path, + struct index_state *istate) +{ + return path_in_sparse_checkout_1(path, istate, 1); +} + static struct path_pattern *last_matching_pattern_from_lists( struct dir_struct *dir, struct index_state *istate, const char *pathname, int pathlen, diff --git a/dir.h b/dir.h index b3e1a54a971..6823312521e 100644 --- a/dir.h +++ b/dir.h @@ -394,6 +394,14 @@ enum pattern_match_result path_matches_pattern_list(const char *pathname, const char *basename, int *dtype, struct pattern_list *pl, struct index_state *istate); + +int init_sparse_checkout_patterns(struct index_state *state); + +int path_in_sparse_checkout(const char *path, + struct index_state *istate); +int path_in_cone_mode_sparse_checkout(const char *path, + struct index_state *istate); + struct dir_entry *dir_add_ignored(struct dir_struct *dir, struct index_state *istate, const char *pathname, int len); diff --git a/sparse-index.c b/sparse-index.c index 880c5f72338..23f7c3bd361 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -33,19 +33,14 @@ static int convert_to_sparse_rec(struct index_state *istate, { int i, can_convert = 1; int start_converted = num_converted; - enum pattern_match_result match; - int dtype = DT_UNKNOWN; struct strbuf child_path = STRBUF_INIT; - struct pattern_list *pl = istate->sparse_checkout_patterns; /* * Is the current path outside of the sparse cone? * Then check if the region can be replaced by a sparse * directory entry (everything is sparse and merged). */ - match = path_matches_pattern_list(ct_path, ct_pathlen, - NULL, &dtype, pl, istate); - if (match != NOT_MATCHED) + if (path_in_sparse_checkout(ct_path, istate)) can_convert = 0; for (i = start; can_convert && i < end; i++) { @@ -152,11 +147,8 @@ int convert_to_sparse(struct index_state *istate) if (!istate->repo->settings.sparse_index) return 0; - if (!istate->sparse_checkout_patterns) { - istate->sparse_checkout_patterns = xcalloc(1, sizeof(struct pattern_list)); - if (get_sparse_checkout_patterns(istate->sparse_checkout_patterns) < 0) - return 0; - } + if (init_sparse_checkout_patterns(istate)) + return 0; /* * We need cone-mode patterns to use sparse-index. If a user edits From patchwork Wed Sep 8 01:42:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12479701 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96C94C433F5 for ; Wed, 8 Sep 2021 01:42:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7ACD260555 for ; Wed, 8 Sep 2021 01:42:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346878AbhIHBny (ORCPT ); Tue, 7 Sep 2021 21:43:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55606 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346698AbhIHBnt (ORCPT ); Tue, 7 Sep 2021 21:43:49 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8466DC0613D9 for ; Tue, 7 Sep 2021 18:42:40 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id q26so726455wrc.7 for ; Tue, 07 Sep 2021 18:42:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=UkV8L6mxUeISHBX+MaekVPfswRsMwROBmHz98kMVj+0=; b=obWf1NYpggSYG79EeXy8vwyTw8viML+sCv4WaVRIe4sBXXb1PBSzMRLT76GE/Bd5FN tFTWg0z+Jwh+LNAm6Nx2/fztiZ7TWqd46j33+EDsd8wgD0H6hrvTqYibKokbYYP3n2ks QjeBxp6UT/0CvS1l39++yU8oNvENlVnL52SUV09AaqQ0V3xdnQvIQ6OA6yAWb+hvPSz4 dG+2qrcc0BUIHfAWZebWuVuDSBeAkiu8jTi1XyQE6kdQc6OM10pPCzOhQLOAXdeN8d1D Icc8Awp245Ko631xIaobnrI+Gq7/Z/CVd/SriiwMTNFLHVC0DTGrQM5Xk0ttcJsajYtY UxyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=UkV8L6mxUeISHBX+MaekVPfswRsMwROBmHz98kMVj+0=; b=JFce9/e85EixD2FbvhflGTv4e81QCfJ/eqmpfVXQasJ95Fr2AhglHOoJVYtLtxzE0p db8GRvYYNDBjsCL+woKE0kKT6WWHdzwSdsm2VTPY/rB2i+TSt7yPEBxHKKmF7MlC/wYc adfsOmYF2dZa/yqhEvnsV8S0kJp9X9q8ZIAAUKBlHZ8VUdTxRKRqRspNpznhAMXiOV/6 9b2Mz1eHahbA6NjDGLrIybMlnOdbvQNiWWEoWZ113Mb+cRcxx/UdYpaD8H244+h/O1p4 d0961+R6jCEFIf3gWPFma3mNjZZkcjuDL0g2TUNTOhJo1uMH1gk8C2TLv+UkJqNT9DT/ GeLw== X-Gm-Message-State: AOAM531oJmhwT4dfZSCU4YR9fmF0Egq0cL7lMygooooM/CBo9AceY77g jORJpkzLFfpv0HErEuMu516JxKF2TvQ= X-Google-Smtp-Source: ABdhPJyK5TgcxF9FSwL6GlCRyv1sZl9WnMTqknUYSDCqEk6vlRSvrPEl4PVGoMd5pEtELzr6Vu6oZw== X-Received: by 2002:adf:f88d:: with SMTP id u13mr1121577wrp.297.1631065359233; Tue, 07 Sep 2021 18:42:39 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id e3sm731308wrv.18.2021.09.07.18.42.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Sep 2021 18:42:38 -0700 (PDT) Message-Id: <6ec3cb2042e1064387dba410c9494a6dc8be0e54.1631065353.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 08 Sep 2021 01:42:31 +0000 Subject: [PATCH v5 7/9] attr: be careful about sparse directories Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, matheus.bernardino@usp.br, stolee@gmail.com, Johannes Schindelin , Eric Sunshine , =?utf-8?b?UmVuw6k=?= Scharfe , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Signed-off-by: Derrick Stolee --- attr.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/attr.c b/attr.c index d029e681f28..79adaa50ea1 100644 --- a/attr.c +++ b/attr.c @@ -14,6 +14,7 @@ #include "utf8.h" #include "quote.h" #include "thread-utils.h" +#include "dir.h" const char git_attr__true[] = "(builtin)true"; const char git_attr__false[] = "\0(builtin)false"; @@ -744,6 +745,20 @@ static struct attr_stack *read_attr_from_index(struct index_state *istate, if (!istate) return NULL; + /* + * The .gitattributes file only applies to files within its + * parent directory. In the case of cone-mode sparse-checkout, + * the .gitattributes file is sparse if and only if all paths + * within that directory are also sparse. Thus, don't load the + * .gitattributes file since it will not matter. + * + * In the case of a sparse index, it is critical that we don't go + * looking for a .gitattributes file, as doing so would cause the + * index to expand. + */ + if (!path_in_cone_mode_sparse_checkout(path, istate)) + return NULL; + buf = read_blob_data_from_index(istate, path, NULL); if (!buf) return NULL; From patchwork Wed Sep 8 01:42:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12479703 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7AD8FC4332F for ; Wed, 8 Sep 2021 01:42:49 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6410161132 for ; Wed, 8 Sep 2021 01:42:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346754AbhIHBnz (ORCPT ); Tue, 7 Sep 2021 21:43:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55612 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346727AbhIHBnt (ORCPT ); Tue, 7 Sep 2021 21:43:49 -0400 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 29DB9C0617A8 for ; Tue, 7 Sep 2021 18:42:41 -0700 (PDT) Received: by mail-wr1-x430.google.com with SMTP id x6so692726wrv.13 for ; Tue, 07 Sep 2021 18:42:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=M+K0Z9WIqh5h1pKa505Bywi5J0nM3YeVcKgYGu5P63g=; b=HrlYllE9PfDjE4VgD4a/ZWl/HeF7nGhNW8ArUvhB3lHpOmzFtXiK9aEk9cjwYVmx0X K2GybSBbvGJWuGao4SU7VZtOaU5ays9Y4u6DkwVWiK3Af/bmCKJpDamzqM9Y4pwevqow EgmNiDWZ/K9JiRJuzdHODJJy0I34Ifc5liC8lNqmkKoLFwP37yhrk2O+0AzgnAywwGcd BqwqO9csWAsh3i6C3AOORVb5QyCVzDR1gZC1zULECSF9G15EiIfa+DBG94tRttkLqCXd xSahae+LvvTiOOynC/UKQusr8k2yYg4nMIYlNT2MJdVZ+IFBg19PyWM4KEmLrSszS6jH kHuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=M+K0Z9WIqh5h1pKa505Bywi5J0nM3YeVcKgYGu5P63g=; b=B7VB+X8hv8oYdzkieRLCeIQg6U6WpQfq91i//e038b9p7R08a1hhkp5fBknNYmpHgE qXe+g/Jzryiw0FGfzF7Xe2U+IRFNAcU5FddwF98tmsZiuasKjU8rWMHW3GHMlSgQxuD6 THu8XnRPehEY6EebOZcSDCPDCdSovPRHGM0Ez4uqlVss3a1awz1AvBHOPcvz/kgcdfID GoP61BES2XI35DPP2LM2s3ibY16WLcUAsQ7zbVRVIhR4Soa04GHX9JXzLDCstuJY/BUO e1CBydkX+vtG6cLOWQ7rThxbT39bJdpRMqqrnvUsK65gp9s4GkAKKwMKXr67sKb1YXUZ csPA== X-Gm-Message-State: AOAM531iYwTON8XfFCB7vNDadwSr1fTk2EzDb1Ui1QmNryUHRqTOMOgn Vn3CaZfbAqHzaRMJ/ednfsR+rGA2UmA= X-Google-Smtp-Source: ABdhPJwA9wkZwd6XkdpKtMNQ1AcMnzHMuHvxBQIkJDfoayov9E4r/objAAwm6oMrAjI/LVn2C5tWoQ== X-Received: by 2002:adf:ec8b:: with SMTP id z11mr1119250wrn.122.1631065359787; Tue, 07 Sep 2021 18:42:39 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id i27sm589611wmb.40.2021.09.07.18.42.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Sep 2021 18:42:39 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 08 Sep 2021 01:42:32 +0000 Subject: [PATCH v5 8/9] sparse-index: add SPARSE_INDEX_MEMORY_ONLY flag Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, matheus.bernardino@usp.br, stolee@gmail.com, Johannes Schindelin , Eric Sunshine , =?utf-8?b?UmVuw6k=?= Scharfe , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The convert_to_sparse() method checks for the GIT_TEST_SPARSE_INDEX environment variable or the "index.sparse" config setting before converting the index to a sparse one. This is for ease of use since all current consumers are preparing to compress the index before writing it to disk. If these settings are not enabled, then convert_to_sparse() silently returns without doing anything. We will add a consumer in the next change that wants to use the sparse index as an in-memory data structure, regardless of whether the on-disk format should be sparse. To that end, create the SPARSE_INDEX_MEMORY_ONLY flag that will skip these config checks when enabled. All current consumers are modified to pass '0' in the new 'flags' parameter. Signed-off-by: Derrick Stolee --- read-cache.c | 4 ++-- sparse-index.c | 37 ++++++++++++++++++++++--------------- sparse-index.h | 3 ++- 3 files changed, 26 insertions(+), 18 deletions(-) diff --git a/read-cache.c b/read-cache.c index 9048ef9e905..f5d4385c408 100644 --- a/read-cache.c +++ b/read-cache.c @@ -3069,7 +3069,7 @@ static int do_write_locked_index(struct index_state *istate, struct lock_file *l int ret; int was_full = !istate->sparse_index; - ret = convert_to_sparse(istate); + ret = convert_to_sparse(istate, 0); if (ret) { warning(_("failed to convert to a sparse-index")); @@ -3182,7 +3182,7 @@ static int write_shared_index(struct index_state *istate, int ret, was_full = !istate->sparse_index; move_cache_to_base_index(istate); - convert_to_sparse(istate); + convert_to_sparse(istate, 0); trace2_region_enter_printf("index", "shared/do_write_index", the_repository, "%s", get_tempfile_path(*temp)); diff --git a/sparse-index.c b/sparse-index.c index 23f7c3bd361..0bc45f60ac5 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -122,30 +122,37 @@ static int index_has_unmerged_entries(struct index_state *istate) return 0; } -int convert_to_sparse(struct index_state *istate) +int convert_to_sparse(struct index_state *istate, int flags) { int test_env; - if (istate->split_index || istate->sparse_index || !istate->cache_nr || + if (istate->sparse_index || !istate->cache_nr || !core_apply_sparse_checkout || !core_sparse_checkout_cone) return 0; if (!istate->repo) istate->repo = the_repository; - /* - * The GIT_TEST_SPARSE_INDEX environment variable triggers the - * index.sparse config variable to be on. - */ - test_env = git_env_bool("GIT_TEST_SPARSE_INDEX", -1); - if (test_env >= 0) - set_sparse_index_config(istate->repo, test_env); + if (!(flags & SPARSE_INDEX_MEMORY_ONLY)) { + /* + * The sparse index is not (yet) integrated with a split index. + */ + if (istate->split_index) + return 0; + /* + * The GIT_TEST_SPARSE_INDEX environment variable triggers the + * index.sparse config variable to be on. + */ + test_env = git_env_bool("GIT_TEST_SPARSE_INDEX", -1); + if (test_env >= 0) + set_sparse_index_config(istate->repo, test_env); - /* - * Only convert to sparse if index.sparse is set. - */ - prepare_repo_settings(istate->repo); - if (!istate->repo->settings.sparse_index) - return 0; + /* + * Only convert to sparse if index.sparse is set. + */ + prepare_repo_settings(istate->repo); + if (!istate->repo->settings.sparse_index) + return 0; + } if (init_sparse_checkout_patterns(istate)) return 0; diff --git a/sparse-index.h b/sparse-index.h index 1115a0d7dd9..9f3d7bc7faf 100644 --- a/sparse-index.h +++ b/sparse-index.h @@ -2,7 +2,8 @@ #define SPARSE_INDEX_H__ struct index_state; -int convert_to_sparse(struct index_state *istate); +#define SPARSE_INDEX_MEMORY_ONLY (1 << 0) +int convert_to_sparse(struct index_state *istate, int flags); /* * Some places in the codebase expect to search for a specific path. From patchwork Wed Sep 8 01:42:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12479705 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C124C433F5 for ; Wed, 8 Sep 2021 01:42:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 33F1C6113C for ; Wed, 8 Sep 2021 01:42:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346842AbhIHBn4 (ORCPT ); Tue, 7 Sep 2021 21:43:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55616 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346771AbhIHBnt (ORCPT ); Tue, 7 Sep 2021 21:43:49 -0400 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BA789C0617AE for ; Tue, 7 Sep 2021 18:42:41 -0700 (PDT) Received: by mail-wr1-x436.google.com with SMTP id q11so721114wrr.9 for ; Tue, 07 Sep 2021 18:42:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=6BhipxpdLHCjlk4DMmYt4bLPZa+qCBdpfxVEfdEvEY0=; b=EHNNOqTzW2MZ6dnTjIC/dYCbxW3fFgiZNEH5/3CI3bdR20PDVhVJESM6D4h9dG+ekF BUv/66kyzZ4lkoeRPDRIxQDZEG1BnkHJyGucJZvpfOJt4qQF80suhzk2M6RRSWC/QxBW yfLtmBr2k5WUSddni88n7Db9A5uvHunAWm7/jpN+8VI8cH0+SSdfrcgV1ERWVRQND8W6 q5eUzUjItTmkFH909llid27GZ2Ey9qgfQbnzRw3N3QMdpkNfAfMbSlPIvUU8RVc6O3NF oYInhd35EW/kdR8p3Am9mYOiXhP7+YrAOsn/lu2XWYqHKedHNAK72FdXrr+BkfgBw7Fq 9jgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=6BhipxpdLHCjlk4DMmYt4bLPZa+qCBdpfxVEfdEvEY0=; b=XsoQOsbU5u+EgPLYRkDj2vVyOuO3QA5gVGTOdb9Dxc82z43yep7ZZCbAWsKgVxgSsc qgndnyvF+negc3d2UPr9qUDWIwuTqGicqeeuHs+hFzrx564dEgHv2lP13XvWELilxqza TXIRaoQ0Rp9N57pzAWPS3T151+Alfye9DVpAs6JdAx1C3Fhv/ypmOpXDdzX1FstqfeTU t5BbHm6zA723lXvVnlbKxhDkjIjnr2dUH3ApLg9+BaGcVv7Lxj8ECRvLJRCpc8RSKBoB X1ZyII/KloN3bf2+rDORyQgBd1nwIFsUn++9x5as7uFwZob23Jr96bN5kuglcODtEtm2 XCSA== X-Gm-Message-State: AOAM533NcsDvqh4G3Vjxpo9YbPBnojpGinwA3T3BFJlV07ne4QRS0vzC olFnJO1kU6SfB2peq8cmpw4sHP700Cc= X-Google-Smtp-Source: ABdhPJx2MoQu1d/td3ld0mC50EmaBV+q/xaKYBsk0pQQZMXnB1oY7rVmWvTPsVmhdOnLQe8wbudeYg== X-Received: by 2002:a5d:56c4:: with SMTP id m4mr206679wrw.225.1631065360297; Tue, 07 Sep 2021 18:42:40 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id y1sm573620wmq.43.2021.09.07.18.42.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Sep 2021 18:42:40 -0700 (PDT) Message-Id: <91b53f20109fe55635b1815f87afd5d5da68a182.1631065354.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 08 Sep 2021 01:42:33 +0000 Subject: [PATCH v5 9/9] sparse-checkout: clear tracked sparse dirs Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, matheus.bernardino@usp.br, stolee@gmail.com, Johannes Schindelin , Eric Sunshine , =?utf-8?b?UmVuw6k=?= Scharfe , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee When changing the scope of a sparse-checkout using cone mode, we might have some tracked directories go out of scope. The current logic removes the tracked files from within those directories, but leaves the ignored files within those directories. This is a bit unexpected to users who have given input to Git saying they don't need those directories anymore. This is something that is new to the cone mode pattern type: the user has explicitly said "I want these directories and _not_ those directories." The typical sparse-checkout patterns more generally apply to "I want files with with these patterns" so it is natural to leave ignored files as they are. This focus on directories in cone mode provides us an opportunity to change the behavior. Leaving these ignored files in the sparse directories makes it impossible to gain performance benefits in the sparse index. When we track into these directories, we need to know if the files are ignored or not, which might depend on the _tracked_ .gitignore file(s) within the sparse directory. This depends on the indexed version of the file, so the sparse directory must be expanded. We must take special care to look for untracked, non-ignored files in these directories before deleting them. We do not want to delete any meaningful work that the users were doing in those directories and perhaps forgot to add and commit before switching sparse-checkout definitions. Since those untracked files might be code files that generated ignored build output, also do not delete any ignored files from these directories in that case. The users can recover their state by resetting their sparse-checkout definition to include that directory and continue. Alternatively, they can see the warning that is presented and delete the directory themselves to regain the performance they expect. By deleting the sparse directories when changing scope (or running 'git sparse-checkout reapply') we regain these performance benefits as if the repository was in a clean state. Since these ignored files are frequently build output or helper files from IDEs, the users should not need the files now that the tracked files are removed. If the tracked files reappear, then they will have newer timestamps than the build artifacts, so the artifacts will need to be regenerated anyway. Use the sparse-index as a data structure in order to find the sparse directories that can be safely deleted. Re-expand the index to a full one if it was full before. Signed-off-by: Derrick Stolee --- Documentation/git-sparse-checkout.txt | 10 +++ builtin/sparse-checkout.c | 94 +++++++++++++++++++++++++++ t/t1091-sparse-checkout-builtin.sh | 59 +++++++++++++++++ 3 files changed, 163 insertions(+) diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt index fdcf43f87cb..42056ee9ff9 100644 --- a/Documentation/git-sparse-checkout.txt +++ b/Documentation/git-sparse-checkout.txt @@ -210,6 +210,16 @@ case-insensitive check. This corrects for case mismatched filenames in the 'git sparse-checkout set' command to reflect the expected cone in the working directory. +When changing the sparse-checkout patterns in cone mode, Git will inspect each +tracked directory that is not within the sparse-checkout cone to see if it +contains any untracked files. If all of those files are ignored due to the +`.gitignore` patterns, then the directory will be deleted. If any of the +untracked files within that directory is not ignored, then no deletions will +occur within that directory and a warning message will appear. If these files +are important, then reset your sparse-checkout definition so they are included, +use `git add` and `git commit` to store them, then remove any remaining files +manually to ensure Git can behave optimally. + SUBMODULES ---------- diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c index 8ba9f13787b..d0f5c4702be 100644 --- a/builtin/sparse-checkout.c +++ b/builtin/sparse-checkout.c @@ -100,6 +100,98 @@ static int sparse_checkout_list(int argc, const char **argv) return 0; } +static void clean_tracked_sparse_directories(struct repository *r) +{ + int i, was_full = 0; + struct strbuf path = STRBUF_INIT; + size_t pathlen; + struct string_list_item *item; + struct string_list sparse_dirs = STRING_LIST_INIT_DUP; + + /* + * If we are not using cone mode patterns, then we cannot + * delete directories outside of the sparse cone. + */ + if (!r || !r->index || !r->worktree) + return; + if (init_sparse_checkout_patterns(r->index) || + !r->index->sparse_checkout_patterns->use_cone_patterns) + return; + + /* + * Use the sparse index as a data structure to assist finding + * directories that are safe to delete. This conversion to a + * sparse index will not delete directories that contain + * conflicted entries or submodules. + */ + if (!r->index->sparse_index) { + /* + * If something, such as a merge conflict or other concern, + * prevents us from converting to a sparse index, then do + * not try deleting files. + */ + if (convert_to_sparse(r->index, SPARSE_INDEX_MEMORY_ONLY)) + return; + was_full = 1; + } + + strbuf_addstr(&path, r->worktree); + strbuf_complete(&path, '/'); + pathlen = path.len; + + /* + * Collect directories that have gone out of scope but also + * exist on disk, so there is some work to be done. We need to + * store the entries in a list before exploring, since that might + * expand the sparse-index again. + */ + for (i = 0; i < r->index->cache_nr; i++) { + struct cache_entry *ce = r->index->cache[i]; + + if (S_ISSPARSEDIR(ce->ce_mode) && + repo_file_exists(r, ce->name)) + string_list_append(&sparse_dirs, ce->name); + } + + for_each_string_list_item(item, &sparse_dirs) { + struct dir_struct dir = DIR_INIT; + struct pathspec p = { 0 }; + struct strvec s = STRVEC_INIT; + + strbuf_setlen(&path, pathlen); + strbuf_addstr(&path, item->string); + + dir.flags |= DIR_SHOW_IGNORED_TOO; + + setup_standard_excludes(&dir); + strvec_push(&s, path.buf); + + parse_pathspec(&p, PATHSPEC_GLOB, 0, NULL, s.v); + fill_directory(&dir, r->index, &p); + + if (dir.nr) { + warning(_("directory '%s' contains untracked files," + " but is not in the sparse-checkout cone"), + item->string); + } else if (remove_dir_recursively(&path, 0)) { + /* + * Removal is "best effort". If something blocks + * the deletion, then continue with a warning. + */ + warning(_("failed to remove directory '%s'"), + item->string); + } + + dir_clear(&dir); + } + + string_list_clear(&sparse_dirs, 0); + strbuf_release(&path); + + if (was_full) + ensure_full_index(r->index); +} + static int update_working_directory(struct pattern_list *pl) { enum update_sparsity_result result; @@ -141,6 +233,8 @@ static int update_working_directory(struct pattern_list *pl) else rollback_lock_file(&lock_file); + clean_tracked_sparse_directories(r); + r->index->sparse_checkout_patterns = NULL; return result; } diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh index 38fc8340f5c..71236981e64 100755 --- a/t/t1091-sparse-checkout-builtin.sh +++ b/t/t1091-sparse-checkout-builtin.sh @@ -642,4 +642,63 @@ test_expect_success MINGW 'cone mode replaces backslashes with slashes' ' check_files repo/deep a deeper1 ' +test_expect_success 'cone mode clears ignored subdirectories' ' + rm repo/.git/info/sparse-checkout && + + git -C repo sparse-checkout init --cone && + git -C repo sparse-checkout set deep/deeper1 && + + cat >repo/.gitignore <<-\EOF && + obj/ + *.o + EOF + + git -C repo add .gitignore && + git -C repo commit -m ".gitignore" && + + mkdir -p repo/obj repo/folder1/obj repo/deep/deeper2/obj && + for file in folder1/obj/a obj/a folder1/file.o folder1.o \ + deep/deeper2/obj/a deep/deeper2/file.o file.o + do + echo ignored >repo/$file || return 1 + done && + + git -C repo status --porcelain=v2 >out && + test_must_be_empty out && + + git -C repo sparse-checkout reapply && + test_path_is_missing repo/folder1 && + test_path_is_missing repo/deep/deeper2 && + test_path_is_dir repo/obj && + test_path_is_file repo/file.o && + + git -C repo status --porcelain=v2 >out && + test_must_be_empty out && + + git -C repo sparse-checkout set deep/deeper2 && + test_path_is_missing repo/deep/deeper1 && + test_path_is_dir repo/deep/deeper2 && + test_path_is_dir repo/obj && + test_path_is_file repo/file.o && + + >repo/deep/deeper2/ignored.o && + >repo/deep/deeper2/untracked && + + # When an untracked file is in the way, all untracked files + # (even ignored files) are preserved. + git -C repo sparse-checkout set folder1 2>err && + grep "contains untracked files" err && + test_path_is_file repo/deep/deeper2/ignored.o && + test_path_is_file repo/deep/deeper2/untracked && + + # The rest of the cone matches expectation + test_path_is_missing repo/deep/deeper1 && + test_path_is_dir repo/obj && + test_path_is_file repo/file.o && + + git -C repo status --porcelain=v2 >out && + echo "? deep/deeper2/untracked" >expect && + test_cmp expect out +' + test_done