From patchwork Tue Jun 29 02:04:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12348821 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0838EC11F65 for ; Tue, 29 Jun 2021 02:05:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E41C361CE5 for ; Tue, 29 Jun 2021 02:05:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231236AbhF2CH1 (ORCPT ); Mon, 28 Jun 2021 22:07:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49912 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230080AbhF2CHY (ORCPT ); Mon, 28 Jun 2021 22:07:24 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1AA82C061760 for ; Mon, 28 Jun 2021 19:04:57 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id p8-20020a7bcc880000b02901dbb595a9f1so1277949wma.2 for ; Mon, 28 Jun 2021 19:04:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=cO98IPJ5A+rusUCsSXexJohE9wZWdkoHGqzHKeEQ398=; b=qfdOlKhb02L1895IRL2vetenfbM26wNgVXi0xpWs98dM4rKjA/AkQEESuWgID1HDYx nw1sabl/EWOoaSFEf3We8BwjJeWbWVK8LtHg1spNyBqY6cWyjr2MWlmqCBG4aRNkOAU3 SZ5rjPDj8vPVACw4w6gVlxzL1Y7g10HyjhmC9EeLDmjJcSjXX/vl8crFo3Mmqa9AsWzj tm8N9LhRZRgHboLLl/iXMEOClVNqe+5Mwe9JQc+1S83PW+JVm/7TvYCiHYB9eG/B+94w y+DfQp3GEyU4Rh/q49SqWkKgGJpey3Upc3kzr+FpXN4nfcuST2I+miADUuyYAdJ+IawN hMEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=cO98IPJ5A+rusUCsSXexJohE9wZWdkoHGqzHKeEQ398=; b=MaH4bd1xLbFEyBJA0NOKAbp9X0VXbkJnusqDF89FqM4FWZF1ZHVjD2jIGxa8Pb/pHX Lqx6vu0KpEl+LetvtFGpwT1zwDnn1dUns5uw5fZSqO62wG5ZaOjy6GFqBc9dCYRm1kQ1 VpWerbkIToqucIuyPthoCMNH8OYK8HiXp2gmJvft/SuvQ24YMeU5Iah1HNVEgI27Dw/k pwuERLjMJs2cnd7/NiKfG/NUh20WIam40iZdnw1Am9jjUeVFR7dvWHU0gkQsH6ti/lqx ntxe9uBJuiLvyY9w+/5GtfSnDWdcHW6PCZSspH4MZv2GUzLisQvKKLvWpbGMxvTGGAn9 BN+A== X-Gm-Message-State: AOAM5326H5lyEX0alESQfB77QCYDtpuMAiS354ygf8tGeFpuojXoOJ7O EgBdTB3jNIWm1fA3cqU6Ue50KI04bZ4= X-Google-Smtp-Source: ABdhPJz3oPPMa0J0rRzjEPkmvtTdYULELTnGWYo42/swc8yFSXy33H/sCgFuuacZhxuVa4O/6vS5KA== X-Received: by 2002:a1c:4c0c:: with SMTP id z12mr29894144wmf.0.1624932295647; Mon, 28 Jun 2021 19:04:55 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id c8sm15936589wri.91.2021.06.28.19.04.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 19:04:55 -0700 (PDT) Message-Id: <2a4a7256304c44f168baf54ff1bdc191be67408f.1624932293.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 29 Jun 2021 02:04:38 +0000 Subject: [PATCH v7 01/16] sparse-index: skip indexes with unmerged entries Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The sparse-index format is designed to be compatible with merge conflicts, even those outside the sparse-checkout definition. The reason is that when converting a full index to a sparse one, a cache entry with nonzero stage will not be collapsed into a sparse directory entry. However, this behavior was not tested, and a different behavior within convert_to_sparse() fails in this scenario. Specifically, cache_tree_update() will fail when unmerged entries exist. convert_to_sparse_rec() uses the cache-tree data to recursively walk the tree structure, but also to compute the OIDs used in the sparse-directory entries. Add an index scan to convert_to_sparse() that will detect if these merge conflict entries exist and skip the conversion before trying to update the cache-tree. This is marked as NEEDSWORK because this can be removed with a suitable update to cache_tree_update() or a similar method that can construct a cache-tree with invalid nodes, but still allow creating the nodes necessary for creating sparse directory entries. It is possible that in the future we will not need to make such an update, since if we do not expand a sparse-index into a full one, this conversion does not need to happen. Thus, this can be deferred until the merge machinery is made to integrate with the sparse-index. Signed-off-by: Derrick Stolee --- sparse-index.c | 18 ++++++++++++++++++ t/t1092-sparse-checkout-compatibility.sh | 22 ++++++++++++++++++++++ 2 files changed, 40 insertions(+) diff --git a/sparse-index.c b/sparse-index.c index affc4048f27..2c695930275 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -116,6 +116,17 @@ int set_sparse_index_config(struct repository *repo, int enable) return res; } +static int index_has_unmerged_entries(struct index_state *istate) +{ + int i; + for (i = 0; i < istate->cache_nr; i++) { + if (ce_stage(istate->cache[i])) + return 1; + } + + return 0; +} + int convert_to_sparse(struct index_state *istate) { int test_env; @@ -152,6 +163,13 @@ int convert_to_sparse(struct index_state *istate) return -1; } + /* + * NEEDSWORK: If we have unmerged entries, then stay full. + * Unmerged entries prevent the cache-tree extension from working. + */ + if (index_has_unmerged_entries(istate)) + return 0; + if (cache_tree_update(istate, 0)) { warning(_("unable to update cache-tree, staying full")); return -1; diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index e9a815ca7aa..ba2fd94adaf 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -352,6 +352,28 @@ test_expect_success 'merge with outside renames' ' done ' +# Sparse-index fails to convert the index in the +# final 'git cherry-pick' command. +test_expect_success 'cherry-pick with conflicts' ' + init_repos && + + write_script edit-conflict <<-\EOF && + echo $1 >conflict + EOF + + test_all_match git checkout -b to-cherry-pick && + run_on_all ../edit-conflict ABC && + test_all_match git add conflict && + test_all_match git commit -m "conflict to pick" && + + test_all_match git checkout -B base HEAD~1 && + run_on_all ../edit-conflict DEF && + test_all_match git add conflict && + test_all_match git commit -m "conflict in base" && + + test_all_match test_must_fail git cherry-pick to-cherry-pick +' + test_expect_success 'clean' ' init_repos && From patchwork Tue Jun 29 02:04:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12348823 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B6B8C11F64 for ; Tue, 29 Jun 2021 02:05:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7726C61C99 for ; Tue, 29 Jun 2021 02:05:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231332AbhF2CH2 (ORCPT ); Mon, 28 Jun 2021 22:07:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49914 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230345AbhF2CHZ (ORCPT ); Mon, 28 Jun 2021 22:07:25 -0400 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B0BBCC061766 for ; Mon, 28 Jun 2021 19:04:57 -0700 (PDT) Received: by mail-wr1-x436.google.com with SMTP id a13so23679207wrf.10 for ; Mon, 28 Jun 2021 19:04:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=iy9XiTv1ciKF1QkjuiSZgTtadS+h+8sdv46VtmcVxm4=; b=Q7lE5Yu8uwiEe3NZhUui0JoVoiTgw36P65E7+LLzjY0CO41RRKijJzb0V3Fq7FMgjh xZ3nZ50j7zkJaLwNh0KJiCJDtyhNdJbI+pevR8l10a4nMuTgx0Ipn5UF0IgA37ZLswxw Ilk5+RKXWBbgJfLAv4gw8KkIhgA+ORyminImfLT6EHLYn3KUR4FVVA2IoZ7hpQ9nNo9v sqt4B5vfhTTRqQgyKiIByu8R7xrUXh/9hNMssErFThRAyj/xnByxO3tugdG4ksPX+Zn8 3DGPU4YUeltnVu6NwVwe0mALxOj/bZ04upBThzN8yXKICjuY0Ysm8LjpfT0aPZWpFDP8 vE/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=iy9XiTv1ciKF1QkjuiSZgTtadS+h+8sdv46VtmcVxm4=; b=XmITuP/WLET0atxN+LRRKmdW7At6PguKOaE/DNvUFus2pIdSxpea3qAHFvyKJgP1XJ ndS80zQHpSKjT0i23dn2OY0foxesm7/5Xb+H3wvkcViwn8OV9a99W8sbJgnfKY6OxWUh 5jiCAJCixhWiVG6MV/hgU3td+JdMiqGsNM6PZXeC1IwFPzmJJrU5Qycw1nio5cyQkUOR KwSVoTBt4ZB6Ra26a5mpKaORXxATl3UdpvKNNyzZuGKC4/eJLkSG+IZTx2moN3DzKQPl sfKtufuDjgDkWJeJmVOf7bTDV4l0S/1wBOZh8mYC8xDYxrrIJFNxreAFXoWrZJuOb7id 73Tg== X-Gm-Message-State: AOAM533iP3LsbvPKQhBsyn5tGPgsPSE59+QRxildZ39kYH1XhgwPrNzP quR+S9e60wb6QVFla4yrBTqGBBH6onI= X-Google-Smtp-Source: ABdhPJzKCUdSLCU1eV12I1e8AOzhXre6ypNOH0hZOc2YuMqd2dTI8+FFdM+wxZ2Prf4R5FWe1KZ/Gw== X-Received: by 2002:a5d:65c1:: with SMTP id e1mr31607912wrw.196.1624932296236; Mon, 28 Jun 2021 19:04:56 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id x81sm1106212wmg.36.2021.06.28.19.04.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 19:04:55 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 29 Jun 2021 02:04:39 +0000 Subject: [PATCH v7 02/16] sparse-index: include EXTENDED flag when expanding Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee When creating a full index from a sparse one, we create cache entries for every blob within a given sparse directory entry. These are correctly marked with the CE_SKIP_WORKTREE flag, but the CE_EXTENDED flag is not included. The CE_EXTENDED flag would exist if we loaded a full index from disk with these entries marked with CE_SKIP_WORKTREE, so we can add the flag here to be consistent. This allows us to directly compare the flags present in cache entries when testing the sparse-index feature, but has no significance to its correctness in the user-facing functionality. Signed-off-by: Derrick Stolee --- sparse-index.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparse-index.c b/sparse-index.c index 2c695930275..ef53bd2198b 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -213,7 +213,7 @@ static int add_path_to_index(const struct object_id *oid, strbuf_addstr(base, path); ce = make_cache_entry(istate, mode, oid, base->buf, 0, 0); - ce->ce_flags |= CE_SKIP_WORKTREE; + ce->ce_flags |= CE_SKIP_WORKTREE | CE_EXTENDED; set_index_entry(istate, istate->cache_nr++, ce); strbuf_setlen(base, len); From patchwork Tue Jun 29 02:04:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12348825 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49CD4C11F66 for ; Tue, 29 Jun 2021 02:05:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3430761CAF for ; Tue, 29 Jun 2021 02:05:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231168AbhF2CH2 (ORCPT ); Mon, 28 Jun 2021 22:07:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49920 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230366AbhF2CHZ (ORCPT ); Mon, 28 Jun 2021 22:07:25 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 28213C061574 for ; Mon, 28 Jun 2021 19:04:58 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id j1so23661747wrn.9 for ; Mon, 28 Jun 2021 19:04:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=M5y7V+S57JkqDlEiYCsuFlFYkP2WRg1BWKCFbReDZCs=; b=k+HKXs+h7vm7e0UJ1cMIgZgsorbZRuEoTy1kmbREGfEwlS2/uxwlPmTRKR8H8WWCCA Eb7Ebm3F2Gyi9eUyqEUeo135Z935ReSdAKVGaJiEhU6u30MJ1wsFAhe1iNyMsdpSdJMN pmDvqG5x0/nTvuYfQd0Kiz37Fl1olGem9Dyx2bek3wT5cXSZWPPlcp4mbfXxLpQkrHSt 5vUkNuaUfvzU0WK2OEJvxxddSOdd2IXzP9+gAKdx/tAth5jcYO7GSOtiRYrn9JF4baqv VHjjYf4K4DNP7IUGftYIkaiOtEiw9YqdKovL187TR0Nu8T/udpTu9KhwhsEpwsXuS2Ly g2CA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=M5y7V+S57JkqDlEiYCsuFlFYkP2WRg1BWKCFbReDZCs=; b=sPNUzw5Zpu/axV67crwOP0hc5ylKEGn01er1jf8UDKkq5uetqBG06o9TK4wKuSn8X0 GufBBk7KAfZYi7dIXPr/GLpD2/OasTgBDb7jce9yVw5XLAK7Tc1UQvS/sLz1nPqdU6LZ zyCfMt2wvlHFu0nIc5vQoexvOCUZtZCST4hIMTCv/1qJIettcV9+F6ndwL12MaWSxA54 jqFT8filOnhRfroWDqdi0XgzktGg/KwhwMjZzs9a2U+f36zsyyXRK26hNskxLjCjnCiA 2Rm38RV9er+cmNt/1WRGt50+eduk+b5uwRxD4Ii3b9wsdi0YvSleyZaYiSmi1Lf6gugl DQIQ== X-Gm-Message-State: AOAM53356Fo2WsgnbspqcilnCfylUufpw4epm6h4hOAvS3tHyFc00/TD iykhTkTMj8cNDr4uj7XZr8suGIC/ehY= X-Google-Smtp-Source: ABdhPJzgnGyvtHIkBA0nUdGKoIrNnjeO3REHiluobhMIWeXXJXHlqgecYaEd3LPsdNszmzKqp91Djw== X-Received: by 2002:a5d:6ac2:: with SMTP id u2mr30507417wrw.104.1624932296775; Mon, 28 Jun 2021 19:04:56 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 12sm1092784wmj.12.2021.06.28.19.04.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 19:04:56 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 29 Jun 2021 02:04:40 +0000 Subject: [PATCH v7 03/16] t1092: replace incorrect 'echo' with 'cat' Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee This fixes the test data shape to be as expected, allowing rename detection to work properly now that the 'larger-content' file actually has meaningful lines. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index ba2fd94adaf..ebbba044f77 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -40,7 +40,7 @@ test_expect_success 'setup' ' done && git checkout -b rename-base base && - echo >folder1/larger-content <<-\EOF && + cat >folder1/larger-content <<-\EOF && matching lines help From patchwork Tue Jun 29 02:04:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12348827 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D918C11F67 for ; Tue, 29 Jun 2021 02:05:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1248F61CE5 for ; Tue, 29 Jun 2021 02:05:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231348AbhF2CH3 (ORCPT ); Mon, 28 Jun 2021 22:07:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49922 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231181AbhF2CH0 (ORCPT ); Mon, 28 Jun 2021 22:07:26 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AF324C061767 for ; Mon, 28 Jun 2021 19:04:58 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id u8so10418309wrq.8 for ; Mon, 28 Jun 2021 19:04:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=uPipHskTT++x5mo1IyP2Wqx3KuP5iqrfQ6d15OcP7qY=; b=ptfjLTULbA26FFJAJOPHu3oSsPNug5vZUwp2gSBuLn1Vj2fmg0WK7kmVXvc8sky+ab eUXwB/tZiW9UNEiX+m3woW1rUhj1O2nopBmVVsi5xM/Dta8H77xnA2aacIqpb0ZhknXg ZvmeDuGhcp4Nu19Bc4ySEiMJed9DvvMYGbkp623gKKhjhZuTraAfW+vTXvd4XJwxoR5y xbFulUrSh6fJhbg5dnzmZ0OkDbhFIbRg0llY+K78E5uixsLS/CIUqFgE8KiTRQewVhfu 3rdE4eTaXA5T001ioz38HkQEU1gaxcw8DDb5yGfLh0gtAOT/IQgavQrua9yp25C5vLHZ k7Jg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=uPipHskTT++x5mo1IyP2Wqx3KuP5iqrfQ6d15OcP7qY=; b=N78DKi0my82LBt8bcTJKaCB8nOMdwNJtQ5j5LjWGtbp0XDndVF6TLjGxJR81jElUaV M/yrkb6UULw8PDRde2SIpvBEST5K8/g3XKc3fWQJl3cDxG08IE8IKXdwekrrIpLBmDeu oqfb3hE8cADE1WvZbogBl51kSx5KtqiRPPcm6I4bOok7gZILKIoVg68csLvTsmyY84WJ 9fXl3lTanBTVE1IFU/BM6fTVmRkG6zE1BO7L0ee65Lfy+e+YPfG5lcUKgwYavxrbdmQi hGLIMyntJO03FZPfUa5Wq6oIPBbIc5rgwjlHfrqjgWHHw+AN056nlkc+KP9yIYl1qI+Y /E0w== X-Gm-Message-State: AOAM5327CTViEwG/AIPax3YnhUzp4e5fL0P7FObkEi59pwEMHTeSL7D/ bRtFf3sCs/InYgG8TgwQWGd7cyIZi1k= X-Google-Smtp-Source: ABdhPJwjRNw4bH+S8U4HYcIQ36DAiS9NSGu4CTOHO5JSNoHbncZjPx4JA2kuAYAGymlc+l5/xcGK1w== X-Received: by 2002:a5d:5919:: with SMTP id v25mr29386377wrd.319.1624932297291; Mon, 28 Jun 2021 19:04:57 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id u12sm16693710wrr.40.2021.06.28.19.04.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 19:04:57 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 29 Jun 2021 02:04:41 +0000 Subject: [PATCH v7 04/16] t1092: expand repository data shape Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee As more features integrate with the sparse-index feature, more and more special cases arise that require different data shapes within the tree structure of the repository in order to demonstrate those cases. Add several interesting special cases all at once instead of sprinkling them across several commits. The interesting cases being added here are: * Add sparse-directory entries on both sides of directories within the sparse-checkout definition. * Add directories outside the sparse-checkout definition who have only one entry and are the first entry of a directory with multiple entries. * Add filenames adjacent to a sparse directory entry that sort before and after the trailing slash. Later tests will take advantage of these shapes, but they also deepen the tests that already exist. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 42 ++++++++++++++++++++++-- 1 file changed, 40 insertions(+), 2 deletions(-) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index ebbba044f77..363d605c209 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -17,7 +17,7 @@ test_expect_success 'setup' ' echo "after folder1" >g && echo "after x" >z && mkdir folder1 folder2 deep x && - mkdir deep/deeper1 deep/deeper2 && + mkdir deep/deeper1 deep/deeper2 deep/before deep/later && mkdir deep/deeper1/deepest && echo "after deeper1" >deep/e && echo "after deepest" >deep/deeper1/e && @@ -25,10 +25,23 @@ test_expect_success 'setup' ' cp a folder2 && cp a x && cp a deep && + cp a deep/before && cp a deep/deeper1 && cp a deep/deeper2 && + cp a deep/later && cp a deep/deeper1/deepest && cp -r deep/deeper1/deepest deep/deeper2 && + mkdir deep/deeper1/0 && + mkdir deep/deeper1/0/0 && + touch deep/deeper1/0/1 && + touch deep/deeper1/0/0/0 && + >folder1- && + >folder1.x && + >folder10 && + cp -r deep/deeper1/0 folder1 && + cp -r deep/deeper1/0 folder2 && + echo >>folder1/0/0/0 && + echo >>folder2/0/1 && git add . && git commit -m "initial commit" && git checkout -b base && @@ -56,11 +69,17 @@ test_expect_success 'setup' ' mv folder1/a folder2/b && mv folder1/larger-content folder2/edited-content && echo >>folder2/edited-content && + echo >>folder2/0/1 && + echo stuff >>deep/deeper1/a && git add . && git commit -m "rename folder1/... to folder2/..." && git checkout -b rename-out-to-in rename-base && mv folder1/a deep/deeper1/b && + echo more stuff >>deep/deeper1/a && + rm folder2/0/1 && + mkdir folder2/0/1 && + echo >>folder2/0/1/1 && mv folder1/larger-content deep/deeper1/edited-content && echo >>deep/deeper1/edited-content && git add . && @@ -68,6 +87,9 @@ test_expect_success 'setup' ' git checkout -b rename-in-to-out rename-base && mv deep/deeper1/a folder1/b && + echo >>folder2/0/1 && + rm -rf folder1/0/0 && + echo >>folder1/0/0 && mv deep/deeper1/larger-content folder1/edited-content && echo >>folder1/edited-content && git add . && @@ -262,13 +284,29 @@ test_expect_success 'diff --staged' ' test_all_match git diff --staged ' -test_expect_success 'diff with renames' ' +test_expect_success 'diff with renames and conflicts' ' init_repos && for branch in rename-out-to-out rename-out-to-in rename-in-to-out do test_all_match git checkout rename-base && test_all_match git checkout $branch -- .&& + test_all_match git status --porcelain=v2 && + test_all_match git diff --staged --no-renames && + test_all_match git diff --staged --find-renames || return 1 + done +' + +test_expect_success 'diff with directory/file conflicts' ' + init_repos && + + for branch in rename-out-to-out rename-out-to-in rename-in-to-out + do + git -C full-checkout reset --hard && + test_sparse_match git reset --hard && + test_all_match git checkout $branch && + test_all_match git checkout rename-base -- . && + test_all_match git status --porcelain=v2 && test_all_match git diff --staged --no-renames && test_all_match git diff --staged --find-renames || return 1 done From patchwork Tue Jun 29 02:04:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12348831 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60170C11F66 for ; Tue, 29 Jun 2021 02:05:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4A6EE61CE5 for ; Tue, 29 Jun 2021 02:05:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231512AbhF2CHc (ORCPT ); Mon, 28 Jun 2021 22:07:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49926 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231133AbhF2CHZ (ORCPT ); Mon, 28 Jun 2021 22:07:25 -0400 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2EF65C061768 for ; Mon, 28 Jun 2021 19:04:59 -0700 (PDT) Received: by mail-wr1-x42f.google.com with SMTP id g7so19090275wri.7 for ; Mon, 28 Jun 2021 19:04:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=0UG3JvnANj8n21Qt83opuTu+nCqIMnauASnFCxaQri4=; b=VrS3rgNld0bAw9sg8rqhGLX2CmoO140VKd/tDbFcbZO+Hod4KsLn0W1dmY+Ty7xPfe Y4N/YngBk5K+rdTb8Nb8Jvib2Mki46FqzJCDDzDhtg6dwHDZiCAHDkN3y3jb5VKHSuZV UHF6DTlmGYnagzrROm5vNWF1Lp3Dm/zAs11cjI3OKcYs9mwWNxRiPQAftPNdLTmR6gBM SmyLJ6gZ/B6wNyWUC58oEW0C6bopWe4RbA9Ag61ieuxVipLh0NRX/0+1Q92EH8uoZe/J TlMEHkkqgDwRLwN9NHmPR0AlAdFXXxsiWy6QvsJM7MpDHtPzmwtkAEvoNKEb9iAzE5wv qxog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=0UG3JvnANj8n21Qt83opuTu+nCqIMnauASnFCxaQri4=; b=CpHzSo9S/9m2AfVxviizGWD5cFqlFXMfRhr93CD8nR2+cip11R2tqHlxB/ylXJZarA MggN9MLNmABBJGysMpc2CltaUnR55Xgd11sPmOwbZ1Q1EBgBLt0hf6OpL1tFV4LwCiVt KXWrun6V7QasFhyPT0ogElKpjiwdUeVnDoL+WWy3cYQMvIVEBytv+D06yW1ZHgN99cCI Df2zZA9jKSRjGIkw7gsgB3BCN0wFLAYpxDOaEW5Vb4B05KgUOa/f+Ic+o6zsbfiETfGv BOs4N5JaF25OnX726VLBxdwyRaG2AHN+BaosthyQYPsDXk+elmoyqBZs9WAuBy0kSSeX fMXw== X-Gm-Message-State: AOAM532qgK1XkY4UHWWGoJ9iPqpjjFI07REEF1OdskTTjeIqZ20hKQkK 7IoSoXVJVXRoFvpFjD7LyX8muz8JVlQ= X-Google-Smtp-Source: ABdhPJygZ292atiG47Yf169lgf1byvU3hSV6mgVsl+bIDLM45T8oHctGhBQTG9TWHjNO6FEyerRrqw== X-Received: by 2002:a5d:6106:: with SMTP id v6mr29607384wrt.278.1624932297836; Mon, 28 Jun 2021 19:04:57 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id s18sm16886203wrw.33.2021.06.28.19.04.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 19:04:57 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 29 Jun 2021 02:04:42 +0000 Subject: [PATCH v7 05/16] t1092: add tests for status/add and sparse files Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Before moving to update 'git status' and 'git add' to work with sparse indexes, add an explicit test that ensures the sparse-index works the same as a normal sparse-checkout when the worktree contains directories and files outside of the sparse cone. Specifically, 'folder1/a' is a file in our test repo, but 'folder1' is not in the sparse cone. When 'folder1/a' is modified, the file is not shown as modified and adding it will fail. This is new behavior as of a20f704 (add: warn when asked to update SKIP_WORKTREE entries, 2021-04-08). Before that change, these adds would be silently ignored. Untracked files are fine: adding new files both with 'git add .' and 'git add folder1/' works just as in a full checkout. This may not be entirely desirable, but we are not intending to change behavior at the moment, only document it. A future change could alter the behavior to be more sensible, and this test could be modified to satisfy the new expected behavior. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 38 ++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 363d605c209..3f61e5686b5 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -254,6 +254,44 @@ test_expect_success 'add, commit, checkout' ' test_all_match git checkout - ' +test_expect_success 'status/add: outside sparse cone' ' + init_repos && + + # adding a "missing" file outside the cone should fail + test_sparse_match test_must_fail git add folder1/a && + + # folder1 is at HEAD, but outside the sparse cone + run_on_sparse mkdir folder1 && + cp initial-repo/folder1/a sparse-checkout/folder1/a && + cp initial-repo/folder1/a sparse-index/folder1/a && + + test_sparse_match git status && + + write_script edit-contents <<-\EOF && + echo text >>$1 + EOF + run_on_sparse ../edit-contents folder1/a && + run_on_all ../edit-contents folder1/new && + + test_sparse_match git status --porcelain=v2 && + + # This "git add folder1/a" fails with a warning + # in the sparse repos, differing from the full + # repo. This is intentional. + test_sparse_match test_must_fail git add folder1/a && + test_sparse_match test_must_fail git add --refresh folder1/a && + test_all_match git status --porcelain=v2 && + + test_all_match git add . && + test_all_match git status --porcelain=v2 && + test_all_match git commit -m folder1/new && + + run_on_all ../edit-contents folder1/newer && + test_all_match git add folder1/ && + test_all_match git status --porcelain=v2 && + test_all_match git commit -m folder1/newer +' + test_expect_success 'checkout and reset --hard' ' init_repos && From patchwork Tue Jun 29 02:04:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12348829 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52711C11F64 for ; Tue, 29 Jun 2021 02:05:05 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3C26061CEB for ; Tue, 29 Jun 2021 02:05:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231402AbhF2CHa (ORCPT ); Mon, 28 Jun 2021 22:07:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49928 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231194AbhF2CH1 (ORCPT ); Mon, 28 Jun 2021 22:07:27 -0400 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AFC1EC061787 for ; Mon, 28 Jun 2021 19:04:59 -0700 (PDT) Received: by mail-wm1-x335.google.com with SMTP id l21-20020a05600c1d15b02901e7513b02dbso717023wms.2 for ; Mon, 28 Jun 2021 19:04:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=20VyvcZukH9mBP0oxKGuPMDI0uq7we8hBvlg4nNNWng=; b=IVqWWAb94YqqSCtCiNGXU0WBuF/FbUrGohhlzceUEUIrwPCSb0JWyS4Lpmqt4P9JuI qCPEw5qVK73VK4mk9JRgHr6ACfZlCC4oHfbtSLqprFnxoxRCgFBGoS78OZjCHJPth/p0 sC4e01X4XGZpGMXju4g2K6G1mcklgiIcS+dyhbjVE9+Q7dqgrZTA054vDSxGkGA2Rin0 CXnLkfHlgzUoGDbxQzfcD3grg456RVQQNw0V8Wzj5NJkpCgo0XwLexwz5+Fs5Ek6JpwN jOxC26L4oR3em/z05TQItsSrhAjGQ1E6gcEm9R6x6ZSa2R+QFW6aUmIh+nRP+2XwziGp W4Ig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=20VyvcZukH9mBP0oxKGuPMDI0uq7we8hBvlg4nNNWng=; b=CVMOfrH4RgCUWdwjKU9u3CYgnznjlIaYH9qSCA0hU4PNX4tjiYhr3fxr0n9jUmsFfh BU0G1DKvTpPzBV+1r3BSGOhYN3oPNejAk2JVzFBcY4yvK7UNublWQDmEddQb55yG2TNF g55pg4TRJFjKU/6mPW2TODB3e+mcRApNtl8iLZW5KsgeE2q4PoI8VxvCwL2eNvMw+gOS exANDh5haYHuU0UTvxEsGuHRQTe14eM8tU6siGEieraX1X9whYKBtJ7qQKzmZf169S+O NkzCRx+VH6iDURpKlvpFe39ay9pI4//q9GVMCkcZGt0H00TcSOz+T1dNoxnNNMl/trr+ oIIw== X-Gm-Message-State: AOAM532yQXMaLPvTdXxCc9AAsrfQk1X0dadeQfWbAubmeKjjhzuqIqu7 VoX+tys1Bdz+eXeq1a3pLOe9ymbJhHw= X-Google-Smtp-Source: ABdhPJz+3ovP30d7fgXA74dwIPy8JzPwRbWcaFAMp1XQYZ96NWh9Ho2EUvBt4Q4Iu/jHj9whlX1XKw== X-Received: by 2002:a7b:ce8e:: with SMTP id q14mr29276003wmj.33.1624932298411; Mon, 28 Jun 2021 19:04:58 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id s5sm16734389wrn.38.2021.06.28.19.04.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 19:04:58 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 29 Jun 2021 02:04:43 +0000 Subject: [PATCH v7 06/16] unpack-trees: preserve cache_bottom Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The cache_bottom member of 'struct unpack_trees_options' is used to track the range of index entries corresponding to a node of the cache tree. While recursing with traverse_by_cache_tree(), this value is preserved on the call stack using a local and then restored as that method returns. The mark_ce_used() method normally modifies the cache_bottom member when it refers to the marked cache entry. However, sparse directory entries are stored as nodes in the cache-tree data structure as of 2de37c53 (cache-tree: integrate with sparse directory entries, 2021-03-30). Thus, the cache_bottom will be modified as the cache-tree walk advances. Do not update it as well within mark_ce_used(). Signed-off-by: Derrick Stolee --- unpack-trees.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/unpack-trees.c b/unpack-trees.c index f88a69f8e71..87c1ed204c8 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -600,6 +600,13 @@ static void mark_ce_used(struct cache_entry *ce, struct unpack_trees_options *o) { ce->ce_flags |= CE_UNPACKED; + /* + * If this is a sparse directory, don't advance cache_bottom. + * That will be advanced later using the cache-tree data. + */ + if (S_ISSPARSEDIR(ce->ce_mode)) + return; + if (o->cache_bottom < o->src_index->cache_nr && o->src_index->cache[o->cache_bottom] == ce) { int bottom = o->cache_bottom; From patchwork Tue Jun 29 02:04:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12348835 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D291EC11F67 for ; Tue, 29 Jun 2021 02:05:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BFB1F61CEB for ; Tue, 29 Jun 2021 02:05:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231618AbhF2CHd (ORCPT ); Mon, 28 Jun 2021 22:07:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49936 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231263AbhF2CH1 (ORCPT ); Mon, 28 Jun 2021 22:07:27 -0400 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 45ACBC0617A6 for ; Mon, 28 Jun 2021 19:05:00 -0700 (PDT) Received: by mail-wr1-x435.google.com with SMTP id j2so23718979wrs.12 for ; Mon, 28 Jun 2021 19:05:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=HDhgtJZsDGyJtVd/z+kWysbOePZ/rRg7q4HZzXdKryk=; b=XiXS9E5d72nTFTLR77JXYd7COWiY8sK9jM+bphCP0SJFcMRiqqRutqC8AfLhhcXLqZ 111WUYHWZZ+5eahJ2TQKTBZZ7zD0wuu8dn1w2ofB3QWQUvbdIIl9/9mE6DVJFNfugLKD zQCIFQclwf1PMbg+VC7cyOjrQMZm0Q7q74AXtGHoTyhWMJVO40DRjnBURlsLmM4cXeYg kLsYTjSsSGqu/LA/JXnbY10nb+sNBMatCRj2Q2oN4nz71WAJ5U+XLp0K0wE8sauiTe/p S7snZxd/8a8LCxLsurZ7qm1WYsfYDR59C+VdV3Zc3q/rsXUNVvWwmyIwNsWSImltMRDu 08nQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=HDhgtJZsDGyJtVd/z+kWysbOePZ/rRg7q4HZzXdKryk=; b=sb6K2DSwyp9NzUzfGdLmziQFfSsnidiCfPulx3k30IjqKjo16zsl2qLY5/nYi/QB2b PmZ+DtrsloQl/XEdJtIC2InZ54/UxclI67zKUz7GJOSERriWBXmcLc7ohExIFHMWCTks /UQjAX2jcgOLdWsblUMvRGU8rHAof/U74cG4QpZWIEdKmjOLW/O06NQQCUanUfPoBmgF HUePrF5vD2s5uo8uCqjZJyoZM+vXJod/ngihzroJPoeaWnONsfrBivqzuAq4CxAfBMBO uMcSrN8BQO0qJU/oIwY/22k/rjkqVJ+lS2BH/it30MIuEqZqGsu3lgLkASxvQn4Cftu0 o8KA== X-Gm-Message-State: AOAM530t+9n25YDUq4osRIfWHUZO7rfCk5IF5bhc3HkgV1gU81J1FpqC YffnlSC7oxwnyZV+i0alklLI4bYMiwA= X-Google-Smtp-Source: ABdhPJyg5ntvrISEYcnteRqTxjv255CuzZum3bUE4QDeZnLiV2j/By8mswZuu+NsXHA5lEzuMnsDlw== X-Received: by 2002:a5d:6d87:: with SMTP id l7mr1223817wrs.222.1624932298968; Mon, 28 Jun 2021 19:04:58 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id z3sm1097740wmi.29.2021.06.28.19.04.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 19:04:58 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 29 Jun 2021 02:04:44 +0000 Subject: [PATCH v7 07/16] unpack-trees: compare sparse directories correctly Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee As we further integrate the sparse-index into unpack-trees, we need to ensure that we compare sparse directory entries correctly with other entries. This affects searching for an exact path as well as sorting index entries. Sparse directory entries contain the trailing directory separator. This is important for the sorting, in particular. Thus, within do_compare_entry() we stop using S_IFREG in all cases, since sparse directories should use S_IFDIR to indicate that the comparison should treat the entry name as a dirctory. Within compare_entry(), it first calls do_compare_entry() to check the leading portion of the name. When the input path is a directory name, we could match exactly already. Thus, we should return 0 if we have an exact string match on a sparse directory entry. The final check is a length comparison between the strings. Signed-off-by: Derrick Stolee --- unpack-trees.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/unpack-trees.c b/unpack-trees.c index 87c1ed204c8..b113cc750f2 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -983,6 +983,7 @@ static int do_compare_entry(const struct cache_entry *ce, int pathlen, ce_len; const char *ce_name; int cmp; + unsigned ce_mode; /* * If we have not precomputed the traverse path, it is quicker @@ -1005,7 +1006,8 @@ static int do_compare_entry(const struct cache_entry *ce, ce_len -= pathlen; ce_name = ce->name + pathlen; - return df_name_compare(ce_name, ce_len, S_IFREG, name, namelen, mode); + ce_mode = S_ISSPARSEDIR(ce->ce_mode) ? S_IFDIR : S_IFREG; + return df_name_compare(ce_name, ce_len, ce_mode, name, namelen, mode); } static int compare_entry(const struct cache_entry *ce, const struct traverse_info *info, const struct name_entry *n) @@ -1014,6 +1016,16 @@ static int compare_entry(const struct cache_entry *ce, const struct traverse_inf if (cmp) return cmp; + /* + * At this point, we know that we have a prefix match. If ce + * is a sparse directory, then allow an exact match. This only + * works when the input name is a directory, since ce->name + * ends in a directory separator. + */ + if (S_ISSPARSEDIR(ce->ce_mode) && + ce->ce_namelen == traverse_path_len(info, tree_entry_len(n)) + 1) + return 0; + /* * Even if the beginning compared identically, the ce should * compare as bigger than a directory leading up to it! From patchwork Tue Jun 29 02:04:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12348833 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43D36C11F65 for ; Tue, 29 Jun 2021 02:05:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2F07361C99 for ; Tue, 29 Jun 2021 02:05:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231559AbhF2CHc (ORCPT ); Mon, 28 Jun 2021 22:07:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49914 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231225AbhF2CH1 (ORCPT ); Mon, 28 Jun 2021 22:07:27 -0400 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D3E46C061760 for ; Mon, 28 Jun 2021 19:05:00 -0700 (PDT) Received: by mail-wr1-x42b.google.com with SMTP id u8so10418384wrq.8 for ; Mon, 28 Jun 2021 19:05:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=O0bk0uMoOfzc+R3V+w3r7b7PaKiQoEFR2JUhLyX6ccg=; b=Yast+wFnNT8WJjGHCAMvvCvJNwi1E9F5rwYxG9fgDHKjG5Mat+OnQyCUN5MCymzUwo Y1T8T70kMqbUWBwcvRURWki7DhiUdaNYAt5/+iLdehKZKeVESZWl5SQJhezNCp3ANQad r+Y2VOK1urs9A9I2iKniCM8gBC8lFXT1vMv7DzKDSroDSMsTl6zahNrpt5LoWBBXFp14 cKFq+5o5KZ263IbSqH03p6EOom9KPMe6pG8ZIYbFAjSfLEDDihdmf7ALNWtXMlYo27ar nUm6wQNIRPvpuRsJqRvknIgRCNyI/a4BuXqt5xTmZg/MDeCBwn0GEDuiqH45S/jsm+PX LixA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=O0bk0uMoOfzc+R3V+w3r7b7PaKiQoEFR2JUhLyX6ccg=; b=APGimeOK0WBbq8qIodTsUdUYR93kfWrs7Kd4Mtn6FBPKifTuPNr1KsvaVRVFFHo8Ew C5xroDCPVOSIB9tDqZJ0mA5HxbSqP0CHWJaq0mQxTuH90jiR6ym0/bY6iV+i8s+TMTrr dRNCDapaOnYI+67my90tQqLu3Oir0+0jP6gMcuyXLT5QJUguDNkjAMazgku9YvWIfi+p pq1ZFAMhgtehLLEUCDTht3XRl1Mtj2FV4wEHcgHyK2uJeUioMaP7V3JMu9FvQMto2vu6 6I1GHOJGeqex7qw1LgA6JFr4V16htrLK8NfpPw2lQeIKkzsBs5GlVbsgDcSa3fEjAS20 QmAQ== X-Gm-Message-State: AOAM530uxXI+9DkoFdojdvsZdchodmfe2LLVVqi6oNG337dVq0nGrRvW cTNNLSSh8XOJ6x91Wl4XIZjMo5T4M08= X-Google-Smtp-Source: ABdhPJx5DogrQVbHoYXr1LRhzKPKQpylDjcRRsfUDCKFPN1T8fKM7keWMyqj3DhktEGfjc9ZVKz8pw== X-Received: by 2002:adf:f606:: with SMTP id t6mr10303292wrp.52.1624932299568; Mon, 28 Jun 2021 19:04:59 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id y17sm15308834wma.48.2021.06.28.19.04.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 19:04:59 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 29 Jun 2021 02:04:45 +0000 Subject: [PATCH v7 08/16] unpack-trees: rename unpack_nondirectories() Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee In the next change, we will use this method to unpack a sparse directory entry, so change the name to unpack_single_entry() so these entries apply. The new name reflects that we will not recurse into trees in order to resolve the conflicts. Signed-off-by: Derrick Stolee --- unpack-trees.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/unpack-trees.c b/unpack-trees.c index b113cc750f2..d26386ce8b2 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -804,7 +804,7 @@ static int traverse_by_cache_tree(int pos, int nr_entries, int nr_names, BUG("We need cache-tree to do this optimization"); /* - * Do what unpack_callback() and unpack_nondirectories() normally + * Do what unpack_callback() and unpack_single_entry() normally * do. But we walk all paths in an iterative loop instead. * * D/F conflicts and higher stage entries are not a concern @@ -1075,11 +1075,11 @@ static struct cache_entry *create_ce_entry(const struct traverse_info *info, * without actually calling it. If you change the logic here you may need to * check and change there as well. */ -static int unpack_nondirectories(int n, unsigned long mask, - unsigned long dirmask, - struct cache_entry **src, - const struct name_entry *names, - const struct traverse_info *info) +static int unpack_single_entry(int n, unsigned long mask, + unsigned long dirmask, + struct cache_entry **src, + const struct name_entry *names, + const struct traverse_info *info) { int i; struct unpack_trees_options *o = info->data; @@ -1322,7 +1322,7 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str } } - if (unpack_nondirectories(n, mask, dirmask, src, names, info) < 0) + if (unpack_single_entry(n, mask, dirmask, src, names, info) < 0) return -1; if (o->merge && src[0]) { From patchwork Tue Jun 29 02:04:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12348837 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57E3FC11F64 for ; Tue, 29 Jun 2021 02:05:08 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 44CBD61CEB for ; Tue, 29 Jun 2021 02:05:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231700AbhF2CHe (ORCPT ); Mon, 28 Jun 2021 22:07:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49940 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231304AbhF2CH2 (ORCPT ); Mon, 28 Jun 2021 22:07:28 -0400 Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8B415C061574 for ; Mon, 28 Jun 2021 19:05:01 -0700 (PDT) Received: by mail-wm1-x336.google.com with SMTP id l1so275235wme.4 for ; Mon, 28 Jun 2021 19:05:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=dBg3vHNNCzcOWR2PJkKRKBYZUfvDuCHqFYdWYLOIH5Q=; b=t984QHVA52kowv+M/rMibnQvaXwAGNC6W0zd11JifSpwp8yEuxn8V27T9GqNSkwlST tmiEhZ4OWoc32FQiEZH5NZkIwzuvAiawd1l+IzRtN0igha5E9ygw0UvE4dSy+fCuHLqM bjiSZOgN80IXdzrz7uiKR5HuEnl/BoyBqdSyMbM5rM4btvA/+5EHaJ2rfnYpXILq9j9Y 3YjGcryE98CsZZX6V+f/O09aeu5jtll6lhqS3t5Kf9VDOZaebTXikvvCaCubeT2SETYF UZ1eu4Lu3WcxnqS3H9Xyij1Ol2dHdRsrs39jnITOdVyhep75l9CKVyMlaLfd15t9W2Qb fDzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=dBg3vHNNCzcOWR2PJkKRKBYZUfvDuCHqFYdWYLOIH5Q=; b=Awf3czm3zt2qEVaopj1hHTFNHfzV21WDwLxFrlV9LoIRk8dXoy/obnRv8x9xTD38rV 4w+evYH3DyftK9bvbdj/nqREkbLEsP/f45Cg6Rn+DfkYMhXxoCBOZgxsMUvJkuIxkEkg jAIYphSzcgqd+/ynuGD+2vpQGl0tAVUJJX5HYZadAY8WmvPhW0TgVqO/K3w1ux3GYD0Q yqG9X0rcfL97M2ZSPaLZIHqFskcKFcg7tOUg+Jxa8G300r4s2JgYBQUVT6WoCe5D8TjA jf3XjtQjV7pE8V1MhCO7cPz0r0ELfMwym6jDkcWp0Ggjgs1RUqZ77JXgvdWWNPALDH7c y0wg== X-Gm-Message-State: AOAM533FBcQAtRP7zd/hx1awlSSlwkGvlWfk2AuZY+hCiMWp4q+sVyrq pt+wed0L2A6oWFYxD45ZA0fTJ81Hn5A= X-Google-Smtp-Source: ABdhPJwFTy+lXUASdiKxnO+sg7Xd8HeJSGXxPm1lp6X95Ix7i4iK1XIvwV/ImM1Y+3metfca24gDoQ== X-Received: by 2002:a7b:c44d:: with SMTP id l13mr29551434wmi.69.1624932300152; Mon, 28 Jun 2021 19:05:00 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id c12sm19335072wrr.90.2021.06.28.19.04.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 19:04:59 -0700 (PDT) Message-Id: <237ccf4e43dd461250b3e6d609a475c1d675ea86.1624932294.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 29 Jun 2021 02:04:46 +0000 Subject: [PATCH v7 09/16] unpack-trees: unpack sparse directory entries Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee During unpack_callback(), index entries are compared against tree entries. These are matched according to names and types. One goal is to decide if we should recurse into subtrees or simply operate on one index entry. In the case of a sparse-directory entry, we do not want to recurse into that subtree and instead simply compare the trees. In some cases, we might want to perform a merge operation on the entry, such as during 'git checkout ' which wants to replace a sparse tree entry with the tree for that path at the target commit. We extend the logic within unpack_single_entry() to create a sparse-directory entry in this case, and then that is sent to call_unpack_fn(). There are some subtleties in this process. For instance, we need to update find_cache_entry() to allow finding a sparse-directory entry that exactly matches a given path. Use the new helper method sparse_dir_matches_path() for this. We also need to ignore conflict markers in the case that the entries correspond to directories and we already have a sparse directory entry. Signed-off-by: Derrick Stolee --- unpack-trees.c | 105 +++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 97 insertions(+), 8 deletions(-) diff --git a/unpack-trees.c b/unpack-trees.c index d26386ce8b2..d141dffbd94 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -1052,13 +1052,15 @@ static struct cache_entry *create_ce_entry(const struct traverse_info *info, const struct name_entry *n, int stage, struct index_state *istate, - int is_transient) + int is_transient, + int is_sparse_directory) { size_t len = traverse_path_len(info, tree_entry_len(n)); + size_t alloc_len = is_sparse_directory ? len + 1 : len; struct cache_entry *ce = is_transient ? - make_empty_transient_cache_entry(len, NULL) : - make_empty_cache_entry(istate, len); + make_empty_transient_cache_entry(alloc_len, NULL) : + make_empty_cache_entry(istate, alloc_len); ce->ce_mode = create_ce_mode(n->mode); ce->ce_flags = create_ce_flags(stage); @@ -1067,6 +1069,13 @@ static struct cache_entry *create_ce_entry(const struct traverse_info *info, /* len+1 because the cache_entry allocates space for NUL */ make_traverse_path(ce->name, len + 1, info, n->path, n->pathlen); + if (is_sparse_directory) { + ce->name[len] = '/'; + ce->name[len + 1] = '\0'; + ce->ce_namelen++; + ce->ce_flags |= CE_SKIP_WORKTREE; + } + return ce; } @@ -1085,10 +1094,17 @@ static int unpack_single_entry(int n, unsigned long mask, struct unpack_trees_options *o = info->data; unsigned long conflicts = info->df_conflicts | dirmask; - /* Do we have *only* directories? Nothing to do */ if (mask == dirmask && !src[0]) return 0; + /* + * When we have a sparse directory entry for src[0], + * then this isn't necessarily a directory-file conflict. + */ + if (mask == dirmask && src[0] && + S_ISSPARSEDIR(src[0]->ce_mode)) + conflicts = 0; + /* * Ok, we've filled in up to any potential index entry in src[0], * now do the rest. @@ -1118,7 +1134,9 @@ static int unpack_single_entry(int n, unsigned long mask, * not stored in the index. otherwise construct the * cache entry from the index aware logic. */ - src[i + o->merge] = create_ce_entry(info, names + i, stage, &o->result, o->merge); + src[i + o->merge] = create_ce_entry(info, names + i, stage, + &o->result, o->merge, + bit & dirmask); } if (o->merge) { @@ -1222,16 +1240,69 @@ static int find_cache_pos(struct traverse_info *info, return -1; } +/* + * Given a sparse directory entry 'ce', compare ce->name to + * info->name + '/' + p->path + '/' if info->name is non-empty. + * Compare ce->name to p->path + '/' otherwise. Note that + * ce->name must end in a trailing '/' because it is a sparse + * directory entry. + */ +static int sparse_dir_matches_path(const struct cache_entry *ce, + struct traverse_info *info, + const struct name_entry *p) +{ + assert(S_ISSPARSEDIR(ce->ce_mode)); + assert(ce->name[ce->ce_namelen - 1] == '/'); + + if (info->namelen) + return ce->ce_namelen == info->namelen + p->pathlen + 2 && + ce->name[info->namelen] == '/' && + !strncmp(ce->name, info->name, info->namelen) && + !strncmp(ce->name + info->namelen + 1, p->path, p->pathlen); + return ce->ce_namelen == p->pathlen + 1 && + !strncmp(ce->name, p->path, p->pathlen); +} + static struct cache_entry *find_cache_entry(struct traverse_info *info, const struct name_entry *p) { + struct cache_entry *ce; int pos = find_cache_pos(info, p->path, p->pathlen); struct unpack_trees_options *o = info->data; if (0 <= pos) return o->src_index->cache[pos]; - else + + /* + * Check for a sparse-directory entry named "path/". + * Due to the input p->path not having a trailing + * slash, the negative 'pos' value overshoots the + * expected position by at least one, hence "-2" here. + */ + pos = -pos - 2; + + if (pos < 0 || pos >= o->src_index->cache_nr) return NULL; + + /* + * We might have multiple entries between 'pos' and + * the actual sparse-directory entry, so start walking + * back until finding it or passing where it would be. + */ + while (pos >= 0) { + ce = o->src_index->cache[pos]; + + if (strncmp(ce->name, p->path, p->pathlen)) + return NULL; + + if (S_ISSPARSEDIR(ce->ce_mode) && + sparse_dir_matches_path(ce, info, p)) + return ce; + + pos--; + } + + return NULL; } static void debug_path(struct traverse_info *info) @@ -1266,6 +1337,21 @@ static void debug_unpack_callback(int n, debug_name_entry(i, names + i); } +/* + * Returns true if and only if the given cache_entry is a + * sparse-directory entry that matches the given name_entry + * from the tree walk at the given traverse_info. + */ +static int is_sparse_directory_entry(struct cache_entry *ce, + struct name_entry *name, + struct traverse_info *info) +{ + if (!ce || !name || !S_ISSPARSEDIR(ce->ce_mode)) + return 0; + + return sparse_dir_matches_path(ce, info, name); +} + /* * Note that traverse_by_cache_tree() duplicates some logic in this function * without actually calling it. If you change the logic here you may need to @@ -1352,9 +1438,12 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str } } - if (traverse_trees_recursive(n, dirmask, mask & ~dirmask, - names, info) < 0) + if (!is_sparse_directory_entry(src[0], names, info) && + traverse_trees_recursive(n, dirmask, mask & ~dirmask, + names, info) < 0) { return -1; + } + return mask; } From patchwork Tue Jun 29 02:04:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12348841 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B02D0C11F66 for ; Tue, 29 Jun 2021 02:05:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9431561C99 for ; Tue, 29 Jun 2021 02:05:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231806AbhF2CHh (ORCPT ); Mon, 28 Jun 2021 22:07:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49948 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231391AbhF2CH3 (ORCPT ); Mon, 28 Jun 2021 22:07:29 -0400 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 23C17C061766 for ; Mon, 28 Jun 2021 19:05:02 -0700 (PDT) Received: by mail-wm1-x335.google.com with SMTP id j11-20020a05600c1c0bb02901e23d4c0977so739991wms.0 for ; Mon, 28 Jun 2021 19:05:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=y16povR6Bo5qH6HHskkHnqLCNjqmRRpyNxlrBNS/WXw=; b=hEqou1q55CYtfOp6uHsZASUFRnMG3511eRpDOEc5IjMOIvkaCyY+bLlH4jrmX4czyg E4ZamItGTNlF2Im0URtfL/hSg7IIQinpqcGwSpwAVznLmznOwMrKTcSC/102BGsbiKxa GtpSLAt1kuDjvlfwCPs2GKG2NkCrnMj7neQy2ELAzAsKqY+TbprN8CTOa1v9rZHhUsip ctb+gpm4itEH1LII+1441GBvsL/3DcqY1LeRjlmYLQpGdtwZhkRdaEYemsoF7Q3iYB2P iM7zAxXOIP2I9p+zg2SW3i1Ij4iHaiqhC5IZsd0kWw0hv88M/X2J0Flo97yeKxh8EAJp zIXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=y16povR6Bo5qH6HHskkHnqLCNjqmRRpyNxlrBNS/WXw=; b=Z3B81TQ2VFt/zdg5Xs4wwIJ7Vi9Gr+lYvx00sirxdcW0M6KWGEoKKl8Lcjor6oYNMN OTV5VTloU8KZM0YXjZ7WBbqeluAlnTj+zuWP9wejYZHUC8yr/ZLBFVcy+9sYepg94X1c pcp/OCdkZkUqVb2DPKqjMRvLALrhDI8CmLu/ajJJz5SqLkaX3wAeol6lm6R+pGSatEz3 vpnBxhzyX8LroAXdO/dyyWAxrKpB3i0S7N4Q8pYOj7ouSZvHB3VTkKIL3qGdfyWIgPRI mZ43oj9vKsOZYgivpxtec/j5FNUzoXOvBg27x0FOSH4bhncUxZSSeRw1fNnPi2D7jBXc RkMA== X-Gm-Message-State: AOAM532Sr6EyCql2XRWq1kxkzOJr1JyijCUsytrKJgaAsYxlzNOMNGGO qv4ztLgNzVn/Du0/piLY1g5j6xAE/mU= X-Google-Smtp-Source: ABdhPJyj7nqVqGo9U2nQQVm83U5LwxeJYx4gpY2qKf8Kqv20r+m01m5uzakzt+piHfbhihSQZ1vxdA== X-Received: by 2002:a05:600c:3b28:: with SMTP id m40mr29151338wms.99.1624932300788; Mon, 28 Jun 2021 19:05:00 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id c10sm8409496wmb.40.2021.06.28.19.05.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 19:05:00 -0700 (PDT) Message-Id: <9f31c691af6780f0ea48bdcb5ff6d56b628f1a81.1624932294.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 29 Jun 2021 02:04:47 +0000 Subject: [PATCH v7 10/16] unpack-trees: handle dir/file conflict of sparse entries Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 25 ++++++++++++++++++++++-- unpack-trees.c | 5 ++++- 2 files changed, 27 insertions(+), 3 deletions(-) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 3f61e5686b5..4e6446e7545 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -95,6 +95,19 @@ test_expect_success 'setup' ' git add . && git commit -m "rename deep/deeper1/... to folder1/..." && + git checkout -b df-conflict base && + rm -rf folder1 && + echo content >folder1 && + git add . && + git commit -m df && + + git checkout -b fd-conflict base && + rm a && + mkdir a && + echo content >a/a && + git add . && + git commit -m fd && + git checkout -b deepest base && echo "updated deepest" >deep/deeper1/deepest/a && git commit -a -m "update deepest" && @@ -325,7 +338,11 @@ test_expect_success 'diff --staged' ' test_expect_success 'diff with renames and conflicts' ' init_repos && - for branch in rename-out-to-out rename-out-to-in rename-in-to-out + for branch in rename-out-to-out \ + rename-out-to-in \ + rename-in-to-out \ + df-conflict \ + fd-conflict do test_all_match git checkout rename-base && test_all_match git checkout $branch -- .&& @@ -338,7 +355,11 @@ test_expect_success 'diff with renames and conflicts' ' test_expect_success 'diff with directory/file conflicts' ' init_repos && - for branch in rename-out-to-out rename-out-to-in rename-in-to-out + for branch in rename-out-to-out \ + rename-out-to-in \ + rename-in-to-out \ + df-conflict \ + fd-conflict do git -C full-checkout reset --hard && test_sparse_match git reset --hard && diff --git a/unpack-trees.c b/unpack-trees.c index d141dffbd94..e63b2dcacbc 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -2617,7 +2617,10 @@ int twoway_merge(const struct cache_entry * const *src, same(current, oldtree) && !same(current, newtree)) { /* 20 or 21 */ return merged_entry(newtree, current, o); - } else + } else if (current && !oldtree && newtree && + S_ISSPARSEDIR(current->ce_mode) != S_ISSPARSEDIR(newtree->ce_mode)) + return merged_entry(newtree, current, o); + else return reject_merge(current, o); } else if (newtree) { From patchwork Tue Jun 29 02:04:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12348839 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85FD9C11F64 for ; Tue, 29 Jun 2021 02:05:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6CF8261CE5 for ; Tue, 29 Jun 2021 02:05:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231770AbhF2CHg (ORCPT ); Mon, 28 Jun 2021 22:07:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49950 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231350AbhF2CH3 (ORCPT ); Mon, 28 Jun 2021 22:07:29 -0400 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C19B0C061767 for ; Mon, 28 Jun 2021 19:05:02 -0700 (PDT) Received: by mail-wm1-x32b.google.com with SMTP id u8-20020a7bcb080000b02901e44e9caa2aso1272057wmj.4 for ; Mon, 28 Jun 2021 19:05:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=KMWoXCHAUCRcXSFT+haQKKmCiVf5GcUjw8ICiM/EavY=; b=W7SlDsAaZN7oQdvmor3CC//lBxVCskBbllRLRgcTkILy7n89LmfOZaOspno5LyILG/ gEcfk6EmyUO8QMfuZJkHPQ3OKDTbmfgneUVInryEcSnmFoawHVSUMq0K9+8pQmJhx43X bVoG4hlMognojAe3SJIwPkSv2knLUwV3elS+JWWKFCNDA5c2oIdMrRb8boQfj45o4yY8 UMzNsZ9aOTbubWv/6/jVAT1DfSRu9tHJCtseIv3e1vdkjjo9IXsnQxNXvufdzY1K1kWF H1KXu/l9Yz+9b/i3p3TJhlhNEdHgafNfzX6ZruEKYbmqz09xXztKQ9LyLrfveUigIhUU 9qjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=KMWoXCHAUCRcXSFT+haQKKmCiVf5GcUjw8ICiM/EavY=; b=fJNGFOZoMPXoDn5NJHVuJnBiGWnEJpAiFWBN4B24r5Byxc5cifn84cqwyVXvRZTl6t DcQwTaDM3TswfkxZbn7YsPx+/lThSdUPlVgRaWOGjrs21G75rJj1w/0Z6uD04/9Hjv0S mHI7oY9l0kPKdy252mP26XOCDT23L8nL64yXlkABENXXFEL6CcZzkBv4/eeN1kkt7yFX VWoM+/HefCcHZ1GVdxUndjOmX+QgZrW/BG0fW+6uaAfnXz+l6yBHsgz/4klhrjVZFWNV Pi9ni5TVekIBrs8x3upc6v2AoKYlK5pE4k28mKav9S+m7GX2XUO2z5MuVDluHJ+Ywvhd AeiQ== X-Gm-Message-State: AOAM533yDmS/sOYm+nyI5aD2MOweaLfimedRXqeMwgj0FHIDsLhX/YNj P/p2SI8QbHgjGH0NIzTzhQS+8OvCAHc= X-Google-Smtp-Source: ABdhPJzSAaNxvP4xNqS+hBieChMydH0puV7pjdL9HPRQie0UhFYg/d8Me8qp+Jv42KVXQ4wh3b1UcQ== X-Received: by 2002:a7b:cf3a:: with SMTP id m26mr5778541wmg.117.1624932301476; Mon, 28 Jun 2021 19:05:01 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id h21sm7552133wmq.38.2021.06.28.19.05.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 19:05:01 -0700 (PDT) Message-Id: <2a43287c47ed361fcae962d508a8a9354c654fcb.1624932294.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 29 Jun 2021 02:04:48 +0000 Subject: [PATCH v7 11/16] dir.c: accept a directory as part of cone-mode patterns Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee When we have sparse directory entries in the index, we want to compare that directory against sparse-checkout patterns. Those pattern matching algorithms are built expecting a file path, not a directory path. This is especially important in the "cone mode" patterns which will match files that exist within the "parent directories" as well as the recursive directory matches. If path_matches_pattern_list() is given a directory, we can add a fake filename ("-") to the directory and get the same results as before, assuming we are in cone mode. Since sparse index requires cone mode patterns, this is an acceptable assumption. Signed-off-by: Derrick Stolee --- dir.c | 24 +++++++++++++++++++----- 1 file changed, 19 insertions(+), 5 deletions(-) diff --git a/dir.c b/dir.c index ebe5ec046e0..0c5264b3b20 100644 --- a/dir.c +++ b/dir.c @@ -1376,7 +1376,7 @@ enum pattern_match_result path_matches_pattern_list( struct path_pattern *pattern; struct strbuf parent_pathname = STRBUF_INIT; int result = NOT_MATCHED; - const char *slash_pos; + size_t slash_pos; if (!pl->use_cone_patterns) { pattern = last_matching_pattern_from_list(pathname, pathlen, basename, @@ -1397,21 +1397,35 @@ enum pattern_match_result path_matches_pattern_list( strbuf_addch(&parent_pathname, '/'); strbuf_add(&parent_pathname, pathname, pathlen); + /* + * Directory entries are matched if and only if a file + * contained immediately within them is matched. For the + * case of a directory entry, modify the path to create + * a fake filename within this directory, allowing us to + * use the file-base matching logic in an equivalent way. + */ + if (parent_pathname.len > 0 && + parent_pathname.buf[parent_pathname.len - 1] == '/') { + slash_pos = parent_pathname.len - 1; + strbuf_add(&parent_pathname, "-", 1); + } else { + const char *slash_ptr = strrchr(parent_pathname.buf, '/'); + slash_pos = slash_ptr ? slash_ptr - parent_pathname.buf : 0; + } + if (hashmap_contains_path(&pl->recursive_hashmap, &parent_pathname)) { result = MATCHED_RECURSIVE; goto done; } - slash_pos = strrchr(parent_pathname.buf, '/'); - - if (slash_pos == parent_pathname.buf) { + if (!slash_pos) { /* include every file in root */ result = MATCHED; goto done; } - strbuf_setlen(&parent_pathname, slash_pos - parent_pathname.buf); + strbuf_setlen(&parent_pathname, slash_pos); if (hashmap_contains_path(&pl->parent_hashmap, &parent_pathname)) { result = MATCHED; From patchwork Tue Jun 29 02:04:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12348843 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB3F9C11F65 for ; Tue, 29 Jun 2021 02:05:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9D2D761C99 for ; Tue, 29 Jun 2021 02:05:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231810AbhF2CHj (ORCPT ); Mon, 28 Jun 2021 22:07:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49954 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231392AbhF2CHa (ORCPT ); Mon, 28 Jun 2021 22:07:30 -0400 Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 61908C061768 for ; Mon, 28 Jun 2021 19:05:03 -0700 (PDT) Received: by mail-wm1-x32c.google.com with SMTP id m41-20020a05600c3b29b02901dcd3733f24so719918wms.1 for ; Mon, 28 Jun 2021 19:05:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=749a8m/Pb2tHSZcl9clKK1mNX+ACw8NI8sZI+4jwZZo=; b=SDzTupqWBpJiCXl62C6IHnOaoiBHK2QN77aqVcNolJHj/G0Roxb9ELpijVB5/0xZ+3 6rBZwdxIc9qIzGmMBLqxHid0Xf0agL4rnqFT9/Ep2NkY2HCC9h2w60Tti/44g4ZM7Sa8 09BmWbBefucdZgovT7VxJ/bs8xLIDBkip/GierkyCsfBtE5ryDHCXZdD31wGQ7NGMc+w JJaq38WeuwC8LcsT2mptWv8yoJ7JAE0rfUEyoqR86QdZ+NkXADUEV8rEWINSJ9LPNLLi nqeGu3rFM6jEyxF+S5vB4IlUq16cwCZQwatdMyTymYv8+Vmddn51M0K/JXUcFVH1Ihxd AS3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=749a8m/Pb2tHSZcl9clKK1mNX+ACw8NI8sZI+4jwZZo=; b=TGsIClfUx4H5A/sn5yg6AFc4n/xMR03Rc1V2WMqMV5/gBzKwOM2LZZfqu6+S3X3vhZ PRpDryRTsLYvEePrgpURbAgX8q5pesjmQURzTctM1nZWZA0+NzbK+jU768klJXzgR4EJ 4E1Qus+0cQGqk6B7nSrZQZDLy0bwVYXQtQ3VfTFxUyUNSz6otJdYxLCq8sJzIdBYKpdN WrbYgldpq5u5DZ48lKUgDDmUjzO6i941h/GE5upHA8lv3sEz8rStjdyvSmnvopaWp+ma KQA7eXTgb6ZFEifZQcy9TeY6c53yXGTbQbAJFzL9QxdphyOVVlNWZcO50ThehuJ9UZVz G3hg== X-Gm-Message-State: AOAM530I1TmaeFz2gzj8NXdTpmdrduErgnhVG3jXwanvOq7S/TK71UPd xV/jgWwNErSAWRQESHiWxmB4TYkRI9k= X-Google-Smtp-Source: ABdhPJz9rWQzatpTYOAcwrGUAIwfQygXdh2hwIvpNU0v+UPipPhadlEWLC+vlMc0aoHMBy8cpK6D2Q== X-Received: by 2002:a7b:c110:: with SMTP id w16mr13836678wmi.178.1624932302015; Mon, 28 Jun 2021 19:05:02 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l7sm9089771wrs.30.2021.06.28.19.05.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 19:05:01 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 29 Jun 2021 02:04:49 +0000 Subject: [PATCH v7 12/16] diff-lib: handle index diffs with sparse dirs Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee While comparing an index to a tree, we may see a sparse directory entry. In this case, we should compare that portion of the tree to the tree represented by that entry. This could include a new tree which needs to be expanded to a full list of added files. It could also include an existing tree, in which case all of the changes inside are important to describe, including the modifications, additions, and deletions. Note that the case where the tree has a path and the index does not remains identical to before: the lack of a cache entry is the same with a sparse index. Use diff_tree_oid() appropriately to compute the diff. Signed-off-by: Derrick Stolee --- diff-lib.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/diff-lib.c b/diff-lib.c index c2ac9250fe9..3f32f038371 100644 --- a/diff-lib.c +++ b/diff-lib.c @@ -325,6 +325,11 @@ static void show_new_file(struct rev_info *revs, unsigned dirty_submodule = 0; struct index_state *istate = revs->diffopt.repo->index; + if (new_file && S_ISSPARSEDIR(new_file->ce_mode)) { + diff_tree_oid(NULL, &new_file->oid, new_file->name, &revs->diffopt); + return; + } + /* * New file in the index: it might actually be different in * the working tree. @@ -347,6 +352,17 @@ static int show_modified(struct rev_info *revs, unsigned dirty_submodule = 0; struct index_state *istate = revs->diffopt.repo->index; + /* + * If both are sparse directory entries, then expand the + * modifications to the file level. + */ + if (old_entry && new_entry && + S_ISSPARSEDIR(old_entry->ce_mode) && + S_ISSPARSEDIR(new_entry->ce_mode)) { + diff_tree_oid(&old_entry->oid, &new_entry->oid, new_entry->name, &revs->diffopt); + return 0; + } + if (get_stat_data(istate, new_entry, &oid, &mode, cached, match_missing, &dirty_submodule, &revs->diffopt) < 0) { if (report_missing) From patchwork Tue Jun 29 02:04:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12348845 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 48F19C11F64 for ; Tue, 29 Jun 2021 02:05:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2B51861CAF for ; Tue, 29 Jun 2021 02:05:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231724AbhF2CHj (ORCPT ); Mon, 28 Jun 2021 22:07:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49956 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231404AbhF2CHb (ORCPT ); Mon, 28 Jun 2021 22:07:31 -0400 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DFA7AC061760 for ; Mon, 28 Jun 2021 19:05:03 -0700 (PDT) Received: by mail-wm1-x32d.google.com with SMTP id v20-20020a05600c2154b02901dcefb16af0so1269005wml.5 for ; Mon, 28 Jun 2021 19:05:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=y0Ogi+qqLnMLGGwroR980d2NMgu2JWWRhDy+7DcqaV0=; b=qV/G0md59ehb8vNR+lwhOeQi0D/QRtvaG2d49EXJOdY6SFPscWXRobrURpkonxq/xd 4U3UE0JpkrEeriW5VkmfOZFTlr4v+wQ1khHNjW/geX/cLL4WvCvoaDB1x20xu0UTg5A9 3zCjredr6MKHkEiplWcoXigNpCPeaaj+yXOQSLqzT7mpRgZgdVgnXs4IIbCfV/f3IPty eAu6CIV/ABAqr8Euz+EpbXYZivQhruKXtb6Un/aWA/jLJwIbsmbI6kh/PnftOkAmPlod 5jgjDBHIblf3k+PF8Ezi0fMZl2I1nUB9fXOoTo1Iye5+bJrRiM4wlr3z45QqqdGGpZ+4 kGRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=y0Ogi+qqLnMLGGwroR980d2NMgu2JWWRhDy+7DcqaV0=; b=b89W386Pu/9vonInuGHloYsddsgY+09TR+PYWVq257Pm/fx5a910EAKLeYkXm1a4VM PTH7iFEOhbBkR/bL9ep2hxIy3B79IKtwzy8pNuqnma0dhlT9EB/H2ifehXxN8K3y1Y3h XOAmzSCj0REJPh81cxugBY4qTnC6JCZsZZqbFGxOhm3x5fLT5Z5kQxd/xmrhr4rDsPHV nMS+jQgcEcHtMpUgP2Tjzu0aHb+O6yLYxnwxGkuX6XlyyxROyZQaknpwPLzvAICMwo94 qW+tO9LTo3j21Wj5TIx7QcI4gNH7/ucM8WG2oc0S1K2gcBGS9rliFc+B5gMTWgCfANhU RPtw== X-Gm-Message-State: AOAM533i0UZjo1I+tlxKpTfYvZhQ3QjIWeKyDwEWkSPvxlbb1M9g2Ne2 k6P0/mEQh4YLqALsDxnJ3HIErmCT+fQ= X-Google-Smtp-Source: ABdhPJyl5ItVkBN4ujRtX5UUEeH5MDqSRtOuhzijyqQ1v2p0oUtJV5pUopyeGUnZFGQxc2ctHISBJA== X-Received: by 2002:a05:600c:4f96:: with SMTP id n22mr29678556wmq.116.1624932302560; Mon, 28 Jun 2021 19:05:02 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id g17sm11766766wrw.31.2021.06.28.19.05.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 19:05:02 -0700 (PDT) Message-Id: <35063ffb8ed6c60c5777d27e4e36a802234200d2.1624932294.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 29 Jun 2021 02:04:50 +0000 Subject: [PATCH v7 13/16] status: skip sparse-checkout percentage with sparse-index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee 'git status' began reporting a percentage of populated paths when sparse-checkout is enabled in 051df3cf (wt-status: show sparse checkout status as well, 2020-07-18). This percentage is incorrect when the index has sparse directories. It would also be expensive to calculate as we would need to parse trees to count the total number of possible paths. Avoid the expensive computation by simplifying the output to only report that a sparse checkout exists, without the percentage. This change is the reason we use 'git status --porcelain=v2' in t1092-sparse-checkout-compatibility.sh. We don't want to ensure that this message is equal across both modes, but instead just the important information about staged, modified, and untracked files are compared. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 8 ++++++++ wt-status.c | 14 +++++++++++--- wt-status.h | 1 + 3 files changed, 20 insertions(+), 3 deletions(-) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 4e6446e7545..d372932cd12 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -231,6 +231,14 @@ test_expect_success 'status with options' ' test_all_match git status --porcelain=v2 -uno ' +test_expect_success 'status reports sparse-checkout' ' + init_repos && + git -C sparse-checkout status >full && + git -C sparse-index status >sparse && + test_i18ngrep "You are in a sparse checkout with " full && + test_i18ngrep "You are in a sparse checkout." sparse +' + test_expect_success 'add, commit, checkout' ' init_repos && diff --git a/wt-status.c b/wt-status.c index 42b67357169..96db3e74962 100644 --- a/wt-status.c +++ b/wt-status.c @@ -1493,9 +1493,12 @@ static void show_sparse_checkout_in_use(struct wt_status *s, if (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_DISABLED) return; - status_printf_ln(s, color, - _("You are in a sparse checkout with %d%% of tracked files present."), - s->state.sparse_checkout_percentage); + if (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_SPARSE_INDEX) + status_printf_ln(s, color, _("You are in a sparse checkout.")); + else + status_printf_ln(s, color, + _("You are in a sparse checkout with %d%% of tracked files present."), + s->state.sparse_checkout_percentage); wt_longstatus_print_trailer(s); } @@ -1653,6 +1656,11 @@ static void wt_status_check_sparse_checkout(struct repository *r, return; } + if (r->index->sparse_index) { + state->sparse_checkout_percentage = SPARSE_CHECKOUT_SPARSE_INDEX; + return; + } + for (i = 0; i < r->index->cache_nr; i++) { struct cache_entry *ce = r->index->cache[i]; if (ce_skip_worktree(ce)) diff --git a/wt-status.h b/wt-status.h index 0d32799b28e..ab9cc9d8f03 100644 --- a/wt-status.h +++ b/wt-status.h @@ -78,6 +78,7 @@ enum wt_status_format { }; #define SPARSE_CHECKOUT_DISABLED -1 +#define SPARSE_CHECKOUT_SPARSE_INDEX -2 struct wt_status_state { int merge_in_progress; From patchwork Tue Jun 29 02:04:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12348851 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7E55EC11F65 for ; Tue, 29 Jun 2021 02:05:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 66AD861C99 for ; Tue, 29 Jun 2021 02:05:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231194AbhF2CHu (ORCPT ); Mon, 28 Jun 2021 22:07:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49986 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231775AbhF2CHh (ORCPT ); Mon, 28 Jun 2021 22:07:37 -0400 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8B7B1C0617A8 for ; Mon, 28 Jun 2021 19:05:04 -0700 (PDT) Received: by mail-wm1-x32e.google.com with SMTP id u8-20020a7bcb080000b02901e44e9caa2aso1272092wmj.4 for ; Mon, 28 Jun 2021 19:05:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=VMrrM2aGM35t7+rKMf9/CmBSSph8xW2UJsNSqfFOQg4=; b=BnPqkR3VEqAtmlVMI2Sn5ID0Zh9A3cS6exoMrMizjR3mqhvHFOyATDXP2glyyhxZF+ ElTVE9/e4diA/digjm2qP2E99Y/cyKQ9EbFWujSZdJVmiBdqjnrFaRh8VpbsKhzoCdqR 6U1iqMcJUc1NWdKjBs8wqDRwhE22yS37efST5It2kHG9U3GoBSuLp6u/iR/Jt8xsxJez tM1nTzxd22axKRNT1VZHfv9ej89Yz5z3IdXp2iSBTxA6x+H9CKw2+sK1L82iF0jXEgYI 9WBx+r6yprMpxJFYUXLMdvU/vPVo8I76j6utI3fUgZi4qXbXbRrVDzSjyOIvea+8NeqY /7qg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=VMrrM2aGM35t7+rKMf9/CmBSSph8xW2UJsNSqfFOQg4=; b=HRqfVnlqtoQMxo7INnSz0uAs0+pE/dfobiT4GrdRx7Crlb08IEF5TWTUaQTme31hws yIdwLnwwCqAkvkBfmOIK9FshbEEl2lxgOPGxWFWpS/BDvlG8WE0UMsnwPZrArDbFx1yF o+igD9pDsbTtDZRRgfgXcuLReIyyCwIy8ZrSjb6g9EANBwOgoHN3BruXgqULC+8nXRd+ pnyPAYmv38g0LCEwNdmWFZFVQ53dLiVLhEMW62yrumo3pEejWulX58sjhcfeXP8O8UpI T8dF28tgde3+549+YGlhnus62XILShpGmMIcXKeSF2ObS2fe849bvS/yK7bYHEdhup01 Vnlw== X-Gm-Message-State: AOAM533/G7aUxBtHWrQ9P9ucB9xtNcjne0yHwVP0YI96zvDblqss6oAO eMRTahIOq8kL6Pp6QJ25QU2gBaQ03Xg= X-Google-Smtp-Source: ABdhPJxvTaCVEVQ0lDP6hkmAPrXh2cLfcsnE13EvlO9z+Ayzvi5IRvLtfISAicDH7PhNcOY4JJ8Jjg== X-Received: by 2002:a7b:c7c8:: with SMTP id z8mr16506634wmk.110.1624932303158; Mon, 28 Jun 2021 19:05:03 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id c16sm7906242wmr.2.2021.06.28.19.05.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 19:05:02 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 29 Jun 2021 02:04:51 +0000 Subject: [PATCH v7 14/16] status: use sparse-index throughout Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee By testing 'git -c core.fsmonitor= status -uno', we can check for the simplest index operations that can be made sparse-aware. The necessary implementation details are already integrated with sparse-checkout, so modify command_requires_full_index to be zero for cmd_status(). In refresh_index(), we loop through the index entries to refresh their stat() information. However, sparse directories have no stat() information to populate. Ignore these entries. This allows 'git status' to no longer expand a sparse index to a full one. This is further tested by dropping the "-uno" option and adding an untracked file into the worktree. The performance test p2000-sparse-checkout-operations.sh demonstrates these improvements: Test HEAD~1 HEAD ----------------------------------------------------------------------------- 2000.2: git status (full-index-v3) 0.31(0.30+0.05) 0.31(0.29+0.06) +0.0% 2000.3: git status (full-index-v4) 0.31(0.29+0.07) 0.34(0.30+0.08) +9.7% 2000.4: git status (sparse-index-v3) 2.35(2.28+0.10) 0.04(0.04+0.05) -98.3% 2000.5: git status (sparse-index-v4) 2.35(2.24+0.15) 0.05(0.04+0.06) -97.9% Note that since HEAD~1 was expanding the sparse index by parsing trees, it was artificially slower than the full index case. Thus, the 98% improvement is misleading, and instead we should celebrate the 0.34s to 0.05s improvement of 85%. This is more indicative of the peformance gains we are expecting by using a sparse index. Note: we are dropping the assignment of core.fsmonitor here. This is not necessary for the test script as we are not altering the config any other way. Correct integration with FS Monitor will be validated in later changes. Signed-off-by: Derrick Stolee --- builtin/commit.c | 3 +++ read-cache.c | 10 ++++++++-- t/t1092-sparse-checkout-compatibility.sh | 13 +++++++++---- 3 files changed, 20 insertions(+), 6 deletions(-) diff --git a/builtin/commit.c b/builtin/commit.c index 190d215d43b..12f51db158a 100644 --- a/builtin/commit.c +++ b/builtin/commit.c @@ -1510,6 +1510,9 @@ int cmd_status(int argc, const char **argv, const char *prefix) if (argc == 2 && !strcmp(argv[1], "-h")) usage_with_options(builtin_status_usage, builtin_status_options); + prepare_repo_settings(the_repository); + the_repository->settings.command_requires_full_index = 0; + status_init_config(&s, git_status_config); argc = parse_options(argc, argv, prefix, builtin_status_options, diff --git a/read-cache.c b/read-cache.c index 1b3c2eb408b..277c2970a03 100644 --- a/read-cache.c +++ b/read-cache.c @@ -1584,8 +1584,7 @@ int refresh_index(struct index_state *istate, unsigned int flags, */ preload_index(istate, pathspec, 0); trace2_region_enter("index", "refresh", NULL); - /* TODO: audit for interaction with sparse-index. */ - ensure_full_index(istate); + for (i = 0; i < istate->cache_nr; i++) { struct cache_entry *ce, *new_entry; int cache_errno = 0; @@ -1600,6 +1599,13 @@ int refresh_index(struct index_state *istate, unsigned int flags, if (ignore_skip_worktree && ce_skip_worktree(ce)) continue; + /* + * If this entry is a sparse directory, then there isn't + * any stat() information to update. Ignore the entry. + */ + if (S_ISSPARSEDIR(ce->ce_mode)) + continue; + if (pathspec && !ce_path_match(istate, ce, pathspec, seen)) filtered = 1; diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index d372932cd12..fed0440bafe 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -532,12 +532,17 @@ test_expect_success 'sparse-index is expanded and converted back' ' GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ git -C sparse-index -c core.fsmonitor="" reset --hard && test_region index convert_to_sparse trace2.txt && - test_region index ensure_full_index trace2.txt && + test_region index ensure_full_index trace2.txt +' - rm trace2.txt && +test_expect_success 'sparse-index is not expanded' ' + init_repos && + + rm -f trace2.txt && + echo >>sparse-index/untracked.txt && GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ - git -C sparse-index -c core.fsmonitor="" status -uno && - test_region index ensure_full_index trace2.txt + git -C sparse-index status && + test_region ! index ensure_full_index trace2.txt ' test_done From patchwork Tue Jun 29 02:04:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12348847 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89B59C11F64 for ; Tue, 29 Jun 2021 02:05:20 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7101961C99 for ; Tue, 29 Jun 2021 02:05:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231761AbhF2CHo (ORCPT ); Mon, 28 Jun 2021 22:07:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49990 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231736AbhF2CHg (ORCPT ); Mon, 28 Jun 2021 22:07:36 -0400 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 12632C0617AE for ; Mon, 28 Jun 2021 19:05:05 -0700 (PDT) Received: by mail-wm1-x32b.google.com with SMTP id j11-20020a05600c1c0bb02901e23d4c0977so740048wms.0 for ; Mon, 28 Jun 2021 19:05:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=p0kTX8dqYf53FOMEeG+jhNu5Ie0uUzMG/x3SJgm7n18=; b=Jd7HzyCeGJoKMJ6A67Tdk+y8avhDhTtVYzpQmreH8+MbuMuKul/FbaZ7sO25SKJaA5 ixjCuWslK3ycwGv57BJ73fSid0gwjlItWvvl144ZGxMDZPq6AWqNXqUGG6IB6BkWyCNa SVp1ErJTcRxMKxNW5bQarOd8kmsf5EoDMKFVMlBVqFA684OX7btMPDg9ulKrPAMAvABG 04l6eMOMB9MBtUAstArg9mqHlUWqc5/clnBS9W9BFiI39ZQdTXZFEHpstNjUkapxGCtE Q5+8yZS/1uzfAoZbBSCncXnWQ3NVTeGrGIy2UO4/Azd2YUtCPR6AjNr6/HDLrTCJEEjB I8gw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=p0kTX8dqYf53FOMEeG+jhNu5Ie0uUzMG/x3SJgm7n18=; b=SF1Tj+yz0x3bgaAyo2aQ9YASv9MhSHJASsHZgYri8RN5WpiAn7UDv/fDwnjuATarCj c5rm1xso6HSfCMZSLN9qIoPKMSVWEf6+CzCTZCxi/X10NCZra2aLDslGbnc9/eLzEvDO rGmMmqs2QiithhynRsy0wEQ4K4pcL06naCaTMaQkyjMxHepYRVp+rjtAKLveCGCgqXHM 0j0zQFFsYfi3alOgd65pPJsK1R+W5iT0YDWGZht3f0KU2xF7x/m/xNDMJ3O/iTODtN+8 DKO52fqXFt2Xsf1AOkGgi1GhLOgG0aF23eMoOVTu6uQD2zkknbc2DakPAoyC8DPLgRxZ gDrg== X-Gm-Message-State: AOAM530HRdHCt78i5hChyWHXwcWNF2IAOgNLgM9U0n4ppS9f2LCeRQfT bNXzuXLrAnG/1xT/Nf0mRjQGVH7WB04= X-Google-Smtp-Source: ABdhPJwts1qJ/kOHj3maktNyqn0xkZUK+FxRNZhaZIJDYYqSYrgVH3iMjJ6y6VMDIpcO4/QMLTbYJg== X-Received: by 2002:a7b:c092:: with SMTP id r18mr6849300wmh.181.1624932303710; Mon, 28 Jun 2021 19:05:03 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id c74sm1140359wme.11.2021.06.28.19.05.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 19:05:03 -0700 (PDT) Message-Id: <717a3f49f97753149d5c435a83b3f1773dd4b1bb.1624932294.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 29 Jun 2021 02:04:52 +0000 Subject: [PATCH v7 15/16] wt-status: expand added sparse directory entries Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee It is difficult, but possible, to get into a state where we intend to add a directory that is outside of the sparse-checkout definition. Add a test to t1092-sparse-checkout-compatibility.sh that demonstrates this using a combination of 'git reset --mixed' and 'git checkout --orphan'. This test failed before because the output of 'git status --porcelain=v2' would not match on the lines for folder1/: * The sparse-checkout repo (with a full index) would output each path name that is intended to be added. * The sparse-index repo would only output that "folder1/" is staged for addition. The status should report the full list of files to be added, and so this sparse-directory entry should be expanded to a full list when reaching it inside the wt_status_collect_changes_initial() method. Use read_tree_at() to assist. Somehow, this loop over the cache entries was not guarded by ensure_full_index() as intended. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 33 +++++++++++++++ wt-status.c | 51 ++++++++++++++++++++++++ 2 files changed, 84 insertions(+) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index fed0440bafe..df217a2d10b 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -545,4 +545,37 @@ test_expect_success 'sparse-index is not expanded' ' test_region ! index ensure_full_index trace2.txt ' +test_expect_success 'reset mixed and checkout orphan' ' + init_repos && + + test_all_match git checkout rename-out-to-in && + + # Sparse checkouts do not agree with full checkouts about + # how to report a directory/file conflict during a reset. + # This command would fail with test_all_match because the + # full checkout reports "T folder1/0/1" while a sparse + # checkout reports "D folder1/0/1". This matches because + # the sparse checkouts skip "adding" the other side of + # the conflict. + test_sparse_match git reset --mixed HEAD~1 && + test_sparse_match test-tool read-cache --table --expand && + test_sparse_match git status --porcelain=v2 && + + # At this point, sparse-checkouts behave differently + # from the full-checkout. + test_sparse_match git checkout --orphan new-branch && + test_sparse_match test-tool read-cache --table --expand && + test_sparse_match git status --porcelain=v2 +' + +test_expect_success 'add everything with deep new file' ' + init_repos && + + run_on_sparse git sparse-checkout set deep/deeper1/deepest && + + run_on_all touch deep/deeper1/x && + test_all_match git add . && + test_all_match git status --porcelain=v2 +' + test_done diff --git a/wt-status.c b/wt-status.c index 96db3e74962..0317baef87e 100644 --- a/wt-status.c +++ b/wt-status.c @@ -657,6 +657,36 @@ static void wt_status_collect_changes_index(struct wt_status *s) clear_pathspec(&rev.prune_data); } +static int add_file_to_list(const struct object_id *oid, + struct strbuf *base, const char *path, + unsigned int mode, void *context) +{ + struct string_list_item *it; + struct wt_status_change_data *d; + struct wt_status *s = context; + struct strbuf full_name = STRBUF_INIT; + + if (S_ISDIR(mode)) + return READ_TREE_RECURSIVE; + + strbuf_add(&full_name, base->buf, base->len); + strbuf_addstr(&full_name, path); + it = string_list_insert(&s->change, full_name.buf); + d = it->util; + if (!d) { + CALLOC_ARRAY(d, 1); + it->util = d; + } + + d->index_status = DIFF_STATUS_ADDED; + /* Leave {mode,oid}_head zero for adds. */ + d->mode_index = mode; + oidcpy(&d->oid_index, oid); + s->committable = 1; + strbuf_release(&full_name); + return 0; +} + static void wt_status_collect_changes_initial(struct wt_status *s) { struct index_state *istate = s->repo->index; @@ -671,6 +701,27 @@ static void wt_status_collect_changes_initial(struct wt_status *s) continue; if (ce_intent_to_add(ce)) continue; + if (S_ISSPARSEDIR(ce->ce_mode)) { + /* + * This is a sparse directory entry, so we want to collect all + * of the added files within the tree. This requires recursively + * expanding the trees to find the elements that are new in this + * tree and marking them with DIFF_STATUS_ADDED. + */ + struct strbuf base = STRBUF_INIT; + struct pathspec ps = { 0 }; + struct tree *tree = lookup_tree(istate->repo, &ce->oid); + + ps.recursive = 1; + ps.has_wildcard = 1; + ps.max_depth = -1; + + strbuf_add(&base, ce->name, ce->ce_namelen); + read_tree_at(istate->repo, tree, &base, &ps, + add_file_to_list, s); + continue; + } + it = string_list_insert(&s->change, ce->name); d = it->util; if (!d) { From patchwork Tue Jun 29 02:04:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12348849 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37C57C11F65 for ; Tue, 29 Jun 2021 02:05:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1FEFF61C99 for ; Tue, 29 Jun 2021 02:05:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231879AbhF2CHq (ORCPT ); Mon, 28 Jun 2021 22:07:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49954 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231741AbhF2CHg (ORCPT ); Mon, 28 Jun 2021 22:07:36 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 854C1C0613A2 for ; Mon, 28 Jun 2021 19:05:05 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id b3so23677747wrm.6 for ; Mon, 28 Jun 2021 19:05:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=tFZIx+WwLaEcHap1rGQysKeAcWD7b272QogaBIbJLog=; b=jQT3yl1x7BUN80qG6cbWp/aFudsgkZzPdFb/Bukf3XCDW6wnIKgR7wH5RNUC6xEg+b QadAt+qNhsa/86efxDFmwUSfTbvPwUmpNLG2cmE5AlGGzpKb8QdzvZ2HAqDMBgVIa0jU KOpTCX/3r6oCwapQV8AbBe9t+/cgRZmHqyPIgxhqm0FKzc8TyPxQ8VN/MP2ZDkeWZdHg MBvHdxiGfJ5rnMhnkS0Ix/9NNl/O2mczxFLU3chazJKyqXKDEkPzDe0qHkKK9fd2BHxg Oq4topf7RGbQfPLf0oJ00GWy/ZQqlSEgHaWyahXTe5P1SnqR6Xow7aT3V8nLiAzvqJBR ftxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=tFZIx+WwLaEcHap1rGQysKeAcWD7b272QogaBIbJLog=; b=ltDSb+HxFJYlawTlT39I6HjOJcF5NjWbLwSpCs5ogdPrOvXQKULbN1TuxAeVL7dEWW bemBlX0LInKW3Fm1sMP/jemkjQfPCUOhH9Ir+RzXv6aDgJYU9W10Q6yYSJJaUIYMqUwG /Cs5AbKezHXrUsxFlzblBZWUtg4i45+PVBEiluOPiEjTobwf4wDbSPApbwVz8K7pXjsF gWsr6gFQLJ4oVqnXOEQJ/i/0Hfc+bngovt1JfCaPoYzzsATdKxA+fNGNcqJ3s0R82yU2 Oos0gjfaJ+0m8zJipFUPZGDA+EmIs0yNkE16ZZljcA/P5+XuAKQDOG3YbDG1Ihk9ot52 DI9w== X-Gm-Message-State: AOAM533hCQb5UYGsbwVYfZseb+gahShv8BAWnR+3xOj6DNgvIiImZ/DF FrO8zo7J7vXMgOGBMJGVG1Q9aQ0N13Q= X-Google-Smtp-Source: ABdhPJx96Vui6vtk1kGxU0GGNZ7psV8k3SaHpc4yx1U9+wAsAxCLxdgw5ob3QjHym+fG9NbRAkh3Vw== X-Received: by 2002:a5d:47cf:: with SMTP id o15mr30770195wrc.6.1624932304184; Mon, 28 Jun 2021 19:05:04 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id e8sm16018514wrq.10.2021.06.28.19.05.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 19:05:03 -0700 (PDT) Message-Id: <1d744848ee6b58ccaf3a30f20abe9797ed5d2ce7.1624932294.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 29 Jun 2021 02:04:53 +0000 Subject: [PATCH v7 16/16] fsmonitor: integrate with sparse index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee If we need to expand a sparse-index into a full one, then the FS Monitor bitmap is going to be incorrect. Ensure that we start fresh at such an event. While this is currently a performance drawback, the eventual hope of the sparse-index feature is that these expansions will be rare and hence we will be able to keep the FS Monitor data accurate across multiple Git commands. These tests are added to demonstrate that the behavior is the same across a full index and a sparse index, but also that file modifications to a tracked directory outside of the sparse cone will trigger ensure_full_index(). Signed-off-by: Derrick Stolee --- sparse-index.c | 7 ++++++ t/t7519-status-fsmonitor.sh | 49 +++++++++++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+) diff --git a/sparse-index.c b/sparse-index.c index ef53bd2198b..53c8f711ccc 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -186,6 +186,10 @@ int convert_to_sparse(struct index_state *istate) cache_tree_free(&istate->cache_tree); cache_tree_update(istate, 0); + istate->fsmonitor_has_run_once = 0; + FREE_AND_NULL(istate->fsmonitor_dirty); + FREE_AND_NULL(istate->fsmonitor_last_update); + istate->sparse_index = 1; trace2_region_leave("index", "convert_to_sparse", istate->repo); return 0; @@ -282,6 +286,9 @@ void ensure_full_index(struct index_state *istate) istate->cache = full->cache; istate->cache_nr = full->cache_nr; istate->cache_alloc = full->cache_alloc; + istate->fsmonitor_has_run_once = 0; + FREE_AND_NULL(istate->fsmonitor_dirty); + FREE_AND_NULL(istate->fsmonitor_last_update); strbuf_release(&base); free(full); diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh index 637391c6ce4..deea88d4431 100755 --- a/t/t7519-status-fsmonitor.sh +++ b/t/t7519-status-fsmonitor.sh @@ -73,6 +73,7 @@ test_expect_success 'setup' ' expect* actual* marker* + trace2* EOF ' @@ -383,4 +384,52 @@ test_expect_success 'status succeeds after staging/unstaging' ' ) ' +# Usage: +# check_sparse_index_behavior [!] +# If "!" is supplied, then we verify that we do not call ensure_full_index +# during a call to 'git status'. Otherwise, we verify that we _do_ call it. +check_sparse_index_behavior () { + git status --porcelain=v2 >expect && + git sparse-checkout init --cone --sparse-index && + git sparse-checkout set dir1 dir2 && + GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ + git status --porcelain=v2 >actual && + test_region $1 index ensure_full_index trace2.txt && + test_region fsm_hook query trace2.txt && + test_cmp expect actual && + rm trace2.txt && + git sparse-checkout disable +} + +test_expect_success 'status succeeds with sparse index' ' + git reset --hard && + + test_config core.fsmonitor "$TEST_DIRECTORY/t7519/fsmonitor-all" && + check_sparse_index_behavior ! && + + write_script .git/hooks/fsmonitor-test<<-\EOF && + printf "last_update_token\0" + EOF + git config core.fsmonitor .git/hooks/fsmonitor-test && + check_sparse_index_behavior ! && + + write_script .git/hooks/fsmonitor-test<<-\EOF && + printf "last_update_token\0" + printf "dir1/modified\0" + EOF + check_sparse_index_behavior ! && + + cp -r dir1 dir1a && + git add dir1a && + git commit -m "add dir1a" && + + # This one modifies outside the sparse-checkout definition + # and hence we expect to expand the sparse-index. + write_script .git/hooks/fsmonitor-test<<-\EOF && + printf "last_update_token\0" + printf "dir1a/modified\0" + EOF + check_sparse_index_behavior +' + test_done