From patchwork Mon Jul 12 17:55:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12371811 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CA4E4C07E9A for ; Mon, 12 Jul 2021 17:56:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AEECB611C1 for ; Mon, 12 Jul 2021 17:56:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235767AbhGLR6t (ORCPT ); Mon, 12 Jul 2021 13:58:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48084 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233424AbhGLR6s (ORCPT ); Mon, 12 Jul 2021 13:58:48 -0400 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2D8D9C0613DD for ; Mon, 12 Jul 2021 10:56:00 -0700 (PDT) Received: by mail-wr1-x430.google.com with SMTP id k4so20482005wrc.8 for ; Mon, 12 Jul 2021 10:56:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=yFW7dZNC+/YWclWUbvDxvwl7KQSIbPaIV7bQgc6a3ok=; b=V+LLhsatwZD02Yh1hzTH92yEWXm38OnD0clMgNDCyXqJl446yAM3fRCVufEiR+LkEC COJB7vcc8UEcBzUkCiFvc5nQ3h3tXKmhQSEylDopoUiQq8eQVeik1ahKx4aDJPG3N0JC 5Rwqq4CTpqRP3dUONy4vLEXD3/upXCi08B4DdXbPZYDwHjTlfjGoUjMzj0jtfnnqhhPA BAX0iC6+9PDkZajA/5LGKsJDZbR/2nc7zJdE6WFKXwAUQxG9aStvDEfFphZbm44pQxpX Q1xCgrlB8BVZZh1n23MPQdzJGZU5fu7LkkQug7H/ruVF12E3ljSM3xdr1fDTQtQFVaC8 zbpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=yFW7dZNC+/YWclWUbvDxvwl7KQSIbPaIV7bQgc6a3ok=; b=LiyI9YmkawRyp00pAhDWWQZ0qqqrXzmcjZfqepDq7SQgF2J9kxDYxDkyk/qhXdF3pG 3hLYuaffhuHguXIOGb650jPMdC+WCORxTU9Q1i4rr8yh25cWgnvyT7rTw47lWgVibVs4 TqIHTmlu1bMbRbw2dGueWw+id/CLoC/hmAeSoT9PBeeKVW0NNPe5Ahgae57CgfAYs0xg ZqeSLCW4/YFwNBA9f4XLvlS6tF+f3RvngRgfIP6M0GtuhTOqhn80Y3QUu7lT6OcJ4GM9 tio9pwTle06buGLI7QqKx0/lA0EFK7TGqm0JxRDZrX3bPVDu3VmTLJA+fuPH19s2pdxF 7TYg== X-Gm-Message-State: AOAM532tamQfdgfMYjhU8Lj19w/rCZE9eSM91aFVxn2RF4CvQvxO1d/i s11NKNRo3yCKUKbilAAJKyXKgxdGWms= X-Google-Smtp-Source: ABdhPJzOkyGwxYaEU1R3FZgpzKcBr1J45KRsZ5u4ifLdqhcOfRf1abAMy8Qac0dyupAjAn9kmQDJNQ== X-Received: by 2002:adf:ed07:: with SMTP id a7mr288702wro.70.1626112558771; Mon, 12 Jul 2021 10:55:58 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id v21sm142989wml.5.2021.07.12.10.55.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Jul 2021 10:55:58 -0700 (PDT) Message-Id: <1815c148e8c36d86de2ac5f27986f5f8d850f6c4.1626112556.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 12 Jul 2021 17:55:42 +0000 Subject: [PATCH v8 01/15] sparse-index: skip indexes with unmerged entries Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The sparse-index format is designed to be compatible with merge conflicts, even those outside the sparse-checkout definition. The reason is that when converting a full index to a sparse one, a cache entry with nonzero stage will not be collapsed into a sparse directory entry. However, this behavior was not tested, and a different behavior within convert_to_sparse() fails in this scenario. Specifically, cache_tree_update() will fail when unmerged entries exist. convert_to_sparse_rec() uses the cache-tree data to recursively walk the tree structure, but also to compute the OIDs used in the sparse-directory entries. Add an index scan to convert_to_sparse() that will detect if these merge conflict entries exist and skip the conversion before trying to update the cache-tree. This is marked as NEEDSWORK because this can be removed with a suitable update to cache_tree_update() or a similar method that can construct a cache-tree with invalid nodes, but still allow creating the nodes necessary for creating sparse directory entries. It is possible that in the future we will not need to make such an update, since if we do not expand a sparse-index into a full one, this conversion does not need to happen. Thus, this can be deferred until the merge machinery is made to integrate with the sparse-index. Signed-off-by: Derrick Stolee --- sparse-index.c | 18 ++++++++++++++++++ t/t1092-sparse-checkout-compatibility.sh | 22 ++++++++++++++++++++++ 2 files changed, 40 insertions(+) diff --git a/sparse-index.c b/sparse-index.c index affc4048f27..2c695930275 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -116,6 +116,17 @@ int set_sparse_index_config(struct repository *repo, int enable) return res; } +static int index_has_unmerged_entries(struct index_state *istate) +{ + int i; + for (i = 0; i < istate->cache_nr; i++) { + if (ce_stage(istate->cache[i])) + return 1; + } + + return 0; +} + int convert_to_sparse(struct index_state *istate) { int test_env; @@ -152,6 +163,13 @@ int convert_to_sparse(struct index_state *istate) return -1; } + /* + * NEEDSWORK: If we have unmerged entries, then stay full. + * Unmerged entries prevent the cache-tree extension from working. + */ + if (index_has_unmerged_entries(istate)) + return 0; + if (cache_tree_update(istate, 0)) { warning(_("unable to update cache-tree, staying full")); return -1; diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index d028b73eba1..b8617ceef71 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -352,6 +352,28 @@ test_expect_success 'merge with outside renames' ' done ' +# Sparse-index fails to convert the index in the +# final 'git cherry-pick' command. +test_expect_success 'cherry-pick with conflicts' ' + init_repos && + + write_script edit-conflict <<-\EOF && + echo $1 >conflict + EOF + + test_all_match git checkout -b to-cherry-pick && + run_on_all ../edit-conflict ABC && + test_all_match git add conflict && + test_all_match git commit -m "conflict to pick" && + + test_all_match git checkout -B base HEAD~1 && + run_on_all ../edit-conflict DEF && + test_all_match git add conflict && + test_all_match git commit -m "conflict in base" && + + test_all_match test_must_fail git cherry-pick to-cherry-pick +' + test_expect_success 'clean' ' init_repos && From patchwork Mon Jul 12 17:55:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12371813 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E28DC07E99 for ; Mon, 12 Jul 2021 17:56:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DAC5D61154 for ; Mon, 12 Jul 2021 17:56:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235783AbhGLR6u (ORCPT ); Mon, 12 Jul 2021 13:58:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48086 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233555AbhGLR6t (ORCPT ); Mon, 12 Jul 2021 13:58:49 -0400 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AE804C0613DD for ; Mon, 12 Jul 2021 10:56:00 -0700 (PDT) Received: by mail-wr1-x42f.google.com with SMTP id i94so26780828wri.4 for ; Mon, 12 Jul 2021 10:56:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=iy9XiTv1ciKF1QkjuiSZgTtadS+h+8sdv46VtmcVxm4=; b=SHelQ3fgxULS5Zsn00hKQDlg77bhLtx+qgfHzUrdDgN9+mM/sKObjIyPRXAcSrFcNA Q8vt5AWsjeAy7ANXSOvzVxfyf4KHylAQ9o3wAzDFFpOpxSOXJgvENn753Wh0HDF+S9wT gRrb0IA2ODEuNpEQRVHYZvgZyG4g+zmPZaNd/bWe6lrQzUnKHO7nRGRRxlt5A5ZNr9p0 Cp6pGXpc/gCx1tN065kLNICOi5TUm1DMMrHd5aoPC6Xk+2QkiqHYyUpeSN0lxmSUXpHT 6FGPtdNNgCc6kEs2Y0Hh7EHGz/6kO+9/Y4Y8zPI5bclwV/No0id8mSUzbmcPJvAKzWAR MXWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=iy9XiTv1ciKF1QkjuiSZgTtadS+h+8sdv46VtmcVxm4=; b=JEpatjlxP+kqwiVXDw1LNFpVL9R93of4UZH8cHTjLxf3yODawa78N8Eo4aGB2yjj+s Ty94AH5tl2taPbRSa83X+OxF+Vg1h0ch+/6w+UoltOTT2HTnTA+XXLRm16cbHb+/Z1ZC WLanW6LJsUEdYxbiz4HMsnlW+NWQ+rsoMejg+N5KQNPVfODSrI5CLvJemVbT8+iTmZi0 CKtCCGpbA8DEsCnGamcCZ4Y+KGj0CqwfuZil3s9yNd0MQdurt8xtVhBGxNC+w+9QlDsZ Y3mC9ya6qEyGT3cF9lBfssIrkuYx6JsTd8sH85QcMyXG0suNYnf1u2lqQucYuAuMMYSA KcXg== X-Gm-Message-State: AOAM532e/nDKkzjrgwi1qEKWfgZ8G0zTzO84EoE5jGZRfxLpzQBQOoou DqeQlcgoasoGWybNptVGiNGd3sFvI5w= X-Google-Smtp-Source: ABdhPJysYBH/4/QTQlE9t8lZDPiOOfZ40OwGl6kVrvm38Qt5p1Yr08QC0/IHYrI0PTAN4dWJTVe8Pw== X-Received: by 2002:a05:6000:1842:: with SMTP id c2mr234680wri.374.1626112559410; Mon, 12 Jul 2021 10:55:59 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id h13sm14959154wrs.68.2021.07.12.10.55.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Jul 2021 10:55:59 -0700 (PDT) Message-Id: <7bcde075d8db1f89c21213fb92f1d280b8adc032.1626112556.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 12 Jul 2021 17:55:43 +0000 Subject: [PATCH v8 02/15] sparse-index: include EXTENDED flag when expanding Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee When creating a full index from a sparse one, we create cache entries for every blob within a given sparse directory entry. These are correctly marked with the CE_SKIP_WORKTREE flag, but the CE_EXTENDED flag is not included. The CE_EXTENDED flag would exist if we loaded a full index from disk with these entries marked with CE_SKIP_WORKTREE, so we can add the flag here to be consistent. This allows us to directly compare the flags present in cache entries when testing the sparse-index feature, but has no significance to its correctness in the user-facing functionality. Signed-off-by: Derrick Stolee --- sparse-index.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparse-index.c b/sparse-index.c index 2c695930275..ef53bd2198b 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -213,7 +213,7 @@ static int add_path_to_index(const struct object_id *oid, strbuf_addstr(base, path); ce = make_cache_entry(istate, mode, oid, base->buf, 0, 0); - ce->ce_flags |= CE_SKIP_WORKTREE; + ce->ce_flags |= CE_SKIP_WORKTREE | CE_EXTENDED; set_index_entry(istate, istate->cache_nr++, ce); strbuf_setlen(base, len); From patchwork Mon Jul 12 17:55:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12371815 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19808C07E9C for ; Mon, 12 Jul 2021 17:56:05 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EEB91611C1 for ; Mon, 12 Jul 2021 17:56:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235802AbhGLR6w (ORCPT ); Mon, 12 Jul 2021 13:58:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48092 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235770AbhGLR6u (ORCPT ); Mon, 12 Jul 2021 13:58:50 -0400 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 57AC5C0613DD for ; Mon, 12 Jul 2021 10:56:01 -0700 (PDT) Received: by mail-wr1-x435.google.com with SMTP id k4so20482101wrc.8 for ; Mon, 12 Jul 2021 10:56:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=SwlaGGwBDbfHE4py4v0EAuLYGfp6NsDfmCTHdty/pqQ=; b=AR/8gyz3BDAza+IGR0U3FDlRuc7lu60dSIZbwQpaYo0gz90Tjt7XovZHhQktv3pNKo vK4YWlLMYBEZEL1Kfc04ox90NX2FS/acL5CNfG35l/9iUGSy8Oo+LofFH8vo9fMMgSis yGRVDp9fHA5GT+WikhBJARwhkHPoFQJxCAmLOukgdaPYyFvUvh5V59wB7g0embFZ7LFJ 8PHdPaIme1rrpVQsuCIFcTkdNW29/YEZ3ObuTyzFFCk0X0IQV/jyXOQpk1vzqQ0QaQ79 CjM5CYQ1suGwWim5LH9aq1XROxmbmCKyylwLqYFXuJOOEx/n1ryFCWysCOMAeWg7gLyO fNww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=SwlaGGwBDbfHE4py4v0EAuLYGfp6NsDfmCTHdty/pqQ=; b=AnlBrZT2y9mqZYHEcLLcij85o7sEhhYXa7aWNhI0kwnn0JBv1RbKh/OoHxL3jYdqhv COHdATCKLp1n4V+tfxfDHxPWBWOrqgmnOuNUO9cbo8v8ugoKbba2urAVJ816Wao0UCMT LzEcKAm/wx8zrY4rmLEeVcqCFH/Ju5je2L+aLjUpjjevxHMK+HrCe1vwnn95RO0AJ6Vz 4gPHVF92zYJyToyKXz2suhg5ENBGnjvNH/TGoKrWPcSq0wVvjGDYumD5jCp7zSVZkxI8 StwoKJexG82unnHiwBTdCW2btAV9W62LB4pjCadhFJrjZPQetdCDfKOD98NQU/sQXfq+ 3yVw== X-Gm-Message-State: AOAM531R67feDJZdPW61awhRy9QOFeCDgaRnz9yvr4X40egpmpl6LWkG 2y+ocMF8zPPtUpTRmJanXoDU6LLCnPM= X-Google-Smtp-Source: ABdhPJwXdN9WSCx1cq+C+HxmD2WNtAfNH/sHlLMo+67yTpWtN46vBK3gY6XZb2dtlt9Ui03/HGwafA== X-Received: by 2002:a5d:66d1:: with SMTP id k17mr268493wrw.110.1626112560012; Mon, 12 Jul 2021 10:56:00 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id v11sm14787300wrs.4.2021.07.12.10.55.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Jul 2021 10:55:59 -0700 (PDT) Message-Id: <05981e30b97a3e668b3a9675508135d42f15450f.1626112556.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 12 Jul 2021 17:55:44 +0000 Subject: [PATCH v8 03/15] t1092: replace incorrect 'echo' with 'cat' Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee This fixes the test data shape to be as expected, allowing rename detection to work properly now that the 'larger-content' file actually has meaningful lines. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index b8617ceef71..87f1014a1c9 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -40,7 +40,7 @@ test_expect_success 'setup' ' done && git checkout -b rename-base base && - echo >folder1/larger-content <<-\EOF && + cat >folder1/larger-content <<-\EOF && matching lines help From patchwork Mon Jul 12 17:55:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12371817 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1F87C07E9A for ; Mon, 12 Jul 2021 17:56:05 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8F0F061206 for ; Mon, 12 Jul 2021 17:56:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235819AbhGLR6x (ORCPT ); Mon, 12 Jul 2021 13:58:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48098 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235800AbhGLR6u (ORCPT ); Mon, 12 Jul 2021 13:58:50 -0400 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 14ABCC0613E5 for ; Mon, 12 Jul 2021 10:56:02 -0700 (PDT) Received: by mail-wr1-x436.google.com with SMTP id r11so21535396wro.9 for ; Mon, 12 Jul 2021 10:56:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=s5F2jg1ozuHBE0uTDnrtvT8FkKzIZbq7avOJxWE7f/I=; b=qECFr/xvwoEvgS839ulEV+btGNnfl+CJhaI6R8PtCjw9cKtI7qNuMJKRdsj0yIaunU gdqcZ4ySjVx5/LF5fJu1xkZolOEXP5ZPDChNd9+Zd9SGqv6X4fsTR507XTz+VVTj8uSg ulNbZjZ6N1iVM6Ieyz1JLrrbkqT9M8NsbWGUTHnDX/INT1Zah55dAZq8d09X4F5U86iz 1jox+JHEioMGXFzJUsDpq+U/5mBKKwwTYXz6J+9+ZJ/aNqsthdXYuKUsOUk9xR1Z0Hk4 Yt7FtpnxO/ja6fFZ4z/TFkpCIdYBFsDV/sAHDg5/4sjJR/6wp3uIVGJEak7e4orC/+BT GdkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=s5F2jg1ozuHBE0uTDnrtvT8FkKzIZbq7avOJxWE7f/I=; b=snp9DK/D0VQTVhNfmThuShF8l+jIZfybkLcnE8fUzB+i6o7fW7+lYHQ9Uhtvc4Dv5W qfRXOH0zEw5fu9hmCOi19AjxcXfqQea9OzwAkSjjxYIYVH+tePF5pgQf9N1wl7SO9Wm1 BX7QFmEUL5kWHWj5+hO51Vn28KlyGc6+nKkdLbYPWsxgcscWPEjrjqW22jtZxdtFfYIP SrJLWZOTWqEViRFnTJSyEhgzUFlhZItPQKqM2vbKFR4HRqZIP6/LfYnhcDFH0LZzyUj9 CRLISlHJwuRCtqFTirpTfNK8LuaOFFFqhN/x4TGWmt5Jrxrd78HFDEfFMywrVK+VSI9B kR+Q== X-Gm-Message-State: AOAM532RYx8CmvAcLtpd0PcDxAHZmZ0bjzdnHjIzyH2/C+OES0t5HNrB 6c9nZqbTE2s/Lgu2r1gxKAT0lqYpYSU= X-Google-Smtp-Source: ABdhPJybxCtBclEUoall4T+jK/E/fTSGx9JbZUGu2V0ycGPaq07fRUa7TZOY/SS0CWs4oxG4PDBAiA== X-Received: by 2002:adf:c44d:: with SMTP id a13mr220594wrg.65.1626112560686; Mon, 12 Jul 2021 10:56:00 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id q5sm13036602wmc.0.2021.07.12.10.56.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Jul 2021 10:56:00 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Mon, 12 Jul 2021 17:55:45 +0000 Subject: [PATCH v8 04/15] t1092: expand repository data shape Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee As more features integrate with the sparse-index feature, more and more special cases arise that require different data shapes within the tree structure of the repository in order to demonstrate those cases. Add several interesting special cases all at once instead of sprinkling them across several commits. The interesting cases being added here are: * Add sparse-directory entries on both sides of directories within the sparse-checkout definition. * Add directories outside the sparse-checkout definition who have only one entry and are the first entry of a directory with multiple entries. * Add filenames adjacent to a sparse directory entry that sort before and after the trailing slash. Later tests will take advantage of these shapes, but they also deepen the tests that already exist. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 42 ++++++++++++++++++++++-- 1 file changed, 40 insertions(+), 2 deletions(-) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 87f1014a1c9..0e71a623619 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -17,7 +17,7 @@ test_expect_success 'setup' ' echo "after folder1" >g && echo "after x" >z && mkdir folder1 folder2 deep x && - mkdir deep/deeper1 deep/deeper2 && + mkdir deep/deeper1 deep/deeper2 deep/before deep/later && mkdir deep/deeper1/deepest && echo "after deeper1" >deep/e && echo "after deepest" >deep/deeper1/e && @@ -25,10 +25,23 @@ test_expect_success 'setup' ' cp a folder2 && cp a x && cp a deep && + cp a deep/before && cp a deep/deeper1 && cp a deep/deeper2 && + cp a deep/later && cp a deep/deeper1/deepest && cp -r deep/deeper1/deepest deep/deeper2 && + mkdir deep/deeper1/0 && + mkdir deep/deeper1/0/0 && + touch deep/deeper1/0/1 && + touch deep/deeper1/0/0/0 && + >folder1- && + >folder1.x && + >folder10 && + cp -r deep/deeper1/0 folder1 && + cp -r deep/deeper1/0 folder2 && + echo >>folder1/0/0/0 && + echo >>folder2/0/1 && git add . && git commit -m "initial commit" && git checkout -b base && @@ -56,11 +69,17 @@ test_expect_success 'setup' ' mv folder1/a folder2/b && mv folder1/larger-content folder2/edited-content && echo >>folder2/edited-content && + echo >>folder2/0/1 && + echo stuff >>deep/deeper1/a && git add . && git commit -m "rename folder1/... to folder2/..." && git checkout -b rename-out-to-in rename-base && mv folder1/a deep/deeper1/b && + echo more stuff >>deep/deeper1/a && + rm folder2/0/1 && + mkdir folder2/0/1 && + echo >>folder2/0/1/1 && mv folder1/larger-content deep/deeper1/edited-content && echo >>deep/deeper1/edited-content && git add . && @@ -68,6 +87,9 @@ test_expect_success 'setup' ' git checkout -b rename-in-to-out rename-base && mv deep/deeper1/a folder1/b && + echo >>folder2/0/1 && + rm -rf folder1/0/0 && + echo >>folder1/0/0 && mv deep/deeper1/larger-content folder1/edited-content && echo >>folder1/edited-content && git add . && @@ -262,13 +284,29 @@ test_expect_success 'diff --staged' ' test_all_match git diff --staged ' -test_expect_success 'diff with renames' ' +test_expect_success 'diff with renames and conflicts' ' init_repos && for branch in rename-out-to-out rename-out-to-in rename-in-to-out do test_all_match git checkout rename-base && test_all_match git checkout $branch -- . && + test_all_match git status --porcelain=v2 && + test_all_match git diff --staged --no-renames && + test_all_match git diff --staged --find-renames || return 1 + done +' + +test_expect_success 'diff with directory/file conflicts' ' + init_repos && + + for branch in rename-out-to-out rename-out-to-in rename-in-to-out + do + git -C full-checkout reset --hard && + test_sparse_match git reset --hard && + test_all_match git checkout $branch && + test_all_match git checkout rename-base -- . && + test_all_match git status --porcelain=v2 && test_all_match git diff --staged --no-renames && test_all_match git diff --staged --find-renames || return 1 done From patchwork Mon Jul 12 17:55:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12371819 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76860C07E99 for ; Mon, 12 Jul 2021 17:56:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6355D611C1 for ; Mon, 12 Jul 2021 17:56:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235812AbhGLR6y (ORCPT ); Mon, 12 Jul 2021 13:58:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48104 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235801AbhGLR6v (ORCPT ); Mon, 12 Jul 2021 13:58:51 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D15AEC0613DD for ; Mon, 12 Jul 2021 10:56:02 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id w13so11969402wmc.3 for ; Mon, 12 Jul 2021 10:56:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=TF6o2tbbkpnAZzSCB71Xp9b8sR9m4FQKmWDgSsgnvdk=; b=WDbHDurPjMRt+CnfLgfPx+bp/JnVbp65GhxryifOMR0mpOBg7Yinm0SxzBkfoTW6IJ 9vr2UbE1YnEYwQe/qD3OlxHWiMCpbFL8LBPUJmPhxZutSNSthdC7PU/2XBL28pZbZG5Z IacK4fsOhT2OXmisj26bS/keBYLpiDg8SrJvFmjxKmQROdLk0UVaovd+CyrSVXK/K5zX iE7W90bqStsYd/eooyo7ETYgcG8V0QdXjZczXnDPe6Yzx4upKAQHh1C2JDhbumIFyMEN OsA4CR6WiSaqNcaxQf6mSok69gh9II9UZpDHmlon/SQLNd9FqvMAnVPwVAZ/j/L+lTQ8 R5gg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=TF6o2tbbkpnAZzSCB71Xp9b8sR9m4FQKmWDgSsgnvdk=; b=FLxv8Ctbw1k5+faHraiEozAw4KJKoLk2Etu25cWKK6nc6bvfkDSSdbei5W5ccRYWkP Wbxay+n8cPCDpJB009OR9p+Vcnxn4A4OMT7PB42hLW365f8NYQ0ycWqBsL396TmPfw/0 1gY0rjDYbT6aXSkKL6CIWHYJa6yyEB8YhHwactBqed0MesxCEu7g0kXE6vdbQjz568Px X7Ln8ptBcLfFm/dxWdJwWsPkTtOoLU+xDTjlH0Vi5jPdjZkeuJhXIm9ghARnBRq/Ikxt 4l2WFrOsTpMebydICWdc3U8wSaBUNAXzNmGVWJb4ZIK+Gdfmkpuk5h/bxqgiTMJihu9A cprw== X-Gm-Message-State: AOAM532M0OByVpioceieIs4xXD2IL1F7u1kAEjl4QN58Pkn2rAoEpAw0 LsVwZPR1Yv2xEcqSoq3VltrXmjH3HnI= X-Google-Smtp-Source: ABdhPJxv6NqacRTcPH9HZak6H9/3n3hL8uMj+/N/AjIyNcKBURCQ+Bxg768+aGQBiZaJ/5rNCpY3UQ== X-Received: by 2002:a05:600c:1911:: with SMTP id j17mr447805wmq.172.1626112561464; Mon, 12 Jul 2021 10:56:01 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id y3sm13882007wrh.16.2021.07.12.10.56.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Jul 2021 10:56:01 -0700 (PDT) Message-Id: <95ddd3abe4e5c85e4c911ba6c2d4bd86099085b4.1626112556.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 12 Jul 2021 17:55:46 +0000 Subject: [PATCH v8 05/15] t1092: add tests for status/add and sparse files Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Before moving to update 'git status' and 'git add' to work with sparse indexes, add an explicit test that ensures the sparse-index works the same as a normal sparse-checkout when the worktree contains directories and files outside of the sparse cone. Specifically, 'folder1/a' is a file in our test repo, but 'folder1' is not in the sparse cone. When 'folder1/a' is modified, the file is not shown as modified and adding it will fail. This is new behavior as of a20f704 (add: warn when asked to update SKIP_WORKTREE entries, 2021-04-08). Before that change, these adds would be silently ignored. Untracked files are fine: adding new files both with 'git add .' and 'git add folder1/' works just as in a full checkout. This may not be entirely desirable, but we are not intending to change behavior at the moment, only document it. A future change could alter the behavior to be more sensible, and this test could be modified to satisfy the new expected behavior. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 38 ++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 0e71a623619..2269f44e033 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -254,6 +254,44 @@ test_expect_success 'add, commit, checkout' ' test_all_match git checkout - ' +test_expect_success 'status/add: outside sparse cone' ' + init_repos && + + # adding a "missing" file outside the cone should fail + test_sparse_match test_must_fail git add folder1/a && + + # folder1 is at HEAD, but outside the sparse cone + run_on_sparse mkdir folder1 && + cp initial-repo/folder1/a sparse-checkout/folder1/a && + cp initial-repo/folder1/a sparse-index/folder1/a && + + test_sparse_match git status && + + write_script edit-contents <<-\EOF && + echo text >>$1 + EOF + run_on_sparse ../edit-contents folder1/a && + run_on_all ../edit-contents folder1/new && + + test_sparse_match git status --porcelain=v2 && + + # This "git add folder1/a" fails with a warning + # in the sparse repos, differing from the full + # repo. This is intentional. + test_sparse_match test_must_fail git add folder1/a && + test_sparse_match test_must_fail git add --refresh folder1/a && + test_all_match git status --porcelain=v2 && + + test_all_match git add . && + test_all_match git status --porcelain=v2 && + test_all_match git commit -m folder1/new && + + run_on_all ../edit-contents folder1/newer && + test_all_match git add folder1/ && + test_all_match git status --porcelain=v2 && + test_all_match git commit -m folder1/newer +' + test_expect_success 'checkout and reset --hard' ' init_repos && From patchwork Mon Jul 12 17:55:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12371823 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66408C07E99 for ; Mon, 12 Jul 2021 17:56:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 51D03611C1 for ; Mon, 12 Jul 2021 17:56:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235835AbhGLR64 (ORCPT ); Mon, 12 Jul 2021 13:58:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48112 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235809AbhGLR6x (ORCPT ); Mon, 12 Jul 2021 13:58:53 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6E701C0613F0 for ; Mon, 12 Jul 2021 10:56:03 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id i94so26781023wri.4 for ; Mon, 12 Jul 2021 10:56:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=20VyvcZukH9mBP0oxKGuPMDI0uq7we8hBvlg4nNNWng=; b=Gz8C5Q4m8V7f/gp8EZKXlmGl1B/OFsGGHqsFIJZjibRmcgZbkVfVZa33BuMbnlU6x2 h+igBPZJ90Ou+vJ/FgXN94rU6f0++DDzY796DMnhQxytuipOfZBDNdzjz75EcEa6uWv9 7i15I6qKT8KjIus1n3xfHMKOIDOuwDrXKpq4+Vem5m4vsJTFjkYuPpyZ4V24C7F/xfrL qxK5jFKy/L+IUJkFro9XT4655rigwhP4EHO9Fnmcw6H2DUd1AhsCyrRi22RLwAfDqUbt 7f77UOgDXLPIMMr7It2QVDzbq6MIimPyxV8TkVgqc02QOEl4IIjlrCApBbfSC+5zf2Se B4aA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=20VyvcZukH9mBP0oxKGuPMDI0uq7we8hBvlg4nNNWng=; b=e0ehmFKCO+BjaQZ66xsxI1PUZHHC2F3m6OfvODeL26nvdoAWfvSVP1fAlvM5hzktMy h4hvcOVO710pmzT755Pa9J+A7ReGfugJpW8Zd2j6YxBB4J52vzlnz2mpfTpuvxnCiyyb BDEg9cL5wpbht+BSoMwkBejH/5aaPcO6tBrAKt7yDCCCOCF3VxCVHUoC5glHDFUq1Mqe 4ZZ2SbjhMurzCX5hDSh6R1fyN0Z7VGoVH/K6eREp27Y6qR3wTDQ0lJByUE73WKLNY1ko YH4g2EyD8hVpbRjD3fIFEqgxhZ9imwJdsxaqTWIVABb9AukXWn+kutlcCYV6qvPa2kIx S4yA== X-Gm-Message-State: AOAM532ewZ2QKDx41uMKK9nEN2IU39CrQ6gRiqnx2aBfsFYXRo/2kuTm t64E2X7JKJ096lyjzmIBwzJ7BqRbNvo= X-Google-Smtp-Source: ABdhPJzrRmZsLXXky7T3BCpCtyBHYuB38weq2T7phHJhlI/wv0jktC8S75DmuZQ+c2Wdt1RJYynxKQ== X-Received: by 2002:a5d:4252:: with SMTP id s18mr266862wrr.88.1626112562094; Mon, 12 Jul 2021 10:56:02 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 129sm6341869wmz.26.2021.07.12.10.56.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Jul 2021 10:56:01 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Mon, 12 Jul 2021 17:55:47 +0000 Subject: [PATCH v8 06/15] unpack-trees: preserve cache_bottom Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The cache_bottom member of 'struct unpack_trees_options' is used to track the range of index entries corresponding to a node of the cache tree. While recursing with traverse_by_cache_tree(), this value is preserved on the call stack using a local and then restored as that method returns. The mark_ce_used() method normally modifies the cache_bottom member when it refers to the marked cache entry. However, sparse directory entries are stored as nodes in the cache-tree data structure as of 2de37c53 (cache-tree: integrate with sparse directory entries, 2021-03-30). Thus, the cache_bottom will be modified as the cache-tree walk advances. Do not update it as well within mark_ce_used(). Signed-off-by: Derrick Stolee --- unpack-trees.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/unpack-trees.c b/unpack-trees.c index f88a69f8e71..87c1ed204c8 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -600,6 +600,13 @@ static void mark_ce_used(struct cache_entry *ce, struct unpack_trees_options *o) { ce->ce_flags |= CE_UNPACKED; + /* + * If this is a sparse directory, don't advance cache_bottom. + * That will be advanced later using the cache-tree data. + */ + if (S_ISSPARSEDIR(ce->ce_mode)) + return; + if (o->cache_bottom < o->src_index->cache_nr && o->src_index->cache[o->cache_bottom] == ce) { int bottom = o->cache_bottom; From patchwork Mon Jul 12 17:55:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12371821 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1284DC07E9C for ; Mon, 12 Jul 2021 17:56:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EA056611C1 for ; Mon, 12 Jul 2021 17:56:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235827AbhGLR6y (ORCPT ); Mon, 12 Jul 2021 13:58:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48106 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235772AbhGLR6x (ORCPT ); Mon, 12 Jul 2021 13:58:53 -0400 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 570DEC0613DD for ; Mon, 12 Jul 2021 10:56:04 -0700 (PDT) Received: by mail-wr1-x435.google.com with SMTP id g16so16901624wrw.5 for ; Mon, 12 Jul 2021 10:56:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=HDhgtJZsDGyJtVd/z+kWysbOePZ/rRg7q4HZzXdKryk=; b=HdXsyucZHjP+cr7nmR/NXldpO1+GWP+nXJpaNzV216WAIj0A1S2fegcH8yx16SwdL5 WGD9k84c4Y5yBc7UDf0tKzgZiKiVGVJos4Ju9zemA3CPTBKWlwVOwcR597CLvY7cmQSI WnuUVQufVc7nZHGl5cYolzVQxfiEXxBh1Xa6a+U9cclFaafX94Hlhpmz6aEfinjlWqgO h9TqZgf4K2QwgTEF84r+T4VOOg9u6iYX+L5zDDbFQTvo8Au6Y4JN+Z5GJh1ad0i87La8 qRiIeJ2OfQGfTfXvd0LPQ6ptkNoIyPtr3BYMXwRSfWlm7/ZFndeZSLfeY2hBOX2ZoY0M b7og== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=HDhgtJZsDGyJtVd/z+kWysbOePZ/rRg7q4HZzXdKryk=; b=ix9R0nVss65BZrlrScrwu3J0bYO6iMClZa8nxFcenY0qjWkafNmL42Z5hdPhgSb1Io bH6+hv8NRxOS7DoQPjpNRObr8MlLaHKh51y1hyFK0cSf/f06UxgPn4MWQBAQkWpL3Hdz 5BMHgPESrXp4d44VqbY0ww9cZ/Q2dN3/lUFrwxmhs674tUIkHFqW0Mt62uIbYW2KyO5c Fiidh6q4CM8Qe/xrnTbk578Aatryy3UAvfcgeYuf6GFRy1KzTguVUqBdXpBkHEqpPx10 t/aJVzt/DQhaV7SQhzF6Jfs4DVbYVOOUTIF08/ExCZf3hOYj25BqVsowxpPZUbc4xVLl e7cA== X-Gm-Message-State: AOAM533QLhJqZZZdSuim92IobaMUD/W2WpvVcxb1RjHrv+DhiaR28AV7 oiErhqC4EefboOQLaAVW+UzSW3BqeSk= X-Google-Smtp-Source: ABdhPJxHC6lqZyaq4KhkaBFSOoBOhZxQQ5VPKBqvEK96QfDmEWr3XiDYsSIJgwJOgtDAwVzV7y1LrQ== X-Received: by 2002:adf:f90d:: with SMTP id b13mr241064wrr.336.1626112562737; Mon, 12 Jul 2021 10:56:02 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id d9sm373606wrx.76.2021.07.12.10.56.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Jul 2021 10:56:02 -0700 (PDT) Message-Id: <988ddce4d45d994488ca074039a9b8a8ed2ca285.1626112556.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 12 Jul 2021 17:55:48 +0000 Subject: [PATCH v8 07/15] unpack-trees: compare sparse directories correctly Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee As we further integrate the sparse-index into unpack-trees, we need to ensure that we compare sparse directory entries correctly with other entries. This affects searching for an exact path as well as sorting index entries. Sparse directory entries contain the trailing directory separator. This is important for the sorting, in particular. Thus, within do_compare_entry() we stop using S_IFREG in all cases, since sparse directories should use S_IFDIR to indicate that the comparison should treat the entry name as a dirctory. Within compare_entry(), it first calls do_compare_entry() to check the leading portion of the name. When the input path is a directory name, we could match exactly already. Thus, we should return 0 if we have an exact string match on a sparse directory entry. The final check is a length comparison between the strings. Signed-off-by: Derrick Stolee --- unpack-trees.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/unpack-trees.c b/unpack-trees.c index 87c1ed204c8..b113cc750f2 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -983,6 +983,7 @@ static int do_compare_entry(const struct cache_entry *ce, int pathlen, ce_len; const char *ce_name; int cmp; + unsigned ce_mode; /* * If we have not precomputed the traverse path, it is quicker @@ -1005,7 +1006,8 @@ static int do_compare_entry(const struct cache_entry *ce, ce_len -= pathlen; ce_name = ce->name + pathlen; - return df_name_compare(ce_name, ce_len, S_IFREG, name, namelen, mode); + ce_mode = S_ISSPARSEDIR(ce->ce_mode) ? S_IFDIR : S_IFREG; + return df_name_compare(ce_name, ce_len, ce_mode, name, namelen, mode); } static int compare_entry(const struct cache_entry *ce, const struct traverse_info *info, const struct name_entry *n) @@ -1014,6 +1016,16 @@ static int compare_entry(const struct cache_entry *ce, const struct traverse_inf if (cmp) return cmp; + /* + * At this point, we know that we have a prefix match. If ce + * is a sparse directory, then allow an exact match. This only + * works when the input name is a directory, since ce->name + * ends in a directory separator. + */ + if (S_ISSPARSEDIR(ce->ce_mode) && + ce->ce_namelen == traverse_path_len(info, tree_entry_len(n)) + 1) + return 0; + /* * Even if the beginning compared identically, the ce should * compare as bigger than a directory leading up to it! From patchwork Mon Jul 12 17:55:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12371825 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 584FEC07E99 for ; Mon, 12 Jul 2021 17:56:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 425F961154 for ; Mon, 12 Jul 2021 17:56:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235865AbhGLR67 (ORCPT ); Mon, 12 Jul 2021 13:58:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48104 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235817AbhGLR6x (ORCPT ); Mon, 12 Jul 2021 13:58:53 -0400 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B7335C0613E9 for ; Mon, 12 Jul 2021 10:56:04 -0700 (PDT) Received: by mail-wr1-x42d.google.com with SMTP id r11so21535546wro.9 for ; Mon, 12 Jul 2021 10:56:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=O0bk0uMoOfzc+R3V+w3r7b7PaKiQoEFR2JUhLyX6ccg=; b=uEDf/4iETF/IUh/+8bjYd9pXMdTRe1gLO73uh8Gkr1z3eB6md6Yol9x9q/ErZMoLff h0OGu32B/IPLd4EYyGKbayjlABy4HnOYBI6noeJa2PTkubfIGpshYMw2Xieu3dUHksFx GttuaFqMdyxEPeD3y0hYj3Bue9z6StLIy191hmbVNOXfXrSt08MJiWJZCd8bI1DDZ0zh 2xPbYnzYQrQq2NWCOqttybgiGL7HvO7khWvtR8sBoyvpqIzSeuR7q2QfV7Dvp9jn2pdf PgSDxWPkt92q2q0qUCTsKDPC3H0uM5JSGKoMz/okC37ZdEKSttWuKdIxCumxOaY+QfMP Ux6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=O0bk0uMoOfzc+R3V+w3r7b7PaKiQoEFR2JUhLyX6ccg=; b=O/wQGXdjZVSIAlzKkUHZzAnfVf/ZJfA9GR0d+6WiRkWNJnEfcP423UY3uFkd8o1Em2 qwMFxkt+UTFH2cnhJhA8VgR6JR6iWjx0PEABkFKympHYKvdj1z91MLEtlp9DlbFJhqh+ RsWfPgHkZYHmyYEClIkOkWyiTQjj8aBgTX0hS8eyxanFDDJnz5L7ffHiwXL7MvBqe6pb ybRDaE6GrUhNM0lDXURHDNzti2rEv1ON1vXsdrez+o0V0fbGhRIULr9kcWp3Es+rIIpD IdbxqaUekK4qsWhRs7vTzdEz2TS+xNcz+1pEaS/FWkPu8x5RXGh1q2JymY+NIMBcHEL5 QzeA== X-Gm-Message-State: AOAM533PZFjHqB8G2OKklASOrHPFC6GaWI5bzW9EAQZeQARxOGBdircL ecpNb1NqTb79l5qK8fUbxde1NMHlQnA= X-Google-Smtp-Source: ABdhPJwxUQMvRlszSad4cKEd3WGGGCbYu0raHs3hJ5q5do6vKz751hapNbOH3OwxSsi1LyqyXiu2Mw== X-Received: by 2002:adf:9d8d:: with SMTP id p13mr246690wre.300.1626112563323; Mon, 12 Jul 2021 10:56:03 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l24sm121207wmi.30.2021.07.12.10.56.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Jul 2021 10:56:03 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Mon, 12 Jul 2021 17:55:49 +0000 Subject: [PATCH v8 08/15] unpack-trees: rename unpack_nondirectories() Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee In the next change, we will use this method to unpack a sparse directory entry, so change the name to unpack_single_entry() so these entries apply. The new name reflects that we will not recurse into trees in order to resolve the conflicts. Signed-off-by: Derrick Stolee --- unpack-trees.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/unpack-trees.c b/unpack-trees.c index b113cc750f2..d26386ce8b2 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -804,7 +804,7 @@ static int traverse_by_cache_tree(int pos, int nr_entries, int nr_names, BUG("We need cache-tree to do this optimization"); /* - * Do what unpack_callback() and unpack_nondirectories() normally + * Do what unpack_callback() and unpack_single_entry() normally * do. But we walk all paths in an iterative loop instead. * * D/F conflicts and higher stage entries are not a concern @@ -1075,11 +1075,11 @@ static struct cache_entry *create_ce_entry(const struct traverse_info *info, * without actually calling it. If you change the logic here you may need to * check and change there as well. */ -static int unpack_nondirectories(int n, unsigned long mask, - unsigned long dirmask, - struct cache_entry **src, - const struct name_entry *names, - const struct traverse_info *info) +static int unpack_single_entry(int n, unsigned long mask, + unsigned long dirmask, + struct cache_entry **src, + const struct name_entry *names, + const struct traverse_info *info) { int i; struct unpack_trees_options *o = info->data; @@ -1322,7 +1322,7 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str } } - if (unpack_nondirectories(n, mask, dirmask, src, names, info) < 0) + if (unpack_single_entry(n, mask, dirmask, src, names, info) < 0) return -1; if (o->merge && src[0]) { From patchwork Mon Jul 12 17:55:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12371827 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 736CAC07E9C for ; Mon, 12 Jul 2021 17:56:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5FA03611C1 for ; Mon, 12 Jul 2021 17:56:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235841AbhGLR7A (ORCPT ); Mon, 12 Jul 2021 13:59:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48126 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235771AbhGLR6x (ORCPT ); Mon, 12 Jul 2021 13:58:53 -0400 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3B94FC0613E5 for ; Mon, 12 Jul 2021 10:56:05 -0700 (PDT) Received: by mail-wr1-x430.google.com with SMTP id r11so21535582wro.9 for ; Mon, 12 Jul 2021 10:56:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=/UNBWPl4t+GAChOB0nnPFbEBS+JT3xS246iyyM1PiPc=; b=eC0F/VZYOno0LO4GCcfVz7p0+ShlSmWNpAKj/nX31/KRXDXu4yGfw1w+/iRAVvXNZk FZp3deJeZh5W3hxNWW14X+zTRDtoLzcBx2vkgFP4iBEIo+uTsIUlOtWxA/dtOirjuVh1 jhh9ocqWGL/BzRAVWmN4GqzdMH5/USQuI+Cj9O6ntC5Uk6sEtJfzRZQL2CjWVETQv1KY pegbnXcHH/iExjahf52tCs8DXaRYj070aPJdszabe/xkb3ptQh9EOxvIaQ0TLdsuj6p9 w0XDUc0gI0a/jwEr35y6n59SYgYCcO6VHFsU2q+r1tl9j1i1te8TsqmWOVcsqsbEVQnW XeKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=/UNBWPl4t+GAChOB0nnPFbEBS+JT3xS246iyyM1PiPc=; b=dQKLNRBReQI4kRP9/5MC2aBFBzc1fj1XJYZue/aQE2rz2tp6UbfHUAumVLRj0N+gz4 aRgU7Tx5R1DRH5ZHPWyBtP9ZhtEywfaHFTbO4T7voF6ofLScH7BmYVhjl+c5oFgRkZC6 dorDRCZyPc6DrHEawBC6HXeZ9a+VoWnxx19pI31dQGs/wCWH+vQ+eURlZT/Y0Yi7blFp XW0L4wGFnw4G0Io1KnkLH7cx8PUEKSk9MjjqadlEMkHFR8c8XeuEEenPwALeZwuzodRK GTyVYqwzd6CNZA7e8Zp/ACbBAZ+908w3sSVMdoFGN1kjepYe7C/oHzH7HWxK8WRUrBmJ EAMg== X-Gm-Message-State: AOAM533u39ErTkmNc9iK6EgCSBaFN7j4IIJezS//UCJqnU0bj5fyyOAv obH9axvVQck4mh5XRFCr/kh9FAywPhc= X-Google-Smtp-Source: ABdhPJzHgzNqPQAMqDYu/ZfYqVPamHTb0AsqLLDZd/eoCT+ue4xQs3Sg5X3KTQ2s16UjGfgetBIXDw== X-Received: by 2002:a05:6000:1a87:: with SMTP id f7mr252739wry.172.1626112563852; Mon, 12 Jul 2021 10:56:03 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id d18sm378656wmp.46.2021.07.12.10.56.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Jul 2021 10:56:03 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Mon, 12 Jul 2021 17:55:50 +0000 Subject: [PATCH v8 09/15] unpack-trees: unpack sparse directory entries Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee During unpack_callback(), index entries are compared against tree entries. These are matched according to names and types. One goal is to decide if we should recurse into subtrees or simply operate on one index entry. In the case of a sparse-directory entry, we do not want to recurse into that subtree and instead simply compare the trees. In some cases, we might want to perform a merge operation on the entry, such as during 'git checkout ' which wants to replace a sparse tree entry with the tree for that path at the target commit. We extend the logic within unpack_single_entry() to create a sparse-directory entry in this case, and then that is sent to call_unpack_fn(). There are some subtleties in this process. For instance, we need to update find_cache_entry() to allow finding a sparse-directory entry that exactly matches a given path. Use the new helper method sparse_dir_matches_path() for this. We also need to ignore conflict markers in the case that the entries correspond to directories and we already have a sparse directory entry. Signed-off-by: Derrick Stolee --- unpack-trees.c | 107 +++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 99 insertions(+), 8 deletions(-) diff --git a/unpack-trees.c b/unpack-trees.c index d26386ce8b2..62ccd5a0ff6 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -1052,13 +1052,15 @@ static struct cache_entry *create_ce_entry(const struct traverse_info *info, const struct name_entry *n, int stage, struct index_state *istate, - int is_transient) + int is_transient, + int is_sparse_directory) { size_t len = traverse_path_len(info, tree_entry_len(n)); + size_t alloc_len = is_sparse_directory ? len + 1 : len; struct cache_entry *ce = is_transient ? - make_empty_transient_cache_entry(len, NULL) : - make_empty_cache_entry(istate, len); + make_empty_transient_cache_entry(alloc_len, NULL) : + make_empty_cache_entry(istate, alloc_len); ce->ce_mode = create_ce_mode(n->mode); ce->ce_flags = create_ce_flags(stage); @@ -1067,6 +1069,13 @@ static struct cache_entry *create_ce_entry(const struct traverse_info *info, /* len+1 because the cache_entry allocates space for NUL */ make_traverse_path(ce->name, len + 1, info, n->path, n->pathlen); + if (is_sparse_directory) { + ce->name[len] = '/'; + ce->name[len + 1] = '\0'; + ce->ce_namelen++; + ce->ce_flags |= CE_SKIP_WORKTREE; + } + return ce; } @@ -1085,10 +1094,17 @@ static int unpack_single_entry(int n, unsigned long mask, struct unpack_trees_options *o = info->data; unsigned long conflicts = info->df_conflicts | dirmask; - /* Do we have *only* directories? Nothing to do */ if (mask == dirmask && !src[0]) return 0; + /* + * When we have a sparse directory entry for src[0], + * then this isn't necessarily a directory-file conflict. + */ + if (mask == dirmask && src[0] && + S_ISSPARSEDIR(src[0]->ce_mode)) + conflicts = 0; + /* * Ok, we've filled in up to any potential index entry in src[0], * now do the rest. @@ -1118,7 +1134,9 @@ static int unpack_single_entry(int n, unsigned long mask, * not stored in the index. otherwise construct the * cache entry from the index aware logic. */ - src[i + o->merge] = create_ce_entry(info, names + i, stage, &o->result, o->merge); + src[i + o->merge] = create_ce_entry(info, names + i, stage, + &o->result, o->merge, + bit & dirmask); } if (o->merge) { @@ -1222,16 +1240,71 @@ static int find_cache_pos(struct traverse_info *info, return -1; } +/* + * Given a sparse directory entry 'ce', compare ce->name to + * info->name + '/' + p->path + '/' if info->name is non-empty. + * Compare ce->name to p->path + '/' otherwise. Note that + * ce->name must end in a trailing '/' because it is a sparse + * directory entry. + */ +static int sparse_dir_matches_path(const struct cache_entry *ce, + struct traverse_info *info, + const struct name_entry *p) +{ + assert(S_ISSPARSEDIR(ce->ce_mode)); + assert(ce->name[ce->ce_namelen - 1] == '/'); + + if (info->namelen) + return ce->ce_namelen == info->namelen + p->pathlen + 2 && + ce->name[info->namelen] == '/' && + !strncmp(ce->name, info->name, info->namelen) && + !strncmp(ce->name + info->namelen + 1, p->path, p->pathlen); + return ce->ce_namelen == p->pathlen + 1 && + !strncmp(ce->name, p->path, p->pathlen); +} + static struct cache_entry *find_cache_entry(struct traverse_info *info, const struct name_entry *p) { + struct cache_entry *ce; int pos = find_cache_pos(info, p->path, p->pathlen); struct unpack_trees_options *o = info->data; if (0 <= pos) return o->src_index->cache[pos]; - else + + /* + * Check for a sparse-directory entry named "path/". + * Due to the input p->path not having a trailing + * slash, the negative 'pos' value overshoots the + * expected position, hence "-2" instead of "-1". + */ + pos = -pos - 2; + + if (pos < 0 || pos >= o->src_index->cache_nr) return NULL; + + /* + * Due to lexicographic sorting and sparse directory + * entried ending with a trailing slash, our path as a + * sparse directory (e.g "subdir/") and our path as a + * file (e.g. "subdir") might be separated by other + * paths (e.g. "subdir-"). + */ + while (pos >= 0) { + ce = o->src_index->cache[pos]; + + if (strncmp(ce->name, p->path, p->pathlen)) + return NULL; + + if (S_ISSPARSEDIR(ce->ce_mode) && + sparse_dir_matches_path(ce, info, p)) + return ce; + + pos--; + } + + return NULL; } static void debug_path(struct traverse_info *info) @@ -1266,6 +1339,21 @@ static void debug_unpack_callback(int n, debug_name_entry(i, names + i); } +/* + * Returns true if and only if the given cache_entry is a + * sparse-directory entry that matches the given name_entry + * from the tree walk at the given traverse_info. + */ +static int is_sparse_directory_entry(struct cache_entry *ce, + struct name_entry *name, + struct traverse_info *info) +{ + if (!ce || !name || !S_ISSPARSEDIR(ce->ce_mode)) + return 0; + + return sparse_dir_matches_path(ce, info, name); +} + /* * Note that traverse_by_cache_tree() duplicates some logic in this function * without actually calling it. If you change the logic here you may need to @@ -1352,9 +1440,12 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str } } - if (traverse_trees_recursive(n, dirmask, mask & ~dirmask, - names, info) < 0) + if (!is_sparse_directory_entry(src[0], names, info) && + traverse_trees_recursive(n, dirmask, mask & ~dirmask, + names, info) < 0) { return -1; + } + return mask; } From patchwork Mon Jul 12 17:55:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12371829 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 07BA6C07E99 for ; Mon, 12 Jul 2021 17:56:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E4E8461154 for ; Mon, 12 Jul 2021 17:56:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235863AbhGLR7E (ORCPT ); Mon, 12 Jul 2021 13:59:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48128 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235801AbhGLR6y (ORCPT ); Mon, 12 Jul 2021 13:58:54 -0400 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CEA07C0613DD for ; Mon, 12 Jul 2021 10:56:05 -0700 (PDT) Received: by mail-wm1-x334.google.com with SMTP id n4so1559102wms.1 for ; Mon, 12 Jul 2021 10:56:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=KMWoXCHAUCRcXSFT+haQKKmCiVf5GcUjw8ICiM/EavY=; b=TZYml90NEwXHceI1TvtaChZBI+QMhnlXTyHeSVDRPWGwzkMK6nu0HAUfaG2j8bggy4 XledFukIJH99JL/qyLjcbBqIuNm5OFZ63tMMO/pSj3tw86kIAyrpEyhXSlpkUUzl74zJ fB0xrGApTnqp3QyjbEe+i7hjHggNBX8awf6rpXv60L0uUGG8uwlcjjbSx4oNKFF/b6Ou kvGwRQ9v31g9MohkZE0lh+PBiYthIbWXGJFJ7upfGO/DwDgSkGpT/+QSLycLe2n07g9f fQ9C3x2jsASJNWzHpjzY62mC7VGrJ/xyoZK6rZP4UQO2YBeKwTAr6I/skVsH2W3UY8SZ qvrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=KMWoXCHAUCRcXSFT+haQKKmCiVf5GcUjw8ICiM/EavY=; b=OKCJKiwRcf23/EBI4IjeNPaNcAueubtZVk5Z1fcdqK1yar34fOr23/T+p2l1HvLW7m KTSbz4igIe7jNum0tqIrensC4bci1KRhJb1y2ix3D3iZjqbakc82z4CBPDXphcFjFLLq Hju4XRtqfs3UkEyVCmQEOtM/xFT5Jtfx+K6IR/7nwfYUQceCjF9oWVJGwVTu/XGsCWz2 yR7+nsgKphXzagBd+0Z63ZextZsr5sZk/iPKRinOoZiec3UktwaJNu9ORU1yhvbRloX7 XA+DWqTZNvcD2hjul+cd9Eu/eKJf3daS9jfkNAngZscQ4bOk4V/QmjXVVnXqToSoxI2a KNnA== X-Gm-Message-State: AOAM530bo+Km5FAIXX1XKOfNTDuRNA/4jy9Nalxtc6l9b4Mpg7b0Uc12 iZf2HDT3ngxQYc33i5FHWisDpXDySZQ= X-Google-Smtp-Source: ABdhPJziIa43uS8/wKNdYgAvCZuSU96YhApB0v8Lhvir/KHuiu/Qfk3DpTbS9UpEgWcKu/64Vm+IvQ== X-Received: by 2002:a1c:59c9:: with SMTP id n192mr426244wmb.81.1626112564513; Mon, 12 Jul 2021 10:56:04 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id t3sm133112wmi.14.2021.07.12.10.56.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Jul 2021 10:56:04 -0700 (PDT) Message-Id: <76c7528f78f0be2ed0a9fc0428f715b36fd451ba.1626112556.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 12 Jul 2021 17:55:51 +0000 Subject: [PATCH v8 10/15] dir.c: accept a directory as part of cone-mode patterns Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee When we have sparse directory entries in the index, we want to compare that directory against sparse-checkout patterns. Those pattern matching algorithms are built expecting a file path, not a directory path. This is especially important in the "cone mode" patterns which will match files that exist within the "parent directories" as well as the recursive directory matches. If path_matches_pattern_list() is given a directory, we can add a fake filename ("-") to the directory and get the same results as before, assuming we are in cone mode. Since sparse index requires cone mode patterns, this is an acceptable assumption. Signed-off-by: Derrick Stolee --- dir.c | 24 +++++++++++++++++++----- 1 file changed, 19 insertions(+), 5 deletions(-) diff --git a/dir.c b/dir.c index ebe5ec046e0..0c5264b3b20 100644 --- a/dir.c +++ b/dir.c @@ -1376,7 +1376,7 @@ enum pattern_match_result path_matches_pattern_list( struct path_pattern *pattern; struct strbuf parent_pathname = STRBUF_INIT; int result = NOT_MATCHED; - const char *slash_pos; + size_t slash_pos; if (!pl->use_cone_patterns) { pattern = last_matching_pattern_from_list(pathname, pathlen, basename, @@ -1397,21 +1397,35 @@ enum pattern_match_result path_matches_pattern_list( strbuf_addch(&parent_pathname, '/'); strbuf_add(&parent_pathname, pathname, pathlen); + /* + * Directory entries are matched if and only if a file + * contained immediately within them is matched. For the + * case of a directory entry, modify the path to create + * a fake filename within this directory, allowing us to + * use the file-base matching logic in an equivalent way. + */ + if (parent_pathname.len > 0 && + parent_pathname.buf[parent_pathname.len - 1] == '/') { + slash_pos = parent_pathname.len - 1; + strbuf_add(&parent_pathname, "-", 1); + } else { + const char *slash_ptr = strrchr(parent_pathname.buf, '/'); + slash_pos = slash_ptr ? slash_ptr - parent_pathname.buf : 0; + } + if (hashmap_contains_path(&pl->recursive_hashmap, &parent_pathname)) { result = MATCHED_RECURSIVE; goto done; } - slash_pos = strrchr(parent_pathname.buf, '/'); - - if (slash_pos == parent_pathname.buf) { + if (!slash_pos) { /* include every file in root */ result = MATCHED; goto done; } - strbuf_setlen(&parent_pathname, slash_pos - parent_pathname.buf); + strbuf_setlen(&parent_pathname, slash_pos); if (hashmap_contains_path(&pl->parent_hashmap, &parent_pathname)) { result = MATCHED; From patchwork Mon Jul 12 17:55:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12371837 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 324EDC07E9A for ; Mon, 12 Jul 2021 17:56:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 20D086120A for ; Mon, 12 Jul 2021 17:56:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235933AbhGLR7S (ORCPT ); Mon, 12 Jul 2021 13:59:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48138 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235821AbhGLR6z (ORCPT ); Mon, 12 Jul 2021 13:58:55 -0400 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 58FB0C0613DD for ; Mon, 12 Jul 2021 10:56:06 -0700 (PDT) Received: by mail-wr1-x42c.google.com with SMTP id g16so16901732wrw.5 for ; Mon, 12 Jul 2021 10:56:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=IX4iFOSyGCMqp4f79yJ4MnYs1wd2+RwP9RtTHQgSjwg=; b=LlzaNPlqb/pX4KNSVLaZLvYT4f2LF0ejvfRDeGPg5W5RSNbfTVYRDdNhduaMdEp2zL a8ZX5PIn9+rCwXXuRoa9FgIeWpAMvNYi8ise786TS7jNiRYw7T4IMOYG7afhLt06JZf4 g4v6+riXAq0nkqd6DtG3zZQ6p2kddMGevMf00Mn3ypTQwbPs/YgN6NYgEnMi/D42Eh/U GX/dWXhcFPmNA1xP8lKkrNE8audcwr1kWYfSz/nd6z6XNkAyTr5crHkxVtlmsvczlBkZ 8lcIUVANg/5yoD/3YtkgWgAo68YRMwhn/6bQHOFlfUG6q44e3AxIkO80as5M3YKsYyG2 eFyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=IX4iFOSyGCMqp4f79yJ4MnYs1wd2+RwP9RtTHQgSjwg=; b=Lt/Kky8lBw8PFuHUyI9X0+/vZgQlQvsUx00Y9p05gKX7Hvgu2dNar3pSSwTDEyQL+u lS+/IWz1QiNOtBgo4+ayHfv5yYaduBRXP85CKg3/g9vZi8Ow8ucWvBazc2trMr2W5evZ ZjF+R4BAd5RNhgiJHIwyxDoJx5sZw45HP4fuba2G1AxmbFBqbQ5Zdgky0K2Km1Hifzeb qocwR+AA8blnvlsrv07iLA9Ox4Jdf2GuzEFYCbj6D2/cQMcG9Og+B3eRl1zMgd9nOXHz 9umKsbyLQUQZaW6MmiwhrwDmlsHw1XZZaDMKvXcsRSapgXp6ownRJcIQhtXUmynMgoSs zyqA== X-Gm-Message-State: AOAM531P4itVtT6rwRcqvmEVumMVzo7C6f3O7ONUYP+toLuuLWcH4j4W a0tdmnJug9zIELCIvn/8HUwAD6+lNp8= X-Google-Smtp-Source: ABdhPJyOg55V24y4i6RVcGP+z4drLp4Y/0Tn79qT5zyhUDVUdtmy0qyCsAGVJA2J2mrJa2Yofgy0vQ== X-Received: by 2002:a5d:4449:: with SMTP id x9mr277092wrr.52.1626112565047; Mon, 12 Jul 2021 10:56:05 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n2sm15293991wrx.40.2021.07.12.10.56.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Jul 2021 10:56:04 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Mon, 12 Jul 2021 17:55:52 +0000 Subject: [PATCH v8 11/15] diff-lib: handle index diffs with sparse dirs Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee While comparing an index to a tree, we may see a sparse directory entry. In this case, we should compare that portion of the tree to the tree represented by that entry. This could include a new tree which needs to be expanded to a full list of added files. It could also include an existing tree, in which case all of the changes inside are important to describe, including the modifications, additions, and deletions. Note that the case where the tree has a path and the index does not remains identical to before: the lack of a cache entry is the same with a sparse index. Use diff_tree_oid() appropriately to compute the diff. Signed-off-by: Derrick Stolee --- diff-lib.c | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/diff-lib.c b/diff-lib.c index c2ac9250fe9..f9eadc4fc1a 100644 --- a/diff-lib.c +++ b/diff-lib.c @@ -325,6 +325,11 @@ static void show_new_file(struct rev_info *revs, unsigned dirty_submodule = 0; struct index_state *istate = revs->diffopt.repo->index; + if (new_file && S_ISSPARSEDIR(new_file->ce_mode)) { + diff_tree_oid(NULL, &new_file->oid, new_file->name, &revs->diffopt); + return; + } + /* * New file in the index: it might actually be different in * the working tree. @@ -347,6 +352,20 @@ static int show_modified(struct rev_info *revs, unsigned dirty_submodule = 0; struct index_state *istate = revs->diffopt.repo->index; + assert(S_ISSPARSEDIR(old_entry->ce_mode) == + S_ISSPARSEDIR(new_entry->ce_mode)); + + /* + * If both are sparse directory entries, then expand the + * modifications to the file level. If only one was a sparse + * directory, then they appear as an add and delete instead of + * a modification. + */ + if (S_ISSPARSEDIR(new_entry->ce_mode)) { + diff_tree_oid(&old_entry->oid, &new_entry->oid, new_entry->name, &revs->diffopt); + return 0; + } + if (get_stat_data(istate, new_entry, &oid, &mode, cached, match_missing, &dirty_submodule, &revs->diffopt) < 0) { if (report_missing) From patchwork Mon Jul 12 17:55:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12371831 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C5A5AC07E9C for ; Mon, 12 Jul 2021 17:56:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A20CB611C1 for ; Mon, 12 Jul 2021 17:56:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235871AbhGLR7G (ORCPT ); Mon, 12 Jul 2021 13:59:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48142 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235834AbhGLR6z (ORCPT ); Mon, 12 Jul 2021 13:58:55 -0400 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EB9C6C0613E9 for ; Mon, 12 Jul 2021 10:56:06 -0700 (PDT) Received: by mail-wm1-x332.google.com with SMTP id i2-20020a05600c3542b02902058529ea07so579209wmq.3 for ; Mon, 12 Jul 2021 10:56:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=7vhfreIm1fgP496Y8tHbzJwdwnJTTb9sQWNvLVkIPvU=; b=na7mHTYLeb5eXEfo5loSp+nDpdpcQPSRbrYEs1hSdnm0vdfUncVZEgIo+la7kEVvck d3B5/v5QkZOdRUBebyc0/BCCnGe5Qb6WKh/PL7E5KGu9CiSmm3l4IXz/49ZPYEQNhBcu ruDFRUgj804fI15eWG4beBciY+w1NiEL9T46aEhI2YqJlIccOc7XoN5RzxQZvy3RMBh6 QribvcTxFM1FPHeEBdm1FUJ7Uf7tVNXePh2rhlF2b3zYoxA2G9iyzSZWOGRulTdPKfS/ IsAKe7lE2+pc6fiqy2/7HrOBWhyFfmHz9aJRpHiDqxxYli2fMtkgTe4zxGeM69qGIdp/ 2adg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=7vhfreIm1fgP496Y8tHbzJwdwnJTTb9sQWNvLVkIPvU=; b=VlxMugafOiBjye4xc03pONNAtNzYcGTvRKu/zEvSN3f88S4zgS4jJ1Q74hwgjTgiUn +8e/GAQUqZtzExfPnNvZHDEopq97+9x2HB73Ww0CT29BZ4vr8C++S4nUtudoTUdEwPrZ FgzLt14H79mLPe+k0V4p4mfMpzQZOkYPLiBhH8Q+LqHWBVHK18zeiAaU95hdl4y+tagr rvDllsG87gNYiun2AfZ2ltuwk13XbCEc5bhgN43nBjX1jns1DYzK1jD/WuEN3pd7c0qa 4pjEx3J73kkK/IycG6HUF9hvhKAP7KoEmpH63sb17nh7TRPMmOnI+KHDUtXb5ieJsxi3 Ku4Q== X-Gm-Message-State: AOAM532IZD/tJ61lCwFGCwEXqPm9Llj7V4Mfe1mSADA1HFJNd1G0cne6 +DD5IUkDrPkCxxg2KsngMj7ZSgXMh3U= X-Google-Smtp-Source: ABdhPJz7j4QLDK5HGrf+z+9hZysjNBcD0cjFmV3sba8gFxRmReeTtyvuVV6Jppm2Q/neZbHloIkhcA== X-Received: by 2002:a1c:e90d:: with SMTP id q13mr405350wmc.163.1626112565650; Mon, 12 Jul 2021 10:56:05 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o29sm11515611wms.13.2021.07.12.10.56.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Jul 2021 10:56:05 -0700 (PDT) Message-Id: <2b72cc2d98505a635d902ab1be66bd1af763a99d.1626112556.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 12 Jul 2021 17:55:53 +0000 Subject: [PATCH v8 12/15] status: skip sparse-checkout percentage with sparse-index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee 'git status' began reporting a percentage of populated paths when sparse-checkout is enabled in 051df3cf (wt-status: show sparse checkout status as well, 2020-07-18). This percentage is incorrect when the index has sparse directories. It would also be expensive to calculate as we would need to parse trees to count the total number of possible paths. Avoid the expensive computation by simplifying the output to only report that a sparse checkout exists, without the percentage. This change is the reason we use 'git status --porcelain=v2' in t1092-sparse-checkout-compatibility.sh. We don't want to ensure that this message is equal across both modes, but instead just the important information about staged, modified, and untracked files are compared. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 8 ++++++++ wt-status.c | 14 +++++++++++--- wt-status.h | 1 + 3 files changed, 20 insertions(+), 3 deletions(-) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 2269f44e033..375b0d35565 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -218,6 +218,14 @@ test_expect_success 'status with options' ' test_all_match git status --porcelain=v2 -uno ' +test_expect_success 'status reports sparse-checkout' ' + init_repos && + git -C sparse-checkout status >full && + git -C sparse-index status >sparse && + test_i18ngrep "You are in a sparse checkout with " full && + test_i18ngrep "You are in a sparse checkout." sparse +' + test_expect_success 'add, commit, checkout' ' init_repos && diff --git a/wt-status.c b/wt-status.c index 42b67357169..96db3e74962 100644 --- a/wt-status.c +++ b/wt-status.c @@ -1493,9 +1493,12 @@ static void show_sparse_checkout_in_use(struct wt_status *s, if (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_DISABLED) return; - status_printf_ln(s, color, - _("You are in a sparse checkout with %d%% of tracked files present."), - s->state.sparse_checkout_percentage); + if (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_SPARSE_INDEX) + status_printf_ln(s, color, _("You are in a sparse checkout.")); + else + status_printf_ln(s, color, + _("You are in a sparse checkout with %d%% of tracked files present."), + s->state.sparse_checkout_percentage); wt_longstatus_print_trailer(s); } @@ -1653,6 +1656,11 @@ static void wt_status_check_sparse_checkout(struct repository *r, return; } + if (r->index->sparse_index) { + state->sparse_checkout_percentage = SPARSE_CHECKOUT_SPARSE_INDEX; + return; + } + for (i = 0; i < r->index->cache_nr; i++) { struct cache_entry *ce = r->index->cache[i]; if (ce_skip_worktree(ce)) diff --git a/wt-status.h b/wt-status.h index 0d32799b28e..ab9cc9d8f03 100644 --- a/wt-status.h +++ b/wt-status.h @@ -78,6 +78,7 @@ enum wt_status_format { }; #define SPARSE_CHECKOUT_DISABLED -1 +#define SPARSE_CHECKOUT_SPARSE_INDEX -2 struct wt_status_state { int merge_in_progress; From patchwork Mon Jul 12 17:55:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12371839 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43964C11F67 for ; Mon, 12 Jul 2021 17:56:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2AD3261222 for ; Mon, 12 Jul 2021 17:56:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235919AbhGLR7R (ORCPT ); Mon, 12 Jul 2021 13:59:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48148 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235733AbhGLR64 (ORCPT ); Mon, 12 Jul 2021 13:58:56 -0400 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 823F2C0613EE for ; Mon, 12 Jul 2021 10:56:07 -0700 (PDT) Received: by mail-wr1-x436.google.com with SMTP id a13so26772194wrf.10 for ; Mon, 12 Jul 2021 10:56:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=jtMJL+eY3PCMeOHMJxHTj2zHIpoZgL49ge9VmbDoBBw=; b=N9wKRsfZXCPukYj41RPNZBzLdJmHE/gbmv1p2QEMHSd5NHHPsKj40OAka+mNycVG7t mxwxgCYXhU1Q3O+N40gXcr9VRgIGaANIYvPGDNOiOiJEhbH5kbH+T6HMoIUe969WcExb F+GRjqL8jl/zGXrUWdxukqxWRN8BJx6k8j+oxhUwAXsVPP8xPBPwLZTcIXC91XGegNGL UKNXYLFdR8Vqu4lGzjAnyr7X0Z9EE87HrsRkepdRoqUU7dNF6usLI984a2fsVp9vEadW ZPXkRNl9rId4LxgeoYFU4zOnori2eJFHvYIwd2pgxYf8yzUYgG00vwAlMJnbMhXiWfK+ VxeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=jtMJL+eY3PCMeOHMJxHTj2zHIpoZgL49ge9VmbDoBBw=; b=bENpaepC30CVYgz2mIsI3fa+ZO07n0aUPTAN6OXqLOOBNjIEIxXbalzWWxThLCqM/q rNvOCrn/hrs0hsLGMX2ONAz4tMIdH0/4xKuwUKu/6y8pezrXLqPUQMlEBFnwzSAxZ0qh kmcW4MHwJSZjv2r6HQm3QsxknIVYMVXiauhVaxfbWsVZfED34tYoMBzOnz07D4n0y+c1 CI+N10B4I+vzhPMNp6Tyr/bNWQSX5gZ7XqxRXh9rgvoYkHkgkfC/LcadXoHJqUMDVbWy Yvt19RF0dNTs95k06OvmM3tyPnn9K9A1ZyaTjdTtc2jgzxeWakgbuf6OwnV3DSH/d6FV RGTQ== X-Gm-Message-State: AOAM533B0f9o9uGXHN2jzCCXKcxyvXbyJhmKY4LNBN+COKeEji5ODGVP ymuOmKblB2oDv1YVkeOSCl4sdWrIgVQ= X-Google-Smtp-Source: ABdhPJyQ8Rtb5Ygid1+LSEiqB4yZiYpWZ/D9B9Od59YV4H8oRmH7wIPoSAWef21cuzYI3EJkM7jp2g== X-Received: by 2002:adf:dfc9:: with SMTP id q9mr215788wrn.117.1626112566193; Mon, 12 Jul 2021 10:56:06 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id t5sm8775311wrw.38.2021.07.12.10.56.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Jul 2021 10:56:05 -0700 (PDT) Message-Id: <1c1feef37339ff19ebfcdaf87a29d7adae73aa06.1626112556.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 12 Jul 2021 17:55:54 +0000 Subject: [PATCH v8 13/15] status: use sparse-index throughout Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee By testing 'git -c core.fsmonitor= status -uno', we can check for the simplest index operations that can be made sparse-aware. The necessary implementation details are already integrated with sparse-checkout, so modify command_requires_full_index to be zero for cmd_status(). In refresh_index(), we loop through the index entries to refresh their stat() information. However, sparse directories have no stat() information to populate. Ignore these entries. This allows 'git status' to no longer expand a sparse index to a full one. This is further tested by dropping the "-uno" option and adding an untracked file into the worktree. The performance test p2000-sparse-checkout-operations.sh demonstrates these improvements: Test HEAD~1 HEAD ----------------------------------------------------------------------------- 2000.2: git status (full-index-v3) 0.31(0.30+0.05) 0.31(0.29+0.06) +0.0% 2000.3: git status (full-index-v4) 0.31(0.29+0.07) 0.34(0.30+0.08) +9.7% 2000.4: git status (sparse-index-v3) 2.35(2.28+0.10) 0.04(0.04+0.05) -98.3% 2000.5: git status (sparse-index-v4) 2.35(2.24+0.15) 0.05(0.04+0.06) -97.9% Note that since HEAD~1 was expanding the sparse index by parsing trees, it was artificially slower than the full index case. Thus, the 98% improvement is misleading, and instead we should celebrate the 0.34s to 0.05s improvement of 85%. This is more indicative of the peformance gains we are expecting by using a sparse index. Note: we are dropping the assignment of core.fsmonitor here. This is not necessary for the test script as we are not altering the config any other way. Correct integration with FS Monitor will be validated in later changes. Signed-off-by: Derrick Stolee --- builtin/commit.c | 3 +++ read-cache.c | 10 ++++++++-- t/t1092-sparse-checkout-compatibility.sh | 13 +++++++++---- 3 files changed, 20 insertions(+), 6 deletions(-) diff --git a/builtin/commit.c b/builtin/commit.c index 190d215d43b..12f51db158a 100644 --- a/builtin/commit.c +++ b/builtin/commit.c @@ -1510,6 +1510,9 @@ int cmd_status(int argc, const char **argv, const char *prefix) if (argc == 2 && !strcmp(argv[1], "-h")) usage_with_options(builtin_status_usage, builtin_status_options); + prepare_repo_settings(the_repository); + the_repository->settings.command_requires_full_index = 0; + status_init_config(&s, git_status_config); argc = parse_options(argc, argv, prefix, builtin_status_options, diff --git a/read-cache.c b/read-cache.c index 0c3ac3cefc0..6a1337cc905 100644 --- a/read-cache.c +++ b/read-cache.c @@ -1585,8 +1585,7 @@ int refresh_index(struct index_state *istate, unsigned int flags, */ preload_index(istate, pathspec, 0); trace2_region_enter("index", "refresh", NULL); - /* TODO: audit for interaction with sparse-index. */ - ensure_full_index(istate); + for (i = 0; i < istate->cache_nr; i++) { struct cache_entry *ce, *new_entry; int cache_errno = 0; @@ -1601,6 +1600,13 @@ int refresh_index(struct index_state *istate, unsigned int flags, if (ignore_skip_worktree && ce_skip_worktree(ce)) continue; + /* + * If this entry is a sparse directory, then there isn't + * any stat() information to update. Ignore the entry. + */ + if (S_ISSPARSEDIR(ce->ce_mode)) + continue; + if (pathspec && !ce_path_match(istate, ce, pathspec, seen)) filtered = 1; diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 375b0d35565..751f397cc7f 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -511,12 +511,17 @@ test_expect_success 'sparse-index is expanded and converted back' ' GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ git -C sparse-index -c core.fsmonitor="" reset --hard && test_region index convert_to_sparse trace2.txt && - test_region index ensure_full_index trace2.txt && + test_region index ensure_full_index trace2.txt +' - rm trace2.txt && +test_expect_success 'sparse-index is not expanded' ' + init_repos && + + rm -f trace2.txt && + echo >>sparse-index/untracked.txt && GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ - git -C sparse-index -c core.fsmonitor="" status -uno && - test_region index ensure_full_index trace2.txt + git -C sparse-index status && + test_region ! index ensure_full_index trace2.txt ' test_done From patchwork Mon Jul 12 17:55:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12371833 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE328C07E99 for ; Mon, 12 Jul 2021 17:56:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AF38C61206 for ; Mon, 12 Jul 2021 17:56:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235877AbhGLR7I (ORCPT ); Mon, 12 Jul 2021 13:59:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48160 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235857AbhGLR66 (ORCPT ); Mon, 12 Jul 2021 13:58:58 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1C2B1C061787 for ; Mon, 12 Jul 2021 10:56:08 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id d12so26171255wre.13 for ; Mon, 12 Jul 2021 10:56:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=hL/gylqY2yIMHmG/78ZgzlK8FtrbEO0n8KCJ7VPSk20=; b=EbFZ84x+/KR16yGPRTZ1iieYodvoxoLny2r67oiLgYkmxn2LKEy3LmigVPUo+6+QIZ ETBNL8V8r7HiUfCdK4QL73zruw0Z/hO6pNc0O545v0EfWmfaS3qx5Jz182LGVar6mM8r uaBSJvT1emh/tynLf0YftzLVIhj94hOCxXA6bTwOv0sNVAQ/fGzBrJijCbujlLixT61n m4b0J7Kvw4qYrMcdu+/JAokbFoEKsUlTkQrg9hjIp8ATQOOeqhA6X1lGZEgmmdyt4G2v RbRZWUpGqQ0MdE2dg8rBwn9jmjHEnq6seJvZJxBPnv95sSvKq9CT2fM1CtWcl2o/dP2W Nokg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=hL/gylqY2yIMHmG/78ZgzlK8FtrbEO0n8KCJ7VPSk20=; b=qRBpwZzFuufGyWku/WY65IEDdKKeRMTD+XPmEZU5pLOPxlOzz7Wt15In/MryMHEOlT +Xh0ADxqbPpuDliQ0BvP9pS+vYLQo9dSa1j6LHCQUPnVrD9NnWidXfzSqdVA63q1yj1k qgmm1OFIltisYhaPxTZ/PXEasaWpLizhFv+rGrqEgCg1ucEGT9y/uTzCkzBrFYXGfFPO 46AJJw2yY/1fNlMgfYfipodjU3+nxzuHdmDkXzxydjJwr0IfuRTRCdW+ByiZC0/+vRF2 OrNpPNQ8ZSXKU2wswQcqZikegtLYlH665gxfUtZHfBQ2lNjNH61Cec7V6Hk1IuTA+qaO yKYA== X-Gm-Message-State: AOAM5322KdQAUlIJ6QH04QRRfloh+7Ms4c8kPCgYKilSm14WvITElo/J tUr1qIOYU+4O2r6K3vAW7U4oRu90Hb8= X-Google-Smtp-Source: ABdhPJxR1jLA8IztJyHI89xz/ejUGjR9/eMS3avy/XhE4GGrT8BKVxQtjxza9kgF0XDPzXiWvA5Sew== X-Received: by 2002:adf:fbce:: with SMTP id d14mr252331wrs.236.1626112566759; Mon, 12 Jul 2021 10:56:06 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id k20sm14062446wrd.70.2021.07.12.10.56.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Jul 2021 10:56:06 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Mon, 12 Jul 2021 17:55:55 +0000 Subject: [PATCH v8 14/15] wt-status: expand added sparse directory entries Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee It is difficult, but possible, to get into a state where we intend to add a directory that is outside of the sparse-checkout definition. Add a test to t1092-sparse-checkout-compatibility.sh that demonstrates this using a combination of 'git reset --mixed' and 'git checkout --orphan'. This test failed before because the output of 'git status --porcelain=v2' would not match on the lines for folder1/: * The sparse-checkout repo (with a full index) would output each path name that is intended to be added. * The sparse-index repo would only output that "folder1/" is staged for addition. The status should report the full list of files to be added, and so this sparse-directory entry should be expanded to a full list when reaching it inside the wt_status_collect_changes_initial() method. Use read_tree_at() to assist. Somehow, this loop over the cache entries was not guarded by ensure_full_index() as intended. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 33 +++++++++++++++ wt-status.c | 51 ++++++++++++++++++++++++ 2 files changed, 84 insertions(+) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 751f397cc7f..2394c36d881 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -524,4 +524,37 @@ test_expect_success 'sparse-index is not expanded' ' test_region ! index ensure_full_index trace2.txt ' +test_expect_success 'reset mixed and checkout orphan' ' + init_repos && + + test_all_match git checkout rename-out-to-in && + + # Sparse checkouts do not agree with full checkouts about + # how to report a directory/file conflict during a reset. + # This command would fail with test_all_match because the + # full checkout reports "T folder1/0/1" while a sparse + # checkout reports "D folder1/0/1". This matches because + # the sparse checkouts skip "adding" the other side of + # the conflict. + test_sparse_match git reset --mixed HEAD~1 && + test_sparse_match test-tool read-cache --table --expand && + test_sparse_match git status --porcelain=v2 && + + # At this point, sparse-checkouts behave differently + # from the full-checkout. + test_sparse_match git checkout --orphan new-branch && + test_sparse_match test-tool read-cache --table --expand && + test_sparse_match git status --porcelain=v2 +' + +test_expect_success 'add everything with deep new file' ' + init_repos && + + run_on_sparse git sparse-checkout set deep/deeper1/deepest && + + run_on_all touch deep/deeper1/x && + test_all_match git add . && + test_all_match git status --porcelain=v2 +' + test_done diff --git a/wt-status.c b/wt-status.c index 96db3e74962..0317baef87e 100644 --- a/wt-status.c +++ b/wt-status.c @@ -657,6 +657,36 @@ static void wt_status_collect_changes_index(struct wt_status *s) clear_pathspec(&rev.prune_data); } +static int add_file_to_list(const struct object_id *oid, + struct strbuf *base, const char *path, + unsigned int mode, void *context) +{ + struct string_list_item *it; + struct wt_status_change_data *d; + struct wt_status *s = context; + struct strbuf full_name = STRBUF_INIT; + + if (S_ISDIR(mode)) + return READ_TREE_RECURSIVE; + + strbuf_add(&full_name, base->buf, base->len); + strbuf_addstr(&full_name, path); + it = string_list_insert(&s->change, full_name.buf); + d = it->util; + if (!d) { + CALLOC_ARRAY(d, 1); + it->util = d; + } + + d->index_status = DIFF_STATUS_ADDED; + /* Leave {mode,oid}_head zero for adds. */ + d->mode_index = mode; + oidcpy(&d->oid_index, oid); + s->committable = 1; + strbuf_release(&full_name); + return 0; +} + static void wt_status_collect_changes_initial(struct wt_status *s) { struct index_state *istate = s->repo->index; @@ -671,6 +701,27 @@ static void wt_status_collect_changes_initial(struct wt_status *s) continue; if (ce_intent_to_add(ce)) continue; + if (S_ISSPARSEDIR(ce->ce_mode)) { + /* + * This is a sparse directory entry, so we want to collect all + * of the added files within the tree. This requires recursively + * expanding the trees to find the elements that are new in this + * tree and marking them with DIFF_STATUS_ADDED. + */ + struct strbuf base = STRBUF_INIT; + struct pathspec ps = { 0 }; + struct tree *tree = lookup_tree(istate->repo, &ce->oid); + + ps.recursive = 1; + ps.has_wildcard = 1; + ps.max_depth = -1; + + strbuf_add(&base, ce->name, ce->ce_namelen); + read_tree_at(istate->repo, tree, &base, &ps, + add_file_to_list, s); + continue; + } + it = string_list_insert(&s->change, ce->name); d = it->util; if (!d) { From patchwork Mon Jul 12 17:55:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12371835 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 27D9CC07E9C for ; Mon, 12 Jul 2021 17:56:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0EDB761206 for ; Mon, 12 Jul 2021 17:56:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235899AbhGLR7L (ORCPT ); Mon, 12 Jul 2021 13:59:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48174 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235772AbhGLR66 (ORCPT ); Mon, 12 Jul 2021 13:58:58 -0400 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9D561C061797 for ; Mon, 12 Jul 2021 10:56:08 -0700 (PDT) Received: by mail-wm1-x32d.google.com with SMTP id n4so1559198wms.1 for ; Mon, 12 Jul 2021 10:56:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=tFZIx+WwLaEcHap1rGQysKeAcWD7b272QogaBIbJLog=; b=VDVEOc4bQ4tPPsjHffCREdLCVpqX5sISmclR4l9TwEGMz5ic91m+jtvwWn9nfOYJqH 0fG4fgFXjone3BKq7qJcuBd/v+JauNI4enR5B/gTfpDJnIbkv4hanVKlSwFrFIrsC9fa H4D6sJJrIxexpt21g1FwFwHmg6HuMAczjvMaJDwAFZnnlwavFJuMTVxqMsGLSkIaZnlJ eoIJoiG2wV6KGMEUCN5jg3q3AXr560+C/m8v4hG37Jh9tNADFKuop6H2CTVtLAiONjTT 89Z3dMNpoMx2TUOUXaYW0FIGJ1WRag2l5rkMWOXvmCc2UhxCO1W0s4lWBUNmSP7kF6M8 MPSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=tFZIx+WwLaEcHap1rGQysKeAcWD7b272QogaBIbJLog=; b=JO6j58tyTI2YWveD5M3/JJo0gA0VkRoklO2zmscmYLhpH8gzuEyMzLbEZ/vcW4HvYs KxqOdmMUu1fq0sAwQNRnCuUydF1y22cIBhbAJM0uIdkQCxs6q0nCCCJP142Q3HUwnuUT 8oKQtjP4MM1IArlhLaDGCtHQicXdL2t5Z910FUQ2U9/V+FkhODHDn9nbkq2CESX3A5tA bPJsS5n1qPZQxbQ5YG9xMoBLZIygqw7aXva0DH6imWTy1SC65v05fG9zGm43ZLZfNIVX EaP+svt9+cokk1gamY1ZYsJIKhCWDJAhqbvk5s2QfN7R3lq4yTlFTg3wsjllKyh1jTtO eppw== X-Gm-Message-State: AOAM5311d/HneNAKQkiV9z02WorXxpj/OJuZSgZqHVedbS/PaMWSefD5 RfKR9sZ6xZdYCgwXxneQHrMH7tgCsW8= X-Google-Smtp-Source: ABdhPJw31XCUsvdm7N3a9nH770PRfEze+eU1PUjpKiPylKQP4MHttwdvYWf+9A9oODboRpcV79KP9Q== X-Received: by 2002:a05:600c:35c1:: with SMTP id r1mr485719wmq.0.1626112567311; Mon, 12 Jul 2021 10:56:07 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id c125sm134053wme.36.2021.07.12.10.56.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 12 Jul 2021 10:56:07 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Mon, 12 Jul 2021 17:55:56 +0000 Subject: [PATCH v8 15/15] fsmonitor: integrate with sparse index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee If we need to expand a sparse-index into a full one, then the FS Monitor bitmap is going to be incorrect. Ensure that we start fresh at such an event. While this is currently a performance drawback, the eventual hope of the sparse-index feature is that these expansions will be rare and hence we will be able to keep the FS Monitor data accurate across multiple Git commands. These tests are added to demonstrate that the behavior is the same across a full index and a sparse index, but also that file modifications to a tracked directory outside of the sparse cone will trigger ensure_full_index(). Signed-off-by: Derrick Stolee --- sparse-index.c | 7 ++++++ t/t7519-status-fsmonitor.sh | 49 +++++++++++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+) diff --git a/sparse-index.c b/sparse-index.c index ef53bd2198b..53c8f711ccc 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -186,6 +186,10 @@ int convert_to_sparse(struct index_state *istate) cache_tree_free(&istate->cache_tree); cache_tree_update(istate, 0); + istate->fsmonitor_has_run_once = 0; + FREE_AND_NULL(istate->fsmonitor_dirty); + FREE_AND_NULL(istate->fsmonitor_last_update); + istate->sparse_index = 1; trace2_region_leave("index", "convert_to_sparse", istate->repo); return 0; @@ -282,6 +286,9 @@ void ensure_full_index(struct index_state *istate) istate->cache = full->cache; istate->cache_nr = full->cache_nr; istate->cache_alloc = full->cache_alloc; + istate->fsmonitor_has_run_once = 0; + FREE_AND_NULL(istate->fsmonitor_dirty); + FREE_AND_NULL(istate->fsmonitor_last_update); strbuf_release(&base); free(full); diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh index 637391c6ce4..deea88d4431 100755 --- a/t/t7519-status-fsmonitor.sh +++ b/t/t7519-status-fsmonitor.sh @@ -73,6 +73,7 @@ test_expect_success 'setup' ' expect* actual* marker* + trace2* EOF ' @@ -383,4 +384,52 @@ test_expect_success 'status succeeds after staging/unstaging' ' ) ' +# Usage: +# check_sparse_index_behavior [!] +# If "!" is supplied, then we verify that we do not call ensure_full_index +# during a call to 'git status'. Otherwise, we verify that we _do_ call it. +check_sparse_index_behavior () { + git status --porcelain=v2 >expect && + git sparse-checkout init --cone --sparse-index && + git sparse-checkout set dir1 dir2 && + GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ + git status --porcelain=v2 >actual && + test_region $1 index ensure_full_index trace2.txt && + test_region fsm_hook query trace2.txt && + test_cmp expect actual && + rm trace2.txt && + git sparse-checkout disable +} + +test_expect_success 'status succeeds with sparse index' ' + git reset --hard && + + test_config core.fsmonitor "$TEST_DIRECTORY/t7519/fsmonitor-all" && + check_sparse_index_behavior ! && + + write_script .git/hooks/fsmonitor-test<<-\EOF && + printf "last_update_token\0" + EOF + git config core.fsmonitor .git/hooks/fsmonitor-test && + check_sparse_index_behavior ! && + + write_script .git/hooks/fsmonitor-test<<-\EOF && + printf "last_update_token\0" + printf "dir1/modified\0" + EOF + check_sparse_index_behavior ! && + + cp -r dir1 dir1a && + git add dir1a && + git commit -m "add dir1a" && + + # This one modifies outside the sparse-checkout definition + # and hence we expect to expand the sparse-index. + write_script .git/hooks/fsmonitor-test<<-\EOF && + printf "last_update_token\0" + printf "dir1a/modified\0" + EOF + check_sparse_index_behavior +' + test_done