From patchwork Tue Apr 13 14:01:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12200595 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D3415C433ED for ; Tue, 13 Apr 2021 14:02:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B600061206 for ; Tue, 13 Apr 2021 14:02:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237589AbhDMOCO (ORCPT ); Tue, 13 Apr 2021 10:02:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53250 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237076AbhDMOCD (ORCPT ); Tue, 13 Apr 2021 10:02:03 -0400 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 594CCC06175F for ; Tue, 13 Apr 2021 07:01:42 -0700 (PDT) Received: by mail-wr1-x42b.google.com with SMTP id m9so3851883wrx.3 for ; Tue, 13 Apr 2021 07:01:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=wy+APj9NNOktVQw1Dg8sosD01WrzUgk01YbHujHnv/M=; b=MeMEm/wQTmjLU4NX0pkv2DYWZyVFVgfjleYo8/Xja3JY064bIUl2TNOQbxAMM9+P33 lmEPAH5jxRfSudXiSboLqPtjjtF1Sb9jJr8shCLPx2Gby9TjwJRRAykBew34CDn8uNLg 02vJWslsj6/gsnSVcHhiXnwga/IdYyMvTIbHa8thK3zvhgeF+jq0SBfRXssJJA6HzZu3 3sUuf4pm+UIfag6w43hNwUuu/SCgcO2oznzuEMljHTG0ybdGbbW+AMAmjoHUDkLteUrP 8i7+pwDnptv3fhjfNQg/ujnW/R+BOMR1kfy/NA3GeXsfsZArNiv8aZyWdG7HBFk8EZrL EAGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=wy+APj9NNOktVQw1Dg8sosD01WrzUgk01YbHujHnv/M=; b=QHlHv+QNIvUWABWUz8CIsPy/4LB31sfaR9FsB1EvOLgy23CPAL3zjW8502BUpoOe+d 6mzh3Avtjti/09VbE/dUbmjPcf8os9qb08pGBFmQMrTJLwe5l6IobYjMFiSRW2xA5UI1 Ma9X+DhKwqYDfPLz3LjDzCWs6CcVHP+eYFSk97rpR5T0DAm++4FS4g+YSUZWAFSH5x+O k/OZXK79r4lOWHNz9hcBIkVuuWFXzsVkv66noI5l2SleKj3IO/LTG73UdUW9bpzmp9JS Y/t8mrkV26B059VEm0Yxvmpw2hfiJwN6Av6cSe5qDpT8btsylytPd6dyHnlrrWnKzbfh 8X2g== X-Gm-Message-State: AOAM532LOteRh+gv/LURAlUsfrmaCvkp5mlYRUOijdMJdXpIFE0VCxed 7rmQqms7hbroQ7pnm3pCOhBd/Z0XakY= X-Google-Smtp-Source: ABdhPJzI9N60MqHz03WFfNFGO6KyUi0A+PBTxu0nRW8JCdNSIgbStAOXx5oRFi76q1Q9vY7xH2Wvsw== X-Received: by 2002:adf:de08:: with SMTP id b8mr10379034wrm.279.1618322499870; Tue, 13 Apr 2021 07:01:39 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f2sm2582184wmp.20.2021.04.13.07.01.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Apr 2021 07:01:39 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 13 Apr 2021 14:01:28 +0000 Subject: [PATCH 01/10] t1092: add tests for status/add and sparse files Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Before moving to update 'git status' and 'git add' to work with sparse indexes, add an explicit test that ensures the sparse-index works the same as a normal sparse-checkout when the worktree contains directories and files outside of the sparse cone. Specifically, 'folder1/a' is a file in our test repo, but 'folder1' is not in the sparse cone. When 'folder1/a' is modified, the file 'folder1/a' is shown as modified, but adding it fails. This is new behavior as of a20f704 (add: warn when asked to update SKIP_WORKTREE entries, 2021-04-08). Before that change, these adds would be silently ignored. Untracked files are fine: adding new files both with 'git add .' and 'git add folder1/' works just as in a full checkout. This may not be entirely desirable, but we are not intending to change behavior at the moment, only document it. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 36 ++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 12e6c453024f..6598c12a2069 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -232,6 +232,42 @@ test_expect_success 'add, commit, checkout' ' test_all_match git checkout - ' +test_expect_success 'status/add: outside sparse cone' ' + init_repos && + + # folder1 is at HEAD, but outside the sparse cone + run_on_sparse mkdir folder1 && + cp initial-repo/folder1/a sparse-checkout/folder1/a && + cp initial-repo/folder1/a sparse-index/folder1/a && + + test_sparse_match git status && + + write_script edit-contents <<-\EOF && + echo text >>$1 + EOF + run_on_all ../edit-contents folder1/a && + run_on_all ../edit-contents folder1/new && + + test_sparse_match git status --porcelain=v2 && + + # This "git add folder1/a" is completely ignored + # by the sparse-checkout repos. It causes the + # full repo to have a different staged environment. + test_must_fail git -C sparse-checkout add folder1/a && + test_must_fail git -C sparse-index add folder1/a && + git -C full-checkout checkout HEAD -- folder1/a && + test_sparse_match git status --porcelain=v2 && + + test_all_match git add . && + test_all_match git status --porcelain=v2 && + test_all_match git commit -m folder1/new && + + run_on_all ../edit-contents folder1/newer && + test_all_match git add folder1/ && + test_all_match git status --porcelain=v2 && + test_all_match git commit -m folder1/newer +' + test_expect_success 'checkout and reset --hard' ' init_repos && From patchwork Tue Apr 13 14:01:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12200591 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6BC0C433B4 for ; Tue, 13 Apr 2021 14:01:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7961B61206 for ; Tue, 13 Apr 2021 14:01:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237401AbhDMOCG (ORCPT ); Tue, 13 Apr 2021 10:02:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53246 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231869AbhDMOCC (ORCPT ); Tue, 13 Apr 2021 10:02:02 -0400 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2265EC061756 for ; Tue, 13 Apr 2021 07:01:42 -0700 (PDT) Received: by mail-wr1-x42d.google.com with SMTP id h4so7553462wrt.12 for ; Tue, 13 Apr 2021 07:01:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=gXqdqUJHSvZv+ScIN0sMWzSKjWEIqKfVgAWYeNbLixQ=; b=TcjuTas1N5MhZwbVOqJyrgLiB3Ti1QImbBBC7lp0NC+MZu6lW8ko4mEGaoqz1v/nUZ MB8UoGxNGvqTV2DVhetuFLNvd+PRqbLeMOllVuHP48KN/Gi/oBWDu9p2IV7YU8XDumYr YpqrYzRJYbEcDmQF+o18leTj+OGAH2uAdFDrbSXepcLBrijx5owk/tUFmo1DCQlvLQGx vo1TKHcgXIlqVduO9WaUuUrTpnazCvZyHw8O+9BlI8MQHvyM/ShoFScIFTplqu79KOHI ugh3c57+QbZcmluf6kRDVoY5j+Oh+kg4iFvsgWp7CChgRkPHnQDvDTj72l4VIabIsV0a buYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=gXqdqUJHSvZv+ScIN0sMWzSKjWEIqKfVgAWYeNbLixQ=; b=gGsqTr3yh7z9YlPpaxll/nqKc7CRTEmOQ5cH+8qlBzLJtuY3UMlzgI6dmbXVhfaynN q9p8fB3ZrQOfXmYC855biTWd//q1wzZXGCCHAkRBwfu9sRLEx1NY5M3vkcx6wmQr4vpQ CLqHHjhDmq1L9yXvJ/ITLaCHq+Xuldy8/mjLwGoL3yWV71ntZ6sEPRORsaXmoKRTzWUF R9o/1wXnGmY9sizXa9J1wbzBm/0qzCr0aR903ow7WbGXIg/HRopwNR0MYlRt5lrvOytL svnr5f6aVBr9FnEIeMPs3yzUSIOn8dJWZehwQl4M/SDY/49ZEiWudjvYOZ3CO9CMcnM3 5wcQ== X-Gm-Message-State: AOAM532i4gadMWQVi34yMOrJQt0vupYYSEU9AHm3YCc/ofolmRGupnha DqieHGiPZPQ4v+EtswOW18IzVhFg1Ns= X-Google-Smtp-Source: ABdhPJynQBdQ72XVgSNVW9NQmOH7Edx1kX7F9S3+5DUIwrAahN/aGxVN597UgPLT7j+31+AtSGO05w== X-Received: by 2002:a5d:65ca:: with SMTP id e10mr36714929wrw.331.1618322500739; Tue, 13 Apr 2021 07:01:40 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id h2sm2668193wmc.24.2021.04.13.07.01.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Apr 2021 07:01:40 -0700 (PDT) Message-Id: <0a3892d2ec9e4acd4cba1c1d0390acc60dc6e50f.1618322497.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 13 Apr 2021 14:01:29 +0000 Subject: [PATCH 02/10] unpack-trees: make sparse aware Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee As a first step to integrate 'git status' and 'git add' with the sparse index, we must start integrating unpack_trees() with sparse directory entries. These changes are currently impossible to trigger because unpack_trees() calls ensure_full_index() if command_requires_full_index is true. This is the case for all commands at the moment. As we expand more commands to be sparse-aware, we might find that more changes are required to unpack_trees(). The current changes will suffice for 'status' and 'add'. unpack_trees() calls the traverse_trees() API using unpack_callback() to decide if we should recurse into a subtree. We must add new abilities to skip a subtree if it corresponds to a sparse directory entry. It is important to be careful about the trailing directory separator that exists in the sparse directory entries but not in the subtree paths. Signed-off-by: Derrick Stolee --- dir.h | 2 +- preload-index.c | 2 ++ read-cache.c | 3 +++ unpack-trees.c | 24 ++++++++++++++++++++++-- 4 files changed, 28 insertions(+), 3 deletions(-) diff --git a/dir.h b/dir.h index 51cb0e217247..9d6666f520f3 100644 --- a/dir.h +++ b/dir.h @@ -503,7 +503,7 @@ static inline int ce_path_match(struct index_state *istate, char *seen) { return match_pathspec(istate, pathspec, ce->name, ce_namelen(ce), 0, seen, - S_ISDIR(ce->ce_mode) || S_ISGITLINK(ce->ce_mode)); + S_ISSPARSEDIR(ce->ce_mode) || S_ISDIR(ce->ce_mode) || S_ISGITLINK(ce->ce_mode)); } static inline int dir_path_match(struct index_state *istate, diff --git a/preload-index.c b/preload-index.c index e5529a586366..35e67057ca9b 100644 --- a/preload-index.c +++ b/preload-index.c @@ -55,6 +55,8 @@ static void *preload_thread(void *_data) continue; if (S_ISGITLINK(ce->ce_mode)) continue; + if (S_ISSPARSEDIR(ce->ce_mode)) + continue; if (ce_uptodate(ce)) continue; if (ce_skip_worktree(ce)) diff --git a/read-cache.c b/read-cache.c index 29ffa9ac5db9..6308234b4838 100644 --- a/read-cache.c +++ b/read-cache.c @@ -1594,6 +1594,9 @@ int refresh_index(struct index_state *istate, unsigned int flags, if (ignore_skip_worktree && ce_skip_worktree(ce)) continue; + if (istate->sparse_index && S_ISSPARSEDIR(ce->ce_mode)) + continue; + if (pathspec && !ce_path_match(istate, ce, pathspec, seen)) filtered = 1; diff --git a/unpack-trees.c b/unpack-trees.c index dddf106d5bd4..9a62e823928a 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -586,6 +586,13 @@ static void mark_ce_used(struct cache_entry *ce, struct unpack_trees_options *o) { ce->ce_flags |= CE_UNPACKED; + /* + * If this is a sparse directory, don't advance cache_bottom. + * That will be advanced later using the cache-tree data. + */ + if (S_ISSPARSEDIR(ce->ce_mode)) + return; + if (o->cache_bottom < o->src_index->cache_nr && o->src_index->cache[o->cache_bottom] == ce) { int bottom = o->cache_bottom; @@ -984,6 +991,9 @@ static int do_compare_entry(const struct cache_entry *ce, ce_len -= pathlen; ce_name = ce->name + pathlen; + /* remove directory separator if a sparse directory entry */ + if (S_ISSPARSEDIR(ce->ce_mode)) + ce_len--; return df_name_compare(ce_name, ce_len, S_IFREG, name, namelen, mode); } @@ -993,6 +1003,10 @@ static int compare_entry(const struct cache_entry *ce, const struct traverse_inf if (cmp) return cmp; + /* If ce is a sparse directory, then allow equality here. */ + if (S_ISSPARSEDIR(ce->ce_mode)) + return 0; + /* * Even if the beginning compared identically, the ce should * compare as bigger than a directory leading up to it! @@ -1243,6 +1257,7 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str struct cache_entry *src[MAX_UNPACK_TREES + 1] = { NULL, }; struct unpack_trees_options *o = info->data; const struct name_entry *p = names; + unsigned recurse = 1; /* Find first entry with a real name (we could use "mask" too) */ while (!p->mode) @@ -1284,12 +1299,16 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str } } src[0] = ce; + + if (S_ISSPARSEDIR(ce->ce_mode)) + recurse = 0; } break; } } - if (unpack_nondirectories(n, mask, dirmask, src, names, info) < 0) + if (recurse && + unpack_nondirectories(n, mask, dirmask, src, names, info) < 0) return -1; if (o->merge && src[0]) { @@ -1319,7 +1338,8 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str } } - if (traverse_trees_recursive(n, dirmask, mask & ~dirmask, + if (recurse && + traverse_trees_recursive(n, dirmask, mask & ~dirmask, names, info) < 0) return -1; return mask; From patchwork Tue Apr 13 14:01:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12200601 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5F0BC433ED for ; Tue, 13 Apr 2021 14:02:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8CB0F613AB for ; Tue, 13 Apr 2021 14:02:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237624AbhDMOC3 (ORCPT ); Tue, 13 Apr 2021 10:02:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53262 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237373AbhDMOCF (ORCPT ); Tue, 13 Apr 2021 10:02:05 -0400 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 635DCC061756 for ; Tue, 13 Apr 2021 07:01:44 -0700 (PDT) Received: by mail-wr1-x42c.google.com with SMTP id r7so4589478wrm.1 for ; Tue, 13 Apr 2021 07:01:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=GV149/jGnSYB9xubiM2m3Ys9KHOjOAzJNLFmGygbCus=; b=hKvNYA8yLOZA0AHCnMHGdOA/4lnxkaDUrB9xhA36jPRuVHmJLhDcd9s7BDHdcTJRfe zS8i24IKHF0ZLS8r5hk103IVFLhe8kevDSCIXRbHNbe7Offxhpi4VyobjwkhHZMmp6M4 q84tXWpjRkhEXgLABlwyp+jgbyVAzrf3Vos7eeJ8MA+JAY8DABAvz5N3aivsxfyR996R 7q4gwPSCbJeY6IWONGT0CkNE1KexNZCI1CwHB9v/fj3wg6A9x54t4YeEm5uWaswwoyQU ZQPEiKLIykkf3wB1vGJPGHlwIEfx4b7dpJpgZcfNG1ArK+q/aoRc0iSgZIAFJGYw/fU6 a/BA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=GV149/jGnSYB9xubiM2m3Ys9KHOjOAzJNLFmGygbCus=; b=EtX3rCaCZlb2K5opDG5ABvhL8biQjTrdAP/rQN66xOvuEjJZ/LLs90TDmyHrajGwY1 wgJbrBHo1RAyow79QMk1gnwixxZUVL4ZU+aRY8rQ+GhJW1FkdMx5LiV0d5DuMFdeEhKq sISEs2iE71WLY/efM4MxtzSC6I12+HEi2JARY3fa5Hdm+wsPs/kz8Rdx5rDSOByClh07 bPKY0JR2lyZActRYWLyWdU91UsABpPdV+cb5ngJ/nFJj8sO1tgUKACDtrfMgb51Q1jtv SCuY1uoMOUof/SHH6KFpE+W8bs8XdHkgi5+UsXypv3oOlkjzyS3lyzDDZBmgQkOKlnyN A9OQ== X-Gm-Message-State: AOAM531xAsa+qfFi2TRn0PnQvzn08fB7/Ei/EgiF2vpSEBjBoGA1Yueu sJOPL0L3QDXQ8f2qImoZn+d96jRcYbc= X-Google-Smtp-Source: ABdhPJxxmAqDzCvCSuGaRWujWu7v75ct6FZfP8kh5gVtjxgfmZYYKopSNfCPxLddqPQVWM6SRzhYjQ== X-Received: by 2002:a5d:484c:: with SMTP id n12mr519939wrs.377.1618322501918; Tue, 13 Apr 2021 07:01:41 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id j6sm2517424wmq.16.2021.04.13.07.01.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Apr 2021 07:01:41 -0700 (PDT) Message-Id: <28ca717e6526f0b66df696f3237d72b9bee2ffc3.1618322497.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 13 Apr 2021 14:01:30 +0000 Subject: [PATCH 03/10] dir.c: accept a directory as part of cone-mode patterns Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee When we have sparse directory entries in the index, we want to compare that directory against sparse-checkout patterns. Those pattern matching algorithms are built expecting a file path, not a directory path. This is especially important in the "cone mode" patterns which will match files that exist within the "parent directories" as well as the recursive directory matches. If path_matches_pattern_list() is given a directory, we can add a fake filename ("-") to the directory and get the same results as before, assuming we are in cone mode. Since sparse index requires cone mode patterns, this is an acceptable assumption. Signed-off-by: Derrick Stolee --- dir.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/dir.c b/dir.c index 166238e79f52..57e22e605cec 100644 --- a/dir.c +++ b/dir.c @@ -1378,6 +1378,11 @@ enum pattern_match_result path_matches_pattern_list( strbuf_addch(&parent_pathname, '/'); strbuf_add(&parent_pathname, pathname, pathlen); + /* Directory requests should be added as if they are a file */ + if (parent_pathname.len > 1 && + parent_pathname.buf[parent_pathname.len - 1] == '/') + strbuf_add(&parent_pathname, "-", 1); + if (hashmap_contains_path(&pl->recursive_hashmap, &parent_pathname)) { result = MATCHED_RECURSIVE; From patchwork Tue Apr 13 14:01:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12200599 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68F18C433ED for ; Tue, 13 Apr 2021 14:02:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4D13B61353 for ; Tue, 13 Apr 2021 14:02:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240709AbhDMOC0 (ORCPT ); Tue, 13 Apr 2021 10:02:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53264 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237343AbhDMOCF (ORCPT ); Tue, 13 Apr 2021 10:02:05 -0400 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E7E53C06138C for ; Tue, 13 Apr 2021 07:01:44 -0700 (PDT) Received: by mail-wr1-x42f.google.com with SMTP id s7so16466921wru.6 for ; Tue, 13 Apr 2021 07:01:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=8VJdwr9CcCAR4dz9bRohUSRZbYTLN3SS0ect0Dufjjk=; b=Z5fpgw+puVzzUrlETeMaorIUAV82unWwZDih1G2xVuwUGO2QIX2WrsJxkLke9j4dXy kuw5BRY+qzrCNP5g2SpDi04pMmAECJ5Y4vtc5wxh429nrPP9bC9Lq5/nsb426jH/5g1T 8gywi2tMF8K/RNpShnr+HPfiqv1rjILBk55NQHW4+6dHIrhQmktIB0uoeabsuXsnLNpK k5v0QxcQjDYHJ0Be7rxGVrdssmAUGkrYug3ISXzcZTO9gPLLLT3DKP7/T04CDQqrkdqW 7JpS46vrsapUxLAJx/RotgGf/EpOUj8sfWtjwCa6yrMeYnXCfK1Fim55ZVWwPnxUe34V T9dA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=8VJdwr9CcCAR4dz9bRohUSRZbYTLN3SS0ect0Dufjjk=; b=uMX4oKVV1uKmh0onSWWPrJiYyWCZivPo802PhCu9hnjp9UYR5CprtZi0GmVLMXypG6 aAGU6o4YYqyDZ4SXmH41VevS9rSQgsupm5z/bK7DHLxt8CiJiB29/9Nwaf+RsjL88YWf 3jilGXojMcnNcarormc42uDPkc2wmwJzjjSJsDUBj/+MROrtKBRNGtqqr0r372ORa7D9 Had3JYI/YJGi14fpTwlJn0+5k7Yb4Y/T8yWrae3usQxClnSI7oVpzd+pepnQb8v/0Et6 xZuhZII1F+qQzvBOZ1yw1LzTsBkv9O8WOHCgIeR36SmnBfelUtG41o1MmCll4jqLvodc tpbg== X-Gm-Message-State: AOAM530t6sh4IaSDDB0Z+OLv0eVWavkdJOhnDs9FI4GiZhfKj0qgpeLB 9kPUZE5yybRLuTwCYjYanG2GU0YvZfo= X-Google-Smtp-Source: ABdhPJyGjS2dObJN8cQ/tDex8ayGUP/81O8NtHdfxCcykRfC3L6gGn58sSdDjGE0FO/5beGF5PISQQ== X-Received: by 2002:adf:f692:: with SMTP id v18mr26935139wrp.206.1618322502521; Tue, 13 Apr 2021 07:01:42 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 24sm2996155wmk.18.2021.04.13.07.01.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Apr 2021 07:01:42 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 13 Apr 2021 14:01:31 +0000 Subject: [PATCH 04/10] status: skip sparse-checkout percentage with sparse-index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee 'git status' began reporting a percentage of populated paths when sparse-checkout is enabled in 051df3cf (wt-status: show sparse checkout status as well, 2020-07-18). This percentage is incorrect when the index has sparse directories. It would also be expensive to calculate as we would need to parse trees to count the total number of possible paths. Avoid the expensive computation by simplifying the output to only report that a sparse checkout exists, without the percentage. This change is the reason we use 'git status --porcelain=v2' in t1092-sparse-checkout-compatibility.sh. We don't want to ensure that this message is equal across both modes, but instead just the important information about staged, modified, and untracked files are compared. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 8 ++++++++ wt-status.c | 14 +++++++++++--- wt-status.h | 1 + 3 files changed, 20 insertions(+), 3 deletions(-) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 6598c12a2069..e488ef9bd941 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -196,6 +196,14 @@ test_expect_success 'status with options' ' test_all_match git status --porcelain=v2 -uno ' +test_expect_success 'status reports sparse-checkout' ' + init_repos && + git -C sparse-checkout status >full && + git -C sparse-index status >sparse && + test_i18ngrep "You are in a sparse checkout with " full && + test_i18ngrep "You are in a sparse checkout." sparse +' + test_expect_success 'add, commit, checkout' ' init_repos && diff --git a/wt-status.c b/wt-status.c index 0c8287a023e4..0425169c1895 100644 --- a/wt-status.c +++ b/wt-status.c @@ -1490,9 +1490,12 @@ static void show_sparse_checkout_in_use(struct wt_status *s, if (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_DISABLED) return; - status_printf_ln(s, color, - _("You are in a sparse checkout with %d%% of tracked files present."), - s->state.sparse_checkout_percentage); + if (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_SPARSE_INDEX) + status_printf_ln(s, color, _("You are in a sparse checkout.")); + else + status_printf_ln(s, color, + _("You are in a sparse checkout with %d%% of tracked files present."), + s->state.sparse_checkout_percentage); wt_longstatus_print_trailer(s); } @@ -1650,6 +1653,11 @@ static void wt_status_check_sparse_checkout(struct repository *r, return; } + if (r->index->sparse_index) { + state->sparse_checkout_percentage = SPARSE_CHECKOUT_SPARSE_INDEX; + return; + } + for (i = 0; i < r->index->cache_nr; i++) { struct cache_entry *ce = r->index->cache[i]; if (ce_skip_worktree(ce)) diff --git a/wt-status.h b/wt-status.h index 0d32799b28e1..ab9cc9d8f032 100644 --- a/wt-status.h +++ b/wt-status.h @@ -78,6 +78,7 @@ enum wt_status_format { }; #define SPARSE_CHECKOUT_DISABLED -1 +#define SPARSE_CHECKOUT_SPARSE_INDEX -2 struct wt_status_state { int merge_in_progress; From patchwork Tue Apr 13 14:01:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12200597 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8156CC433ED for ; Tue, 13 Apr 2021 14:02:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5FCDF61206 for ; Tue, 13 Apr 2021 14:02:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232475AbhDMOCX (ORCPT ); Tue, 13 Apr 2021 10:02:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53260 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237201AbhDMOCF (ORCPT ); Tue, 13 Apr 2021 10:02:05 -0400 Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 47BA9C061574 for ; Tue, 13 Apr 2021 07:01:44 -0700 (PDT) Received: by mail-wm1-x32c.google.com with SMTP id w7-20020a1cdf070000b0290125f388fb34so8245334wmg.0 for ; Tue, 13 Apr 2021 07:01:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=igbZPETxXaQSENv8XIc9P3Kyj/wn+H2Pe1/dZBLGnEk=; b=Y2j+GqBsHJBYfSXrNF2LZYui9m3WdUJh3wXP6aRZCwuMx70NpyYKuIlX2l5V4Kn57e FNi6ztUFhWh3MB0cAzsBGbBZuY6fCPxH4XgjrgilKH+MDubdM2NnKGIDa/nRF/IK3QRW KNBoWZZC09qJ0rA7OQKg3GSpAzBfRnQLx2xTgMx4HueN4yUlcd2mdr7tivIEwWKhjrna A7doul5Ssg774O/mnDL3OG20tltSK0gLMwRKLxy0/4ICPaAtWXpps32V79uRmamljO8X doDdInx/UeUuEflt5nWcYxtSfZDoIh9xYqfR2Tb9mmh3LDvN89AZecZqOluc+WwuwDxo a45w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=igbZPETxXaQSENv8XIc9P3Kyj/wn+H2Pe1/dZBLGnEk=; b=lOiQgAxnPPq0RjFlw4eaG5dwunMPi/JrCkxdPGjnx1bfXHQCjGc6dBiHgvqxuIKV+0 +/Wp1lKp2kCekbliCqKNHlZgGhUBsj/O59hVFrWit6zYLLDMOU/iyW1w0jT1AKx38VQN gRZhgsBTzOsnP+w41U/0ka7EjDsLDgtIpIk9egSZQm53GRetESLXwrW/7kEHMRDmiZVk FQDjonRZWNwkHb706cQHljapjpmY0tyyyoEHidYQr/+CB2hSoECco9PiTmpf3AJjKtjy tcbeZ3YJo+iMy7oTpBU09QDrINyFsYCayd8jUR6mefRxXam+gyt7VVGjkNgKik3MR/Nr 0l0w== X-Gm-Message-State: AOAM531Iwl8WqY8Gi4LVxnyPDTK6iPig5d+CQd4BB7EkbphvVOQCYfUb BXiXLqA5uUv6ipWmv9orF4HoKHZO6Mo= X-Google-Smtp-Source: ABdhPJznB4qIbEsQjk/uE6SpxPtCNvXkfn0lvjDYvCg4deZN9VL1fRNJxbJowDzMu7C0ZDjvc1TmWg== X-Received: by 2002:a1c:f204:: with SMTP id s4mr185728wmc.8.1618322503027; Tue, 13 Apr 2021 07:01:43 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id b187sm2645687wmh.17.2021.04.13.07.01.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Apr 2021 07:01:42 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 13 Apr 2021 14:01:32 +0000 Subject: [PATCH 05/10] status: use sparse-index throughout Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee By testing 'git -c core.fsmonitor= status -uno', we can check for the simplest index operations that can be made sparse-aware. The necessary implementation details are already integrated with sparse-checkout, so modify command_requires_full_index to be zero for cmd_status(). By running the debugger for 'git status -uno' after that change, we find two instances of ensure_full_index() that were added for extra safety, but can be removed without issue. In refresh_index(), we loop through the index entries. The refresh_cache_ent() method copies the sparse directories into the refreshed index without issue. The loop within run_diff_files() skips things that are in stage 0 and have skip-worktree enabled, so seems safe to disable ensure_full_index() here. This allows some cases of 'git status' to no longer expand a sparse index to a full one, giving the following performance improvements for p2000-sparse-checkout-operations.sh: Test HEAD~1 HEAD ----------------------------------------------------------------------------- 2000.2: git status (full-index-v3) 0.38(0.36+0.07) 0.37(0.31+0.10) -2.6% 2000.3: git status (full-index-v4) 0.38(0.29+0.12) 0.37(0.30+0.11) -2.6% 2000.4: git status (sparse-index-v3) 2.43(2.33+0.14) 0.04(0.05+0.04) -98.4% 2000.5: git status (sparse-index-v4) 2.44(2.35+0.13) 0.05(0.04+0.07) -98.0% Note that since HEAD~1 was expanding the sparse index by parsing trees, it was artificially slower than the full index case. Thus, the 98% improvement is misleading, and instead we should celebrate the 0.37s to 0.05s improvement of 82%. This is more indicative of the peformance gains we are expecting by using a sparse index. Note: we are dropping the assignment of core.fsmonitor here. This is not necessary for the test script as we are not altering the config any other way. Correct integration with FS Monitor will be validated in later changes. Signed-off-by: Derrick Stolee --- builtin/commit.c | 3 +++ read-cache.c | 2 -- t/t1092-sparse-checkout-compatibility.sh | 12 ++++++++---- 3 files changed, 11 insertions(+), 6 deletions(-) diff --git a/builtin/commit.c b/builtin/commit.c index cf0c36d1dcb2..e529da7beadd 100644 --- a/builtin/commit.c +++ b/builtin/commit.c @@ -1404,6 +1404,9 @@ int cmd_status(int argc, const char **argv, const char *prefix) if (argc == 2 && !strcmp(argv[1], "-h")) usage_with_options(builtin_status_usage, builtin_status_options); + prepare_repo_settings(the_repository); + the_repository->settings.command_requires_full_index = 0; + status_init_config(&s, git_status_config); argc = parse_options(argc, argv, prefix, builtin_status_options, diff --git a/read-cache.c b/read-cache.c index 6308234b4838..83e6bdef7604 100644 --- a/read-cache.c +++ b/read-cache.c @@ -1578,8 +1578,6 @@ int refresh_index(struct index_state *istate, unsigned int flags, */ preload_index(istate, pathspec, 0); trace2_region_enter("index", "refresh", NULL); - /* TODO: audit for interaction with sparse-index. */ - ensure_full_index(istate); for (i = 0; i < istate->cache_nr; i++) { struct cache_entry *ce, *new_entry; int cache_errno = 0; diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index e488ef9bd941..380a085f8ec4 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -449,12 +449,16 @@ test_expect_success 'sparse-index is expanded and converted back' ' GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ git -C sparse-index -c core.fsmonitor="" reset --hard && test_region index convert_to_sparse trace2.txt && - test_region index ensure_full_index trace2.txt && + test_region index ensure_full_index trace2.txt +' - rm trace2.txt && +test_expect_success 'sparse-index is not expanded' ' + init_repos && + + rm -f trace2.txt && GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ - git -C sparse-index -c core.fsmonitor="" status -uno && - test_region index ensure_full_index trace2.txt + git -C sparse-index status -uno && + test_region ! index ensure_full_index trace2.txt ' test_done From patchwork Tue Apr 13 14:01:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12200613 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC54FC433B4 for ; Tue, 13 Apr 2021 14:02:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B1C4A613B7 for ; Tue, 13 Apr 2021 14:02:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344932AbhDMODE (ORCPT ); Tue, 13 Apr 2021 10:03:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53274 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237585AbhDMOCI (ORCPT ); Tue, 13 Apr 2021 10:02:08 -0400 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 035CCC06138D for ; Tue, 13 Apr 2021 07:01:46 -0700 (PDT) Received: by mail-wm1-x335.google.com with SMTP id k128so8803214wmk.4 for ; Tue, 13 Apr 2021 07:01:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=UWeNrfEJxLnaBSFF3+fKnKM+OImF8teOvdm3wXKtY7w=; b=QNPpsVaoLfvHllg9FVSyHitDqfDVP23voZfOzR2RRoPMA8B7oiekaWJC0JAkwraGYZ C/cuLb9oFHJanEpmHJ2yccdEKyU7Zy1xBarifge1mhKSG0kalq9hAm+YtB55AjM4i/aw ze750qWFh/KlSMUt16L3X7nOTlpdfLhaDRsAHqEvGan3/9YfDny7zkyAJ9M43VHb1nQa pjKThle6QS+iwBro2xl6i9mHkRN4DAhPQrhPEdETIJUmAgyht5bJvgqMU7LT4MgQjh0f uRK8jKPGPh+RdEPXmCNDQruFu/KR8QFAUlQcwr4b2Y+K8GAyFqy9zrXdUKMm2litLfXI W1uw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=UWeNrfEJxLnaBSFF3+fKnKM+OImF8teOvdm3wXKtY7w=; b=gGdZrUe+TD+7TQiFRwUZ73lwr8X8FEXfCjwq6xa94NX6djhX4diocRQYTBFF58fQEG 5lrs3H8BpQTXERVj9mjGhh48YmQvGcfJTpzT4HNfirn4nSgXFiJ5QNiW4msPiFjf0oKS JT/7nqJKiehBnBwUbsi+MMd35X+PTthWuAbcPym84XosTgSijUTjSpKmdyuGU6xXajzG DShJ3nJB4LX39a/IEJN2tNqHCkXRK5SraHpBypZmftuPMoi+hDT101lpv+r/M5gQdbHG /W2OqQzGyhyTrO88Q0Moh0Rr6mvnHYRDhyJeCE/lP74gId1WzKU4bU40oBmntrJsXBTf +mqg== X-Gm-Message-State: AOAM530Wa+G1L7yzf9s0CQUp31TQljzzGKK4HgRbtDnowSfr/ofcbNOL x2wDrj/n5RfnCFraf3m1t/6o1EbeW4g= X-Google-Smtp-Source: ABdhPJyR5s9ZdKG8yHD9Elf34SLvzz0WbAN/L29zes6GmYCJFOUtyVZ93UBjnlZKC0V/vgt14fE04Q== X-Received: by 2002:a1c:f407:: with SMTP id z7mr181090wma.20.1618322503555; Tue, 13 Apr 2021 07:01:43 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id q18sm17489030wrs.25.2021.04.13.07.01.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Apr 2021 07:01:43 -0700 (PDT) Message-Id: <434306541613cbd0b9bb4ebb3102d97e3df9eb94.1618322497.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 13 Apr 2021 14:01:33 +0000 Subject: [PATCH 06/10] dir: use expand_to_path() for sparse directories Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The recently-implemented expand_to_path() method can supply position queries a faster response if they are specifically asking for a path within the sparse cone. Since this is the most-common scenario, this provides a significant speedup. Update t1092-sparse-checkout-compatibility.sh to fully ensure that 'git status' does not expand a sparse index to a full one, even when there exist untracked files. The performance test script p2000-sparse-operations.sh demonstrates that this is the final hole to fill to allow 'git status' to speed up when using a sparse index: Test HEAD~1 HEAD ------------------------------------------------------------------------------ 2000.4: git status (sparse-index-v3) 1.50(1.43+0.10) 0.04(0.04+0.03) -97.3% 2000.5: git status (sparse-index-v4) 1.50(1.43+0.10) 0.04(0.03+0.04) -97.3% Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 380a085f8ec4..b937d7096afd 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -456,8 +456,9 @@ test_expect_success 'sparse-index is not expanded' ' init_repos && rm -f trace2.txt && + echo >>sparse-index/untracked.txt && GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ - git -C sparse-index status -uno && + git -C sparse-index status && test_region ! index ensure_full_index trace2.txt ' From patchwork Tue Apr 13 14:01:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12200607 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5F42C43460 for ; Tue, 13 Apr 2021 14:02:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8850D613B1 for ; Tue, 13 Apr 2021 14:02:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239357AbhDMOCk (ORCPT ); Tue, 13 Apr 2021 10:02:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53276 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237436AbhDMOCG (ORCPT ); Tue, 13 Apr 2021 10:02:06 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8A736C06138E for ; Tue, 13 Apr 2021 07:01:46 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id p6so9871880wrn.9 for ; Tue, 13 Apr 2021 07:01:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=WvaZ3YS8coAgjJkm1yddD68D6kGxSzaGXo2N92oNDOY=; b=AtYenlkFZ64WD6VI9zw6lN7KlF6Fy1V02qLbEGdjejqywQCxrnrstt+2owJlfZ+PiB fpOBqDFDrLunFFxGAttoYgHt0LVkuwXuY3JIhNOo3ffwnPmI8MWsRd6zZboKwu8oKwfS cdQ+eNEXABhS2d1Qj1ZeABqstnSDUvkHLbg2+gXEhlUJao5sGCtI5n7tSuwcdX+j2WCQ hTvs8WU2T7pDmNuxQZkrd4Dj+5BzmDgz1EaUk5QVjqATcWNmGSnFZ1b4m1P8Wjg8IJlz AjMhKv1VkT4qVxDWE3X+rqE5WPayNClWyfCU7G13eAHf9j7t4FesDJc8xqALUbxg9leq eE8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=WvaZ3YS8coAgjJkm1yddD68D6kGxSzaGXo2N92oNDOY=; b=hPWViBzxJF1fkXS7+PHgGfwEQbgzQk2ubc8qXHAzQQT4sWH0FjXvpCa7RTIIFMRRoW +SJQVpN0e/R/qFof86O2j+DgVI/wbnWLbcfvC1Co/q9OLqO09ecxokciyUwDxjFhD2he U2ZaWHnkvPRZr7eSNyFQa5bgBPZZuU0qcq289ahCZCcYS+P3RVKjhi/dChHd8A6i2qVy /woi2IYf0vW2J6gfbxddm7lGXx5Zi6z1an1CLUPrLEOm+qnWGDTq0242nI76W9KX7Quv JJCMeEn2wWZPa87RFaqPOOiRvGOKNzJWHlNYdTBQYunUQUBC7XzNAXJDza2PSTdAjF0+ /y8Q== X-Gm-Message-State: AOAM532y2oqSIw+Y9g4t6ajymSZhO88Ap4RQ4Wl8u8jI0xKsYiAFi8XU aWrDF+ExzaRqA3zfH3xk10y69wXelhc= X-Google-Smtp-Source: ABdhPJxoqu9REEFmafm5nIHUXz0TW6ApF6i245TF6ypsztT6qLzyDJ//gb1zkSlNfh6kqil6Oc1pMQ== X-Received: by 2002:adf:fa07:: with SMTP id m7mr34492488wrr.42.1618322504076; Tue, 13 Apr 2021 07:01:44 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id y31sm2686739wmp.46.2021.04.13.07.01.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Apr 2021 07:01:43 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 13 Apr 2021 14:01:34 +0000 Subject: [PATCH 07/10] add: allow operating on a sparse-only index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Disable command_requires_full_index for 'git add'. This does not require any additional removals of ensure_full_index(). The main reason is that 'git add' discovers changes based on the pathspec and the worktree itself. These are then inserted into the index directly, and calls to index_name_pos() or index_file_exists() already call expand_to_path() at the appropriate time to support a sparse-index. Add a test to check that 'git add -A' and 'git add ' does not expand the index at all, as long as is not within a sparse directory. This does not help the global 'git add .' case. We can measure the improvement using p2000-sparse-operations.sh with these results: Test HEAD~1 HEAD ------------------------------------------------------------------------------ 2000.6: git add -A (full-index-v3) 1.35(1.00+0.20) 1.33(0.98+0.19) -1.5% 2000.7: git add -A (full-index-v4) 1.25(0.97+0.17) 1.23(0.96+0.16) -1.6% 2000.8: git add -A (sparse-index-v3) 2.38(2.28+0.13) 0.06(0.04+0.08) -97.5% 2000.9: git add -A (sparse-index-v4) 2.39(2.25+0.18) 0.06(0.04+0.07) -97.5% While the 97% improvement seems impressive, it's important to recognize that previously we had significant overhead for expanding the sparse-index. Comparing to the full index case, 'git add -A' goes from 1.33s to 0.06s, which is "only" a 95% improvement. Signed-off-by: Derrick Stolee --- builtin/add.c | 3 +++ t/t1092-sparse-checkout-compatibility.sh | 12 ++++++++++++ 2 files changed, 15 insertions(+) diff --git a/builtin/add.c b/builtin/add.c index 58ee3f954ef7..0572d0344065 100644 --- a/builtin/add.c +++ b/builtin/add.c @@ -526,6 +526,9 @@ int cmd_add(int argc, const char **argv, const char *prefix) add_new_files = !take_worktree_changes && !refresh_only && !add_renormalize; require_pathspec = !(take_worktree_changes || (0 < addremove_explicit)); + prepare_repo_settings(the_repository); + the_repository->settings.command_requires_full_index = 0; + hold_locked_index(&lock_file, LOCK_DIE_ON_ERROR); /* diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index b937d7096afd..c210dba78067 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -459,6 +459,18 @@ test_expect_success 'sparse-index is not expanded' ' echo >>sparse-index/untracked.txt && GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ git -C sparse-index status && + test_region ! index ensure_full_index trace2.txt && + + rm trace2.txt && + echo >>sparse-index/README.md && + GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ + git -C sparse-index add -A && + test_region ! index ensure_full_index trace2.txt && + + rm trace2.txt && + echo >>sparse-index/extra.txt && + GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ + git -C sparse-index add extra.txt && test_region ! index ensure_full_index trace2.txt ' From patchwork Tue Apr 13 14:01:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12200609 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 50F60C433B4 for ; Tue, 13 Apr 2021 14:02:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 364E161353 for ; Tue, 13 Apr 2021 14:02:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237436AbhDMOCx (ORCPT ); Tue, 13 Apr 2021 10:02:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53272 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237575AbhDMOCI (ORCPT ); Tue, 13 Apr 2021 10:02:08 -0400 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EBEFDC06175F for ; Tue, 13 Apr 2021 07:01:45 -0700 (PDT) Received: by mail-wm1-x329.google.com with SMTP id o20-20020a05600c4fd4b0290114265518afso8804482wmq.4 for ; Tue, 13 Apr 2021 07:01:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=SrgOTEjinxEkmRzJNt4NPuTMuL2Eyhqp5grWczNHIq4=; b=Iaj0ZL4pB6nKPiMhftItDUnf6QRPb/SQQDegi9x7SWIis5duukE0do3MG097u27HIT 2cnR0LtI1CWkHhoqbUtQwVUJNLnswjHpIS0775QiM4S2T40puCpsU1SYGrtvx/EdUkxr AA11N7EWD8zqygZeEQNbdc+HN3myUxqlYzqEbjjCrvu4qYI2T6tSIV8NEDWYvvVheq2p PDBA0FjxMQWP5NW0SqAHtMiSD6CLj58Tu6a68G/MBtugbURzAN60Hj89vtgcriovgtmX +dQTVE1k4XmpmZDJ2LxifE1uNQ5HYvzJh9JW06tAMgMY0bGy83nezfFcmXSrGH7GJIBB tcYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=SrgOTEjinxEkmRzJNt4NPuTMuL2Eyhqp5grWczNHIq4=; b=YDh1trCTbPR+6ScbFv7+/jBdvkJXoWFwdKBR7jq4vXAcG88d4SBmz91O5vOsvCQbxh BoiSDN0jZKJeD5TQS4OwpqYrVa70tESPLhqZ5usWS7dyoO6NU69XR2ZnVkDkYys+aNqp Ib7jk4+CxwPsAifP4zoIZwrMNV6ZxTzAfhvMO3h83d2XeFF8z4a0Njig1D6iVY1jnuVe 4EOj2Qnj1P+5SMzXa9P3d4eKTOgDTdcL9mc0ITG/G921rM7A3hdxzcJQO4xeevUtKt1l rUvYxx6sV5v44HVDR9HhEazR5BiEIYhn2RxFzZ1v5870IEcI9sSq9jclPWf+GAqXRlEQ X7Bw== X-Gm-Message-State: AOAM532o7Fmw4ItBPhItnVckNHntp8vP+gtSq/oGU7Jvr+mAA7z9cn0l iqjgcJ3bez4xPeX1SaOjzPvuZHtFaxY= X-Google-Smtp-Source: ABdhPJznoy2ISpEUULU+qyRaLoxnzuOq+ddx5JByeOn7dt0nrvwO7ZNqGstx2/hn+QjRVWvWKzN5jw== X-Received: by 2002:a7b:cf27:: with SMTP id m7mr136509wmg.95.1618322504653; Tue, 13 Apr 2021 07:01:44 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id z15sm14034280wrv.39.2021.04.13.07.01.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Apr 2021 07:01:44 -0700 (PDT) Message-Id: <6d7f30f2b90ac2151645a29c6a12639074c8f2d3.1618322498.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 13 Apr 2021 14:01:35 +0000 Subject: [PATCH 08/10] pathspec: stop calling ensure_full_index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The add_pathspec_matches_against_index() focuses on matching a pathspec to file entries in the index. This already works correctly for its only use: checking if untracked files exist in the index. The compatibility checks in t1092 already test that 'git add ' works for a directory outside of the sparse cone. That provides coverage for removing this guard. This finalizes our ability to run 'git add .' without expanding a sparse index to a full one. This is evidenced by an update to t1092 and by these performance numbers for p2000-sparse-operations.sh: Test HEAD~1 HEAD -------------------------------------------------------------------------------- 2000.10: git add . (full-index-v3) 1.37(1.02+0.18) 1.38(1.01+0.20) +0.7% 2000.11: git add . (full-index-v4) 1.26(1.00+0.15) 1.27(0.99+0.17) +0.8% 2000.12: git add . (sparse-index-v3) 2.39(2.29+0.14) 0.06(0.05+0.07) -97.5% 2000.13: git add . (sparse-index-v4) 2.42(2.32+0.14) 0.06(0.05+0.06) -97.5% While the 97% improvement is shown by the test results, it is worth noting that expanding the sparse index was adding overhead in previous commits. Comparing to the full index case, we see the performance go from 1.27s to 0.06s, a 95% improvement. Signed-off-by: Derrick Stolee --- pathspec.c | 2 -- t/t1092-sparse-checkout-compatibility.sh | 6 ++++++ 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/pathspec.c b/pathspec.c index 54813c0c4e8e..b51b48471fe6 100644 --- a/pathspec.c +++ b/pathspec.c @@ -37,8 +37,6 @@ void add_pathspec_matches_against_index(const struct pathspec *pathspec, num_unmatched++; if (!num_unmatched) return; - /* TODO: audit for interaction with sparse-index. */ - ensure_full_index(istate); for (i = 0; i < istate->cache_nr; i++) { const struct cache_entry *ce = istate->cache[i]; if (sw_action == PS_IGNORE_SKIP_WORKTREE && ce_skip_worktree(ce)) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index c210dba78067..738013b00191 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -471,6 +471,12 @@ test_expect_success 'sparse-index is not expanded' ' echo >>sparse-index/extra.txt && GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ git -C sparse-index add extra.txt && + test_region ! index ensure_full_index trace2.txt && + + rm trace2.txt && + echo >>sparse-index/untracked.txt && + GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ + git -C sparse-index add . && test_region ! index ensure_full_index trace2.txt ' From patchwork Tue Apr 13 14:01:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12200611 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 144D4C433ED for ; Tue, 13 Apr 2021 14:02:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id ED79F61206 for ; Tue, 13 Apr 2021 14:02:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243527AbhDMOC7 (ORCPT ); Tue, 13 Apr 2021 10:02:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53286 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231927AbhDMOCI (ORCPT ); Tue, 13 Apr 2021 10:02:08 -0400 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B8903C061574 for ; Tue, 13 Apr 2021 07:01:47 -0700 (PDT) Received: by mail-wr1-x42f.google.com with SMTP id a4so16601831wrr.2 for ; Tue, 13 Apr 2021 07:01:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=bDJHZj1SEW4NF6/qiK9Pt7qfHcqQUwE1hZUVJoNYJFI=; b=vPt83GjB5k8HynTZQh+ytanxvf5EHQz2t6hIOhgsgcDInpcz9P662REP1ybO2VkviJ s7M4scxkjqnxT4cL9IWkzLLVDyR6Bpx9JCoeFbWFkIvPYqtF+VkB1OAP+ghfYA/0sbbj UfP5j7JPfoVX88K/SplBBMXkUguuk4AUnupS859zIgdWfIpq352ZAiCAf08bQtzz8Co5 T7Z/tgjlPy48/ygmjw6RzJ5IdzDFquTOM/2+a7Og1H46Hc3msLEBgXNgM9LG97ZCqHrV gPt9NESoBmLQ0OYxwuiQMYSVYuIvppApKLem7dz8+x2w9h8RSbbtYzXzvbZNuprcZPgF myzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=bDJHZj1SEW4NF6/qiK9Pt7qfHcqQUwE1hZUVJoNYJFI=; b=bLMaVA2cP7Yv7htYf5S4LViQY18rhvFdYQg0NLkmS8fuMgtKf5IrHx0d1o8Bzimbth nrOaEE+tO8GZh7Kc21qGlWJbgn5pNtKdQvapfq1C1+e4yrt14OyDPQ7duVwpIX0g3YnF oLozVFTLcH+gM5pSkeBa6EWmCydrbBpaqWkJ0eVXKUHJS4L0TUCSnQSrNJQnLV5wRx5c cIYQ2s9z5Gt+PYGXrMwNu3Rhrq/3wOHQxLhr86LkxHBLtxOoZZWZ4AKwroeWdn1ath3D 5Iq5x5iwC/L1oRAVkg/YPCXxl2Swqte7yEd9cQqqNTYaP3bUo+SztYZT0vUemZivfNkr DfAw== X-Gm-Message-State: AOAM531H7X/wCl5gQ/zuVIhLYBRDHGhDT+maYmT5dF7UYjNJPV9HkTtS QvotfTMuyTKoGBokHDUMUKIghWjhiVY= X-Google-Smtp-Source: ABdhPJwj72BQlw180TYGXUmjRtsqlM0lxHsmntEtpWbIMqRiFXOiPDMjaukMZjV8OFMtBP7lV0+awg== X-Received: by 2002:a05:6000:18d2:: with SMTP id w18mr37235590wrq.88.1618322505267; Tue, 13 Apr 2021 07:01:45 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id a7sm21846541wrn.50.2021.04.13.07.01.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Apr 2021 07:01:45 -0700 (PDT) Message-Id: <75199bbe8ca179b7d2e2b09831af0d479bd28e6e.1618322498.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 13 Apr 2021 14:01:36 +0000 Subject: [PATCH 09/10] t7519: add sparse directories to FS monitor tests Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The File System Monitor (FS Monitor) tests in t7519 demonstrate some important interactions with the index and the response from the FS Monitor hook. Later changes will integrate the FS Monitor extension in the index with the existence of sparse directory entries in a sparse index. To do so, we need to include directories outside of the sparse checkout definition. Add a new directory, dir1a, between dir1 and dir2 in the test repo used by this script. By inserting it in the middle, we are more likely to trigger incorrect behavior when the fsmonitor_dirty bitmap is involved with sparse directories changing the position of cache entries. I could have modified the test to create two repos, one sparse and one not, but that causes confusion in the expected output. Further, it makes the test take twice as long. With this approach, we can validate that FS Monitor works with the sparse index feature using the GIT_TEST_SPARSE_INDEX=1 environment variable. The test currently fails with that environment variable because FS Monitor is disabled when a sparse index exists. The following changes will update this behavior. Signed-off-by: Derrick Stolee --- t/t7519-status-fsmonitor.sh | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh index 45d025f96010..23879d967297 100755 --- a/t/t7519-status-fsmonitor.sh +++ b/t/t7519-status-fsmonitor.sh @@ -62,11 +62,16 @@ test_expect_success 'setup' ' mkdir dir1 && : >dir1/tracked && : >dir1/modified && + mkdir dir1a && + : >dir1a/a && + : >dir1a/b && mkdir dir2 && : >dir2/tracked && : >dir2/modified && git -c core.fsmonitor= add . && git -c core.fsmonitor= commit -m initial && + git sparse-checkout init --cone --no-sparse-index && + git sparse-checkout set dir1 dir2 && git config core.fsmonitor .git/hooks/fsmonitor-test && cat >.gitignore <<-\EOF .gitignore @@ -99,6 +104,8 @@ test_expect_success 'update-index --no-fsmonitor" removes the fsmonitor extensio cat >expect <expect <expect <expect <expect <expect < X-Patchwork-Id: 12200603 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8254C433B4 for ; Tue, 13 Apr 2021 14:02:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AB914613AB for ; Tue, 13 Apr 2021 14:02:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237908AbhDMOCb (ORCPT ); Tue, 13 Apr 2021 10:02:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53278 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237548AbhDMOCH (ORCPT ); Tue, 13 Apr 2021 10:02:07 -0400 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 196B6C06138F for ; Tue, 13 Apr 2021 07:01:47 -0700 (PDT) Received: by mail-wm1-x329.google.com with SMTP id y20-20020a1c4b140000b029011f294095d3so10740223wma.3 for ; Tue, 13 Apr 2021 07:01:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=KGRjbi4uVmoUD8RSU3rrrD/0eMwup9huQ9SZfnCO0NI=; b=lVlbP1rYJUmsvuM/pruoyICbrCbc1W385IgvmRScyjArDnAToiYOhuAUqBNuiZIqqT uuixrLgVDmCkHmAzAG+gIiI+mJoyeYMW0tZ6FD2NTu6/rEebVQInrGXU3zrV/BWpZ3ju vcuZQ4+ZOBaFQWXI6j8WdF/ugIf0EbtcsYsBM1jetb7p57eQJCSZ7ELc8S1wSZTFLGHQ iPA2DB0NtmkosHFZMIItPsSP/5B5c7ZgXVUptYzNeBvJ+8rgApxdWhj96k3Ys38ilt7G DcAWVPQdcrtSHmZox+nFwxwDzafAU4WIAMKEj5INLkr7M9nja4StvBcSBv+3Oq22XpKe QwNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=KGRjbi4uVmoUD8RSU3rrrD/0eMwup9huQ9SZfnCO0NI=; b=O4fkTgrIjA9me8MKVMgBue9NmXvKHMDVF0AwMYLnqy4RLtI/AI2YN+DFqfWKc0v6jC rqG4Iro3AcDIs82BqioNsTycttRU1PRl609ZQQsnAekCtOxs8UfHQ1mXB+FVcjRl0Dpq TYrM84OulzalyOtb/jx7f7ftPDFBa5A+7qPRW4e/nR1tp/Ahm1+1x7uk6JGbb8PDIdvh XbhhSZLfOJjKg8M3iXrdVi1VZssMh3v/b1/t4WjgL1e9E3tNjd90KBo/NzrinUrLVif4 3gQ/lmhMSopcKycf90M8U0sOzKk2YxFv68BER6M9ZMnS12VUtBU8EEKdNWsi5K7M8g63 0UVQ== X-Gm-Message-State: AOAM531D9WwPvYKIPPXX6R8rrUX3EihYGcMTm/5RtNl+Mt/tA9GrTtse YbeB4CPjwQvHois3+maJRra3ezPfKrQ= X-Google-Smtp-Source: ABdhPJzw6pp2RiTtoI2Ojzc9k4zIPoELpMIsy7nZJZeExAZuHcWenOTZrUOXcCD/NOhzlMnE1ospUA== X-Received: by 2002:a7b:c303:: with SMTP id k3mr173288wmj.100.1618322505816; Tue, 13 Apr 2021 07:01:45 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id a15sm20190174wrr.53.2021.04.13.07.01.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Apr 2021 07:01:45 -0700 (PDT) Message-Id: <9d1183ddd280b1e9a580dcbfe9761ceec8d0bc80.1618322498.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 13 Apr 2021 14:01:37 +0000 Subject: [PATCH 10/10] fsmonitor: test with sparse index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee During the effort to protect uses of the index to operate on a full index, we did not modify fsmonitor.c. This is because it already works effectively with only the change to index_name_stage_pos(). The only thing left to do is to test that it works correctly. These tests are added to demonstrate that the behavior is the same across a full index and a sparse index, but also that file modifications to a tracked directory outside of the sparse cone will trigger ensure_full_index(). Signed-off-by: Derrick Stolee --- t/t7519-status-fsmonitor.sh | 48 +++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh index 23879d967297..306157d48abf 100755 --- a/t/t7519-status-fsmonitor.sh +++ b/t/t7519-status-fsmonitor.sh @@ -78,6 +78,7 @@ test_expect_success 'setup' ' expect* actual* marker* + trace2* EOF ' @@ -400,4 +401,51 @@ test_expect_success 'status succeeds after staging/unstaging' ' ) ' +test_expect_success 'status succeeds with sparse index' ' + test_config core.fsmonitor "$TEST_DIRECTORY/t7519/fsmonitor-all" && + git status --porcelain=v2 >expect && + git sparse-checkout init --cone --sparse-index && + GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ + git status --porcelain=v2 >actual && + test_region ! index ensure_full_index trace2.txt && + test_cmp expect actual && + rm trace2.txt && + + write_script .git/hooks/fsmonitor-test<<-\EOF && + printf "last_update_token\0" + EOF + git config core.fsmonitor .git/hooks/fsmonitor-test && + git status --porcelain=v2 >expect && + git sparse-checkout init --cone --sparse-index && + GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ + git status --porcelain=v2 >actual && + test_region ! index ensure_full_index trace2.txt && + test_cmp expect actual && + rm trace2.txt && + + write_script .git/hooks/fsmonitor-test<<-\EOF && + printf "last_update_token\0" + printf "dir1/modified\0" + EOF + git config core.fsmonitor .git/hooks/fsmonitor-test && + git status --porcelain=v2 >expect && + git sparse-checkout init --cone --sparse-index && + GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ + git status --porcelain=v2 >actual && + test_region ! index ensure_full_index trace2.txt && + test_cmp expect actual && + + write_script .git/hooks/fsmonitor-test<<-\EOF && + printf "last_update_token\0" + printf "dir1a/modified\0" + EOF + git config core.fsmonitor .git/hooks/fsmonitor-test && + git status --porcelain=v2 >expect && + git sparse-checkout init --cone --sparse-index && + GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ + git status --porcelain=v2 >actual && + test_region index ensure_full_index trace2.txt && + test_cmp expect actual +' + test_done