From patchwork Tue Jun 29 01:51:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12348757 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5843FC11F64 for ; Tue, 29 Jun 2021 01:52:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3EBAE61CCE for ; Tue, 29 Jun 2021 01:52:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231849AbhF2Byp (ORCPT ); Mon, 28 Jun 2021 21:54:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46988 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231429AbhF2ByX (ORCPT ); Mon, 28 Jun 2021 21:54:23 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 43708C0617AD for ; Mon, 28 Jun 2021 18:51:52 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id b3so23641493wrm.6 for ; Mon, 28 Jun 2021 18:51:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=ECbNMnswesiYtXXqYlcprEFedHSup0mr+SkRjMRoLXE=; b=vTe+RLbBeRGKatg0dqwDAHHmyIY39gWQm4S9cerUrM/mg5y415Ph5U9xb5yJV7ZHSk BiHglw4urHkQdSw7tJnhUyVZvdbBMeHxeGPM9/Vyb6Hr+WznMEWcTYD0JetMFwiLVniT ggI4okTPbDJ74IW5V2pwDFx0WQN9cxGRY7rB2ilEPryoSwmkB8KNcOLVjBo/jrxCCHoe NSfi6v/N0mGOQZ1qws4dy8lZVeSL5V3mRS9Gnb+JLmmp2t/l5XUj3qIEmjL6tpGPAd7g zmXIlyFmSQEpVMcvykfInXA/ZniMFlQctueS8LLw7PLBWiZrbr9xsz+KImiGTaFIG5Cg cJnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=ECbNMnswesiYtXXqYlcprEFedHSup0mr+SkRjMRoLXE=; b=h0nvXqcF0QBwcb3dR+AWflfs9uDyTXXRPayEO5LpSMeJJC4n1kA80XG9Fk5yPz5VXH GMmzwme7nVhL91XptdK7PpUa1JxpIpV38nOwfwmfAS+rg9B4s4LNT3igDr76sIkb+XAX P2PrOl0/2s7kDRlW+VLLAMx1i8/6i5GMY3l8DyFDFBwLR4qIkNzLObH4H7a6gQKtmEYy XTAdMzsq6ZCXc5m2ujbccYaGmbvqPDqRmzJYB/IMfdZnNlCSAIsYeojHJa68iB4IPo2s lnKD6osWPeLA9OHdUkmVBwW0iN4tu0zG2dd4624UkJsDfwd72Mv71yGeJCU/GtmVUke2 ILgQ== X-Gm-Message-State: AOAM5313mZCtklGPe7XVCz0+uRu/GZXU0QkqZfr1Ikijiysoj3nz9OdR pB0UcZwsPBq3/IS6VoBYoWD+bZKKKtU= X-Google-Smtp-Source: ABdhPJzBHjg2Kh0oeMhFVSAIYu7FA2v1PFZxQ7tGiybpCWVoBbCh+giC6eGB3W9RH0gue79RSpZWBg== X-Received: by 2002:a5d:6da3:: with SMTP id u3mr15388034wrs.394.1624931510884; Mon, 28 Jun 2021 18:51:50 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 9sm1631096wmf.3.2021.06.28.18.51.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 18:51:50 -0700 (PDT) Message-Id: <3499105eb67f8514589b5839c0661b4049714fb7.1624931502.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 29 Jun 2021 01:51:40 +0000 Subject: [PATCH v6 12/14] status: use sparse-index throughout Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee By testing 'git -c core.fsmonitor= status -uno', we can check for the simplest index operations that can be made sparse-aware. The necessary implementation details are already integrated with sparse-checkout, so modify command_requires_full_index to be zero for cmd_status(). In refresh_index(), we loop through the index entries to refresh their stat() information. However, sparse directories have no stat() information to populate. Ignore these entries. This allows 'git status' to no longer expand a sparse index to a full one. This is further tested by dropping the "-uno" option and adding an untracked file into the worktree. The performance test p2000-sparse-checkout-operations.sh demonstrates these improvements: Test HEAD~1 HEAD ----------------------------------------------------------------------------- 2000.2: git status (full-index-v3) 0.31(0.30+0.05) 0.31(0.29+0.06) +0.0% 2000.3: git status (full-index-v4) 0.31(0.29+0.07) 0.34(0.30+0.08) +9.7% 2000.4: git status (sparse-index-v3) 2.35(2.28+0.10) 0.04(0.04+0.05) -98.3% 2000.5: git status (sparse-index-v4) 2.35(2.24+0.15) 0.05(0.04+0.06) -97.9% Note that since HEAD~1 was expanding the sparse index by parsing trees, it was artificially slower than the full index case. Thus, the 98% improvement is misleading, and instead we should celebrate the 0.34s to 0.05s improvement of 85%. This is more indicative of the peformance gains we are expecting by using a sparse index. Note: we are dropping the assignment of core.fsmonitor here. This is not necessary for the test script as we are not altering the config any other way. Correct integration with FS Monitor will be validated in later changes. Signed-off-by: Derrick Stolee --- builtin/commit.c | 3 +++ read-cache.c | 10 ++++++++-- t/t1092-sparse-checkout-compatibility.sh | 13 +++++++++---- 3 files changed, 20 insertions(+), 6 deletions(-) diff --git a/builtin/commit.c b/builtin/commit.c index 190d215d43b..12f51db158a 100644 --- a/builtin/commit.c +++ b/builtin/commit.c @@ -1510,6 +1510,9 @@ int cmd_status(int argc, const char **argv, const char *prefix) if (argc == 2 && !strcmp(argv[1], "-h")) usage_with_options(builtin_status_usage, builtin_status_options); + prepare_repo_settings(the_repository); + the_repository->settings.command_requires_full_index = 0; + status_init_config(&s, git_status_config); argc = parse_options(argc, argv, prefix, builtin_status_options, diff --git a/read-cache.c b/read-cache.c index 1b3c2eb408b..277c2970a03 100644 --- a/read-cache.c +++ b/read-cache.c @@ -1584,8 +1584,7 @@ int refresh_index(struct index_state *istate, unsigned int flags, */ preload_index(istate, pathspec, 0); trace2_region_enter("index", "refresh", NULL); - /* TODO: audit for interaction with sparse-index. */ - ensure_full_index(istate); + for (i = 0; i < istate->cache_nr; i++) { struct cache_entry *ce, *new_entry; int cache_errno = 0; @@ -1600,6 +1599,13 @@ int refresh_index(struct index_state *istate, unsigned int flags, if (ignore_skip_worktree && ce_skip_worktree(ce)) continue; + /* + * If this entry is a sparse directory, then there isn't + * any stat() information to update. Ignore the entry. + */ + if (S_ISSPARSEDIR(ce->ce_mode)) + continue; + if (pathspec && !ce_path_match(istate, ce, pathspec, seen)) filtered = 1; diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 9035adcb7db..1e9737cb4b7 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -508,12 +508,17 @@ test_expect_success 'sparse-index is expanded and converted back' ' GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ git -C sparse-index -c core.fsmonitor="" reset --hard && test_region index convert_to_sparse trace2.txt && - test_region index ensure_full_index trace2.txt && + test_region index ensure_full_index trace2.txt +' - rm trace2.txt && +test_expect_success 'sparse-index is not expanded' ' + init_repos && + + rm -f trace2.txt && + echo >>sparse-index/untracked.txt && GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ - git -C sparse-index -c core.fsmonitor="" status -uno && - test_region index ensure_full_index trace2.txt + git -C sparse-index status && + test_region ! index ensure_full_index trace2.txt ' test_done