From patchwork Mon Jan 25 17:41:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12043855 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 940D2C433DB for ; Mon, 25 Jan 2021 17:52:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6A90622D50 for ; Mon, 25 Jan 2021 17:52:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730795AbhAYRvq (ORCPT ); Mon, 25 Jan 2021 12:51:46 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60142 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731000AbhAYRuH (ORCPT ); Mon, 25 Jan 2021 12:50:07 -0500 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8C23FC06178A for ; Mon, 25 Jan 2021 09:42:18 -0800 (PST) Received: by mail-wm1-x32d.google.com with SMTP id u14so38584wml.4 for ; Mon, 25 Jan 2021 09:42:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=u5K9PFa5Vfs4PYusDpkPAqQOyigt8N/TlkTHkWaruOk=; b=U8K5wG6K36HGSdduah12yIYbBTJhLTFcCFyZsyxu+1XYd3CoGEG7qdut71uCdkfxT3 c/Hw2MtW/aM9id3+3ABpTrpK8FtS3MkjSUC8ZW+S8qyo9xWgWxOyECjV76sLg4D1/MGC gJa1isRUgWQx9qR3bd99oihXFCfdGPLIzzVkR8kYFRwEd/nUnbuD04Bogt1K+d2wkbTB hicpb5PyT9zDdWY1/+6tljq7MmKM4E73pJz2+v7Xl9nrcbrYLWln0pt6qEENd4Sy4mvu T9JEtz9BZHCDxOqdldhKfyitLdcLxRbYyYVP6CeQ9zu3ik9yP2AG3NHfqi73530tY7JV SrzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=u5K9PFa5Vfs4PYusDpkPAqQOyigt8N/TlkTHkWaruOk=; b=HeGTzp6MQ+l49D3q5vZxZN/uhID7uXY3BQ4j45OnQ7rz0RAhdMHAsr6qI48ciiQ1cS p5/zoNUp0p4M5+yPP6Qn9QMPrtzi84JSj3RbpqxzTLBFihdIZ7kSR93vBgZD/Gy47xwa mbJf4ZwuovZny394FIZqym9k6MA08k37BwePYFuM/vnHok00yoDzctOf6QVrGZ/y6lwH gLEM9XTVfaNtaRWaCRQjo/+kkzbhNr9j91Rbks6ZnlJBqnwZP+hfANssz8wS4l6dsi+3 jQFhLBqbU81JzYqf5VyZgpplec14IdPUFZOIwAsOK0LV6sdYjlMPkPbcs3hsCawAPbTP 15/A== X-Gm-Message-State: AOAM533VgH5uZdHR1AR+LkH/gJ+vDWXnQ1e1uQlaG9DIjRwHTJcxulsL 1Z/YN687nGVlb7i6IY2qdpgmfZ2zrIc= X-Google-Smtp-Source: ABdhPJw1BjrarTqyxTe8PlcVVg5XG0yrzHPDGrSV59x9qJdv1l5bXhuOrk93cP/hjgxIuWNv9mlmgw== X-Received: by 2002:a1c:a549:: with SMTP id o70mr1137845wme.71.1611596537063; Mon, 25 Jan 2021 09:42:17 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l14sm9274276wrq.87.2021.01.25.09.42.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 09:42:16 -0800 (PST) Message-Id: <3c31623f6aeef0b00fc51b4476c503da607c2ae5.1611596534.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 25 Jan 2021 17:41:47 +0000 Subject: [PATCH 01/27] sparse-index: add guard to ensure full index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, peff@peff.net, jrnieder@gmail.com, sunshine@sunshineco.com, pclouds@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Upcoming changes will introduce modifications to the index format that allow sparse directories. It will be useful to have a mechanism for converting those sparse index files into full indexes by walking the tree at those sparse directories. Name this method ensure_full_index() as it will guarantee that the index is fully expanded. This method is not implemented yet, and instead we focus on the scaffolding to declare it and call it at the appropriate time. Add a 'command_requires_full_index' member to struct repo_settings. This will be an indicator that we need the index in full mode to do certain index operations. This starts as being true for every command, then we will set it to false as some commands integrate with sparse indexes. If 'command_requires_full_index' is true, then we will immediately expand a sparse index to a full one upon reading from disk. This suffices for now, but we will want to add more callers to ensure_full_index() later. Signed-off-by: Derrick Stolee --- Makefile | 1 + repo-settings.c | 8 ++++++++ repository.c | 11 ++++++++++- repository.h | 2 ++ sparse-index.c | 8 ++++++++ sparse-index.h | 7 +++++++ 6 files changed, 36 insertions(+), 1 deletion(-) create mode 100644 sparse-index.c create mode 100644 sparse-index.h diff --git a/Makefile b/Makefile index 7b64106930a..77564ae3b78 100644 --- a/Makefile +++ b/Makefile @@ -999,6 +999,7 @@ LIB_OBJS += sha1-name.o LIB_OBJS += shallow.o LIB_OBJS += sideband.o LIB_OBJS += sigchain.o +LIB_OBJS += sparse-index.o LIB_OBJS += split-index.o LIB_OBJS += stable-qsort.o LIB_OBJS += strbuf.o diff --git a/repo-settings.c b/repo-settings.c index f7fff0f5ab8..d63569e4041 100644 --- a/repo-settings.c +++ b/repo-settings.c @@ -77,4 +77,12 @@ void prepare_repo_settings(struct repository *r) UPDATE_DEFAULT_BOOL(r->settings.core_untracked_cache, UNTRACKED_CACHE_KEEP); UPDATE_DEFAULT_BOOL(r->settings.fetch_negotiation_algorithm, FETCH_NEGOTIATION_DEFAULT); + + /* + * This setting guards all index reads to require a full index + * over a sparse index. After suitable guards are placed in the + * codebase around uses of the index, this setting will be + * removed. + */ + r->settings.command_requires_full_index = 1; } diff --git a/repository.c b/repository.c index c98298acd01..a8acae002f7 100644 --- a/repository.c +++ b/repository.c @@ -10,6 +10,7 @@ #include "object.h" #include "lockfile.h" #include "submodule-config.h" +#include "sparse-index.h" /* The main repository */ static struct repository the_repo; @@ -261,6 +262,8 @@ void repo_clear(struct repository *repo) int repo_read_index(struct repository *repo) { + int res; + if (!repo->index) repo->index = xcalloc(1, sizeof(*repo->index)); @@ -270,7 +273,13 @@ int repo_read_index(struct repository *repo) else if (repo->index->repo != repo) BUG("repo's index should point back at itself"); - return read_index_from(repo->index, repo->index_file, repo->gitdir); + res = read_index_from(repo->index, repo->index_file, repo->gitdir); + + prepare_repo_settings(repo); + if (repo->settings.command_requires_full_index) + ensure_full_index(repo->index); + + return res; } int repo_hold_locked_index(struct repository *repo, diff --git a/repository.h b/repository.h index b385ca3c94b..e06a2301569 100644 --- a/repository.h +++ b/repository.h @@ -41,6 +41,8 @@ struct repo_settings { enum fetch_negotiation_setting fetch_negotiation_algorithm; int core_multi_pack_index; + + unsigned command_requires_full_index:1; }; struct repository { diff --git a/sparse-index.c b/sparse-index.c new file mode 100644 index 00000000000..82183ead563 --- /dev/null +++ b/sparse-index.c @@ -0,0 +1,8 @@ +#include "cache.h" +#include "repository.h" +#include "sparse-index.h" + +void ensure_full_index(struct index_state *istate) +{ + /* intentionally left blank */ +} diff --git a/sparse-index.h b/sparse-index.h new file mode 100644 index 00000000000..8dda92032e2 --- /dev/null +++ b/sparse-index.h @@ -0,0 +1,7 @@ +#ifndef SPARSE_INDEX_H__ +#define SPARSE_INDEX_H__ + +struct index_state; +void ensure_full_index(struct index_state *istate); + +#endif \ No newline at end of file From patchwork Mon Jan 25 17:41:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12043841 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EAEAFC433DB for ; Mon, 25 Jan 2021 17:50:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B51C122D04 for ; Mon, 25 Jan 2021 17:50:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731193AbhAYRuX (ORCPT ); Mon, 25 Jan 2021 12:50:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:60148 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730979AbhAYRuH (ORCPT ); Mon, 25 Jan 2021 12:50:07 -0500 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 68A2AC061797 for ; Mon, 25 Jan 2021 09:42:19 -0800 (PST) Received: by mail-wr1-x432.google.com with SMTP id h9so3568961wrr.9 for ; Mon, 25 Jan 2021 09:42:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=bM7h5tZklpRmsUG1HJvppNFZIAHSL84NVJpJrjtxJEE=; b=CpG46/A65PkH7pO1dsjXvT3YYUcqu9f3Rl0r1p7xgNpI/bgLg5H+I7Ed/t6s6G2d4Y efE+SxvTZCNq0F3W5aGQDri7cH/1zJh6JaVkuRDi1UdFcNcfCbIaIL2oCf63X2ibKYN+ x+QWFDaCuAjlZXgbJ4Y/ZKuM0WBjvk9ENKlXa0SV+9hljimbOY0i+VTTBkdDm7hf94RI e3GiUddpWUFrCz5BEZuGznSNvmt8mT8JtyrIfcOt51IyNeF8bEpOF4uJDwOa07/0DPgt fAIgblq7DsslC6q/ud6E2xTlvVrz3mQqiSOBmgbIg8QNQBaAbcgFZ/5jpIV9sWb4M9Kc sXnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=bM7h5tZklpRmsUG1HJvppNFZIAHSL84NVJpJrjtxJEE=; b=MoF1M7c+ets657tzIJMMRI3o/BvCQQL6jum/NYnvCuQf9nBDKgtYjhsk64G1EtJ4VE A2IGhQxk6ltdH+jcdGwk/TaM0bDxNLPBykKpQZs1PExh63FaS8ap/z1XNQqb8XJH37FM /NYCCtZp50g7RFudh8FrAUikzb1c77bdW1fQsvkfIAZPFQPCQy9UkCmjgRwTMlW003Je zSY6hSlPCf9iWcu6seugxmrDmFbczwfk9zgSKHfV762gndoaC7My+c2az6o0cJcMu6BL F42GUwExEXilz94VnjNLn/KZlmue6mA46ChM95p/rg0icC5jtsxq/ZgSYKA04EZowX1A s9UA== X-Gm-Message-State: AOAM533fg4F9RxDWn6WWktuQYxKYNJGvNpZkU5GyWPsAjZMxYymL5uny MkD9DzoA7zqRGtkpR7b7IM2xCV9hDL8= X-Google-Smtp-Source: ABdhPJzqCszeCOK7PNxC/Fk3TmgKrvLDLQ7sBRD/jLAGV1Y6hMCDOaKdu6yklWjvfXYDtoesA15LMw== X-Received: by 2002:a5d:654f:: with SMTP id z15mr2261463wrv.46.1611596537999; Mon, 25 Jan 2021 09:42:17 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id h1sm20453349wrr.73.2021.01.25.09.42.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 09:42:17 -0800 (PST) Message-Id: In-Reply-To: References: Date: Mon, 25 Jan 2021 17:41:48 +0000 Subject: [PATCH 02/27] sparse-index: implement ensure_full_index() Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, peff@peff.net, jrnieder@gmail.com, sunshine@sunshineco.com, pclouds@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee We will mark an in-memory index_state as having sparse directory entries with the sparse_index bit. These currently cannot exist, but we will add a mechanism for collapsing a full index to a sparse one in a later change. That will happen at write time, so we must first allow parsing the format before writing it. Commands or methods that require a full index in order to operate can call ensure_full_index() to expand that index in-memory. This requires parsing trees using that index's repository. Sparse directory entries have a specific 'ce_mode' value. The macro S_ISSPARSEDIR(ce) can check if a cache_entry 'ce' has this type. This ce_mode is not possible with the existing index formats, so we don't also verify all properties of a sparse-directory entry, which are: 1. ce->ce_mode == 01000755 2. ce->flags & CE_SKIP_WORKTREE is true 3. ce->name[ce->namelen - 1] == '/' (ends in dir separator) 4. ce->oid references a tree object. These are all semi-enforced in ensure_full_index() to some extent. Any deviation will cause a warning at minimum or a failure in the worst case. Signed-off-by: Derrick Stolee --- cache.h | 11 +++++- read-cache.c | 9 +++++ sparse-index.c | 94 +++++++++++++++++++++++++++++++++++++++++++++++++- sparse-index.h | 1 + 4 files changed, 113 insertions(+), 2 deletions(-) diff --git a/cache.h b/cache.h index f9c7a603841..884046ca5b8 100644 --- a/cache.h +++ b/cache.h @@ -204,6 +204,10 @@ struct cache_entry { #error "CE_EXTENDED_FLAGS out of range" #endif +#define CE_MODE_SPARSE_DIRECTORY 01000755 +#define SPARSE_DIR_MODE 0100 +#define S_ISSPARSEDIR(m) ((m)->ce_mode == CE_MODE_SPARSE_DIRECTORY) + /* Forward structure decls */ struct pathspec; struct child_process; @@ -249,6 +253,8 @@ static inline unsigned int create_ce_mode(unsigned int mode) { if (S_ISLNK(mode)) return S_IFLNK; + if (mode == SPARSE_DIR_MODE) + return CE_MODE_SPARSE_DIRECTORY; if (S_ISDIR(mode) || S_ISGITLINK(mode)) return S_IFGITLINK; return S_IFREG | ce_permissions(mode); @@ -319,7 +325,8 @@ struct index_state { drop_cache_tree : 1, updated_workdir : 1, updated_skipworktree : 1, - fsmonitor_has_run_once : 1; + fsmonitor_has_run_once : 1, + sparse_index : 1; struct hashmap name_hash; struct hashmap dir_hash; struct object_id oid; @@ -721,6 +728,8 @@ int read_index_from(struct index_state *, const char *path, const char *gitdir); int is_index_unborn(struct index_state *); +void ensure_full_index(struct index_state *istate); + /* For use with `write_locked_index()`. */ #define COMMIT_LOCK (1 << 0) #define SKIP_IF_UNCHANGED (1 << 1) diff --git a/read-cache.c b/read-cache.c index ecf6f689940..1097ecbf132 100644 --- a/read-cache.c +++ b/read-cache.c @@ -101,6 +101,9 @@ static const char *alternate_index_output; static void set_index_entry(struct index_state *istate, int nr, struct cache_entry *ce) { + if (S_ISSPARSEDIR(ce)) + istate->sparse_index = 1; + istate->cache[nr] = ce; add_name_hash(istate, ce); } @@ -2255,6 +2258,12 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist) trace2_data_intmax("index", the_repository, "read/cache_nr", istate->cache_nr); + if (!istate->repo) + istate->repo = the_repository; + prepare_repo_settings(istate->repo); + if (istate->repo->settings.command_requires_full_index) + ensure_full_index(istate); + return istate->cache_nr; unmap: diff --git a/sparse-index.c b/sparse-index.c index 82183ead563..1e70244dc13 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -1,8 +1,100 @@ #include "cache.h" #include "repository.h" #include "sparse-index.h" +#include "tree.h" +#include "pathspec.h" +#include "trace2.h" + +static void set_index_entry(struct index_state *istate, int nr, struct cache_entry *ce) +{ + ALLOC_GROW(istate->cache, nr + 1, istate->cache_alloc); + + istate->cache[nr] = ce; + add_name_hash(istate, ce); +} + +static int add_path_to_index(const struct object_id *oid, + struct strbuf *base, const char *path, + unsigned int mode, int stage, void *context) +{ + struct index_state *istate = (struct index_state *)context; + struct cache_entry *ce; + size_t len = base->len; + + if (S_ISDIR(mode)) + return READ_TREE_RECURSIVE; + + strbuf_addstr(base, path); + + ce = make_cache_entry(istate, mode, oid, base->buf, 0, 0); + ce->ce_flags |= CE_SKIP_WORKTREE; + set_index_entry(istate, istate->cache_nr++, ce); + + strbuf_setlen(base, len); + return 0; +} void ensure_full_index(struct index_state *istate) { - /* intentionally left blank */ + int i; + struct index_state *full; + + if (!istate || !istate->sparse_index) + return; + + if (!istate->repo) + istate->repo = the_repository; + + trace2_region_enter("index", "ensure_full_index", istate->repo); + + /* initialize basics of new index */ + full = xcalloc(1, sizeof(struct index_state)); + memcpy(full, istate, sizeof(struct index_state)); + + /* then change the necessary things */ + full->sparse_index = 0; + full->cache_alloc = (3 * istate->cache_alloc) / 2; + full->cache_nr = 0; + ALLOC_ARRAY(full->cache, full->cache_alloc); + + for (i = 0; i < istate->cache_nr; i++) { + struct cache_entry *ce = istate->cache[i]; + struct tree *tree; + struct pathspec ps; + + if (!S_ISSPARSEDIR(ce)) { + set_index_entry(full, full->cache_nr++, ce); + continue; + } + if (!(ce->ce_flags & CE_SKIP_WORKTREE)) + warning(_("index entry is a directory, but not sparse (%08x)"), + ce->ce_flags); + + /* recursively walk into cd->name */ + tree = lookup_tree(istate->repo, &ce->oid); + + memset(&ps, 0, sizeof(ps)); + ps.recursive = 1; + ps.has_wildcard = 1; + ps.max_depth = -1; + + read_tree_recursive(istate->repo, tree, + ce->name, strlen(ce->name), + 0, &ps, + add_path_to_index, full); + + /* free directory entries. full entries are re-used */ + discard_cache_entry(ce); + } + + /* Copy back into original index. */ + memcpy(&istate->name_hash, &full->name_hash, sizeof(full->name_hash)); + istate->sparse_index = 0; + istate->cache = full->cache; + istate->cache_nr = full->cache_nr; + istate->cache_alloc = full->cache_alloc; + + free(full); + + trace2_region_leave("index", "ensure_full_index", istate->repo); } diff --git a/sparse-index.h b/sparse-index.h index 8dda92032e2..a2777dcac59 100644 --- a/sparse-index.h +++ b/sparse-index.h @@ -3,5 +3,6 @@ struct index_state; void ensure_full_index(struct index_state *istate); +int convert_to_sparse(struct index_state *istate); #endif \ No newline at end of file From patchwork Mon Jan 25 17:41:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12043883 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ECC5FC433DB for ; Mon, 25 Jan 2021 17:58:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AD68322B3B for ; Mon, 25 Jan 2021 17:58:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731344AbhAYR5F (ORCPT ); Mon, 25 Jan 2021 12:57:05 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33062 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731326AbhAYRz2 (ORCPT ); Mon, 25 Jan 2021 12:55:28 -0500 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 442EAC0617A9 for ; Mon, 25 Jan 2021 09:42:20 -0800 (PST) Received: by mail-wm1-x32a.google.com with SMTP id m2so63342wmm.1 for ; Mon, 25 Jan 2021 09:42:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=ACG9h4H0pFWVZC1W+bCjnciJP/rNO8RkdW8tGnDOLZY=; b=LLYQYh7gMbprME1/kbN1dNhNApHzLy0Q7nme0zOqiyXBgumYppDE54DozHuhIe8kyq NJQIcUixCmpnVsWb9dEV1NUnnNVJwNcIa2tZ0B87LcGT4JL+p1axL7zHtkCHbk/Od1jS EdxWM/8k6wnZMT2C22tftcT6KLIzFzdbdKi3Ni2aASTyStOdjucx/3EspHpWH2fNRYjs yHtr6J1YsCY6iFfyA6DJxaITCNfxCiN0FaGmR3RGCi9ma6BgyWrMqpov+Zcl5n/nlJDF ppLf/BB4OT4lBwP01k7/mSe7q8zaxYil+FubA3zKpeYg2ml7OZsPL5n39lFdeUhF4qtd zoVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=ACG9h4H0pFWVZC1W+bCjnciJP/rNO8RkdW8tGnDOLZY=; b=OVn8VazJIFZATGmHDeBUjbb0FWQfksdgnRRrxAVjwrHxUKcrBsKLDi+nQV+Pe0XnQW UHRPbdVsgG8OjXsVQM1xryJlEE8rb7uAWh0VNc/sjtKz9iERk8xFiVD75HqA5gE4VQ87 A4nhb+ZvYUNzet24Y8l0HF66yod+sw+HS6/9RXEW4I+YlvhT6V7i/g7Qp1H8vych3Qbq iIYv0Jd7fGYAgLUx7zrDUwEw9xLBN7OHgInNCR8He0b46maPDYoCHpcqfdA5gZ39+By7 4Zxrm9T9HAeYaeRThKtZgWM0orbdy2ydSUh/ltEqDp3wbjWLItfC1IxIZtZokyXLu8UN HBjA== X-Gm-Message-State: AOAM5325EAv3xIzbwOx+wVwEtBIEe2IHOOi5YQ6S+Z/4ECqw4Ok9Y+8Z rDuoM0eliWuh+tQtZyr4y67DC9xre4A= X-Google-Smtp-Source: ABdhPJwTSa5fvixQhHsEhLsaCNsW/1qq1/YMSn97NJCEeNlk00O1FGGGE7M/Q04mBdhmXl9Ex5kUtA== X-Received: by 2002:a1c:b7d6:: with SMTP id h205mr1188893wmf.182.1611596538831; Mon, 25 Jan 2021 09:42:18 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id j9sm24283113wrm.14.2021.01.25.09.42.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 09:42:18 -0800 (PST) Message-Id: In-Reply-To: References: Date: Mon, 25 Jan 2021 17:41:49 +0000 Subject: [PATCH 03/27] t1092: compare sparse-checkout to sparse-index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, peff@peff.net, jrnieder@gmail.com, sunshine@sunshineco.com, pclouds@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Add a new 'sparse-index' repo alongside the 'full-checkout' and 'sparse-checkout' repos in t1092-sparse-checkout-compatibility.sh. Also add run_on_sparse and test_sparse_match helpers. These helpers will be used when the sparse index is implemented. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 29 ++++++++++++++++++++---- 1 file changed, 25 insertions(+), 4 deletions(-) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 8cd3e5a8d22..8876eae0fe3 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -7,6 +7,7 @@ test_description='compare full workdir to sparse workdir' test_expect_success 'setup' ' git init initial-repo && ( + (GIT_TEST_SPARSE_INDEX=0 && export GIT_TEST_SPARSE_INDEX) && cd initial-repo && echo a >a && echo "after deep" >e && @@ -87,23 +88,32 @@ init_repos () { cp -r initial-repo sparse-checkout && git -C sparse-checkout reset --hard && - git -C sparse-checkout sparse-checkout init --cone && + + cp -r initial-repo sparse-index && + git -C sparse-index reset --hard && # initialize sparse-checkout definitions - git -C sparse-checkout sparse-checkout set deep + git -C sparse-checkout sparse-checkout init --cone && + git -C sparse-checkout sparse-checkout set deep && + GIT_TEST_SPARSE_INDEX=1 git -C sparse-index sparse-checkout init --cone && + GIT_TEST_SPARSE_INDEX=1 git -C sparse-index sparse-checkout set deep } run_on_sparse () { ( cd sparse-checkout && - $* >../sparse-checkout-out 2>../sparse-checkout-err + GIT_TEST_SPARSE_INDEX=0 $* >../sparse-checkout-out 2>../sparse-checkout-err + ) && + ( + cd sparse-index && + $* >../sparse-index-out 2>../sparse-index-err ) } run_on_all () { ( cd full-checkout && - $* >../full-checkout-out 2>../full-checkout-err + GIT_TEST_SPARSE_INDEX=0 $* >../full-checkout-out 2>../full-checkout-err ) && run_on_sparse $* } @@ -114,6 +124,17 @@ test_all_match () { test_cmp full-checkout-err sparse-checkout-err } +test_sparse_match () { + run_on_sparse $* && + test_cmp sparse-checkout-out sparse-index-out && + test_cmp sparse-checkout-err sparse-index-err +} + +test_expect_success 'expanded in-memory index matches full index' ' + init_repos && + test_sparse_match test-tool read-cache --expand --table-no-stat +' + test_expect_success 'status with options' ' init_repos && test_all_match git status --porcelain=v2 && From patchwork Mon Jan 25 17:41:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12043879 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70B15C433E0 for ; Mon, 25 Jan 2021 17:57:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 41D3C22583 for ; Mon, 25 Jan 2021 17:57:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731363AbhAYR5c (ORCPT ); Mon, 25 Jan 2021 12:57:32 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33052 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731238AbhAYR4A (ORCPT ); Mon, 25 Jan 2021 12:56:00 -0500 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 19789C0617AA for ; Mon, 25 Jan 2021 09:42:21 -0800 (PST) Received: by mail-wm1-x32d.google.com with SMTP id u14so60073wmq.4 for ; Mon, 25 Jan 2021 09:42:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Rs6uqC7Asx9Upezrgq7UeiN/r211M1U3vC2roWojKMg=; b=KpqNJ0JQEYGpcSk41FhlO8WLOMeUsl2inbtOmztR0W5urmruMc47jYmn2oxoJ8xisV 61wwFNo9tDJv/yO4YnfEJYsTYUmnN31bqkpaHPaqMJgvQlUXBhpxgWqpZGF3Xp1p/ygs g7u+3irknhjgPwDCjjRV5oYDF2Og9wm6dyj+InFUgkeiGpbVuAx/9QVnh7z21A7eYjp5 Ww9bA4uKIL8kdYVoL07101ET3E6kCRNAe4iXWLfYrsZwJnCfXHiG4yMZmhSjpKoV+wBd RluQSoQYNvJQitd0L83ulO/d/tu5HurqxF+rulkGLE7V41YfHo9Kn8q+eVRlaln8YzUH gStw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Rs6uqC7Asx9Upezrgq7UeiN/r211M1U3vC2roWojKMg=; b=RHvSinwD+nu+3wmmN5yRI8H5QWGMy/YXT5ZEHG3rgbndVKnzOk7GnoDt0Dcw6i06Of IlnRky2jni7ye+ea0hac9T1Yptr//RFU8pbavnm4KSuu/XSdlcPl0/ZEcFoyBvnU+UbP gTdMLQBpkeBRooPbiUfstiB9dqcTgLabHodRbWmYNri3I6CR93mW1qxvymLqxv4ZrWyX t0ceNmKU6axGIIQRC7zoXwInluwQMLRtKrZI5Ba6e7xBU6lISGp1pzhbuhINoGslkEQS 2pUuxzdyjOVq1i3ty62P22eCZUxo1itmJgh1BQQs46NxosBANh45uvNAasSIYp6iUa/G FIrg== X-Gm-Message-State: AOAM533tkNuJ1XdGyeg0FMAOw6GRfA8wX28+j/uT4CwPI3iS/n6C5Wo3 slYe10Hq2TeY9G15mBCyUdYRJBhjPGs= X-Google-Smtp-Source: ABdhPJyZZw3NLquacgNmNYYAy5j2wuUrZPKhPv7YFMiR4vl7z8OZfOzEkyjtQ8YezKmNWhH4CTG06Q== X-Received: by 2002:a7b:c1c6:: with SMTP id a6mr1155486wmj.23.1611596539726; Mon, 25 Jan 2021 09:42:19 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l7sm6510wmg.41.2021.01.25.09.42.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 09:42:19 -0800 (PST) Message-Id: In-Reply-To: References: Date: Mon, 25 Jan 2021 17:41:50 +0000 Subject: [PATCH 04/27] test-read-cache: print cache entries with --table Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, peff@peff.net, jrnieder@gmail.com, sunshine@sunshineco.com, pclouds@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee This table is helpful for discovering data in the index to ensure it is being written correctly, especially as we build and test the sparse-index. To make the option parsing slightly more robust, wrap the string comparisons in a loop adapted from test-dir-iterator.c. Care must be taken with the final check for the 'cnt' variable. We continue the expectation that the numerical value is the final argument. Signed-off-by: Derrick Stolee --- t/helper/test-read-cache.c | 49 ++++++++++++++++++++++++++++++++++---- 1 file changed, 44 insertions(+), 5 deletions(-) diff --git a/t/helper/test-read-cache.c b/t/helper/test-read-cache.c index 244977a29bd..cd7d106a675 100644 --- a/t/helper/test-read-cache.c +++ b/t/helper/test-read-cache.c @@ -2,18 +2,55 @@ #include "cache.h" #include "config.h" +static void print_cache_entry(struct cache_entry *ce) +{ + /* stat info */ + printf("%08x %08x %08x %08x %08x %08x ", + ce->ce_stat_data.sd_ctime.sec, + ce->ce_stat_data.sd_ctime.nsec, + ce->ce_stat_data.sd_mtime.sec, + ce->ce_stat_data.sd_mtime.nsec, + ce->ce_stat_data.sd_dev, + ce->ce_stat_data.sd_ino); + + /* mode in binary */ + printf("0b%d%d%d%d ", + (ce->ce_mode >> 15) & 1, + (ce->ce_mode >> 14) & 1, + (ce->ce_mode >> 13) & 1, + (ce->ce_mode >> 12) & 1); + + /* output permissions? */ + printf("%04o ", ce->ce_mode & 01777); + + printf("%s ", oid_to_hex(&ce->oid)); + + printf("%s\n", ce->name); +} + +static void print_cache(struct index_state *cache) +{ + int i; + for (i = 0; i < the_index.cache_nr; i++) + print_cache_entry(the_index.cache[i]); +} + int cmd__read_cache(int argc, const char **argv) { int i, cnt = 1; const char *name = NULL; + int table = 0; - if (argc > 1 && skip_prefix(argv[1], "--print-and-refresh=", &name)) { - argc--; - argv++; + for (++argv, --argc; *argv && starts_with(*argv, "--"); ++argv, --argc) { + if (skip_prefix(*argv, "--print-and-refresh=", &name)) + continue; + if (!strcmp(*argv, "--table")) { + table = 1; + } } - if (argc == 2) - cnt = strtol(argv[1], NULL, 0); + if (argc == 1) + cnt = strtol(argv[0], NULL, 0); setup_git_directory(); git_config(git_default_config, NULL); for (i = 0; i < cnt; i++) { @@ -30,6 +67,8 @@ int cmd__read_cache(int argc, const char **argv) ce_uptodate(the_index.cache[pos]) ? "" : " not"); write_file(name, "%d\n", i); } + if (table) + print_cache(&the_index); discard_cache(); } return 0; From patchwork Mon Jan 25 17:41:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12043907 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EBB27C433DB for ; Mon, 25 Jan 2021 18:05:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BC83422B3F for ; Mon, 25 Jan 2021 18:05:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730683AbhAYSFA (ORCPT ); Mon, 25 Jan 2021 13:05:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33224 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731307AbhAYRzw (ORCPT ); Mon, 25 Jan 2021 12:55:52 -0500 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0D549C0617AB for ; Mon, 25 Jan 2021 09:42:22 -0800 (PST) Received: by mail-wm1-x335.google.com with SMTP id 190so70478wmz.0 for ; Mon, 25 Jan 2021 09:42:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=SpreOg0UdiWIA47xLMGMTTmZvzsRrDk/5CpRa4HSDDE=; b=isbbZCKzNyfykAmrO/Yiq7jWH4S+M1uvPx9nbqHufVjdICu7L2fJyQcdJpZhdfrer6 ZKN+g8zrRKg+HzSzTukh47uFZY5v0vhYW7XlwrQkIjH1tOlyPEP2LrdzSIFwpNrywrLb aXUF3TtIuCbqmo19KAjIJkStd/lv5fvoxPy4uxuuB4YSvXaL3KESqF1Zz+JhBwzJ4WNz /oHoL+DVDCemklHhI9NQ7wCRWLC19ZvW+c/21sQzdVVrcur4lkYgpg9Teiv+/ZXOMtJa sDZoytvPuVO6BERRikMG2VZn0KneP2N+vkJSI/5sD5X7KEsKS3A5wvvXaE//bG2/7z2E 6S1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=SpreOg0UdiWIA47xLMGMTTmZvzsRrDk/5CpRa4HSDDE=; b=dX33xroWCkWYxVeA/svs4WhfEcGlbwNXeIQ5hL6rVDGEc20GFzgzBQnSB6fJLZ6a1q ewHr4URsbuIZTRoLxGAKEijl7bHqPD3YAI9Frm5AARja8H6qIpsxNO9z+1s3nC23+VtJ VwMdh6CwtzbDNEi4Tj9JQLTEDzEJ58ElW5PqVvq5s559XZHOMrXK73i2kl9EYPIFq/92 RMGg0p/dhzVor51SpMiZZIdP+rZFaCqrKPBVrM5D/Tb7OyvVhTHGoTKWzKnKgsEj2WsP PsnjD2NNXwVsEeOUZA2O42UeTK3AUNnkIyGScDbo4NZo+B3M/KNzcWkF/usAJ0HXQyLZ kHEw== X-Gm-Message-State: AOAM5300N4iTDTMvnYGcsTXrdCT+w7rtN0mNLS0DGtPgdl7XSDffXmlL tzYgepQu8LxMCY2zAeqO0i/BH8cYe1M= X-Google-Smtp-Source: ABdhPJwgLQWRQBbPoIiaMe0ZCrzz9Z4V84ZwTa3X1PoMmQ14DhVSdioN89SbFJPPuHQlyJNxG+Drdw== X-Received: by 2002:a1c:25c2:: with SMTP id l185mr1184538wml.62.1611596540654; Mon, 25 Jan 2021 09:42:20 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id t25sm8439wmj.39.2021.01.25.09.42.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 09:42:20 -0800 (PST) Message-Id: In-Reply-To: References: Date: Mon, 25 Jan 2021 17:41:51 +0000 Subject: [PATCH 05/27] test-tool: read-cache --table --no-stat Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, peff@peff.net, jrnieder@gmail.com, sunshine@sunshineco.com, pclouds@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The 'test-tool read-cache --table' output is helpful to understand the full contents of the index entries on-disk. This is particularly helpful when trying to diagnose issues with a real repository example. However, for test cases we might want to compare the index contents of two repositories that were updated in similar ways, but will not actually share the same stat data. Add the '--no-stat' option to remove the timestamps and other stat data from the output. This allows us to compare index contents directly. Signed-off-by: Derrick Stolee --- t/helper/test-read-cache.c | 44 ++++++++++++++---------- t/t1092-sparse-checkout-compatibility.sh | 2 +- 2 files changed, 26 insertions(+), 20 deletions(-) diff --git a/t/helper/test-read-cache.c b/t/helper/test-read-cache.c index cd7d106a675..f858d0d0a0c 100644 --- a/t/helper/test-read-cache.c +++ b/t/helper/test-read-cache.c @@ -2,16 +2,18 @@ #include "cache.h" #include "config.h" -static void print_cache_entry(struct cache_entry *ce) +static void print_cache_entry(struct cache_entry *ce, unsigned stat) { - /* stat info */ - printf("%08x %08x %08x %08x %08x %08x ", - ce->ce_stat_data.sd_ctime.sec, - ce->ce_stat_data.sd_ctime.nsec, - ce->ce_stat_data.sd_mtime.sec, - ce->ce_stat_data.sd_mtime.nsec, - ce->ce_stat_data.sd_dev, - ce->ce_stat_data.sd_ino); + if (stat) { + /* stat info */ + printf("%08x %08x %08x %08x %08x %08x ", + ce->ce_stat_data.sd_ctime.sec, + ce->ce_stat_data.sd_ctime.nsec, + ce->ce_stat_data.sd_mtime.sec, + ce->ce_stat_data.sd_mtime.nsec, + ce->ce_stat_data.sd_dev, + ce->ce_stat_data.sd_ino); + } /* mode in binary */ printf("0b%d%d%d%d ", @@ -28,48 +30,52 @@ static void print_cache_entry(struct cache_entry *ce) printf("%s\n", ce->name); } -static void print_cache(struct index_state *cache) +static void print_cache(struct index_state *cache, unsigned stat) { int i; for (i = 0; i < the_index.cache_nr; i++) - print_cache_entry(the_index.cache[i]); + print_cache_entry(the_index.cache[i], stat); } int cmd__read_cache(int argc, const char **argv) { + struct repository *r = the_repository; int i, cnt = 1; const char *name = NULL; int table = 0; + int stat = 1; for (++argv, --argc; *argv && starts_with(*argv, "--"); ++argv, --argc) { if (skip_prefix(*argv, "--print-and-refresh=", &name)) continue; - if (!strcmp(*argv, "--table")) { + if (!strcmp(*argv, "--table")) table = 1; - } + else if (!strcmp(*argv, "--no-stat")) + stat = 0; } if (argc == 1) cnt = strtol(argv[0], NULL, 0); setup_git_directory(); git_config(git_default_config, NULL); + for (i = 0; i < cnt; i++) { - read_cache(); + repo_read_index(r); if (name) { int pos; - refresh_index(&the_index, REFRESH_QUIET, + refresh_index(r->index, REFRESH_QUIET, NULL, NULL, NULL); - pos = index_name_pos(&the_index, name, strlen(name)); + pos = index_name_pos(r->index, name, strlen(name)); if (pos < 0) die("%s not in index", name); printf("%s is%s up to date\n", name, - ce_uptodate(the_index.cache[pos]) ? "" : " not"); + ce_uptodate(r->index->cache[pos]) ? "" : " not"); write_file(name, "%d\n", i); } if (table) - print_cache(&the_index); - discard_cache(); + print_cache(r->index, stat); + discard_index(r->index); } return 0; } diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 8876eae0fe3..3aa9b0d21b4 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -132,7 +132,7 @@ test_sparse_match () { test_expect_success 'expanded in-memory index matches full index' ' init_repos && - test_sparse_match test-tool read-cache --expand --table-no-stat + test_sparse_match test-tool read-cache --expand --table --no-stat ' test_expect_success 'status with options' ' From patchwork Mon Jan 25 17:41:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12043863 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4631DC433DB for ; Mon, 25 Jan 2021 17:56:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1F39022D04 for ; Mon, 25 Jan 2021 17:56:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731331AbhAYRzh (ORCPT ); Mon, 25 Jan 2021 12:55:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33052 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731307AbhAYRzK (ORCPT ); Mon, 25 Jan 2021 12:55:10 -0500 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EB320C061351 for ; Mon, 25 Jan 2021 09:42:22 -0800 (PST) Received: by mail-wr1-x42f.google.com with SMTP id 7so13822194wrz.0 for ; Mon, 25 Jan 2021 09:42:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=DLHpNtZdXcw+qiTCBYSCYZeRA+wJGG0YpncTEfBc3+g=; b=gz5AeZ12Hqole3lnOpyOaDMWBltJOZSD9nI8lsRquRUxJc9oy8d2a0fnvsXuuWFGHA 4Yl83BbNpyqL2r3pEsBELRVqKk2IopxdCvzK0tnfoKm6ZP8m/0E7SH+Cxtvm/+uVr/Us 9xT/Y1jIlLE5NwT60GCHjgNCsOcus/9cHo1M1XvturuBqQyxU/0Yyc4C6oDWQf+XxhzR bLs3c/CZ1wd4KBqniZEsv+t69ADKS7QJmPbHlI1Y4yRsMVnJlKxVHW8ZrsWtBFcAra8J qzV344hcxfyiWHPLR95qclmGYq9JUE1fXV5Fc+z6aaahy6sT9MNYooa1f13q09KvQo00 U2ow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=DLHpNtZdXcw+qiTCBYSCYZeRA+wJGG0YpncTEfBc3+g=; b=ABsyrqvvp02FIHqZshmJ1XUEgpHz3yfMG46NwoOBw2HuihfoFevvhqOrEeDQusvEr/ GI0+VE5RzGTDKVha/sRqeafII6EoF3e0kZExu8o7DA12E10aN4VXy1krIKCkeKd3n+dR VcUmLcJBZvH8hhpzcEe/b5UXWSCnPhYenUUl1WJbnuOY0Q52YavZUpPhCQhZ4RjwUoXq JOqxsubYdK69rkHEzKbkeS6JAYRHJWdJxb67j/2yYMDEYGQDTUF4jxZG/JQjLmPv3Z3Z xQ999wQR9YcLBcSnELc3IySa/sa3yJq/5E/0ef4vk8aDtd5oVVBUh4W0gY6Rn5Vp9zPv FL0w== X-Gm-Message-State: AOAM532lyfTjv917MW0xVdjtbtlBATvYUpomOKtMjAq0JMl6tR8smNCd mXh4r1guLfruCPUec1uMqSfV6RY8Rss= X-Google-Smtp-Source: ABdhPJwoHF62iC6J4O2afuN2PIN9KQmKXQ9xDYjlsxyBp+P4hXRT4rBoo6RBwpMruh+A9Dlqqbi4tw== X-Received: by 2002:adf:b1db:: with SMTP id r27mr2241419wra.125.1611596541625; Mon, 25 Jan 2021 09:42:21 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 15sm55172wmk.3.2021.01.25.09.42.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 09:42:21 -0800 (PST) Message-Id: <8bbd1cd59ba04720cf5f77903a1883a25eb8606b.1611596534.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 25 Jan 2021 17:41:52 +0000 Subject: [PATCH 06/27] test-tool: don't force full index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, peff@peff.net, jrnieder@gmail.com, sunshine@sunshineco.com, pclouds@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee We will use 'test-tool read-cache --table' to check that a sparse index is written as part of init_repos. Since we will no longer always expand a sparse index into a full index, add an '--expand' parameter that adds a call to ensure_full_index() so we can compare a sparse index directly against a full index, or at least what the in-memory index looks like when expanded in this way. Signed-off-by: Derrick Stolee --- t/helper/test-read-cache.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/t/helper/test-read-cache.c b/t/helper/test-read-cache.c index f858d0d0a0c..3c45dfeb3cb 100644 --- a/t/helper/test-read-cache.c +++ b/t/helper/test-read-cache.c @@ -1,6 +1,7 @@ #include "test-tool.h" #include "cache.h" #include "config.h" +#include "sparse-index.h" static void print_cache_entry(struct cache_entry *ce, unsigned stat) { @@ -44,6 +45,11 @@ int cmd__read_cache(int argc, const char **argv) const char *name = NULL; int table = 0; int stat = 1; + int expand = 0; + + initialize_the_repository(); + prepare_repo_settings(r); + r->settings.command_requires_full_index = 0; for (++argv, --argc; *argv && starts_with(*argv, "--"); ++argv, --argc) { if (skip_prefix(*argv, "--print-and-refresh=", &name)) @@ -52,6 +58,8 @@ int cmd__read_cache(int argc, const char **argv) table = 1; else if (!strcmp(*argv, "--no-stat")) stat = 0; + else if (!strcmp(*argv, "--expand")) + expand = 1; } if (argc == 1) @@ -61,6 +69,10 @@ int cmd__read_cache(int argc, const char **argv) for (i = 0; i < cnt; i++) { repo_read_index(r); + + if (expand) + ensure_full_index(r->index); + if (name) { int pos; From patchwork Mon Jan 25 17:41:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12043903 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1D144C433E9 for ; Mon, 25 Jan 2021 18:03:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E26932255F for ; Mon, 25 Jan 2021 18:03:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731321AbhAYSDG (ORCPT ); Mon, 25 Jan 2021 13:03:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33232 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731338AbhAYRzx (ORCPT ); Mon, 25 Jan 2021 12:55:53 -0500 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D018EC061352 for ; Mon, 25 Jan 2021 09:42:23 -0800 (PST) Received: by mail-wr1-x431.google.com with SMTP id g10so13836180wrx.1 for ; Mon, 25 Jan 2021 09:42:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=eHAMjNv/BeGtNrvMBA7TfpjsHOcFjXTnK0ibqxh84Uk=; b=sAR+4HWiy/Fu9ubqDsufC9EYPTgvDCSKW4BWp6fDXjVzPq/zVJDnqTadSbRyRkCuM9 Sv2klBIF8/V4DkZMhRxaDxT3IkpgR8wsraSNcAGYKu+KmeTd8myT7iribZCHO/g0hYmX YrMuhPGALVEKxX+Hk/fN/mMVDbuJRf0bdEm6ms4xB58de5joyOhqqWrNehF0RQQ/DW4i Exp8joT+jsGtbTlZDYWryOTa2baSx4XiLVBAJSLddvxVObuVdJQ3FU66FsgDUFdk7rLl xyndeJoENGtlyVaW4f7JPaDgHo37nE6JitvW0TyK+tOgY2uwFPCvGF5VxANCqLsjdf60 klwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=eHAMjNv/BeGtNrvMBA7TfpjsHOcFjXTnK0ibqxh84Uk=; b=VFBLNVUvB6KIViNf+nZNrdZEFxbE9sfx966LPSA0Evc93PK43ewlk7Kfo2LxLa9dqT AJVzQ7KMYdUjqhG2aO56NGAx/TB2Buw2eIbjsTryfoZH9fhUhbnkNuAPJuAAvjnPLfWT MG34j8MNV/uRufGu0ydHE1A88d0fslr6s5hL4OHMkJ2KF78kyrqm9U1N4CYFJ4krQhM4 vZHBtzPxBkRbctj7bkzPOGryjmJdzDtr5bsYZpecLEmc4HacS26owTdqt9ghgduhH+Ra pK3yORELn/+sj9yjAwnDauMgahgnGk9LVitL2fgc7uiIBbUv6GBFesuSox8QZ5KnwdYk lfBA== X-Gm-Message-State: AOAM531JAAxIWZNekKxqZYyWWhBHkkaf7eheGKmnsDDM1jQ//XopCjuR dm6sOEbjUSeJew0AQR+3qZz4eaArvYQ= X-Google-Smtp-Source: ABdhPJyOHHbLovGQGvXZruTFxxWeiv2sMHZ2GeImcIiNC5KIGELrH38d7fIYAwX5SyMcqZgeu5kUtA== X-Received: by 2002:a5d:4d84:: with SMTP id b4mr2269344wru.361.1611596542435; Mon, 25 Jan 2021 09:42:22 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o17sm2872026wrm.52.2021.01.25.09.42.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 09:42:21 -0800 (PST) Message-Id: <175c3c62543f89144b03b3bdff750ad29d17ba03.1611596534.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 25 Jan 2021 17:41:53 +0000 Subject: [PATCH 07/27] unpack-trees: ensure full index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, peff@peff.net, jrnieder@gmail.com, sunshine@sunshineco.com, pclouds@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The next change will translate full indexes into sparse indexes at write time. The existing logic provides a way for every sparse index to be expanded to a full index at read time. However, there are cases where an index is written and then continues to be used in-memory to perform further updates. unpack_trees() is frequently called after such a write. In particular, commands like 'git reset' do this double-update of the index. Ensure that we have a full index when entering unpack_trees(), but only when command_requires_full_index is true. This is always true at the moment, but we will later relax that after unpack_trees() is updated to handle sparse directory entries. Signed-off-by: Derrick Stolee --- unpack-trees.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/unpack-trees.c b/unpack-trees.c index f5f668f532d..4dd99219073 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -1567,6 +1567,7 @@ static int verify_absent(const struct cache_entry *, */ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options *o) { + struct repository *repo = the_repository; int i, ret; static struct cache_entry *dfc; struct pattern_list pl; @@ -1578,6 +1579,12 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options trace_performance_enter(); trace2_region_enter("unpack_trees", "unpack_trees", the_repository); + prepare_repo_settings(repo); + if (repo->settings.command_requires_full_index) { + ensure_full_index(o->src_index); + ensure_full_index(o->dst_index); + } + if (!core_apply_sparse_checkout || !o->update) o->skip_sparse_checkout = 1; if (!o->skip_sparse_checkout && !o->pl) { From patchwork Mon Jan 25 17:41:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12043861 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9AF56C433E6 for ; Mon, 25 Jan 2021 17:55:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5C67422583 for ; Mon, 25 Jan 2021 17:55:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731316AbhAYRzQ (ORCPT ); Mon, 25 Jan 2021 12:55:16 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33062 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731310AbhAYRzL (ORCPT ); Mon, 25 Jan 2021 12:55:11 -0500 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 97D27C061353 for ; Mon, 25 Jan 2021 09:42:24 -0800 (PST) Received: by mail-wr1-x42d.google.com with SMTP id v15so13808444wrx.4 for ; Mon, 25 Jan 2021 09:42:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=vpAX72lNJ35KIs0L6NecQ5GsW2/fj8OPGN09bLouxHo=; b=Rb0ohfbbhQZAUtknHUVfO5eJmrGbru3ec1iIhdhJlOGPCs6TPiBjfp16GQT2MGlWSA wDfcZpGMIwWeKDNU3/3O9FV8a/rUTADEYeYLWc82PQZ/L+aKgAAqYppjAzXlxpJ9GbEL 0zloGJwm99VifRXyIMiTL6lxl4UxQSNgowni0iMMKl2xArg/J5l7SAedQ2tPqdrY2+oj xXLmaNAXOhxqfQC+6iBepBK0i8oGgzdJ7ybTbQ4NpIHMmUPmwt4Hjw7gIK5f4YmQfe96 ZirTe7k3Cep+0ToXcNa5F+CJAAf5ZA/h81JTARid3MXR/zzpyKdmYBtmI+nIgbjESrUe vg9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=vpAX72lNJ35KIs0L6NecQ5GsW2/fj8OPGN09bLouxHo=; b=fMoD5+H1q3S/6tbVjyAZMwQYVYgfCOhNqwQKP1ZnrhPvQDeVM2+f3ZE3ILcUFStNHq B7wZ0eZeF32orKuYBOfVuIgZj1mjhpQaOEr/1nNi0PW+NUqQi5YTVbPGx17yzxG0xlN0 L5t+6U4ET2JYwWApZe6htiaBo7QjKxec6wMMkQIoiqRkRS6Dvo0LDUL0j06pnHdUgKmE Uf0EDwldSbyTGkuZjKt741eBXw064rECb4NDHW5RdsskNE65QEhBzerHHxn8IxCJLKjH elMZAKj5vk6+G5zbXW6yY6urc+aVbf5/Ad+PB3xTDVrVNndRePfqrnfU115hUDnyDNop Piuw== X-Gm-Message-State: AOAM532GjaVNhlyziezaJTOu8zbFT/ouohIndFFDUayRpqdCoG2PHJjH 4yRuqu17Pp2TXVVD31yzdYKgyW2PtlI= X-Google-Smtp-Source: ABdhPJwxlgfXJcXdIliDxUAqppOsUE5/7bUEbBXRt7Yi7B4+lVgfxEL5Ege0ruTZppgI0jCG7vUrEQ== X-Received: by 2002:adf:fb52:: with SMTP id c18mr2223302wrs.186.1611596543256; Mon, 25 Jan 2021 09:42:23 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id s2sm42848wmc.12.2021.01.25.09.42.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 09:42:22 -0800 (PST) Message-Id: <80aac5b8b71da96d6287f19e30f4ee8f786ca07b.1611596534.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 25 Jan 2021 17:41:54 +0000 Subject: [PATCH 08/27] sparse-checkout: hold pattern list in index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, peff@peff.net, jrnieder@gmail.com, sunshine@sunshineco.com, pclouds@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee As we modify the sparse-checkout definition, we perform index operations on a pattern_list that only exists in-memory. This allows easy backing out in case the index update fails. However, if the index write itself cares about the sparse-checkout pattern set, we need access to that in-memory copy. Place a pointer to a 'struct pattern_list' in the index so we can access this on-demand. This will be used in the next change which uses the sparse-checkout definition to filter out directories that are outsie the sparse cone. Signed-off-by: Derrick Stolee --- builtin/sparse-checkout.c | 17 ++++++++++------- cache.h | 2 ++ 2 files changed, 12 insertions(+), 7 deletions(-) diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c index 2306a9ad98e..e00b82af727 100644 --- a/builtin/sparse-checkout.c +++ b/builtin/sparse-checkout.c @@ -110,6 +110,8 @@ static int update_working_directory(struct pattern_list *pl) if (is_index_unborn(r->index)) return UPDATE_SPARSITY_SUCCESS; + r->index->sparse_checkout_patterns = pl; + memset(&o, 0, sizeof(o)); o.verbose_update = isatty(2); o.update = 1; @@ -138,6 +140,7 @@ static int update_working_directory(struct pattern_list *pl) else rollback_lock_file(&lock_file); + r->index->sparse_checkout_patterns = NULL; return result; } @@ -517,19 +520,18 @@ static int modify_pattern_list(int argc, const char **argv, enum modify_type m) { int result; int changed_config = 0; - struct pattern_list pl; - memset(&pl, 0, sizeof(pl)); + struct pattern_list *pl = xcalloc(1, sizeof(*pl)); switch (m) { case ADD: if (core_sparse_checkout_cone) - add_patterns_cone_mode(argc, argv, &pl); + add_patterns_cone_mode(argc, argv, pl); else - add_patterns_literal(argc, argv, &pl); + add_patterns_literal(argc, argv, pl); break; case REPLACE: - add_patterns_from_input(&pl, argc, argv); + add_patterns_from_input(pl, argc, argv); break; } @@ -539,12 +541,13 @@ static int modify_pattern_list(int argc, const char **argv, enum modify_type m) changed_config = 1; } - result = write_patterns_and_update(&pl); + result = write_patterns_and_update(pl); if (result && changed_config) set_config(MODE_NO_PATTERNS); - clear_pattern_list(&pl); + clear_pattern_list(pl); + free(pl); return result; } diff --git a/cache.h b/cache.h index 884046ca5b8..b05341cc687 100644 --- a/cache.h +++ b/cache.h @@ -311,6 +311,7 @@ static inline unsigned int canon_mode(unsigned int mode) struct split_index; struct untracked_cache; struct progress; +struct pattern_list; struct index_state { struct cache_entry **cache; @@ -336,6 +337,7 @@ struct index_state { struct mem_pool *ce_mem_pool; struct progress *progress; struct repository *repo; + struct pattern_list *sparse_checkout_patterns; }; /* Name hashing */ From patchwork Mon Jan 25 17:41:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12043859 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5695AC433DB for ; Mon, 25 Jan 2021 17:55:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1787922583 for ; Mon, 25 Jan 2021 17:55:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730795AbhAYRzT (ORCPT ); Mon, 25 Jan 2021 12:55:19 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33060 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731309AbhAYRzL (ORCPT ); Mon, 25 Jan 2021 12:55:11 -0500 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9BCD3C061354 for ; Mon, 25 Jan 2021 09:42:25 -0800 (PST) Received: by mail-wr1-x430.google.com with SMTP id v15so13808527wrx.4 for ; Mon, 25 Jan 2021 09:42:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=YlF5+pQCsy7HQb9ujN5KU3ddV/rOxfPIDB1cbahQdb8=; b=WVKfyHFnXdqVVz5uYxZSUaK3KC7rrjkFo9RyUm0ojRw/XVA8LuoOFAK+jwUFedgiaD xmqxlhJvow9tpzGMwV3XvYBptyIY06VdBvqauq4fw6Ibka+y8OW808g2gTUuRpTDWxBB imgjRmhkNz/jZVpi0dhwTGHuOKfLJl3ZhOmoe3qmOZZXzXu6v8Zi9L0vgJ6xPPxAIhQy pArUloMtrwlm1INCPImb6v78Ugc+i59FRYTOZp1stOv4WfkGAenSl5rK1rj8WEMacSZw uY7oBEtmLM41UwXe9RxoAjry8NbzdDnYvkbfs5gGvt4dRn3nzdoF+iWv9fOlu+EeK8cF WA+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=YlF5+pQCsy7HQb9ujN5KU3ddV/rOxfPIDB1cbahQdb8=; b=N966ZfIlQ/eImuSovuIiYbGV/KxH3RE5Mjfo+tcUKOR4w1dirWV2PEcsFeoacC/Zcf qX6z0z2dUfluyAkg5n6t1xDG4WVN/0mRwsgvVNgRrou6aM/5E9luyB7StDsnRGusKZkl nlco1Ct5XmCCa1bgVO/Hx5eEZBBQIX209m3IeVKebXybH+lGgv12Kuh7DYS5MC7aVFCy 9dP7ON3vCrPz2w718UaKq+zLdo6QskSt2bFFdAeEGxXvjF5PhSNzs0REGjZm7XaGtBtI JLc5hjJlol98+ZemdA0+hI/XrHm0GK+3zHCC9tXugpsX+zhopkH4rlsxcoOnbzS9pcAj 6Liw== X-Gm-Message-State: AOAM531uSAv2seOp1sCXrklRLLv3iXRcQQ1Nd5sUHdJ9y+XjB0bISYMs TpT8/AZ9LUPzb9ap8Z7kQSdKvUJlnaw= X-Google-Smtp-Source: ABdhPJxYQZyn/1GDPoRVyYfKMsOmomuidKg+IdowgnROs8tJRSYOPIL/g8S2+v7Eq2BGqy68dssZuA== X-Received: by 2002:adf:ee43:: with SMTP id w3mr2353252wro.200.1611596544069; Mon, 25 Jan 2021 09:42:24 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l8sm38907wmi.8.2021.01.25.09.42.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 09:42:23 -0800 (PST) Message-Id: In-Reply-To: References: Date: Mon, 25 Jan 2021 17:41:55 +0000 Subject: [PATCH 09/27] sparse-index: convert from full to sparse Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, peff@peff.net, jrnieder@gmail.com, sunshine@sunshineco.com, pclouds@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee If we have a full index, then we can convert it to a sparse index by replacing directories outside of the sparse cone with sparse directory entries. The convert_to_sparse() method does this, when the situation is appropriate. For now, we avoid converting the index to a sparse index if: 1. the index is split. 2. the index is already sparse. 3. sparse-checkout is disabled. 4. sparse-checkout does not use cone mode. Finally, we currently limit the conversion to when the GIT_TEST_SPARSE_INDEX environment variable is enabled. A mode using Git config will be added in a later change. The trickiest thing about this conversion is that we might not be able to mark a directory as a sparse directory just because it is outside the sparse cone. There might be unmerged files within that directory, so we need to look for those. Also, if there is some strange reason why a file is not marked with CE_SKIP_WORKTREE, then we should give up on converting that directory. There is still hope that some of its subdirectories might be able to convert to sparse, so we keep looking deeper. The conversion process is assisted by the cache-tree extension. This is calculated from the full index if it does not already exist. We then abandon the cache-tree as it no longer applies to the newly-sparse index. Thus, this cache-tree will be recalculated in every sparse-full-sparse round-trip until we integrate the cache-tree extension with the sparse index. We can compare the behavior of the sparse-index in t1092-sparse-checkout-compability.sh by using GIT_TEST_SPARSE_INDEX=1 when operating on the 'sparse-index' repo. We can also compare the two sparse repos directly, such as comparing their indexes (when expanded to full in the case of the 'sparse-index' repo). We also verify that the index is actually populated with sparse directory entries. The 'checkout and reset (mixed)' test is marked for failure when comparing a sparse repo to a full repo, but we can compare the two sparse-checkout cases directly to ensure that we are not changing the behavior when using a sparse index. Signed-off-by: Derrick Stolee --- cache-tree.c | 3 + read-cache.c | 18 ++- sparse-index.c | 139 +++++++++++++++++++++++ t/t1092-sparse-checkout-compatibility.sh | 63 +++++++++- 4 files changed, 218 insertions(+), 5 deletions(-) diff --git a/cache-tree.c b/cache-tree.c index 2fb483d3c08..5f07a39e501 100644 --- a/cache-tree.c +++ b/cache-tree.c @@ -6,6 +6,7 @@ #include "object-store.h" #include "replace-object.h" #include "promisor-remote.h" +#include "sparse-index.h" #ifndef DEBUG_CACHE_TREE #define DEBUG_CACHE_TREE 0 @@ -442,6 +443,8 @@ int cache_tree_update(struct index_state *istate, int flags) if (i) return i; + ensure_full_index(istate); + if (!istate->cache_tree) istate->cache_tree = cache_tree(); diff --git a/read-cache.c b/read-cache.c index 1097ecbf132..0522260416e 100644 --- a/read-cache.c +++ b/read-cache.c @@ -25,6 +25,7 @@ #include "fsmonitor.h" #include "thread-utils.h" #include "progress.h" +#include "sparse-index.h" /* Mask for the name length in ce_flags in the on-disk index */ @@ -1002,8 +1003,15 @@ int verify_path(const char *path, unsigned mode) c = *path++; if ((c == '.' && !verify_dotfile(path, mode)) || - is_dir_sep(c) || c == '\0') + is_dir_sep(c)) return 0; + /* + * allow terminating directory separators for + * sparse directory enries. + */ + if (c == '\0') + return mode == CE_MODE_SPARSE_DIRECTORY || + mode == SPARSE_DIR_MODE; } else if (c == '\\' && protect_ntfs) { if (is_ntfs_dotgit(path)) return 0; @@ -3062,6 +3070,13 @@ static int do_write_locked_index(struct index_state *istate, struct lock_file *l { int ret; + ret = convert_to_sparse(istate); + + if (ret) { + warning(_("failed to convert to a sparse-index")); + return ret; + } + /* * TODO trace2: replace "the_repository" with the actual repo instance * that is associated with the given "istate". @@ -3165,6 +3180,7 @@ static int write_shared_index(struct index_state *istate, int ret; move_cache_to_base_index(istate); + convert_to_sparse(istate); trace2_region_enter_printf("index", "shared/do_write_index", the_repository, "%s", (*temp)->filename.buf); diff --git a/sparse-index.c b/sparse-index.c index 1e70244dc13..d8f1a5a13d7 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -4,6 +4,145 @@ #include "tree.h" #include "pathspec.h" #include "trace2.h" +#include "cache-tree.h" +#include "config.h" +#include "dir.h" +#include "fsmonitor.h" + +static struct cache_entry *construct_sparse_dir_entry( + struct index_state *istate, + const char *sparse_dir, + struct cache_tree *tree) +{ + struct cache_entry *de; + + de = make_cache_entry(istate, SPARSE_DIR_MODE, &tree->oid, sparse_dir, 0, 0); + + de->ce_flags |= CE_SKIP_WORKTREE; + return de; +} + +/* + * Returns the number of entries "inserted" into the index. + */ +static int convert_to_sparse_rec(struct index_state *istate, + int num_converted, + int start, int end, + const char *ct_path, size_t ct_pathlen, + struct cache_tree *ct) +{ + int i, can_convert = 1; + int start_converted = num_converted; + enum pattern_match_result match; + int dtype; + struct strbuf child_path = STRBUF_INIT; + struct pattern_list *pl = istate->sparse_checkout_patterns; + + /* + * Is the current path outside of the sparse cone? + * Then check if the region can be replaced by a sparse + * directory entry (everything is sparse and merged). + */ + match = path_matches_pattern_list(ct_path, ct_pathlen, + NULL, &dtype, pl, istate); + if (match != NOT_MATCHED) + can_convert = 0; + + for (i = start; can_convert && i < end; i++) { + struct cache_entry *ce = istate->cache[i]; + + if (ce_stage(ce) || + !(ce->ce_flags & CE_SKIP_WORKTREE)) + can_convert = 0; + } + + if (can_convert) { + struct cache_entry *se; + se = construct_sparse_dir_entry(istate, ct_path, ct); + + istate->cache[num_converted++] = se; + return 1; + } + + for (i = start; i < end; ) { + int count, span, pos = -1; + const char *base, *slash; + struct cache_entry *ce = istate->cache[i]; + + /* + * Detect if this is a normal entry oustide of any subtree + * entry. + */ + base = ce->name + ct_pathlen; + slash = strchr(base, '/'); + + if (slash) + pos = cache_tree_subtree_pos(ct, base, slash - base); + + if (pos < 0) { + istate->cache[num_converted++] = ce; + i++; + continue; + } + + strbuf_setlen(&child_path, 0); + strbuf_add(&child_path, ce->name, slash - ce->name + 1); + + span = ct->down[pos]->cache_tree->entry_count; + count = convert_to_sparse_rec(istate, + num_converted, i, i + span, + child_path.buf, child_path.len, + ct->down[pos]->cache_tree); + num_converted += count; + i += span; + } + + strbuf_release(&child_path); + return num_converted - start_converted; +} + +int convert_to_sparse(struct index_state *istate) +{ + if (istate->split_index || istate->sparse_index || + !core_apply_sparse_checkout || !core_sparse_checkout_cone) + return 0; + + /* + * For now, only create a sparse index with the + * GIT_TEST_SPARSE_INDEX environment variable. We will relax + * this once we have a proper way to opt-in (and later still, + * opt-out). + */ + if (!git_env_bool("GIT_TEST_SPARSE_INDEX", 0)) + return 0; + + if (!istate->sparse_checkout_patterns) { + istate->sparse_checkout_patterns = xcalloc(1, sizeof(struct pattern_list)); + if (get_sparse_checkout_patterns(istate->sparse_checkout_patterns) < 0) + return 0; + } + + if (!istate->sparse_checkout_patterns->use_cone_patterns) { + warning(_("attempting to use sparse-index without cone mode")); + return -1; + } + + if (cache_tree_update(istate, 0)) { + warning(_("unable to update cache-tree, staying full")); + return -1; + } + + remove_fsmonitor(istate); + + trace2_region_enter("index", "convert_to_sparse", istate->repo); + istate->cache_nr = convert_to_sparse_rec(istate, + 0, 0, istate->cache_nr, + "", 0, istate->cache_tree); + istate->drop_cache_tree = 1; + istate->sparse_index = 1; + trace2_region_leave("index", "convert_to_sparse", istate->repo); + return 0; +} static void set_index_entry(struct index_state *istate, int nr, struct cache_entry *ce) { diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 3aa9b0d21b4..22becbaca2e 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -2,6 +2,9 @@ test_description='compare full workdir to sparse workdir' +GIT_TEST_CHECK_CACHE_TREE=0 +GIT_TEST_SPLIT_INDEX=0 + . ./test-lib.sh test_expect_success 'setup' ' @@ -106,7 +109,7 @@ run_on_sparse () { ) && ( cd sparse-index && - $* >../sparse-index-out 2>../sparse-index-err + GIT_TEST_SPARSE_INDEX=1 $* >../sparse-index-out 2>../sparse-index-err ) } @@ -121,7 +124,9 @@ run_on_all () { test_all_match () { run_on_all $* && test_cmp full-checkout-out sparse-checkout-out && - test_cmp full-checkout-err sparse-checkout-err + test_cmp full-checkout-out sparse-index-out && + test_cmp full-checkout-err sparse-checkout-err && + test_cmp full-checkout-err sparse-index-err } test_sparse_match () { @@ -130,6 +135,38 @@ test_sparse_match () { test_cmp sparse-checkout-err sparse-index-err } +test_expect_success 'sparse-index contents' ' + init_repos && + + test-tool -C sparse-index read-cache --table --no-stat >cache && + for dir in folder1 folder2 x + do + TREE=$(git -C sparse-index rev-parse HEAD:$dir) && + grep "0b0000 0755 $TREE $dir/" cache \ + || return 1 + done && + + GIT_TEST_SPARSE_INDEX=1 git -C sparse-index sparse-checkout set folder1 && + + test-tool -C sparse-index read-cache --table --no-stat >cache && + for dir in deep folder2 x + do + TREE=$(git -C sparse-index rev-parse HEAD:$dir) && + grep "0b0000 0755 $TREE $dir/" cache \ + || return 1 + done && + + GIT_TEST_SPARSE_INDEX=1 git -C sparse-index sparse-checkout set deep/deeper1 && + + test-tool -C sparse-index read-cache --table --no-stat >cache && + for dir in deep/deeper2 folder1 folder2 x + do + TREE=$(git -C sparse-index rev-parse HEAD:$dir) && + grep "0b0000 0755 $TREE $dir/" cache \ + || return 1 + done +' + test_expect_success 'expanded in-memory index matches full index' ' init_repos && test_sparse_match test-tool read-cache --expand --table --no-stat @@ -137,6 +174,7 @@ test_expect_success 'expanded in-memory index matches full index' ' test_expect_success 'status with options' ' init_repos && + test_sparse_match ls && test_all_match git status --porcelain=v2 && test_all_match git status --porcelain=v2 -z -u && test_all_match git status --porcelain=v2 -uno && @@ -169,7 +207,7 @@ test_expect_success 'add, commit, checkout' ' test_all_match git add -A && test_all_match git status --porcelain=v2 && - test_all_match git commit -m "Extend README.md" && + test_all_match git commit -m "Extend-README.md" && test_all_match git checkout HEAD~1 && test_all_match git checkout - && @@ -273,6 +311,17 @@ test_expect_failure 'checkout and reset (mixed)' ' test_all_match git reset update-folder2 ' +# Ensure that sparse-index behaves identically to +# sparse-checkout with a full index. +test_expect_success 'checkout and reset (mixed) [sparse]' ' + init_repos && + + test_sparse_match git checkout -b reset-test update-deep && + test_sparse_match git reset deepest && + test_sparse_match git reset update-folder1 && + test_sparse_match git reset update-folder2 +' + test_expect_success 'merge' ' init_repos && @@ -309,14 +358,20 @@ test_expect_success 'clean' ' test_all_match git status --porcelain=v2 && test_all_match git clean -f && test_all_match git status --porcelain=v2 && + test_sparse_match ls && + test_sparse_match ls folder1 && test_all_match git clean -xf && test_all_match git status --porcelain=v2 && + test_sparse_match ls && + test_sparse_match ls folder1 && test_all_match git clean -xdf && test_all_match git status --porcelain=v2 && + test_sparse_match ls && + test_sparse_match ls folder1 && - test_path_is_dir sparse-checkout/folder1 + test_sparse_match test_path_is_dir folder1 ' test_done From patchwork Mon Jan 25 17:41:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12043895 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 73CADC433E0 for ; Mon, 25 Jan 2021 18:01:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3A70822B3B for ; Mon, 25 Jan 2021 18:01:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731214AbhAYSBR (ORCPT ); Mon, 25 Jan 2021 13:01:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33062 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731237AbhAYR4A (ORCPT ); Mon, 25 Jan 2021 12:56:00 -0500 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5F15CC061355 for ; Mon, 25 Jan 2021 09:42:26 -0800 (PST) Received: by mail-wr1-x435.google.com with SMTP id q7so13802337wre.13 for ; Mon, 25 Jan 2021 09:42:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=kZFR1isx3QPYkeyF9f202HOAtWRotawyy8Lv6DfXCFY=; b=bxGn7H49iYHdodS3nGQOYnt7O513xRdZ/Ukw6aL8WfZprlnLWumNM2XyeyoOWQkvIr m8qCTn2C9sKGbdnix6FsGCqu15WOtjndUe2OsXZgGVxNSJl6DSpdokptNQfD8rEfmvpq kSjWGZB7BIzn1TBNKhin+vTVVhTr2H41bxZAUXKQKm+04My1RNvq15FWwj9OhZk+L6lq 7QDxmEVRohnyi3hv7SKhqgRJK+T7+rWcK5y/SCzncsO7wYlDE208EDySjz0z9uNVjIdr t4SJoL9mURnMT3NdHqnaJOhcqYLe/Znzt5rvzDLeNxTlXNEWSsEYTHrgBuQSjSgVVL/S 1+mw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=kZFR1isx3QPYkeyF9f202HOAtWRotawyy8Lv6DfXCFY=; b=KL0Udmbp/mPmzX1BgqXqZ5c7B1LEE9BbJ535mggq/d3eI0gFnhZPlskEqPgVxdiKHm uaGsRO+ORu6AysH5Q/iYitNuiIdEyrg7qmGyMG3QHr1Q2j3mKOjuBQKdFKMwQMlIWnmu nNWP6zTxHe72YjwWpyCHcGFvScsQiMqXFpSCE5Eq5HbHQFnX7Yn+lYT887FzRvTxG8Tf lIkRE8236VVWfkGKrFYGdcPZl2oFwjt3ayUtO8FjlfzhAH6p90lDs8B0dvRwwrxO/5d+ Wvta/+90r1a29nMJCt4um/WwFZfIwTjy9+Sf1CFbxH9k2O/yhQKQOyjszQMc4q442v9K nP/A== X-Gm-Message-State: AOAM531juRpLzSDr+txpSpY/1zdu8CBDFOU0hsY/173V3pVxqIL0yMf9 6c4D64zZ1M30XGIqr5sZqkJjiPXhzWk= X-Google-Smtp-Source: ABdhPJzrZtHEcPB71nGpmrkzZGRED9gJcpcEw5yDQIN2tOx+HBrtYMpaB+2AZVeX8tc66EDzfCAp6w== X-Received: by 2002:a05:6000:41:: with SMTP id k1mr2225352wrx.386.1611596544950; Mon, 25 Jan 2021 09:42:24 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id x25sm27835wmk.20.2021.01.25.09.42.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 09:42:24 -0800 (PST) Message-Id: <9cd38fb0e9e67109940b73489364b2f2099002ba.1611596534.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 25 Jan 2021 17:41:56 +0000 Subject: [PATCH 10/27] submodule: sparse-index should not collapse links Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, peff@peff.net, jrnieder@gmail.com, sunshine@sunshineco.com, pclouds@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee A submodule is stored as a "Git link" that actually points to a commit within a submodule. Submodules are populated or not depending on submodule configuration, not sparse-checkout. To ensure that the sparse-index feature integrates correctly with submodules, we should not collapse a directory if there is a Git link within its range. This allows us to remove ensure_full_index() from die_path_inside_submodule() because a sparse-index will not remove the entries for Git links. The loop already 'continue's if the cache entry is not a Git link. Signed-off-by: Derrick Stolee --- sparse-index.c | 1 + 1 file changed, 1 insertion(+) diff --git a/sparse-index.c b/sparse-index.c index d8f1a5a13d7..5dd0b835b9d 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -52,6 +52,7 @@ static int convert_to_sparse_rec(struct index_state *istate, struct cache_entry *ce = istate->cache[i]; if (ce_stage(ce) || + S_ISGITLINK(ce->ce_mode) || !(ce->ce_flags & CE_SKIP_WORKTREE)) can_convert = 0; } From patchwork Mon Jan 25 17:41:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12043931 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6A59C433E0 for ; Mon, 25 Jan 2021 18:13:52 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8D3BF22C9E for ; Mon, 25 Jan 2021 18:13:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731128AbhAYSN1 (ORCPT ); Mon, 25 Jan 2021 13:13:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33050 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731109AbhAYRzW (ORCPT ); Mon, 25 Jan 2021 12:55:22 -0500 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 282FBC061356 for ; Mon, 25 Jan 2021 09:42:27 -0800 (PST) Received: by mail-wr1-x434.google.com with SMTP id d16so13147125wro.11 for ; Mon, 25 Jan 2021 09:42:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=ZLJFWeKJGZmF+8ekApYC8LfnEGt9a2eaYNT3MSN9DOM=; b=E40ybx291ptWFZxcQnwZm3IbEQB/h0OxwZ1ta6bfoco3MLYybCwnx1B984A/s90mAq M4A4u7v32Wu1SRVhpnggy6hNFVkRcZvbwPDx2PZ7MJETN2QC49QtHpOQfZKNhQoHjDAs a3i+jL05vFcLS0k5YNSxcDwlG+Zj+oEc8bEWG6qwTCKP+s927rPTzzOaCEySl2Tl5wlY XpJzXt1tkCb587jvCEOiBqU+afq2YB8qrjjajSchU8v6gdCVNzXORd/RE2noiQMQZ3Ub sOiC7BnNbJZzbNNYLUBqOR+nFfyEorfbKfbq/m7umAcuV73Ayj/q5hZaa4kDOf/ojfyM 2Ccg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=ZLJFWeKJGZmF+8ekApYC8LfnEGt9a2eaYNT3MSN9DOM=; b=EeCsZyTWOBHJFH3TElWUkOScddrRyafYS5aKCRlT90TR6vgOe2CmnXYa9B2SuazBn0 hlNmIbu98nqMApJNoQIeseq6rbOWGNWsD9dMQywpPoInNoi76UKWOfqjJBOh9z856eyP m3VKoPKDJIuZWGHoxBhQXBbE48VreChP7fOn6xddnrdnSZlW99GgIiLvkBDJQ0ND7ZeP 0yem7AAGaFOMn/R3qIfXRaotoFZ2MxbMxZviUQNCv1Rc8qzuyIZLdbx0DlmUUMPBVVSX tl8qADLWLIGfkOZ72GWv0KugWJRYTJInR9OTsfNURhK3/HBrjGkmZDmc6au9gKMqTQo1 bc3w== X-Gm-Message-State: AOAM533PvJRST5rL6cyKtNEhjmMaGy+EocusBqJLjbcNynhQBug5ad9a EvwgGbp+jFJS4n6zC56EdIIaP6ONPt4= X-Google-Smtp-Source: ABdhPJyXpftGXS9mZyraCNIhMmorQ17CbkqOoX94CBbG/aR/hWqTwHqEXTy2rWSIOxw4zblsX9RcEg== X-Received: by 2002:adf:ba0c:: with SMTP id o12mr2210839wrg.322.1611596545806; Mon, 25 Jan 2021 09:42:25 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id u7sm11259152wrr.80.2021.01.25.09.42.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 09:42:25 -0800 (PST) Message-Id: <09893b4a6bbe13e61395411bdae57ce041829042.1611596534.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 25 Jan 2021 17:41:57 +0000 Subject: [PATCH 11/27] unpack-trees: allow sparse directories Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, peff@peff.net, jrnieder@gmail.com, sunshine@sunshineco.com, pclouds@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The index_pos_by_traverse_info() currently throws a BUG() when a directory entry exists exactly in the index. We need to consider that it is possible to have a directory in a sparse index as long as that entry is itself marked with the skip-worktree bit. The negation of the 'pos' variable must be conditioned to only when it starts as negative. This is identical behavior as before when the index is full. The starts_with() condition matches because our name.buf terminates with a directory separator, just like our sparse directory entries. Signed-off-by: Derrick Stolee --- unpack-trees.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/unpack-trees.c b/unpack-trees.c index 4dd99219073..b324eec2a5d 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -746,9 +746,12 @@ static int index_pos_by_traverse_info(struct name_entry *names, strbuf_make_traverse_path(&name, info, names->path, names->pathlen); strbuf_addch(&name, '/'); pos = index_name_pos(o->src_index, name.buf, name.len); - if (pos >= 0) - BUG("This is a directory and should not exist in index"); - pos = -pos - 1; + if (pos >= 0) { + if (!o->src_index->sparse_index || + !(o->src_index->cache[pos]->ce_flags & CE_SKIP_WORKTREE)) + BUG("This is a directory and should not exist in index"); + } else + pos = -pos - 1; if (pos >= o->src_index->cache_nr || !starts_with(o->src_index->cache[pos]->name, name.buf) || (pos > 0 && starts_with(o->src_index->cache[pos-1]->name, name.buf))) From patchwork Mon Jan 25 17:41:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12043865 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62ECBC433E6 for ; Mon, 25 Jan 2021 17:56:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 36EB922D50 for ; Mon, 25 Jan 2021 17:56:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731236AbhAYR4A (ORCPT ); Mon, 25 Jan 2021 12:56:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33058 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731308AbhAYRzK (ORCPT ); Mon, 25 Jan 2021 12:55:10 -0500 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E9BE7C0612F2 for ; Mon, 25 Jan 2021 09:42:27 -0800 (PST) Received: by mail-wr1-x434.google.com with SMTP id v15so13808691wrx.4 for ; Mon, 25 Jan 2021 09:42:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=eUgXhnq3iSGakbidVqTrAi+YK05jsh83OknbN9FyXNM=; b=qsABUqoxSzqbjUePubAyvOfnNUEDzgQyNW9U0Yk88Vj1+Bzjq6mv6/r2+8394wxZ5n cLBOKzbeAQUj+VxC1AWFoS4OfbxyBBFwGUIQIhAVSUq3hq0/6EQwody0BC7Xv8gmJPIB 0Kv/Q5xcVn0w0HoukBWe0o1k1+l5VDOCv9bzihzVGU4H0rMQZTJH6X1oDGfuOV9meCGp ZxQbzboRmF3iAjOiC6kt4dr/WbCNx5XRTOZ2qTwRPwQOaN+zp+8YIM62bVnHNerHNklv EFvWOiy0dpGvrT3tN0RxD/SoA3i+ElXCx4R/37wJ/39dIb9kJs3JPFtf1ic9kAr+6h8U O7BA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=eUgXhnq3iSGakbidVqTrAi+YK05jsh83OknbN9FyXNM=; b=dORG3BianPmGkDUyfJWnAb8rGphtl2V2WZlyFzkkLBltROXQm2Aw9k3NTbHridsMh9 v1eUDKFe4QIakWD56D2SrntL88gV5+yUkgFesC6P0B4ym1vzHIbGUzAKHDTy5gLQEnU+ 18m4yB6PSvEfaeFQR4h8oHmD6z38KJi3+g76aoMWEBhUYKXHmARCWB9iRu5usPMQVzEt W+oqp/taLnA9VkfPKY9W/+atxnd8SvCbX6a5fBg93T3S3EkOYld+kaSRYs9GauePdwvo fcUiBS+R/rjusQH8CNM6tf7n6Llv702Ei80WmJa3ZeD2R0DMh/r2py4e+RfgI01sj0nK BLig== X-Gm-Message-State: AOAM530nYh1dJULhGpZu7lGJkSPsi2dOtrX9/61xIO8p2ZaFEaAyPICx 6pbV9OUU+n8mY0kLcAQP5QHOm+oMG5E= X-Google-Smtp-Source: ABdhPJxD6XR3M35MJ2dF3vyEh7DFYA6Lc070UNuRRFI4+68R7LqrMyZv4XQ6ZGL4KVc5wYdBnq3eug== X-Received: by 2002:adf:a2ca:: with SMTP id t10mr2216169wra.370.1611596546601; Mon, 25 Jan 2021 09:42:26 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id u16sm23445753wrn.68.2021.01.25.09.42.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 09:42:26 -0800 (PST) Message-Id: <9b2ae1bb6c49b67cc17691d58044bbb8efb114a3.1611596534.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 25 Jan 2021 17:41:58 +0000 Subject: [PATCH 12/27] sparse-index: check index conversion happens Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, peff@peff.net, jrnieder@gmail.com, sunshine@sunshineco.com, pclouds@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Add a test case that uses test_region to ensure that we are truly expanding a sparse index to a full one, then converting back to sparse when writing the index. As we integrate more Git commands with the sparse index, we will convert these commands to check that we do _not_ convert the sparse index to a full index and instead stay sparse the entire time. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 22becbaca2e..a22def89e37 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -374,4 +374,21 @@ test_expect_success 'clean' ' test_sparse_match test_path_is_dir folder1 ' +test_expect_success 'sparse-index is expanded and converted back' ' + init_repos && + + ( + (GIT_TEST_SPARSE_INDEX=1 && export GIT_TEST_SPARSE_INDEX) && + GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ + git -C sparse-index -c core.fsmonitor="" reset --hard && + test_region index convert_to_sparse trace2.txt && + test_region index ensure_full_index trace2.txt && + + rm trace2.txt && + GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ + git -C sparse-index -c core.fsmonitor="" status -uno && + test_region index ensure_full_index trace2.txt + ) +' + test_done From patchwork Mon Jan 25 17:41:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12043901 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EE864C433DB for ; Mon, 25 Jan 2021 18:03:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B509E2251F for ; Mon, 25 Jan 2021 18:03:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731343AbhAYSDK (ORCPT ); Mon, 25 Jan 2021 13:03:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33234 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731339AbhAYRzx (ORCPT ); Mon, 25 Jan 2021 12:55:53 -0500 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DD4CFC06121C for ; Mon, 25 Jan 2021 09:42:28 -0800 (PST) Received: by mail-wm1-x32b.google.com with SMTP id u14so39233wml.4 for ; Mon, 25 Jan 2021 09:42:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=GJ4WeDWijjlg9pmh5tttLlHgELqjWYp/L8J8l1rm5LI=; b=bkU5WVMlGCWTNdIS4XIrPqcUOl28M5ddKHvqZEHE1wzuZn+3amfoFCLgoXCgXQwybH Cs1Eev/7slnC9sTDCiHps+vbS3HWhasAULS47jM8mYYcUfeA6Gp6vimtyxQGOBMgly0B UB6gbj80xRTZYWx4EQo38WNdYXtZMwtP4D+dYdIqwFvLWycFNjT4VinuIAzDPM5vKbz2 /5JPshtT3MjUjcEN0dN0v7LyP1T431VJXMXpgs8twhyuc8GNTUxHKMbph3OFO9eiYd/N PtpA5T0OHqzWLdM2lBMXZqd7SDAisMnImTN18L2ojC720cdT+xVPsUwVbMOhn9pxGU42 3nkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=GJ4WeDWijjlg9pmh5tttLlHgELqjWYp/L8J8l1rm5LI=; b=jroHtbaHw+RYMuSnPP1PfJk1VXFE519vTsW70brls0JqfwqNdbX2PKa8JEJWFD5vRi QzCm0J4vjBJsUlVO75zI/hhtImka9dcEMqLIu3cMdQfmdjDvKiqB/gq/+8Qb4MeL6GJd IxQHB/AA4o4yHXSX2fVOIE3A0gatjNIeaoM2o49R9/nl2GmoY4jvQjkzGWMnF9EMyGP2 wkhAyTrZO0Sa55XyPcNIoutSZPnpqiFO8zQWxhtX87osLnIX7thzaJsQbTEq9D/nK4ya z7/w6Haa6s3NrkVyCButNIKjbGlQEIqgGQzMsfbrhl32cCLf/TKrxmAA0YRElrwk8OcD 8MTA== X-Gm-Message-State: AOAM533xcxLoBGehIS7w2mNbvAl0E86jtJjHCNJKeT/y+yCK3wqCxlrS iwRRFU1vfuicgap30J4dMu+AEKnCuOw= X-Google-Smtp-Source: ABdhPJysMujhwxphLyuDXQc0UFhcQJ5Fa3kHVpxu2DMfaSuSwgJ49BzJd3Cz8BMYb/PHNTbxqvhE4Q== X-Received: by 2002:a05:600c:1986:: with SMTP id t6mr1150328wmq.93.1611596547378; Mon, 25 Jan 2021 09:42:27 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id q2sm50617wma.6.2021.01.25.09.42.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 09:42:26 -0800 (PST) Message-Id: In-Reply-To: References: Date: Mon, 25 Jan 2021 17:41:59 +0000 Subject: [PATCH 13/27] sparse-index: create extension for compatibility Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, peff@peff.net, jrnieder@gmail.com, sunshine@sunshineco.com, pclouds@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Previously, we enabled the sparse index format only using GIT_TEST_SPARSE_INDEX=1. This is not a feasible direction for users to actually select this mode. Further, sparse directory entries are not understood by the index formats as advertised. We _could_ add a new index version that explicitly adds these capabilities, but there are nuances to index formats 2, 3, and 4 that are still valuable to select as options. For now, create a repo extension, "extensions.sparseIndex", that specifies that the tool reading this repository must understand sparse directory entries. This change only encodes the extension and enables it when GIT_TEST_SPARSE_INDEX=1. Later, we will add a more user-friendly CLI mechanism. Signed-off-by: Derrick Stolee --- Documentation/config/extensions.txt | 7 ++++++ cache.h | 1 + repo-settings.c | 7 ++++++ repository.h | 3 ++- setup.c | 3 +++ sparse-index.c | 38 +++++++++++++++++++++++++---- 6 files changed, 53 insertions(+), 6 deletions(-) diff --git a/Documentation/config/extensions.txt b/Documentation/config/extensions.txt index 4e23d73cdca..5c86b364873 100644 --- a/Documentation/config/extensions.txt +++ b/Documentation/config/extensions.txt @@ -6,3 +6,10 @@ extensions.objectFormat:: Note that this setting should only be set by linkgit:git-init[1] or linkgit:git-clone[1]. Trying to change it after initialization will not work and will produce hard-to-diagnose issues. + +extensions.sparseIndex:: + When combined with `core.sparseCheckout=true` and + `core.sparseCheckoutCone=true`, the index may contain entries + corresponding to directories outside of the sparse-checkout + definition. Versions of Git that do not understand this extension + do not expect directory entries in the index. diff --git a/cache.h b/cache.h index b05341cc687..dcf089b7006 100644 --- a/cache.h +++ b/cache.h @@ -1054,6 +1054,7 @@ struct repository_format { int worktree_config; int is_bare; int hash_algo; + int sparse_index; char *work_tree; struct string_list unknown_extensions; struct string_list v1_only_extensions; diff --git a/repo-settings.c b/repo-settings.c index d63569e4041..9677d50f923 100644 --- a/repo-settings.c +++ b/repo-settings.c @@ -85,4 +85,11 @@ void prepare_repo_settings(struct repository *r) * removed. */ r->settings.command_requires_full_index = 1; + + /* + * Initialize this as off. + */ + r->settings.sparse_index = 0; + if (!repo_config_get_bool(r, "extensions.sparseindex", &value) && value) + r->settings.sparse_index = 1; } diff --git a/repository.h b/repository.h index e06a2301569..a45f7520fd9 100644 --- a/repository.h +++ b/repository.h @@ -42,7 +42,8 @@ struct repo_settings { int core_multi_pack_index; - unsigned command_requires_full_index:1; + unsigned command_requires_full_index:1, + sparse_index:1; }; struct repository { diff --git a/setup.c b/setup.c index c04cd25a30d..cd839456461 100644 --- a/setup.c +++ b/setup.c @@ -500,6 +500,9 @@ static enum extension_result handle_extension(const char *var, return error("invalid value for 'extensions.objectformat'"); data->hash_algo = format; return EXTENSION_OK; + } else if (!strcmp(ext, "sparseindex")) { + data->sparse_index = 1; + return EXTENSION_OK; } return EXTENSION_UNKNOWN; } diff --git a/sparse-index.c b/sparse-index.c index 5dd0b835b9d..71544095267 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -102,19 +102,47 @@ static int convert_to_sparse_rec(struct index_state *istate, return num_converted - start_converted; } +static int enable_sparse_index(struct repository *repo) +{ + const char *config_path = repo_git_path(repo, "config.worktree"); + + if (upgrade_repository_format(1) < 0) { + warning(_("unable to upgrade repository format to enable sparse-index")); + return -1; + } + git_config_set_in_file_gently(config_path, + "extensions.sparseIndex", + "true"); + + prepare_repo_settings(repo); + repo->settings.sparse_index = 1; + return 0; +} + int convert_to_sparse(struct index_state *istate) { if (istate->split_index || istate->sparse_index || !core_apply_sparse_checkout || !core_sparse_checkout_cone) return 0; + if (!istate->repo) + istate->repo = the_repository; + + /* + * The GIT_TEST_SPARSE_INDEX environment variable triggers the + * extensions.sparseIndex config variable to be on. + */ + if (git_env_bool("GIT_TEST_SPARSE_INDEX", 0)) { + int err = enable_sparse_index(istate->repo); + if (err < 0) + return err; + } + /* - * For now, only create a sparse index with the - * GIT_TEST_SPARSE_INDEX environment variable. We will relax - * this once we have a proper way to opt-in (and later still, - * opt-out). + * Only convert to sparse if extensions.sparseIndex is set. */ - if (!git_env_bool("GIT_TEST_SPARSE_INDEX", 0)) + prepare_repo_settings(istate->repo); + if (!istate->repo->settings.sparse_index) return 0; if (!istate->sparse_checkout_patterns) { From patchwork Mon Jan 25 17:42:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12043875 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 973EFC433E0 for ; Mon, 25 Jan 2021 17:57:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6085F22B3F for ; Mon, 25 Jan 2021 17:57:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730369AbhAYR4v (ORCPT ); Mon, 25 Jan 2021 12:56:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33058 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731324AbhAYRz2 (ORCPT ); Mon, 25 Jan 2021 12:55:28 -0500 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CDC5CC06121D for ; Mon, 25 Jan 2021 09:42:29 -0800 (PST) Received: by mail-wr1-x435.google.com with SMTP id c12so13832925wrc.7 for ; Mon, 25 Jan 2021 09:42:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Cm85K2WgDg8t8LZNq5tEFsz8C0DHDAk09/x86vaB230=; b=qkfIQFAbqTQPy5SwK2IvUUdR+XQu2ke3PfxDh6pHwLX2h3tM3xkNwH0hFAEwHizyvj zitt9Rcnbmbpl1W9XZiIixuIJmDeQRsX3hFrjNU6pafzbZYacdZQMxkCDD2VwTlyfWMV zdwSU915/Cq038BemaKN70JssJC5mwfabulD8zbfGYRSvx5hCEl/rrqTtvLGgELrHmwa dCgEpw8bzXb4lBGEjzy0c9d9oLsDy1V983o1x0XCKYkS0bdul7+lTEshtpnEMXaaN5lx Fal7aWgNbJCyIorVTp7hEslFjECYooapBZQ3k34NlkXTZFCV3Q2iZ2C5l0+1lN3nnRwG MGQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Cm85K2WgDg8t8LZNq5tEFsz8C0DHDAk09/x86vaB230=; b=uGK73e2F0vYAd3x/wvrzst6SFaCV34qHPaz+zGgUCiHP6bJJbHIxLGMVVZNPkIwP+c IEsSRaWF7YQUEPZpJzl6xFywzonVZWM1gEzPXyPT4DqIkbGZgfTpvRTLlhsRaRThvs9d frsX7Xsz5H8qg2vtkzCAZDXMyYuEyOApaYLarspp2UlD5XVXLaH2yqdGp4BEZhUOBiZA Bv/ima4jzAz2sM334NFc5RQe6EWj9kwyDmLS01y/jMtYeq3YCVZQSyq/gxcszznOBJbP 5B2WpbPHIDdoiAATryd0pi8k7PxYdlm1dJq4YcRLy1lWWoMhUOUADue1iJ7u3lfuDJMW Uy3w== X-Gm-Message-State: AOAM531DA7jznb4nQkAI8ljdnzrXqWDKMVcI7n4cfBhmrqTYd/vX3i+K yvOvKECqiqpWcDLQLFwCtEqxj5BwWi8= X-Google-Smtp-Source: ABdhPJwAC9L8taxh2RimN1Np6rtHeEkQjH8/X+YfgtMmxgwK6/yzODQ4wqx4PmG7r35Lz7XHC77NSQ== X-Received: by 2002:a05:6000:1142:: with SMTP id d2mr2265896wrx.307.1611596548344; Mon, 25 Jan 2021 09:42:28 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id d13sm23446289wrx.93.2021.01.25.09.42.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 09:42:27 -0800 (PST) Message-Id: <54ca484b057b5caa9871cffe47a0a60617e0c5cf.1611596534.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 25 Jan 2021 17:42:00 +0000 Subject: [PATCH 14/27] sparse-checkout: toggle sparse index from builtin Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, peff@peff.net, jrnieder@gmail.com, sunshine@sunshineco.com, pclouds@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The sparse index extension is used to signal that index writes should be in sparse mode. This was only updated using GIT_TEST_SPARSE_INDEX=1. Add a '--[no-]sparse-index' option to 'git sparse-checkout init' that specifies if the sparse index should be used. It also updates the index to use the correct format, either way. Add a warning in the documentation that the use of a repository extension might reduce compatibility with third-party tools. 'git sparse-checkout init' already sets extension.worktreeConfig, which places most sparse-checkout users outside of the scope of most third-party tools. Update t1092-sparse-checkout-compatibility.sh to use this CLI instead of GIT_TEST_SPARSE_INDEX=1. Signed-off-by: Derrick Stolee --- Documentation/git-sparse-checkout.txt | 14 +++++++++ builtin/sparse-checkout.c | 17 ++++++++++- sparse-index.c | 38 ++++++++++++++++-------- sparse-index.h | 3 ++ t/t1092-sparse-checkout-compatibility.sh | 33 ++++++++++---------- 5 files changed, 75 insertions(+), 30 deletions(-) diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt index a0eeaeb02ee..b51b8450cfd 100644 --- a/Documentation/git-sparse-checkout.txt +++ b/Documentation/git-sparse-checkout.txt @@ -45,6 +45,20 @@ To avoid interfering with other worktrees, it first enables the When `--cone` is provided, the `core.sparseCheckoutCone` setting is also set, allowing for better performance with a limited set of patterns (see 'CONE PATTERN SET' below). ++ +Use the `--[no-]sparse-index` option to toggle the use of the sparse +index format. This reduces the size of the index to be more closely +aligned with your sparse-checkout definition. This can have significant +performance advantages for commands such as `git status` or `git add`. +This feature is still experimental. Some commands might be slower with +a sparse index until they are properly integrated with the feature. ++ +**WARNING:** Using a sparse index requires modifying the index in a way +that is not completely understood by other tools. Enabling sparse index +enables the `extensions.spareseIndex` config value, which might cause +other tools to stop working with your repository. If you have trouble with +this compatibility, then run `git sparse-checkout sparse-index disable` to +remove this config and rewrite your index to not be sparse. 'set':: Write a set of patterns to the sparse-checkout file, as given as diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c index e00b82af727..ca63e2c64e9 100644 --- a/builtin/sparse-checkout.c +++ b/builtin/sparse-checkout.c @@ -14,6 +14,7 @@ #include "unpack-trees.h" #include "wt-status.h" #include "quote.h" +#include "sparse-index.h" static const char *empty_base = ""; @@ -283,12 +284,13 @@ static int set_config(enum sparse_checkout_mode mode) } static char const * const builtin_sparse_checkout_init_usage[] = { - N_("git sparse-checkout init [--cone]"), + N_("git sparse-checkout init [--cone] [--[no-]sparse-index]"), NULL }; static struct sparse_checkout_init_opts { int cone_mode; + int sparse_index; } init_opts; static int sparse_checkout_init(int argc, const char **argv) @@ -303,11 +305,15 @@ static int sparse_checkout_init(int argc, const char **argv) static struct option builtin_sparse_checkout_init_options[] = { OPT_BOOL(0, "cone", &init_opts.cone_mode, N_("initialize the sparse-checkout in cone mode")), + OPT_BOOL(0, "sparse-index", &init_opts.sparse_index, + N_("toggle the use of a sparse index")), OPT_END(), }; repo_read_index(the_repository); + init_opts.sparse_index = -1; + argc = parse_options(argc, argv, NULL, builtin_sparse_checkout_init_options, builtin_sparse_checkout_init_usage, 0); @@ -326,6 +332,15 @@ static int sparse_checkout_init(int argc, const char **argv) sparse_filename = get_sparse_checkout_filename(); res = add_patterns_from_file_to_list(sparse_filename, "", 0, &pl, NULL); + if (init_opts.sparse_index >= 0) { + if (set_sparse_index_config(the_repository, init_opts.sparse_index) < 0) + die(_("failed to modify sparse-index config")); + + /* force an index rewrite */ + repo_read_index(the_repository); + the_repository->index->updated_workdir = 1; + } + /* If we already have a sparse-checkout file, use it. */ if (res >= 0) { free(sparse_filename); diff --git a/sparse-index.c b/sparse-index.c index 71544095267..3552f88fb03 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -104,23 +104,38 @@ static int convert_to_sparse_rec(struct index_state *istate, static int enable_sparse_index(struct repository *repo) { - const char *config_path = repo_git_path(repo, "config.worktree"); + int res; if (upgrade_repository_format(1) < 0) { warning(_("unable to upgrade repository format to enable sparse-index")); return -1; } - git_config_set_in_file_gently(config_path, - "extensions.sparseIndex", - "true"); + res = git_config_set_gently("extensions.sparseindex", "true"); prepare_repo_settings(repo); repo->settings.sparse_index = 1; - return 0; + return res; +} + +int set_sparse_index_config(struct repository *repo, int enable) +{ + int res; + + if (enable) + return enable_sparse_index(repo); + + /* Don't downgrade repository format, just remove the extension. */ + res = git_config_set_multivar_gently("extensions.sparseindex", NULL, "", + CONFIG_FLAGS_MULTI_REPLACE); + + prepare_repo_settings(repo); + repo->settings.sparse_index = 0; + return res; } int convert_to_sparse(struct index_state *istate) { + int test_env; if (istate->split_index || istate->sparse_index || !core_apply_sparse_checkout || !core_sparse_checkout_cone) return 0; @@ -129,14 +144,13 @@ int convert_to_sparse(struct index_state *istate) istate->repo = the_repository; /* - * The GIT_TEST_SPARSE_INDEX environment variable triggers the - * extensions.sparseIndex config variable to be on. + * If GIT_TEST_SPARSE_INDEX=1, then trigger extensions.sparseIndex + * to be fully enabled. If GIT_TEST_SPARSE_INDEX=0 (set explicitly), + * then purposefully disable the setting. */ - if (git_env_bool("GIT_TEST_SPARSE_INDEX", 0)) { - int err = enable_sparse_index(istate->repo); - if (err < 0) - return err; - } + test_env = git_env_bool("GIT_TEST_SPARSE_INDEX", -1); + if (test_env >= 0) + set_sparse_index_config(istate->repo, test_env); /* * Only convert to sparse if extensions.sparseIndex is set. diff --git a/sparse-index.h b/sparse-index.h index a2777dcac59..ca936e95d11 100644 --- a/sparse-index.h +++ b/sparse-index.h @@ -5,4 +5,7 @@ struct index_state; void ensure_full_index(struct index_state *istate); int convert_to_sparse(struct index_state *istate); +struct repository; +int set_sparse_index_config(struct repository *repo, int enable); + #endif \ No newline at end of file diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index a22def89e37..c6b7e8b8891 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -4,6 +4,7 @@ test_description='compare full workdir to sparse workdir' GIT_TEST_CHECK_CACHE_TREE=0 GIT_TEST_SPLIT_INDEX=0 +GIT_TEST_SPARSE_INDEX= . ./test-lib.sh @@ -98,8 +99,9 @@ init_repos () { # initialize sparse-checkout definitions git -C sparse-checkout sparse-checkout init --cone && git -C sparse-checkout sparse-checkout set deep && - GIT_TEST_SPARSE_INDEX=1 git -C sparse-index sparse-checkout init --cone && - GIT_TEST_SPARSE_INDEX=1 git -C sparse-index sparse-checkout set deep + git -C sparse-index sparse-checkout init --cone --sparse-index && + test_cmp_config -C sparse-index true extensions.sparseindex && + git -C sparse-index sparse-checkout set deep } run_on_sparse () { @@ -109,7 +111,7 @@ run_on_sparse () { ) && ( cd sparse-index && - GIT_TEST_SPARSE_INDEX=1 $* >../sparse-index-out 2>../sparse-index-err + $* >../sparse-index-out 2>../sparse-index-err ) } @@ -146,7 +148,7 @@ test_expect_success 'sparse-index contents' ' || return 1 done && - GIT_TEST_SPARSE_INDEX=1 git -C sparse-index sparse-checkout set folder1 && + git -C sparse-index sparse-checkout set folder1 && test-tool -C sparse-index read-cache --table --no-stat >cache && for dir in deep folder2 x @@ -156,7 +158,7 @@ test_expect_success 'sparse-index contents' ' || return 1 done && - GIT_TEST_SPARSE_INDEX=1 git -C sparse-index sparse-checkout set deep/deeper1 && + git -C sparse-index sparse-checkout set deep/deeper1 && test-tool -C sparse-index read-cache --table --no-stat >cache && for dir in deep/deeper2 folder1 folder2 x @@ -377,18 +379,15 @@ test_expect_success 'clean' ' test_expect_success 'sparse-index is expanded and converted back' ' init_repos && - ( - (GIT_TEST_SPARSE_INDEX=1 && export GIT_TEST_SPARSE_INDEX) && - GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ - git -C sparse-index -c core.fsmonitor="" reset --hard && - test_region index convert_to_sparse trace2.txt && - test_region index ensure_full_index trace2.txt && - - rm trace2.txt && - GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ - git -C sparse-index -c core.fsmonitor="" status -uno && - test_region index ensure_full_index trace2.txt - ) + GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ + git -C sparse-index -c core.fsmonitor="" reset --hard && + test_region index convert_to_sparse trace2.txt && + test_region index ensure_full_index trace2.txt && + + rm trace2.txt && + GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ + git -C sparse-index -c core.fsmonitor="" status -uno && + test_region index ensure_full_index trace2.txt ' test_done From patchwork Mon Jan 25 17:42:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12043867 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 120E0C433DB for ; Mon, 25 Jan 2021 17:56:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D20DC22C9E for ; Mon, 25 Jan 2021 17:56:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731193AbhAYR4V (ORCPT ); Mon, 25 Jan 2021 12:56:21 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33054 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731322AbhAYRz1 (ORCPT ); Mon, 25 Jan 2021 12:55:27 -0500 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0AA55C06121E for ; Mon, 25 Jan 2021 09:42:31 -0800 (PST) Received: by mail-wr1-x42b.google.com with SMTP id l12so13818992wry.2 for ; Mon, 25 Jan 2021 09:42:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=tqoNH9biFM0sjLiauW4wB6Ql2LjvjVZRLf3VhPK4ZxI=; b=AAVToAEjh9zUqj6oFDWaDQKquChe64cdram/hFRdefqTuMutvCHSekqHz4OjqPnAk+ c5Y2UF0oz2CltWDdk+mQBgnuIZt6P2AKoix/gJJxSZEsuywbMK4iY8B8SKa7sVtnnuog oZDPST4vflTo1gYnr+VD6MG0C3OeaJzITdTMIzL0fm8dxMrnPgJhiGWBAtK/mbUgVwQR BXp3dF0vpVGWRmZpE273kAggO8IveGv/eAv7BMsKV9MQMB8KDXFLzt0h5qavO2XtF0xf +vMY/4PtH32i4LNqEFYuc/5vli/hkR59GJKcNZlbbtygKUufwmZaLZfIsHkVbg9Vpiqm 9prg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=tqoNH9biFM0sjLiauW4wB6Ql2LjvjVZRLf3VhPK4ZxI=; b=XS5Z0mAdBb4c8eFDCocdL3ImK3ECHqIe/qyLVrxlPs9gQm9dH79hPZ6aZvnHKbqvU2 4HgGP1KTFdr7N1MDfPsRfKBwv6cZ+rVhGKyvnBm8wT9R4WjQN/ZgXkEYk8uqG05wxHN5 isYLtDWPJ+/ZWjlrgf4In8yM4ETFC7wtVtVhLov+HQjdutea7Rtj/owQVuFQa0/b+DNd DRIPzGwwq4MXJRYnnaHpMUxhzfxOoi7DDlfly/1n7TyzuXPUXKIx3UDTeWS5WpOxfq+H Ds0zT8XzbNTLdTPpAsOwUeBFTWPmHRF4WMjX1nVVqux/MiarPmwwLChFO+fno5TWy4qH fh+g== X-Gm-Message-State: AOAM531vwXncGDzg6IoqhAtBUV0rhbCnAGJIeWOhSG/fmMXCf7/pQKmF EvgO0oXfX+tnl6Btz2mt5E3/r8/BtFY= X-Google-Smtp-Source: ABdhPJxgU2+kcNksxkomdTbqwyDwmK0PA2zHiS/QmCIOxNod1XQveQJCkHEKaAqAMifUSbKpcnoL7w== X-Received: by 2002:a5d:440a:: with SMTP id z10mr2277673wrq.266.1611596549190; Mon, 25 Jan 2021 09:42:29 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id b2sm5418wmd.41.2021.01.25.09.42.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 09:42:28 -0800 (PST) Message-Id: <71b33bb2498d53e5818f3a597240b530976152fb.1611596534.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 25 Jan 2021 17:42:01 +0000 Subject: [PATCH 15/27] [RFC-VERSION] *: ensure full index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, peff@peff.net, jrnieder@gmail.com, sunshine@sunshineco.com, pclouds@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee This giant patch is not intended for actual review. I have a branch that has these changes split out in a sane way with some commentary in each file that is modified. The idea here is to guard certain portions of the codebase that do not know how to handle sparse indexes by ensuring that the index is expanded to a full index before proceeding with the logic. This also provides a good mechanism for testing which code needs updating to enable the sparse index in a Git builtin. The builtin can set the_repository->settings.command_requires_full_index to zero and then we can debug the command with a breakpoint on ensure_full_index(). That identifies the portion of code that needs adjusting before enabling sparse indexes for that command. Some index operations must be changed to operate on a non-const pointer, since ensuring a full index will modify the index itself. There are likely some gaps to these protections, which is why it will be important to carefully test each scenario as we relax the requirements. I expect that to be a long effort. Signed-off-by: Derrick Stolee --- apply.c | 10 +++++++++- blame.c | 7 ++++++- builtin/checkout-index.c | 5 ++++- builtin/grep.c | 2 ++ builtin/ls-files.c | 9 ++++++++- builtin/merge-index.c | 2 ++ builtin/mv.c | 2 ++ builtin/rm.c | 2 ++ builtin/sparse-checkout.c | 1 + builtin/update-index.c | 2 ++ cache.h | 1 + diff-lib.c | 2 ++ diff.c | 2 ++ dir.c | 14 +++++++++++++- entry.c | 2 ++ fsmonitor.c | 11 ++++++++++- merge-recursive.c | 22 +++++++++++++++++++--- name-hash.c | 6 ++++++ pathspec.c | 5 +++-- pathspec.h | 4 ++-- read-cache.c | 19 +++++++++++++++++-- rerere.c | 2 ++ resolve-undo.c | 6 ++++++ sha1-name.c | 3 +++ split-index.c | 2 ++ submodule.c | 24 +++++++++++++++++++----- submodule.h | 6 +++--- tree.c | 2 ++ wt-status.c | 7 +++++++ 29 files changed, 159 insertions(+), 23 deletions(-) diff --git a/apply.c b/apply.c index 668b16e9893..5bfbd928b38 100644 --- a/apply.c +++ b/apply.c @@ -3523,6 +3523,8 @@ static int load_current(struct apply_state *state, if (!patch->is_new) BUG("patch to %s is not a creation", patch->old_name); + ensure_full_index(state->repo->index); + pos = index_name_pos(state->repo->index, name, strlen(name)); if (pos < 0) return error(_("%s: does not exist in index"), name); @@ -3692,7 +3694,11 @@ static int check_preimage(struct apply_state *state, } if (state->check_index && !previous) { - int pos = index_name_pos(state->repo->index, old_name, + int pos; + + ensure_full_index(state->repo->index); + + pos = index_name_pos(state->repo->index, old_name, strlen(old_name)); if (pos < 0) { if (patch->is_new < 0) @@ -3751,6 +3757,8 @@ static int check_to_create(struct apply_state *state, if (state->check_index && (!ok_if_exists || !state->cached)) { int pos; + ensure_full_index(state->repo->index); + pos = index_name_pos(state->repo->index, new_name, strlen(new_name)); if (pos >= 0) { struct cache_entry *ce = state->repo->index->cache[pos]; diff --git a/blame.c b/blame.c index a5044fcfaa6..0aa368a35cf 100644 --- a/blame.c +++ b/blame.c @@ -108,6 +108,7 @@ static void verify_working_tree_path(struct repository *r, return; } + ensure_full_index(r->index); pos = index_name_pos(r->index, path, strlen(path)); if (pos >= 0) ; /* path is in the index */ @@ -277,7 +278,11 @@ static struct commit *fake_working_tree_commit(struct repository *r, len = strlen(path); if (!mode) { - int pos = index_name_pos(r->index, path, len); + int pos; + + ensure_full_index(r->index); + + pos = index_name_pos(r->index, path, len); if (0 <= pos) mode = r->index->cache[pos]->ce_mode; else diff --git a/builtin/checkout-index.c b/builtin/checkout-index.c index 4bbfc92dce5..24c85b1c125 100644 --- a/builtin/checkout-index.c +++ b/builtin/checkout-index.c @@ -48,11 +48,14 @@ static void write_tempfile_record(const char *name, const char *prefix) static int checkout_file(const char *name, const char *prefix) { int namelen = strlen(name); - int pos = cache_name_pos(name, namelen); + int pos; int has_same_name = 0; int did_checkout = 0; int errs = 0; + ensure_full_index(the_repository->index); + pos = index_name_pos(the_repository->index, name, namelen); + if (pos < 0) pos = -pos - 1; diff --git a/builtin/grep.c b/builtin/grep.c index ca259af4416..e53cf817204 100644 --- a/builtin/grep.c +++ b/builtin/grep.c @@ -506,6 +506,8 @@ static int grep_cache(struct grep_opt *opt, if (repo_read_index(repo) < 0) die(_("index file corrupt")); + ensure_full_index(repo->index); + for (nr = 0; nr < repo->index->cache_nr; nr++) { const struct cache_entry *ce = repo->index->cache[nr]; strbuf_setlen(&name, name_base_len); diff --git a/builtin/ls-files.c b/builtin/ls-files.c index c8eae899b82..933e259cdbe 100644 --- a/builtin/ls-files.c +++ b/builtin/ls-files.c @@ -150,7 +150,7 @@ static void show_other_files(const struct index_state *istate, } } -static void show_killed_files(const struct index_state *istate, +static void show_killed_files(struct index_state *istate, const struct dir_struct *dir) { int i; @@ -159,6 +159,8 @@ static void show_killed_files(const struct index_state *istate, char *cp, *sp; int pos, len, killed = 0; + ensure_full_index(istate); + for (cp = ent->name; cp - ent->name < ent->len; cp = sp + 1) { sp = strchr(cp, '/'); if (!sp) { @@ -313,6 +315,7 @@ static void show_files(struct repository *repo, struct dir_struct *dir) show_killed_files(repo->index, dir); } if (show_cached || show_stage) { + ensure_full_index(repo->index); for (i = 0; i < repo->index->cache_nr; i++) { const struct cache_entry *ce = repo->index->cache[i]; @@ -332,6 +335,7 @@ static void show_files(struct repository *repo, struct dir_struct *dir) } } if (show_deleted || show_modified) { + ensure_full_index(repo->index); for (i = 0; i < repo->index->cache_nr; i++) { const struct cache_entry *ce = repo->index->cache[i]; struct stat st; @@ -368,6 +372,7 @@ static void prune_index(struct index_state *istate, if (!prefix || !istate->cache_nr) return; + ensure_full_index(istate); pos = index_name_pos(istate, prefix, prefixlen); if (pos < 0) pos = -pos-1; @@ -428,6 +433,8 @@ void overlay_tree_on_index(struct index_state *istate, if (!tree) die("bad tree-ish %s", tree_name); + ensure_full_index(istate); + /* Hoist the unmerged entries up to stage #3 to make room */ for (i = 0; i < istate->cache_nr; i++) { struct cache_entry *ce = istate->cache[i]; diff --git a/builtin/merge-index.c b/builtin/merge-index.c index 38ea6ad6ca2..3e1ddabd650 100644 --- a/builtin/merge-index.c +++ b/builtin/merge-index.c @@ -80,6 +80,8 @@ int cmd_merge_index(int argc, const char **argv, const char *prefix) read_cache(); + ensure_full_index(&the_index); + i = 1; if (!strcmp(argv[i], "-o")) { one_shot = 1; diff --git a/builtin/mv.c b/builtin/mv.c index 7dac714af90..2ab6416fce9 100644 --- a/builtin/mv.c +++ b/builtin/mv.c @@ -145,6 +145,8 @@ int cmd_mv(int argc, const char **argv, const char *prefix) if (read_cache() < 0) die(_("index file corrupt")); + ensure_full_index(&the_index); + source = internal_prefix_pathspec(prefix, argv, argc, 0); modes = xcalloc(argc, sizeof(enum update_mode)); /* diff --git a/builtin/rm.c b/builtin/rm.c index 4858631e0f0..2db4fcd22d9 100644 --- a/builtin/rm.c +++ b/builtin/rm.c @@ -291,6 +291,8 @@ int cmd_rm(int argc, const char **argv, const char *prefix) refresh_index(&the_index, REFRESH_QUIET|REFRESH_UNMERGED, &pathspec, NULL, NULL); + ensure_full_index(&the_index); + seen = xcalloc(pathspec.nr, 1); for (i = 0; i < active_nr; i++) { diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c index ca63e2c64e9..14022b5e182 100644 --- a/builtin/sparse-checkout.c +++ b/builtin/sparse-checkout.c @@ -123,6 +123,7 @@ static int update_working_directory(struct pattern_list *pl) o.pl = pl; setup_work_tree(); + ensure_full_index(r->index); repo_hold_locked_index(r, &lock_file, LOCK_DIE_ON_ERROR); diff --git a/builtin/update-index.c b/builtin/update-index.c index 79087bccea4..521a6c23c75 100644 --- a/builtin/update-index.c +++ b/builtin/update-index.c @@ -1088,6 +1088,8 @@ int cmd_update_index(int argc, const char **argv, const char *prefix) the_index.updated_skipworktree = 1; + ensure_full_index(&the_index); + /* * Custom copy of parse_options() because we want to handle * filename arguments as they come. diff --git a/cache.h b/cache.h index dcf089b7006..306eab444b9 100644 --- a/cache.h +++ b/cache.h @@ -346,6 +346,7 @@ void add_name_hash(struct index_state *istate, struct cache_entry *ce); void remove_name_hash(struct index_state *istate, struct cache_entry *ce); void free_name_hash(struct index_state *istate); +void ensure_full_index(struct index_state *istate); /* Cache entry creation and cleanup */ diff --git a/diff-lib.c b/diff-lib.c index b73cc1859a4..3743e4463b4 100644 --- a/diff-lib.c +++ b/diff-lib.c @@ -96,6 +96,8 @@ int run_diff_files(struct rev_info *revs, unsigned int option) uint64_t start = getnanotime(); struct index_state *istate = revs->diffopt.repo->index; + ensure_full_index(istate); + diff_set_mnemonic_prefix(&revs->diffopt, "i/", "w/"); refresh_fsmonitor(istate); diff --git a/diff.c b/diff.c index 2253ec88029..02fafee8587 100644 --- a/diff.c +++ b/diff.c @@ -3901,6 +3901,8 @@ static int reuse_worktree_file(struct index_state *istate, if (!want_file && would_convert_to_git(istate, name)) return 0; + ensure_full_index(istate); + len = strlen(name); pos = index_name_pos(istate, name, len); if (pos < 0) diff --git a/dir.c b/dir.c index d153a63bbd1..ad6eb033cb1 100644 --- a/dir.c +++ b/dir.c @@ -892,13 +892,15 @@ void add_pattern(const char *string, const char *base, add_pattern_to_hashsets(pl, pattern); } -static int read_skip_worktree_file_from_index(const struct index_state *istate, +static int read_skip_worktree_file_from_index(struct index_state *istate, const char *path, size_t *size_out, char **data_out, struct oid_stat *oid_stat) { int pos, len; + ensure_full_index(istate); + len = strlen(path); pos = index_name_pos(istate, path, len); if (pos < 0) @@ -1088,6 +1090,10 @@ static int add_patterns(const char *fname, const char *base, int baselen, close(fd); if (oid_stat) { int pos; + + if (istate) + ensure_full_index(istate); + if (oid_stat->valid && !match_stat_data_racy(istate, &oid_stat->stat, &st)) ; /* no content change, oid_stat->oid still good */ @@ -1696,6 +1702,8 @@ static enum exist_status directory_exists_in_index(struct index_state *istate, if (ignore_case) return directory_exists_in_index_icase(istate, dirname, len); + ensure_full_index(istate); + pos = index_name_pos(istate, dirname, len); if (pos < 0) pos = -pos-1; @@ -2050,6 +2058,8 @@ static int get_index_dtype(struct index_state *istate, int pos; const struct cache_entry *ce; + ensure_full_index(istate); + ce = index_file_exists(istate, path, len, 0); if (ce) { if (!ce_uptodate(ce)) @@ -3536,6 +3546,8 @@ static void connect_wt_gitdir_in_nested(const char *sub_worktree, if (repo_read_index(&subrepo) < 0) die(_("index file corrupt in repo %s"), subrepo.gitdir); + ensure_full_index(subrepo.index); + for (i = 0; i < subrepo.index->cache_nr; i++) { const struct cache_entry *ce = subrepo.index->cache[i]; diff --git a/entry.c b/entry.c index a0532f1f000..d505e6f2c6e 100644 --- a/entry.c +++ b/entry.c @@ -412,6 +412,8 @@ static void mark_colliding_entries(const struct checkout *state, ce->ce_flags |= CE_MATCHED; + ensure_full_index(state->istate); + for (i = 0; i < state->istate->cache_nr; i++) { struct cache_entry *dup = state->istate->cache[i]; diff --git a/fsmonitor.c b/fsmonitor.c index fe9e9d7baf4..7b8cd3975b9 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -97,6 +97,9 @@ int read_fsmonitor_extension(struct index_state *istate, const void *data, void fill_fsmonitor_bitmap(struct index_state *istate) { unsigned int i, skipped = 0; + + ensure_full_index(istate); + istate->fsmonitor_dirty = ewah_new(); for (i = 0; i < istate->cache_nr; i++) { if (istate->cache[i]->ce_flags & CE_REMOVE) @@ -158,7 +161,11 @@ static int query_fsmonitor(int version, const char *last_update, struct strbuf * static void fsmonitor_refresh_callback(struct index_state *istate, const char *name) { - int pos = index_name_pos(istate, name, strlen(name)); + int pos; + + ensure_full_index(istate); + + pos = index_name_pos(istate, name, strlen(name)); if (pos >= 0) { struct cache_entry *ce = istate->cache[pos]; @@ -330,6 +337,8 @@ void tweak_fsmonitor(struct index_state *istate) if (istate->fsmonitor_dirty) { if (fsmonitor_enabled) { + ensure_full_index(istate); + /* Mark all entries valid */ for (i = 0; i < istate->cache_nr; i++) { istate->cache[i]->ce_flags |= CE_FSMONITOR_VALID; diff --git a/merge-recursive.c b/merge-recursive.c index f736a0f6323..12109f37723 100644 --- a/merge-recursive.c +++ b/merge-recursive.c @@ -522,6 +522,8 @@ static struct string_list *get_unmerged(struct index_state *istate) unmerged->strdup_strings = 1; + ensure_full_index(istate); + for (i = 0; i < istate->cache_nr; i++) { struct string_list_item *item; struct stage_data *e; @@ -762,6 +764,8 @@ static int dir_in_way(struct index_state *istate, const char *path, strbuf_addstr(&dirpath, path); strbuf_addch(&dirpath, '/'); + ensure_full_index(istate); + pos = index_name_pos(istate, dirpath.buf, dirpath.len); if (pos < 0) @@ -785,9 +789,13 @@ static int dir_in_way(struct index_state *istate, const char *path, static int was_tracked_and_matches(struct merge_options *opt, const char *path, const struct diff_filespec *blob) { - int pos = index_name_pos(&opt->priv->orig_index, path, strlen(path)); + int pos; struct cache_entry *ce; + ensure_full_index(&opt->priv->orig_index); + + pos = index_name_pos(&opt->priv->orig_index, path, strlen(path)); + if (0 > pos) /* we were not tracking this path before the merge */ return 0; @@ -802,7 +810,11 @@ static int was_tracked_and_matches(struct merge_options *opt, const char *path, */ static int was_tracked(struct merge_options *opt, const char *path) { - int pos = index_name_pos(&opt->priv->orig_index, path, strlen(path)); + int pos; + + ensure_full_index(&opt->priv->orig_index); + + pos = index_name_pos(&opt->priv->orig_index, path, strlen(path)); if (0 <= pos) /* we were tracking this path before the merge */ @@ -814,6 +826,9 @@ static int was_tracked(struct merge_options *opt, const char *path) static int would_lose_untracked(struct merge_options *opt, const char *path) { struct index_state *istate = opt->repo->index; + int pos; + + ensure_full_index(istate); /* * This may look like it can be simplified to: @@ -832,7 +847,7 @@ static int would_lose_untracked(struct merge_options *opt, const char *path) * update_file()/would_lose_untracked(); see every comment in this * file which mentions "update_stages". */ - int pos = index_name_pos(istate, path, strlen(path)); + pos = index_name_pos(istate, path, strlen(path)); if (pos < 0) pos = -1 - pos; @@ -3086,6 +3101,7 @@ static int handle_content_merge(struct merge_file_info *mfi, * flag to avoid making the file appear as if it were * deleted by the user. */ + ensure_full_index(&opt->priv->orig_index); pos = index_name_pos(&opt->priv->orig_index, path, strlen(path)); ce = opt->priv->orig_index.cache[pos]; if (ce_skip_worktree(ce)) { diff --git a/name-hash.c b/name-hash.c index 4e03fac9bb1..0f6d4fcca5a 100644 --- a/name-hash.c +++ b/name-hash.c @@ -679,6 +679,8 @@ int index_dir_exists(struct index_state *istate, const char *name, int namelen) { struct dir_entry *dir; + ensure_full_index(istate); + lazy_init_name_hash(istate); dir = find_dir_entry(istate, name, namelen); return dir && dir->nr; @@ -689,6 +691,8 @@ void adjust_dirname_case(struct index_state *istate, char *name) const char *startPtr = name; const char *ptr = startPtr; + ensure_full_index( istate); + lazy_init_name_hash(istate); while (*ptr) { while (*ptr && *ptr != '/') @@ -712,6 +716,8 @@ struct cache_entry *index_file_exists(struct index_state *istate, const char *na struct cache_entry *ce; unsigned int hash = memihash(name, namelen); + ensure_full_index(istate); + lazy_init_name_hash(istate); ce = hashmap_get_entry_from_hash(&istate->name_hash, hash, NULL, diff --git a/pathspec.c b/pathspec.c index 7a229d8d22f..9b105855483 100644 --- a/pathspec.c +++ b/pathspec.c @@ -20,7 +20,7 @@ * to use find_pathspecs_matching_against_index() instead. */ void add_pathspec_matches_against_index(const struct pathspec *pathspec, - const struct index_state *istate, + struct index_state *istate, char *seen) { int num_unmatched = 0, i; @@ -36,6 +36,7 @@ void add_pathspec_matches_against_index(const struct pathspec *pathspec, num_unmatched++; if (!num_unmatched) return; + ensure_full_index(istate); for (i = 0; i < istate->cache_nr; i++) { const struct cache_entry *ce = istate->cache[i]; ce_path_match(istate, ce, pathspec, seen); @@ -51,7 +52,7 @@ void add_pathspec_matches_against_index(const struct pathspec *pathspec, * given pathspecs achieves against all items in the index. */ char *find_pathspecs_matching_against_index(const struct pathspec *pathspec, - const struct index_state *istate) + struct index_state *istate) { char *seen = xcalloc(pathspec->nr, 1); add_pathspec_matches_against_index(pathspec, istate, seen); diff --git a/pathspec.h b/pathspec.h index 454ce364fac..f19c5dcf022 100644 --- a/pathspec.h +++ b/pathspec.h @@ -150,10 +150,10 @@ static inline int ps_strcmp(const struct pathspec_item *item, } void add_pathspec_matches_against_index(const struct pathspec *pathspec, - const struct index_state *istate, + struct index_state *istate, char *seen); char *find_pathspecs_matching_against_index(const struct pathspec *pathspec, - const struct index_state *istate); + struct index_state *istate); int match_pathspec_attrs(const struct index_state *istate, const char *name, int namelen, const struct pathspec_item *item); diff --git a/read-cache.c b/read-cache.c index 0522260416e..65679d70d7c 100644 --- a/read-cache.c +++ b/read-cache.c @@ -622,7 +622,11 @@ void remove_marked_cache_entries(struct index_state *istate, int invalidate) int remove_file_from_index(struct index_state *istate, const char *path) { - int pos = index_name_pos(istate, path, strlen(path)); + int pos; + + ensure_full_index(istate); + + pos = index_name_pos(istate, path, strlen(path)); if (pos < 0) pos = -pos-1; cache_tree_invalidate_path(istate, path); @@ -640,9 +644,12 @@ static int compare_name(struct cache_entry *ce, const char *path, int namelen) static int index_name_pos_also_unmerged(struct index_state *istate, const char *path, int namelen) { - int pos = index_name_pos(istate, path, namelen); + int pos; struct cache_entry *ce; + ensure_full_index(istate); + + pos = index_name_pos(istate, path, namelen); if (pos >= 0) return pos; @@ -717,6 +724,8 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st, int hash_flags = HASH_WRITE_OBJECT; struct object_id oid; + ensure_full_index(istate); + if (flags & ADD_CACHE_RENORMALIZE) hash_flags |= HASH_RENORMALIZE; @@ -1095,6 +1104,8 @@ static int has_dir_name(struct index_state *istate, size_t len_eq_last; int cmp_last = 0; + ensure_full_index(istate); + /* * We are frequently called during an iteration on a sorted * list of pathnames and while building a new index. Therefore, @@ -1338,6 +1349,8 @@ int add_index_entry(struct index_state *istate, struct cache_entry *ce, int opti { int pos; + ensure_full_index(istate); + if (option & ADD_CACHE_JUST_APPEND) pos = istate->cache_nr; else { @@ -1547,6 +1560,8 @@ int refresh_index(struct index_state *istate, unsigned int flags, * we only have to do the special cases that are left. */ preload_index(istate, pathspec, 0); + + ensure_full_index(istate); for (i = 0; i < istate->cache_nr; i++) { struct cache_entry *ce, *new_entry; int cache_errno = 0; diff --git a/rerere.c b/rerere.c index 9281131a9f1..1836a6cfbcf 100644 --- a/rerere.c +++ b/rerere.c @@ -962,6 +962,8 @@ static int handle_cache(struct index_state *istate, struct rerere_io_mem io; int marker_size = ll_merge_marker_size(istate, path); + ensure_full_index(istate); + /* * Reproduce the conflicted merge in-core */ diff --git a/resolve-undo.c b/resolve-undo.c index 236320f179c..a4265834977 100644 --- a/resolve-undo.c +++ b/resolve-undo.c @@ -125,6 +125,8 @@ int unmerge_index_entry_at(struct index_state *istate, int pos) if (!istate->resolve_undo) return pos; + ensure_full_index(istate); + ce = istate->cache[pos]; if (ce_stage(ce)) { /* already unmerged */ @@ -172,6 +174,8 @@ void unmerge_marked_index(struct index_state *istate) if (!istate->resolve_undo) return; + ensure_full_index(istate); + for (i = 0; i < istate->cache_nr; i++) { const struct cache_entry *ce = istate->cache[i]; if (ce->ce_flags & CE_MATCHED) @@ -186,6 +190,8 @@ void unmerge_index(struct index_state *istate, const struct pathspec *pathspec) if (!istate->resolve_undo) return; + ensure_full_index(istate); + for (i = 0; i < istate->cache_nr; i++) { const struct cache_entry *ce = istate->cache[i]; if (!ce_path_match(istate, ce, pathspec, NULL)) diff --git a/sha1-name.c b/sha1-name.c index 0b23b86ceb4..c2f17e526ab 100644 --- a/sha1-name.c +++ b/sha1-name.c @@ -1734,6 +1734,8 @@ static void diagnose_invalid_index_path(struct repository *r, if (!prefix) prefix = ""; + ensure_full_index(r->index); + /* Wrong stage number? */ pos = index_name_pos(istate, filename, namelen); if (pos < 0) @@ -1854,6 +1856,7 @@ static enum get_oid_result get_oid_with_context_1(struct repository *repo, if (!repo->index || !repo->index->cache) repo_read_index(repo); + ensure_full_index(repo->index); pos = index_name_pos(repo->index, cp, namelen); if (pos < 0) pos = -pos - 1; diff --git a/split-index.c b/split-index.c index c0e8ad670d0..3150fa6476a 100644 --- a/split-index.c +++ b/split-index.c @@ -4,6 +4,8 @@ struct split_index *init_split_index(struct index_state *istate) { + ensure_full_index(istate); + if (!istate->split_index) { istate->split_index = xcalloc(1, sizeof(*istate->split_index)); istate->split_index->refcount = 1; diff --git a/submodule.c b/submodule.c index b3bb59f0664..f80cfddbd52 100644 --- a/submodule.c +++ b/submodule.c @@ -33,9 +33,13 @@ static struct oid_array ref_tips_after_fetch; * will be disabled because we can't guess what might be configured in * .gitmodules unless the user resolves the conflict. */ -int is_gitmodules_unmerged(const struct index_state *istate) +int is_gitmodules_unmerged(struct index_state *istate) { - int pos = index_name_pos(istate, GITMODULES_FILE, strlen(GITMODULES_FILE)); + int pos; + + ensure_full_index(istate); + + pos = index_name_pos(istate, GITMODULES_FILE, strlen(GITMODULES_FILE)); if (pos < 0) { /* .gitmodules not found or isn't merged */ pos = -1 - pos; if (istate->cache_nr > pos) { /* there is a .gitmodules */ @@ -77,7 +81,11 @@ int is_writing_gitmodules_ok(void) */ int is_staging_gitmodules_ok(struct index_state *istate) { - int pos = index_name_pos(istate, GITMODULES_FILE, strlen(GITMODULES_FILE)); + int pos; + + ensure_full_index(istate); + + pos = index_name_pos(istate, GITMODULES_FILE, strlen(GITMODULES_FILE)); if ((pos >= 0) && (pos < istate->cache_nr)) { struct stat st; @@ -301,7 +309,7 @@ int is_submodule_populated_gently(const char *path, int *return_error_code) /* * Dies if the provided 'prefix' corresponds to an unpopulated submodule */ -void die_in_unpopulated_submodule(const struct index_state *istate, +void die_in_unpopulated_submodule(struct index_state *istate, const char *prefix) { int i, prefixlen; @@ -311,6 +319,8 @@ void die_in_unpopulated_submodule(const struct index_state *istate, prefixlen = strlen(prefix); + ensure_full_index(istate); + for (i = 0; i < istate->cache_nr; i++) { struct cache_entry *ce = istate->cache[i]; int ce_len = ce_namelen(ce); @@ -331,11 +341,13 @@ void die_in_unpopulated_submodule(const struct index_state *istate, /* * Dies if any paths in the provided pathspec descends into a submodule */ -void die_path_inside_submodule(const struct index_state *istate, +void die_path_inside_submodule(struct index_state *istate, const struct pathspec *ps) { int i, j; + ensure_full_index(istate); + for (i = 0; i < istate->cache_nr; i++) { struct cache_entry *ce = istate->cache[i]; int ce_len = ce_namelen(ce); @@ -1420,6 +1432,8 @@ static int get_next_submodule(struct child_process *cp, { struct submodule_parallel_fetch *spf = data; + ensure_full_index(spf->r->index); + for (; spf->count < spf->r->index->cache_nr; spf->count++) { const struct cache_entry *ce = spf->r->index->cache[spf->count]; const char *default_argv; diff --git a/submodule.h b/submodule.h index 4ac6e31cf1f..84640c49c11 100644 --- a/submodule.h +++ b/submodule.h @@ -39,7 +39,7 @@ struct submodule_update_strategy { }; #define SUBMODULE_UPDATE_STRATEGY_INIT {SM_UPDATE_UNSPECIFIED, NULL} -int is_gitmodules_unmerged(const struct index_state *istate); +int is_gitmodules_unmerged(struct index_state *istate); int is_writing_gitmodules_ok(void); int is_staging_gitmodules_ok(struct index_state *istate); int update_path_in_gitmodules(const char *oldpath, const char *newpath); @@ -60,9 +60,9 @@ int is_submodule_active(struct repository *repo, const char *path); * Otherwise the return error code is the same as of resolve_gitdir_gently. */ int is_submodule_populated_gently(const char *path, int *return_error_code); -void die_in_unpopulated_submodule(const struct index_state *istate, +void die_in_unpopulated_submodule(struct index_state *istate, const char *prefix); -void die_path_inside_submodule(const struct index_state *istate, +void die_path_inside_submodule(struct index_state *istate, const struct pathspec *ps); enum submodule_update_type parse_submodule_update_type(const char *value); int parse_submodule_update_strategy(const char *value, diff --git a/tree.c b/tree.c index e76517f6b18..60f575440c8 100644 --- a/tree.c +++ b/tree.c @@ -170,6 +170,8 @@ int read_tree(struct repository *r, struct tree *tree, int stage, * to matter. */ + ensure_full_index(istate); + /* * See if we have cache entry at the stage. If so, * do it the original slow way, otherwise, append and then diff --git a/wt-status.c b/wt-status.c index 7074bbdd53c..5366d336938 100644 --- a/wt-status.c +++ b/wt-status.c @@ -509,6 +509,8 @@ static int unmerged_mask(struct index_state *istate, const char *path) int pos, mask; const struct cache_entry *ce; + ensure_full_index(istate); + pos = index_name_pos(istate, path, strlen(path)); if (0 <= pos) return 0; @@ -657,6 +659,8 @@ static void wt_status_collect_changes_initial(struct wt_status *s) struct index_state *istate = s->repo->index; int i; + ensure_full_index(istate); + for (i = 0; i < istate->cache_nr; i++) { struct string_list_item *it; struct wt_status_change_data *d; @@ -2295,6 +2299,9 @@ static void wt_porcelain_v2_print_unmerged_entry( */ memset(stages, 0, sizeof(stages)); sum = 0; + + ensure_full_index(istate); + pos = index_name_pos(istate, it->string, strlen(it->string)); assert(pos < 0); pos = -pos-1; From patchwork Mon Jan 25 17:42:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12043881 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95602C433DB for ; Mon, 25 Jan 2021 17:57:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5C65A22B3F for ; Mon, 25 Jan 2021 17:57:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731094AbhAYR5m (ORCPT ); Mon, 25 Jan 2021 12:57:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33060 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731240AbhAYR4A (ORCPT ); Mon, 25 Jan 2021 12:56:00 -0500 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5834EC06121F for ; Mon, 25 Jan 2021 09:42:31 -0800 (PST) Received: by mail-wr1-x42f.google.com with SMTP id v15so13808921wrx.4 for ; Mon, 25 Jan 2021 09:42:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=bgdoJjG5Sk7TG4mfT3ECXJqwtfoumNFy7HwY4DtEyH8=; b=K64x8U2x1//HycKNHwNliIp8D0duP+G4KVfyWghVPPNEmokKm/sfqT9bXWFiFB6mwO gwIPnAsXUS/rWPWSpRuaXym5eeLGG6MybyuQObgC2l9iNnmAT4UbQzCpYz+nwTI+FGHz u4UnxHgNEnwIGnuTS8iong2Y9oSvEE1oKtg4uU3p+liE5EI1LFA1uDgMtYuBE44fNpWO DcQ5tIdYuSMpRF9ZXhFvmCPZHj2EEE2dsqieUQ9XGlWhVT7QqRyYKQmmZIinTordZE29 VktIahM53VzT7waOcVFUI5VbaB97pfWgn3RBriRa5XRTg/rUs4uGrpJnQ9rQVRBqQ0GM e+Iw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=bgdoJjG5Sk7TG4mfT3ECXJqwtfoumNFy7HwY4DtEyH8=; b=UysDp83D1sjvcP8lkHktwc0Cx7ziYxwfI0oJqy29neycgWUMuQGaY9jbN/j1ezv0/W P9ARCJEKsADMJ2uvQ93Il5MQY5LsqHepJ7xSaA7PpMMC1NFIc6zd4Jh9GdqzRbKBtCSo sj8kx8DvrQWCJ5A0IoT4C5OdNA9AcI5taBD7fPaV/Yu6PKghPgud0CG3KKRnEk+unU7I IzWAn4HJXUW1zEmlq7go2UxjzH4py5wrhxyRDzr6jjDEZj+Zt2ajqJ2UE6Hp4QQdsrlj 50d3W3O89y6BkwpZG5lygLJ+L67fMYOQwU0AKz3G4LCHhYKn9GjndTGFQ9oSlRxB46UD CS0w== X-Gm-Message-State: AOAM530mzevyrJO+6C4gCkqrA4tJMHpOwGAkHgVqnb9k8oeScJI1XqUk 1QPE6UwsJrb08o+52Gfvvn8mwQLlJPg= X-Google-Smtp-Source: ABdhPJy4Eh/xcxJznGUlmYoifq2Xu7BtErHiUdvmrEUP1M4Z4MyN1NqRFGsICMTyEY9jAIPS3GAJmg== X-Received: by 2002:a05:6000:18ac:: with SMTP id b12mr2241656wri.77.1611596549946; Mon, 25 Jan 2021 09:42:29 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id y11sm2931197wrh.16.2021.01.25.09.42.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 09:42:29 -0800 (PST) Message-Id: <45cf57c9c40bebb7383b8aab19c82fc4e41d2cd3.1611596534.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 25 Jan 2021 17:42:02 +0000 Subject: [PATCH 16/27] unpack-trees: make sparse aware Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, peff@peff.net, jrnieder@gmail.com, sunshine@sunshineco.com, pclouds@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee As a first step to integrate 'git status' and 'git add' with the sparse index, we must start integrating unpack_trees() with sparse directory entries. These changes are currently impossible to trigger because unpack_trees() calls ensure_full_index() if command_requires_full_index is true. This is the case for all commands at the moment. As we expand more commands to be sparse-aware, we might find that more changes are required to unpack_trees(). The current changes will suffice for 'status' and 'add'. unpack_trees() calls the traverse_trees() API using unpack_callback() to decide if we should recurse into a subtree. We must add new abilities to skip a subtree if it corresponds to a sparse directory entry. It is important to be careful about the trailing directory separator that exists in the sparse directory entries but not in the subtree paths. Signed-off-by: Derrick Stolee --- dir.h | 2 +- preload-index.c | 2 ++ read-cache.c | 3 +++ unpack-trees.c | 24 ++++++++++++++++++++++-- 4 files changed, 28 insertions(+), 3 deletions(-) diff --git a/dir.h b/dir.h index facfae47402..300305ec335 100644 --- a/dir.h +++ b/dir.h @@ -503,7 +503,7 @@ static inline int ce_path_match(const struct index_state *istate, char *seen) { return match_pathspec(istate, pathspec, ce->name, ce_namelen(ce), 0, seen, - S_ISDIR(ce->ce_mode) || S_ISGITLINK(ce->ce_mode)); + S_ISSPARSEDIR(ce) || S_ISDIR(ce->ce_mode) || S_ISGITLINK(ce->ce_mode)); } static inline int dir_path_match(const struct index_state *istate, diff --git a/preload-index.c b/preload-index.c index ed6eaa47388..323fc8c5100 100644 --- a/preload-index.c +++ b/preload-index.c @@ -54,6 +54,8 @@ static void *preload_thread(void *_data) continue; if (S_ISGITLINK(ce->ce_mode)) continue; + if (S_ISSPARSEDIR(ce)) + continue; if (ce_uptodate(ce)) continue; if (ce_skip_worktree(ce)) diff --git a/read-cache.c b/read-cache.c index 65679d70d7c..ab0c2b86ec0 100644 --- a/read-cache.c +++ b/read-cache.c @@ -1572,6 +1572,9 @@ int refresh_index(struct index_state *istate, unsigned int flags, if (ignore_submodules && S_ISGITLINK(ce->ce_mode)) continue; + if (istate->sparse_index && S_ISSPARSEDIR(ce)) + continue; + if (pathspec && !ce_path_match(istate, ce, pathspec, seen)) filtered = 1; diff --git a/unpack-trees.c b/unpack-trees.c index b324eec2a5d..90644856a80 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -583,6 +583,13 @@ static void mark_ce_used(struct cache_entry *ce, struct unpack_trees_options *o) { ce->ce_flags |= CE_UNPACKED; + /* + * If this is a sparse directory, don't advance cache_bottom. + * That will be advanced later using the cache-tree data. + */ + if (S_ISSPARSEDIR(ce)) + return; + if (o->cache_bottom < o->src_index->cache_nr && o->src_index->cache[o->cache_bottom] == ce) { int bottom = o->cache_bottom; @@ -980,6 +987,9 @@ static int do_compare_entry(const struct cache_entry *ce, ce_len -= pathlen; ce_name = ce->name + pathlen; + /* remove directory separator if a sparse directory entry */ + if (S_ISSPARSEDIR(ce)) + ce_len--; return df_name_compare(ce_name, ce_len, S_IFREG, name, namelen, mode); } @@ -989,6 +999,10 @@ static int compare_entry(const struct cache_entry *ce, const struct traverse_inf if (cmp) return cmp; + /* If ce is a sparse directory, then allow equality here. */ + if (S_ISSPARSEDIR(ce)) + return 0; + /* * Even if the beginning compared identically, the ce should * compare as bigger than a directory leading up to it! @@ -1239,6 +1253,7 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str struct cache_entry *src[MAX_UNPACK_TREES + 1] = { NULL, }; struct unpack_trees_options *o = info->data; const struct name_entry *p = names; + unsigned recurse = 1; /* Find first entry with a real name (we could use "mask" too) */ while (!p->mode) @@ -1280,12 +1295,16 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str } } src[0] = ce; + + if (S_ISSPARSEDIR(ce)) + recurse = 0; } break; } } - if (unpack_nondirectories(n, mask, dirmask, src, names, info) < 0) + if (recurse && + unpack_nondirectories(n, mask, dirmask, src, names, info) < 0) return -1; if (o->merge && src[0]) { @@ -1315,7 +1334,8 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str } } - if (traverse_trees_recursive(n, dirmask, mask & ~dirmask, + if (recurse && + traverse_trees_recursive(n, dirmask, mask & ~dirmask, names, info) < 0) return -1; return mask; From patchwork Mon Jan 25 17:42:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12043897 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3369C433E0 for ; Mon, 25 Jan 2021 18:02:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8E9F722B3B for ; Mon, 25 Jan 2021 18:02:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731193AbhAYSBy (ORCPT ); Mon, 25 Jan 2021 13:01:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33238 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731341AbhAYRzx (ORCPT ); Mon, 25 Jan 2021 12:55:53 -0500 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 36E8DC061220 for ; Mon, 25 Jan 2021 09:42:32 -0800 (PST) Received: by mail-wr1-x436.google.com with SMTP id a9so13816140wrt.5 for ; Mon, 25 Jan 2021 09:42:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=lvFlNEHR515oVqS/y4SFBq68nw4FbVcN5LYMHfggzqs=; b=t/rHO+a9ovwtnIShrTMBZHXKTTI7sDL0zlAXXPqf79+dMCjOk4Jg8zmdclf24lwaJJ LU8pNmUbyMBrLQmNmHXMKuSVE3rGCx9de7TRW0VJxxZOlRRfuDfx8OreT+qrjGdzh+9g LSy4wksjw2u0drp6znF3VlO/golJoukP47Sdvmvg2eTth9bdWZTH7Rb/hsomNpfzXFGJ 83Kd0mINsYnJbf3JqZq3EuzB4QunQSw+cIvaExC2X6pGljJ8E/oy3jyEO7yhxIyvQAF4 pLvJhGWrW84j9Eidan++s1vEP9Qp7HrmcAN+lmL0BRNRfp6F0jvFSy83OCTE+2KgV2C5 hbcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=lvFlNEHR515oVqS/y4SFBq68nw4FbVcN5LYMHfggzqs=; b=Jk8AcjcoIK498FfEdXGMj8Me+uaNrfTGzdyJqftouoBA+e72GOtRtj56p1i+AH0tyS 9vDfEuL7SQ/BmCmeLmZF5Q6yJXRgwVnM+LBdJog74Ab9Z21kL8+IUS+5f1lVG1JLjQcT Kv5v3zyUkDgKPhMDb9L/sPcBtb2wS2QXIz5uNfewnduH+Q+YYOuyt3aoZ+qKC1n5CeBZ gQUiyx/OpTzSohVGyUI5kMXCRhWT0evphSogLwbQHyPstKdTs8Ztt26PUnfAMQakEVkE enqEkNZOhp96nJWS2EVIu6iY+F4qEZpovjz6XiKBPzLKGmCTFLACpK/SqovQ4OkcjKM+ yo7Q== X-Gm-Message-State: AOAM533FA7//f/P0UHVSPIfvHBn2jnGmrlpLj5d1QrX2HSX9KxknpKlr jC+gCc1tlQdIkzbJ57Sj56orvB1uEuQ= X-Google-Smtp-Source: ABdhPJxgAoD3CxxL7e2J+hipf+V6ATOigUwUZ5tv8v4RS5MCqhvJYBMShSR1M6V8kW1SVXUGr/Aa8g== X-Received: by 2002:adf:fb05:: with SMTP id c5mr2200392wrr.69.1611596550777; Mon, 25 Jan 2021 09:42:30 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id p18sm14203wmc.31.2021.01.25.09.42.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 09:42:30 -0800 (PST) Message-Id: <036653cac368c6c04b439f5352d70a5dcc3c5feb.1611596534.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 25 Jan 2021 17:42:03 +0000 Subject: [PATCH 17/27] dir.c: accept a directory as part of cone-mode patterns Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, peff@peff.net, jrnieder@gmail.com, sunshine@sunshineco.com, pclouds@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee When we have sparse directory entries in the index, we want to compare that directory against sparse-checkout patterns. Those pattern matching algorithms are built expecting a file path, not a directory path. This is especially important in the "cone mode" patterns which will match files that exist within the "parent directories" as well as the recursive directory matches. If path_matches_pattern_list() is given a directory, we can add a bogus filename ("-") to the directory and get the same results as before, assuming we are in cone mode. Since sparse index requires cone mode patterns, this is an acceptable assumption. Signed-off-by: Derrick Stolee --- dir.c | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/dir.c b/dir.c index ad6eb033cb1..c786fa98d0e 100644 --- a/dir.c +++ b/dir.c @@ -1384,6 +1384,11 @@ enum pattern_match_result path_matches_pattern_list( strbuf_addch(&parent_pathname, '/'); strbuf_add(&parent_pathname, pathname, pathlen); + /* Directory requests should be added as if they are a file */ + if (parent_pathname.len > 1 && + parent_pathname.buf[parent_pathname.len - 1] == '/') + strbuf_add(&parent_pathname, "-", 1); + if (hashmap_contains_path(&pl->recursive_hashmap, &parent_pathname)) { result = MATCHED_RECURSIVE; From patchwork Mon Jan 25 17:42:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12043887 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C0C9DC433DB for ; Mon, 25 Jan 2021 18:00:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8C9D722B3B for ; Mon, 25 Jan 2021 18:00:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731246AbhAYSAF (ORCPT ); Mon, 25 Jan 2021 13:00:05 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33366 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731258AbhAYR4Z (ORCPT ); Mon, 25 Jan 2021 12:56:25 -0500 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1B390C061221 for ; Mon, 25 Jan 2021 09:42:33 -0800 (PST) Received: by mail-wm1-x32e.google.com with SMTP id y187so74074wmd.3 for ; Mon, 25 Jan 2021 09:42:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=itRwluwYkYmkHZlSCa6+txpiwJvR8IeJetvlxdAepUA=; b=MLtVWJc0Az9ltfuc+XIk+6WHtMS8nyD/9MIOIIYVFVJqVXyY3vRlSWWLMsHB8rM9eK gRygMB461EsBh3rzHjFNW1koD9dJkujzE3lUiynvBfo/6SzPyu5FPhya6OXVLAHAKzcV C8nRtCQIwag9vbLVoFff9TuHRVscPeJoSO7q4RS3tUjAlQ8ZTGm2kh6Y5JgC1nqkGK5v /S04TEI0tmnnn6I1WHnWiOXnwQmHWE08J8ePeieYaDNtKN3OEIXiCWKxys/A14lH6HkZ yBMz+x5PjOPzrDXCaVFqWHkqmWmjGoSv5jDuQwlSvn38mThFE3Az/vgqzjBwBrBQOAF3 WP/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=itRwluwYkYmkHZlSCa6+txpiwJvR8IeJetvlxdAepUA=; b=gT74wLGdhMaF1Bd4/v+bSBVb/JVpcNIreTKY62X8F/4FGf982/jnuk43AqFfW6twjv WW3Po840HkdEmwCL+ffOxUcj4eEi0LvQXfvmzU7CavsyVSF8kmC7IabTAbbNfMLELqz6 vfnnfTpcmxi971BktQh9NHyagEoGjNXJ1XxB0BEgI99PxAHGF3jdlJEulU9JA+3Zivv+ 0WBoc15F8e1pXD6zXWwBxBO/S3HKHmsWDb26uQ1e/clbxEMMjUdap5CepwqeXGFDiyD0 CYyONe4ied7ZlyjH71xe9pqORNEnEKpNY3JsDP2kqtbNmowvysNmgbnfAVKlwcyfxEof 395g== X-Gm-Message-State: AOAM533KCDYmMsCc1LbBjMBPj7KFZLVcRWqXPyJ6QXm7qh9G1fs7o3G1 0PBczOijUP/5MwMEJhYJ9U2ptawGfhw= X-Google-Smtp-Source: ABdhPJySg+KBmMNUE5zsg+imFym9r3HNKj/ML8exLIseZfCjWGi12PV+QZPG0G10FUZw5Lt74l8BnQ== X-Received: by 2002:a05:600c:618:: with SMTP id o24mr1203300wmm.82.1611596551641; Mon, 25 Jan 2021 09:42:31 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id x81sm5265wmg.40.2021.01.25.09.42.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 09:42:31 -0800 (PST) Message-Id: <30e36850fd5f429e81cdb6fe89576acf7eb74571.1611596534.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 25 Jan 2021 17:42:04 +0000 Subject: [PATCH 18/27] status: use sparse-index throughout Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, peff@peff.net, jrnieder@gmail.com, sunshine@sunshineco.com, pclouds@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee By testing 'git -c core.fsmonitor= status -uno', we can check for the simplest index operations that can be made sparse-aware. The necessary implementation details are already integrated with sparse-checkout, so modify command_requires_full_index to be zero for cmd_status(). By running the debugger for 'git status -uno' after that change, we find two instances of ensure_full_index() that were added for extra safety, but can be removed without issue. In refresh_index(), we loop through the index entries. The refresh_cache_ent() method copies the sparse directories into the refreshed index without issue. The loop within run_diff_files() skips things that are in stage 0 and have skip-worktree enabled, so seems safe to disable ensure_full_index() here. While this change avoids calling ensure_full_index(), it actually slows 'git status' because we do not have the cache-tree extension to help us. Signed-off-by: Derrick Stolee --- builtin/commit.c | 3 +++ diff-lib.c | 2 -- read-cache.c | 1 - t/t1092-sparse-checkout-compatibility.sh | 10 +++++++--- 4 files changed, 10 insertions(+), 6 deletions(-) diff --git a/builtin/commit.c b/builtin/commit.c index 505fe60956d..543aa0caeae 100644 --- a/builtin/commit.c +++ b/builtin/commit.c @@ -1400,6 +1400,9 @@ int cmd_status(int argc, const char **argv, const char *prefix) if (argc == 2 && !strcmp(argv[1], "-h")) usage_with_options(builtin_status_usage, builtin_status_options); + prepare_repo_settings(the_repository); + the_repository->settings.command_requires_full_index = 0; + status_init_config(&s, git_status_config); argc = parse_options(argc, argv, prefix, builtin_status_options, diff --git a/diff-lib.c b/diff-lib.c index 3743e4463b4..b73cc1859a4 100644 --- a/diff-lib.c +++ b/diff-lib.c @@ -96,8 +96,6 @@ int run_diff_files(struct rev_info *revs, unsigned int option) uint64_t start = getnanotime(); struct index_state *istate = revs->diffopt.repo->index; - ensure_full_index(istate); - diff_set_mnemonic_prefix(&revs->diffopt, "i/", "w/"); refresh_fsmonitor(istate); diff --git a/read-cache.c b/read-cache.c index ab0c2b86ec0..78910d8f1b7 100644 --- a/read-cache.c +++ b/read-cache.c @@ -1561,7 +1561,6 @@ int refresh_index(struct index_state *istate, unsigned int flags, */ preload_index(istate, pathspec, 0); - ensure_full_index(istate); for (i = 0; i < istate->cache_nr; i++) { struct cache_entry *ce, *new_entry; int cache_errno = 0; diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index c6b7e8b8891..a3521cdc310 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -382,12 +382,16 @@ test_expect_success 'sparse-index is expanded and converted back' ' GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ git -C sparse-index -c core.fsmonitor="" reset --hard && test_region index convert_to_sparse trace2.txt && - test_region index ensure_full_index trace2.txt && + test_region index ensure_full_index trace2.txt +' - rm trace2.txt && +test_expect_success 'sparse-index is not expanded' ' + init_repos && + + rm -f trace2.txt && GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ git -C sparse-index -c core.fsmonitor="" status -uno && - test_region index ensure_full_index trace2.txt + test_region ! index ensure_full_index trace2.txt ' test_done From patchwork Mon Jan 25 17:42:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12043877 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1B01AC433E6 for ; Mon, 25 Jan 2021 17:57:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DD19922583 for ; Mon, 25 Jan 2021 17:57:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731308AbhAYR5M (ORCPT ); Mon, 25 Jan 2021 12:57:12 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33060 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731323AbhAYRz2 (ORCPT ); Mon, 25 Jan 2021 12:55:28 -0500 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CCC82C061222 for ; Mon, 25 Jan 2021 09:42:33 -0800 (PST) Received: by mail-wr1-x431.google.com with SMTP id m1so13119825wrq.12 for ; Mon, 25 Jan 2021 09:42:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=ekNEm1K7+IoPighnLWAgxDWfFsc/lZtfxyrUwGCkduY=; b=EGcLn7V9TD7CjVz4UY84AU/uTX2RVy/rYViuwcK/kZ25WaSAZEauMPlS6byhUW5eYW PwosV3T0UvgnF3LyRU2I5+80l4hLagSQhps0Y1vv2NrStH6xJOZGjuFcb88A1+0d+gkv jcOT5pfYEN0ZP/trH2XTV5Vt2Tm5dbcBTEMIweUibivLx45nZGWWq7b06DTDdQh8pHPa aR2SSPoe2Nf/H2y1wlaE4Y+dnuydmUFH1ZazCKcUlqil1PoKmyNpXECyR1SBl1TJuavJ fylN6SOcx9cIzU0qw0YkXimqkLL0airJTmlcfc4hRKxhpm4rNizhqvq1qxsyMm0lDOy1 Y4HA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=ekNEm1K7+IoPighnLWAgxDWfFsc/lZtfxyrUwGCkduY=; b=ITF9PDAASV5u9Zf5XqFPc4028jyd2cbJQ7Y7Tas6UN/zePAwYMSoZ5TcCZrsVHBvWK c9xOzUYwgoO1IYOEwkAp7RSS4+LE3dZrBM0CyGuaQyr0qkId+/4uoSN1q9tq8HGrKxHp K0KHTY8as4O/NS5yYJgo9y8N5SIbcEm8nkkDPoS76NoOog6a55snbGQbFn9ElqtSyO1l h+JfMzzCtsohAviTtHCL+e1Ks66NYOgbtDJVJoXQDmk9OHtvb6t82bzjcV7xhqdeFNJB AWGmiOGenQGL7Mi6GTDBFAbwHcAFCo6oUYA7Pk4IA8w7tCw+8ycVrx7eQlkboSsA1wkf 42qw== X-Gm-Message-State: AOAM533GyfxLsVCeriXuBZlViBiV+IIwLK8us0B7BsFplk+MkaxFupVJ zVYtIEb0eauKuApGTSZ9VDpM61onNc4= X-Google-Smtp-Source: ABdhPJyBi5zce1KT2VfxEo8JPIQda3rICeex9iuPO72SD/81qNfOr+eN+L8Nu4mkUcyBcOAPqVcaEw== X-Received: by 2002:adf:92a6:: with SMTP id 35mr2218017wrn.193.1611596552430; Mon, 25 Jan 2021 09:42:32 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n193sm51458wmb.0.2021.01.25.09.42.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 09:42:32 -0800 (PST) Message-Id: In-Reply-To: References: Date: Mon, 25 Jan 2021 17:42:05 +0000 Subject: [PATCH 19/27] status: skip sparse-checkout percentage with sparse-index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, peff@peff.net, jrnieder@gmail.com, sunshine@sunshineco.com, pclouds@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee 'git status' began reporting a percentage of populated paths when sparse-checkout is enabled in 051df3cf (wt-status: show sparse checkout status as well, 2020-07-18). This percentage is incorrect when the index has sparse directories. It would also be expensive to calculate as we would need to parse trees to count the total number of possible paths. Avoid the expensive computation by simplifying the output to only report that a sparse checkout exists, without the percentage. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 8 ++++++++ wt-status.c | 14 +++++++++++--- wt-status.h | 1 + 3 files changed, 20 insertions(+), 3 deletions(-) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index a3521cdc310..09650f0755c 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -190,6 +190,14 @@ test_expect_success 'status with options' ' test_all_match git status --porcelain=v2 -uno ' +test_expect_success 'status reports sparse-checkout' ' + init_repos && + git -C sparse-checkout status >full && + git -C sparse-index status >sparse && + test_i18ngrep "You are in a sparse checkout with " full && + test_i18ngrep "You are in a sparse checkout." sparse +' + test_expect_success 'add, commit, checkout' ' init_repos && diff --git a/wt-status.c b/wt-status.c index 5366d336938..46c9d71068e 100644 --- a/wt-status.c +++ b/wt-status.c @@ -1492,9 +1492,12 @@ static void show_sparse_checkout_in_use(struct wt_status *s, if (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_DISABLED) return; - status_printf_ln(s, color, - _("You are in a sparse checkout with %d%% of tracked files present."), - s->state.sparse_checkout_percentage); + if (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_SPARSE_INDEX) + status_printf_ln(s, color, _("You are in a sparse checkout.")); + else + status_printf_ln(s, color, + _("You are in a sparse checkout with %d%% of tracked files present."), + s->state.sparse_checkout_percentage); wt_longstatus_print_trailer(s); } @@ -1652,6 +1655,11 @@ static void wt_status_check_sparse_checkout(struct repository *r, return; } + if (r->index->sparse_index) { + state->sparse_checkout_percentage = SPARSE_CHECKOUT_SPARSE_INDEX; + return; + } + for (i = 0; i < r->index->cache_nr; i++) { struct cache_entry *ce = r->index->cache[i]; if (ce_skip_worktree(ce)) diff --git a/wt-status.h b/wt-status.h index 35b44c388ed..3cb0c200244 100644 --- a/wt-status.h +++ b/wt-status.h @@ -80,6 +80,7 @@ enum wt_status_format { #define HEAD_DETACHED_AT _("HEAD detached at ") #define HEAD_DETACHED_FROM _("HEAD detached from ") #define SPARSE_CHECKOUT_DISABLED -1 +#define SPARSE_CHECKOUT_SPARSE_INDEX -2 struct wt_status_state { int merge_in_progress; From patchwork Mon Jan 25 17:42:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12043891 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 65331C433DB for ; Mon, 25 Jan 2021 18:00:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2A48B22B3B for ; Mon, 25 Jan 2021 18:00:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731079AbhAYSAu (ORCPT ); Mon, 25 Jan 2021 13:00:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33336 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731119AbhAYR4V (ORCPT ); Mon, 25 Jan 2021 12:56:21 -0500 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A0E20C061223 for ; Mon, 25 Jan 2021 09:42:34 -0800 (PST) Received: by mail-wr1-x42d.google.com with SMTP id c12so13833314wrc.7 for ; Mon, 25 Jan 2021 09:42:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=uWvh1uq57/MSzGVd3Bb90wwkbavYqjng6CV9Jz8ctUk=; b=fRdRPaJoxPS2sVIKBGbYA3YLHTWQjPZfXovB4Ph1pnEKhr7TLy6z5ToqbvNnHKvi0O EVmuiBfq9oHzUNPdNunVnUvNCNycUp2cQgt6XbO9soxnwjAAqi7p7ICIFkiDBPYAe7w1 xZf/QYKX4oqtGNrAxaCbzsoFz/PKmUZ1PqRxYLmNVcGf9QQDT3BcPd/q7C2NM8dVElgQ IyaBU9uE5Ewkm/pl/zLaMZOBTX8RFFwe1mm4SwXFjqi64LPgWVkD2y5ewmwJmoJ9ilf9 4jL04GVT8Zc+Rw5CpINkBW30wg6mbHRs8fw8N81Iza6nA0asn4PFj75CW/BeYpgq8/xr o7JA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=uWvh1uq57/MSzGVd3Bb90wwkbavYqjng6CV9Jz8ctUk=; b=cvJLfC7QQhdxgrz3lla5a0aODz1nzYMZSvn5fq4k0HAVThCVtQoNh2r6yVI4TJkoga T54pplaylSaQkoFKyhpZDjg6Y4VYpbD4Q20nNHhUpwBICQLASmxJi1O8iNd5C1GMLJWn Lpm1ue715soE2nBS4OR9OY0MLQ3GP20pEcNU+iJgKn53xm+miKU7sRVwlV00TBm2ucUO 8RXC1bsYfc0C79CqzhAQItVFB+AgLDvjWPqGU6w+T6t6ErUz4azcKlQB/tkGoAxPrPGu 6x0lsYvJOS0jx28dPIguWcgvkcmhoaUNmS5wOL8mC317UruI3O7+YXMjiajF8sTrdSz+ YVeA== X-Gm-Message-State: AOAM531XI8odNdhWvm4ImOap5zczyFs6FSEp7N2O+bNNAdzEaItbEKef L3aUG0UG8VQ5WzKMJUc1X6TcFsWuE+g= X-Google-Smtp-Source: ABdhPJyfs0uCZGKkK1S6chYf4knT54wVcC3F5Ftm9cB6I3lf95CHTYEzToh1fgfvCJNaStwYgSP0Vw== X-Received: by 2002:adf:b257:: with SMTP id y23mr2175449wra.371.1611596553284; Mon, 25 Jan 2021 09:42:33 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id h1sm20454046wrr.73.2021.01.25.09.42.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 09:42:32 -0800 (PST) Message-Id: In-Reply-To: References: Date: Mon, 25 Jan 2021 17:42:06 +0000 Subject: [PATCH 20/27] sparse-index: expand_to_path() trivial implementation Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, peff@peff.net, jrnieder@gmail.com, sunshine@sunshineco.com, pclouds@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Before we check if a specific file or directory exists in the index, it would be good to see if a leading directory is a sparse-directory. If so, we will want to expand the index _just enough_ to be sure that the paths we are interested in are in the index. The actually-interesting implementation will follow in a later change. For now, simply call ensure_full_index() to expand every directory simultaneously. Calls like index_dir_exists(), adjust_dirname_case(), and index_file_exists() in name-hash.c can trust the name hash if the index was properly expanded for the requested names. These methods can transition from ensure_full_index() to expand_to_path(). Signed-off-by: Derrick Stolee --- name-hash.c | 10 ++++------ sparse-index.c | 7 +++++++ sparse-index.h | 12 ++++++++++++ 3 files changed, 23 insertions(+), 6 deletions(-) diff --git a/name-hash.c b/name-hash.c index 0f6d4fcca5a..641f6900a7c 100644 --- a/name-hash.c +++ b/name-hash.c @@ -8,6 +8,7 @@ #include "cache.h" #include "thread-utils.h" #include "trace2.h" +#include "sparse-index.h" struct dir_entry { struct hashmap_entry ent; @@ -679,9 +680,8 @@ int index_dir_exists(struct index_state *istate, const char *name, int namelen) { struct dir_entry *dir; - ensure_full_index(istate); - lazy_init_name_hash(istate); + expand_to_path(istate, name, namelen, 0); dir = find_dir_entry(istate, name, namelen); return dir && dir->nr; } @@ -691,9 +691,8 @@ void adjust_dirname_case(struct index_state *istate, char *name) const char *startPtr = name; const char *ptr = startPtr; - ensure_full_index( istate); - lazy_init_name_hash(istate); + expand_to_path(istate, name, strlen(name), 0); while (*ptr) { while (*ptr && *ptr != '/') ptr++; @@ -716,9 +715,8 @@ struct cache_entry *index_file_exists(struct index_state *istate, const char *na struct cache_entry *ce; unsigned int hash = memihash(name, namelen); - ensure_full_index(istate); - lazy_init_name_hash(istate); + expand_to_path(istate, name, namelen, icase); ce = hashmap_get_entry_from_hash(&istate->name_hash, hash, NULL, struct cache_entry, ent); diff --git a/sparse-index.c b/sparse-index.c index 3552f88fb03..dd1a06dfdd3 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -280,3 +280,10 @@ void ensure_full_index(struct index_state *istate) trace2_region_leave("index", "ensure_full_index", istate->repo); } + +void expand_to_path(struct index_state *istate, + const char *path, size_t pathlen, int icase) +{ + /* for now, do the obviously-correct, slow thing */ + ensure_full_index(istate); +} diff --git a/sparse-index.h b/sparse-index.h index ca936e95d11..549e4171f1a 100644 --- a/sparse-index.h +++ b/sparse-index.h @@ -4,6 +4,18 @@ struct index_state; void ensure_full_index(struct index_state *istate); int convert_to_sparse(struct index_state *istate); +/* + * Some places in the codebase expect to search for a specific path. + * This path might be outside of the sparse-checkout definition, in + * which case a sparse-index may not contain a path for that index. + * + * Given an index and a path, check to see if a leading directory for + * 'path' exists in the index as a sparse directory. In that case, + * expand that sparse directory to a full range of cache entries and + * populate the index accordingly. + */ +void expand_to_path(struct index_state *istate, + const char *path, size_t pathlen, int icase); struct repository; int set_sparse_index_config(struct repository *repo, int enable); From patchwork Mon Jan 25 17:42:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12043893 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 03818C433DB for ; Mon, 25 Jan 2021 18:01:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B6AD122B3B for ; Mon, 25 Jan 2021 18:01:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730729AbhAYSA7 (ORCPT ); Mon, 25 Jan 2021 13:00:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33334 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731009AbhAYR4V (ORCPT ); Mon, 25 Jan 2021 12:56:21 -0500 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 75008C061224 for ; Mon, 25 Jan 2021 09:42:35 -0800 (PST) Received: by mail-wr1-x42e.google.com with SMTP id a9so13816483wrt.5 for ; Mon, 25 Jan 2021 09:42:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=C4K9HzfTJGIydN06TzEfi5PvtGP8wNfAxqle/0fCqkI=; b=UBFFoIjDtBRDV14Nyk4Adz/C5bJW/8tglfat9j6nB6hi2v0zEbet1Dqevd2zGKpJFy mQfgD9s+fOsPrdacvWfNvH0FnKLHKomDfMZTFwLak/o3vYZ4Pq9bTrCaAHuMThfrTk3u MiweuHzjxQvUMH0VFUa/pu+gg+9KllL86v6NnJ8O3Y/3hkuTKga4WgbJ9fRLNZFYLde0 FE88DyElCSOBhMvIWdPyBFTjx6X7rB3/hoGDROtkyflXw6Q7xxCQzS3Ec0HmcVgTV2jS hTacxRD81Bbngbk27YewwtU8Y6bTgB8gIM3b1tJM4aKpt5+z5UuOaRfuR/lT529H9wdp lXbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=C4K9HzfTJGIydN06TzEfi5PvtGP8wNfAxqle/0fCqkI=; b=gIA+aacsi33LIkqtOOG1rAVcDzecRDkyKFv5dUkMzFOau4krxr4BdAuriR6JKRfGs5 QVEpRgdf6EWNaB41yh77HM27D8AVGvCmoGm9Zjx4vA5aKX8Q8XEIfafi3W2WhLNuovSz 6CDrrHQTd9Jkt1VoQ2aSlJRcBRxDaCFhDQrIiMtGXNIHB5L8lIIdKcwqIGDG6Sspq+tK zs8dTBUfHunGRGg7oZEBIWTX5LBgU4uexceXxEC3qtzk0iEdLQCDZ5ZIXASjpFGe2RbU j+cDlXymqv26SX2A2WUvDeUrKaeaatlUKCBfTM8W9iAYtKFwvRCNzOJ9P4lq2O/p/w19 vdGw== X-Gm-Message-State: AOAM531o1XpRYon/pc+RhJkJsGYAUtXPCSLX8QQvOUmb2TL2Ff8pjbOH 5Fm1+5FhyztwByNOSke8NdNZUctbDvI= X-Google-Smtp-Source: ABdhPJwrNi0btRo4Yr2S2CA//e3HY1VROY5T13TqPG3AZzY/BdXtc+QtvinC8KZ9Qt5yC/+F1H8WfA== X-Received: by 2002:a5d:538b:: with SMTP id d11mr2285784wrv.334.1611596554060; Mon, 25 Jan 2021 09:42:34 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m82sm17920wmf.29.2021.01.25.09.42.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 09:42:33 -0800 (PST) Message-Id: In-Reply-To: References: Date: Mon, 25 Jan 2021 17:42:07 +0000 Subject: [PATCH 21/27] sparse-index: expand_to_path no-op if path exists Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, peff@peff.net, jrnieder@gmail.com, sunshine@sunshineco.com, pclouds@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee We need to check the file hashmap first, then look to see if the directory signals a non-sparse directory entry. In such a case, we can rely on the contents of the sparse-index. We still use ensure_full_index() in the case that we hit a path that is within a sparse-directory entry. Signed-off-by: Derrick Stolee --- name-hash.c | 6 ++++++ sparse-index.c | 53 ++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 59 insertions(+) diff --git a/name-hash.c b/name-hash.c index 641f6900a7c..cb0f316f652 100644 --- a/name-hash.c +++ b/name-hash.c @@ -110,6 +110,12 @@ static void hash_index_entry(struct index_state *istate, struct cache_entry *ce) if (ce->ce_flags & CE_HASHED) return; ce->ce_flags |= CE_HASHED; + + if (ce->ce_mode == CE_MODE_SPARSE_DIRECTORY) { + add_dir_entry(istate, ce); + return; + } + hashmap_entry_init(&ce->ent, memihash(ce->name, ce_namelen(ce))); hashmap_add(&istate->name_hash, &ce->ent); diff --git a/sparse-index.c b/sparse-index.c index dd1a06dfdd3..bf8dce9a09b 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -281,9 +281,62 @@ void ensure_full_index(struct index_state *istate) trace2_region_leave("index", "ensure_full_index", istate->repo); } +static int in_expand_to_path = 0; + void expand_to_path(struct index_state *istate, const char *path, size_t pathlen, int icase) { + struct strbuf path_as_dir = STRBUF_INIT; + int pos; + + /* prevent extra recursion */ + if (in_expand_to_path) + return; + + if (!istate || !istate->sparse_index) + return; + + if (!istate->repo) + istate->repo = the_repository; + + in_expand_to_path = 1; + + /* + * We only need to actually expand a region if the + * following are both true: + * + * 1. 'path' is not already in the index. + * 2. Some parent directory of 'path' is a sparse directory. + */ + + strbuf_add(&path_as_dir, path, pathlen); + strbuf_addch(&path_as_dir, '/'); + + /* in_expand_to_path prevents infinite recursion here */ + if (index_file_exists(istate, path, pathlen, icase)) + goto cleanup; + + pos = index_name_pos(istate, path_as_dir.buf, path_as_dir.len); + + if (pos < 0) + pos = -pos - 1; + + /* + * Even if the path doesn't exist, if the value isn't exactly a + * sparse-directory entry, then there is no need to expand the + * index. + */ + if (istate->cache[pos]->ce_mode != CE_MODE_SPARSE_DIRECTORY) + goto cleanup; + + trace2_region_enter("index", "expand_to_path", istate->repo); + /* for now, do the obviously-correct, slow thing */ ensure_full_index(istate); + + trace2_region_leave("index", "expand_to_path", istate->repo); + +cleanup: + strbuf_release(&path_as_dir); + in_expand_to_path = 0; } From patchwork Mon Jan 25 17:42:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12043889 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C400DC433E0 for ; Mon, 25 Jan 2021 18:00:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8FA1822B3B for ; Mon, 25 Jan 2021 18:00:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731043AbhAYSAs (ORCPT ); Mon, 25 Jan 2021 13:00:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33332 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730597AbhAYR4V (ORCPT ); Mon, 25 Jan 2021 12:56:21 -0500 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 62E39C061225 for ; Mon, 25 Jan 2021 09:42:36 -0800 (PST) Received: by mail-wm1-x32a.google.com with SMTP id j18so48028wmi.3 for ; Mon, 25 Jan 2021 09:42:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=dbe4mlp/ovuriEiQ0skOC1I1wriOtqEa/5rVRQODJfY=; b=W8XuZKsGWAQC8/qjQyKj0WbQ6NqOrrFygulHqXOdChae1PqLccW/0Dy5OBdQAwGxx5 OujlKzb4lGhb0B6EGUrr0tBFRocC7kXxns4dPjL5GIsmSiikCkqhZkEhkgAk+4h8fIm0 Xuly0oE1ls/0TzaVS561WDZ09ztvnLZ//rBlBDLZoTMY3XPp2E6wE+mD2jahgL2VBKKx hCKRC6PpFBNud7GfTaMEpc1adWGkJAGi8aoFFlmnSUlB8BwmC+7FQnV1ostgaJGYwgBW Me8BkBt9EdxaqSXO1usIoB5izCFVsNeo7Wh8Cd2dS8NU5X33+7OjD2hBLf9Ylapf4GEg FkKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=dbe4mlp/ovuriEiQ0skOC1I1wriOtqEa/5rVRQODJfY=; b=t0InW7NJAek4kyq/wmTgFA7C6GwTyeVbqE6TytxIMzxsvs2J57DH3EXt/GGpRI79jH UVJgngybRxntLloBpJjpeha4rX9is81mNNmqyArWkWLDkuZF6O8BFlTROTGQdIq5LeaD 6ACmJMfoMXyGV59kWwIKfUw5OwIo5FwvgmVqFlmQS3syufUB3mB/SoyCxQ840znSX+Xz hto12hGFIpQ/U47urlhk5K+UIfdOy4ljTumKJav/AAqw3c0zpyB/2JhsUuJa5AoL/lhl Y///sjjQBSHGvbwPvv4vcfXhIWE5e4c0Y497zDqC+GioYmwbMnVo1S7o/CjOww35CLDs sFMg== X-Gm-Message-State: AOAM5309YuKR9vtpcXlDimfO4qNUFoiKbkxAAQNwJE8XZgck0Vgu/BkS I/qrwhCHS8vRsdAdOICJtBXj8pN06Tg= X-Google-Smtp-Source: ABdhPJx8NsYij8TgDvZs2PC00D7oTtyU594wPRozWotrU3yv2/QEAqZq79IvQbPwD5wfEcZdNsymJw== X-Received: by 2002:a1c:de09:: with SMTP id v9mr1255612wmg.0.1611596554946; Mon, 25 Jan 2021 09:42:34 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m12sm43399wmc.10.2021.01.25.09.42.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 09:42:34 -0800 (PST) Message-Id: In-Reply-To: References: Date: Mon, 25 Jan 2021 17:42:08 +0000 Subject: [PATCH 22/27] add: allow operating on a sparse-only index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, peff@peff.net, jrnieder@gmail.com, sunshine@sunshineco.com, pclouds@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Replace enough callers to ensure_full_index() to instead call expand_to_path() to reduce how often 'git add' expands a sparse index in memory (before writing a sparse index again). One non-obvious case is index_name_pos_also_unmerged() which is only hit on the Windows platform (in my tests). Use expand_to_path() instead of ensure_full_index(). Add a test to check that 'git add -A' and 'git add ' does not expand the index at all, as long as is not within a sparse directory. Signed-off-by: Derrick Stolee --- builtin/add.c | 3 +++ dir.c | 8 ++++---- read-cache.c | 10 +++++----- sparse-index.c | 18 ++++++++++++++---- t/t1092-sparse-checkout-compatibility.sh | 14 ++++++++++++++ 5 files changed, 40 insertions(+), 13 deletions(-) diff --git a/builtin/add.c b/builtin/add.c index a825887c503..b73f8d51de6 100644 --- a/builtin/add.c +++ b/builtin/add.c @@ -491,6 +491,9 @@ int cmd_add(int argc, const char **argv, const char *prefix) add_new_files = !take_worktree_changes && !refresh_only && !add_renormalize; require_pathspec = !(take_worktree_changes || (0 < addremove_explicit)); + prepare_repo_settings(the_repository); + the_repository->settings.command_requires_full_index = 0; + hold_locked_index(&lock_file, LOCK_DIE_ON_ERROR); /* diff --git a/dir.c b/dir.c index c786fa98d0e..21998c7c4b7 100644 --- a/dir.c +++ b/dir.c @@ -18,6 +18,7 @@ #include "ewah/ewok.h" #include "fsmonitor.h" #include "submodule-config.h" +#include "sparse-index.h" /* * Tells read_directory_recursive how a file or directory should be treated. @@ -899,9 +900,9 @@ static int read_skip_worktree_file_from_index(struct index_state *istate, { int pos, len; - ensure_full_index(istate); - len = strlen(path); + + expand_to_path(istate, path, len, 0); pos = index_name_pos(istate, path, len); if (pos < 0) return -1; @@ -1707,8 +1708,7 @@ static enum exist_status directory_exists_in_index(struct index_state *istate, if (ignore_case) return directory_exists_in_index_icase(istate, dirname, len); - ensure_full_index(istate); - + expand_to_path(istate, dirname, len, 0); pos = index_name_pos(istate, dirname, len); if (pos < 0) pos = -pos-1; diff --git a/read-cache.c b/read-cache.c index 78910d8f1b7..8c974829497 100644 --- a/read-cache.c +++ b/read-cache.c @@ -647,7 +647,7 @@ static int index_name_pos_also_unmerged(struct index_state *istate, int pos; struct cache_entry *ce; - ensure_full_index(istate); + expand_to_path(istate, path, namelen, 0); pos = index_name_pos(istate, path, namelen); if (pos >= 0) @@ -724,8 +724,6 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st, int hash_flags = HASH_WRITE_OBJECT; struct object_id oid; - ensure_full_index(istate); - if (flags & ADD_CACHE_RENORMALIZE) hash_flags |= HASH_RENORMALIZE; @@ -733,6 +731,8 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st, return error(_("%s: can only add regular files, symbolic links or git-directories"), path); namelen = strlen(path); + expand_to_path(istate, path, namelen, 0); + if (S_ISDIR(st_mode)) { if (resolve_gitlink_ref(path, "HEAD", &oid) < 0) return error(_("'%s' does not have a commit checked out"), path); @@ -1104,7 +1104,7 @@ static int has_dir_name(struct index_state *istate, size_t len_eq_last; int cmp_last = 0; - ensure_full_index(istate); + expand_to_path(istate, ce->name, ce->ce_namelen, 0); /* * We are frequently called during an iteration on a sorted @@ -1349,7 +1349,7 @@ int add_index_entry(struct index_state *istate, struct cache_entry *ce, int opti { int pos; - ensure_full_index(istate); + expand_to_path(istate, ce->name, ce->ce_namelen, 0); if (option & ADD_CACHE_JUST_APPEND) pos = istate->cache_nr; diff --git a/sparse-index.c b/sparse-index.c index bf8dce9a09b..a201f3b905c 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -286,6 +286,7 @@ static int in_expand_to_path = 0; void expand_to_path(struct index_state *istate, const char *path, size_t pathlen, int icase) { + struct cache_entry *ce = NULL; struct strbuf path_as_dir = STRBUF_INIT; int pos; @@ -320,13 +321,22 @@ void expand_to_path(struct index_state *istate, if (pos < 0) pos = -pos - 1; + if (pos < istate->cache_nr) + ce = istate->cache[pos]; /* - * Even if the path doesn't exist, if the value isn't exactly a - * sparse-directory entry, then there is no need to expand the - * index. + * If we didn't land on a sparse directory, then there is + * nothing to expand. */ - if (istate->cache[pos]->ce_mode != CE_MODE_SPARSE_DIRECTORY) + if (ce && !S_ISSPARSEDIR(ce)) + goto cleanup; + /* + * If that sparse directory is not a prefix of the path we + * are looking for, then we don't need to expand. + */ + if (ce && + (ce->ce_namelen >= path_as_dir.len || + strncmp(ce->name, path_as_dir.buf, ce->ce_namelen))) goto cleanup; trace2_region_enter("index", "expand_to_path", istate->repo); diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 09650f0755c..ae594ab880c 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -390,6 +390,20 @@ test_expect_success 'sparse-index is expanded and converted back' ' GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ git -C sparse-index -c core.fsmonitor="" reset --hard && test_region index convert_to_sparse trace2.txt && + test_region index ensure_full_index trace2.txt && + + rm trace2.txt && + echo >>sparse-index/README.md && + GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ + git -C sparse-index -c core.fsmonitor="" add -A && + test_region index convert_to_sparse trace2.txt && + test_region index ensure_full_index trace2.txt && + + rm trace2.txt && + echo >>sparse-index/extra.txt && + GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ + git -C sparse-index -c core.fsmonitor="" add extra.txt && + test_region index convert_to_sparse trace2.txt && test_region index ensure_full_index trace2.txt ' From patchwork Mon Jan 25 17:42:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12043933 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91B74C433E0 for ; Mon, 25 Jan 2021 18:14:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4E9AD22583 for ; Mon, 25 Jan 2021 18:14:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731308AbhAYSNv (ORCPT ); Mon, 25 Jan 2021 13:13:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33202 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730723AbhAYRzr (ORCPT ); Mon, 25 Jan 2021 12:55:47 -0500 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 290ABC0611C0 for ; Mon, 25 Jan 2021 09:42:37 -0800 (PST) Received: by mail-wr1-x435.google.com with SMTP id 7so13823457wrz.0 for ; Mon, 25 Jan 2021 09:42:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=sGydUQb2UhpH1D9fSdUXsEtwvQkgf63UmO/XmLlG58Q=; b=nrVxuFXv1jSNqCWL3ojs2GMb6uP4Z9C0gMgF+ymI6lFXuYtSV3zI8CWfyj7/OmPUAW yg94N80Wh1Li08uskACsVFdryLhkqpkcY6W4zoib3y01MeIsfRqQtIe0WxBZN06hYuD3 2Hwgh+agnBBjcULtsFPF7LOv7uTm3VRfxFaVuCdce+sN6y+cVlZTd8SfmaDsfqVmIiuW TeAYUyiyBpryl///somROsxAd928nL6ihKpMUp89iAXZ1wASvfr3gFbERldZqQ+QbwFA PhgDbqIFPjMZWU3OsIk/gwlc94yAchnBbJLaiscX1jEfQMPvrqGo0TWfD/jkAStw2r89 dkvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=sGydUQb2UhpH1D9fSdUXsEtwvQkgf63UmO/XmLlG58Q=; b=adEa3THeIrVvXu6p2LtuEn+WUhpukSecJff8IjFGzOO45GTFzY9pgYhuEUr4XlwcM6 QDFil0iaM2kmvYf4WR0eKOfBLnGfn11lUvrG26/AyTKZmTzIrzSC6Hzjr/giydr7HWcc Md4LzXxmCUb0aKsbsg215DbTvaV+ccSmY6WMQISLM1L5s7YgxXQUk0zEHHdkuDpLkmw/ 4bRp+chMz+0AzQrbEAWVLBZqP+Wh/qcPlGi/cR3ZujiJzGrL9tZhlMPmplWAajZVYICm azl9YhLP1zDcUsencM9jI4e0RoxnFCIJ0xFd02W2ujWbg9IpEMEVvo2mZM3xn5229dl2 zYMg== X-Gm-Message-State: AOAM532oJow6uXQ1FYZDT7lnZtEFqeicaZW+TfxocTQ3hPGT4I1SCib2 JBL82lv6PHQWAhRp16j8b3Ie/zoZf2M= X-Google-Smtp-Source: ABdhPJzPSwyY5jlmpLzEuXSGY+17z/CipZdb70VrJDkSizsFtNKoRtW78PETbhaXVjocWHUOk3hjAQ== X-Received: by 2002:a5d:4c4f:: with SMTP id n15mr2290669wrt.124.1611596555807; Mon, 25 Jan 2021 09:42:35 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id c5sm7999483wrn.77.2021.01.25.09.42.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 09:42:35 -0800 (PST) Message-Id: <6b9f935b87aec1806f48294574924fc26c4abbc7.1611596534.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 25 Jan 2021 17:42:09 +0000 Subject: [PATCH 23/27] submodule: die_path_inside_submodule is sparse aware Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, peff@peff.net, jrnieder@gmail.com, sunshine@sunshineco.com, pclouds@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Since we already do not collapse a sparse directory if it contains a submodule, we don't need to expand to a full index in die_path_inside_submodule(). A simple scan of the index entries is sufficient. Signed-off-by: Derrick Stolee --- submodule.c | 2 -- t/t1092-sparse-checkout-compatibility.sh | 24 +++++++++++------------- 2 files changed, 11 insertions(+), 15 deletions(-) diff --git a/submodule.c b/submodule.c index f80cfddbd52..487d083e4ef 100644 --- a/submodule.c +++ b/submodule.c @@ -346,8 +346,6 @@ void die_path_inside_submodule(struct index_state *istate, { int i, j; - ensure_full_index(istate); - for (i = 0; i < istate->cache_nr; i++) { struct cache_entry *ce = istate->cache[i]; int ce_len = ce_namelen(ce); diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index ae594ab880c..2e8efe6ab37 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -390,29 +390,27 @@ test_expect_success 'sparse-index is expanded and converted back' ' GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ git -C sparse-index -c core.fsmonitor="" reset --hard && test_region index convert_to_sparse trace2.txt && - test_region index ensure_full_index trace2.txt && + test_region index ensure_full_index trace2.txt +' + +test_expect_success 'sparse-index is not expanded' ' + init_repos && + + rm -f trace2.txt && + GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ + git -C sparse-index -c core.fsmonitor="" status -uno && + test_region ! index ensure_full_index trace2.txt && rm trace2.txt && echo >>sparse-index/README.md && GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ git -C sparse-index -c core.fsmonitor="" add -A && - test_region index convert_to_sparse trace2.txt && - test_region index ensure_full_index trace2.txt && + test_region ! index ensure_full_index trace2.txt && rm trace2.txt && echo >>sparse-index/extra.txt && GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ git -C sparse-index -c core.fsmonitor="" add extra.txt && - test_region index convert_to_sparse trace2.txt && - test_region index ensure_full_index trace2.txt -' - -test_expect_success 'sparse-index is not expanded' ' - init_repos && - - rm -f trace2.txt && - GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ - git -C sparse-index -c core.fsmonitor="" status -uno && test_region ! index ensure_full_index trace2.txt ' From patchwork Mon Jan 25 17:42:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12043905 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 77288C433E0 for ; Mon, 25 Jan 2021 18:05:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 285C322B3F for ; Mon, 25 Jan 2021 18:05:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731335AbhAYSEc (ORCPT ); Mon, 25 Jan 2021 13:04:32 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33230 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731333AbhAYRzw (ORCPT ); Mon, 25 Jan 2021 12:55:52 -0500 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0EE57C0611C1 for ; Mon, 25 Jan 2021 09:42:38 -0800 (PST) Received: by mail-wr1-x42c.google.com with SMTP id v15so13809562wrx.4 for ; Mon, 25 Jan 2021 09:42:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=jQ96vWOniBW8PZNv61/9806PkJgXsRsQRxZXXKQWvDQ=; b=dX+JjGIRXPfcQ3y2rS7eHH341hd6cTOLnGB++IdByOE+ca1aZLOj6uA3BGnBaAc9nr pFWBVmft7m4onJDP7qAUwqJuU3U0nhRtqo8L22GGnPdralrnw5sA1TZ9wmiCImTxpLSZ ++Fdv5SgD8FkaW+e7HjTN/4UTagGqJG/yVQ7cTZ4vAkYrxgdom3i1sNrfVj341ZVdl5n D6nyjyxBwAnc6IiwWsFwwV/AUtWmJD7x8qESH5z6xwEl0QuW+caSMU26KeDHzERRAb60 42JMXOA6WDw6eDA2cKpfh9ijFre5VNJxJYkBkht5+YzEJ39CW8fxth/kQYaVU2wnxcgp rtdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=jQ96vWOniBW8PZNv61/9806PkJgXsRsQRxZXXKQWvDQ=; b=ZpUVWiH7HdqDOkNW//QRu2RtiNs5/p6RKPUFfFpiKpObADJfJv+JMgTNxHD1ra+Doi KmapK4eEvWUq/cRIWeO9afRzutlPBMYyvIV+akr+PPzQuRf0vouEc2WKDBGMx2ee7uCE XOCmHVi1YSu+vkXsVTTRODpHjnXGg9t/UKZqFez9yVKDo2j2RKBXHyiP9m8SeiA1tTuK L0XwRUmNDQyEK1SACVjbE+/ZwIKCG0iAmR6pAZCNW3HCipBXMG4ozNPLKc82KEfAoBWq O+N8ow0j8iSahaD4thfPqd6zEPI7Jo+xFIPV40087FDdkUxAkTXQXr7eY5icir9hkWrZ WgAg== X-Gm-Message-State: AOAM533T63HWuhLXUzgS1xDypyCMhEJt1POcOpV9Q7IDK1oX6VCOhs01 jBgGlOdYLOR2GQHav+ZNZcUzZ/sVr/U= X-Google-Smtp-Source: ABdhPJyvAZqzlUUKvBUQZZ6/c743L6W6tMO6Qc+rhwilxFY4tYaX1eaBdOh093AanGbL2v2ulvGezg== X-Received: by 2002:adf:cd83:: with SMTP id q3mr2208157wrj.225.1611596556697; Mon, 25 Jan 2021 09:42:36 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id y63sm29684wmd.21.2021.01.25.09.42.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 09:42:36 -0800 (PST) Message-Id: <926a2e12cf7534339e1f9bedff50d97bf251ffa2.1611596534.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 25 Jan 2021 17:42:10 +0000 Subject: [PATCH 24/27] dir: use expand_to_path in add_patterns() Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, peff@peff.net, jrnieder@gmail.com, sunshine@sunshineco.com, pclouds@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The add_patterns() method has a way to extract a pattern file from the index. If this pattern file is sparse and within a sparse directory entry, then we need to expand the index before looking for that entry as a file path. For now, convert ensure_full_index() into expand_to_path() to only expand this way when absolutely necessary. Signed-off-by: Derrick Stolee --- dir.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dir.c b/dir.c index 21998c7c4b7..7df8d3b1da0 100644 --- a/dir.c +++ b/dir.c @@ -1093,7 +1093,7 @@ static int add_patterns(const char *fname, const char *base, int baselen, int pos; if (istate) - ensure_full_index(istate); + expand_to_path(istate, fname, strlen(fname), 0); if (oid_stat->valid && !match_stat_data_racy(istate, &oid_stat->stat, &st)) From patchwork Mon Jan 25 17:42:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12043899 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1982DC433E0 for ; Mon, 25 Jan 2021 18:03:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E06ED22242 for ; Mon, 25 Jan 2021 18:03:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731197AbhAYSDC (ORCPT ); Mon, 25 Jan 2021 13:03:02 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33236 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731340AbhAYRzx (ORCPT ); Mon, 25 Jan 2021 12:55:53 -0500 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D96C2C0611C2 for ; Mon, 25 Jan 2021 09:42:38 -0800 (PST) Received: by mail-wm1-x32d.google.com with SMTP id c127so49696wmf.5 for ; Mon, 25 Jan 2021 09:42:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=cMsFnS14xgniOaBIK70q63JnbTK9euVHtZQ1xBvUskA=; b=hxP5GXBMHTaji4s+R2ZOKnHgvnknWNbGC9UDV6HAG2/HCxvAfaVgmbhUGo9g6pI2Mr uuGYNSk63Z1perkOuuu0WTTW60UgTBg5KmgMXg89U9l5qHY3nVtDVVykElakgPzQY6mF s9sPmBDpkRfA0/8EeSXiQzHB7H/P5g7wM0IJeyZMjxGN6a53naGxX4LDComSCryu5zOl A+SWvaBuQcnZlHG46zERkLlqDBRSKCgxoL5P81NbNCpUhMGkjyBMgeL8IyyOOUuVEIun +frSfVN5X4wTKCD+W+NMKKxdQNFE5O6PdwyO1ZAyPTto0eKeeX6DqMHOXonn82mqX3Mg Ki5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=cMsFnS14xgniOaBIK70q63JnbTK9euVHtZQ1xBvUskA=; b=gu5kvOIdtWzsH075a2oZMZHAjvgQ3bzn2hl4dZeiAfXc/NjEV56FdnTkptkUotdN1/ 7X3Mk29c5e3XB0b3KPwvh82bSRh0A3YmZUK4cp9UMHwLpqytdrwXO2O/KmJWsy0J1TgA I226tIoz6kL8RhdZwZzwZMBs43Uxr7FvPShercpeXtK0awD8+5kj+dnO7jRKGxFp4+KS Fc+mcAZ5G6oisFJTqXz8Nf5BfrvfhiwOW0ry9aZUWcmYd+Et7r6BlggraXdWEu2HCSzL SdsjTEODyMx5J5ZrAV8XDnnp8yGZyvPHn91/pKuIIWc9gJG3zF31LpW5dha1hk+dX/aP z9DA== X-Gm-Message-State: AOAM530oH+rCeA4BovnYtKSS7LuofFGBaG1OMgpFr6It4f4jgxYEQqlc oLCb6xxyvPEWx+lC3K+PmxyAVKQtzek= X-Google-Smtp-Source: ABdhPJxWw9pBj1gIM77iwYO7XdQvvPnyUM8VCleLZlgdTLJeiMIT/7n1OuyiZINP4XaBO4y5Ve6IOA== X-Received: by 2002:a1c:7413:: with SMTP id p19mr1148583wmc.39.1611596557433; Mon, 25 Jan 2021 09:42:37 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id b12sm16364178wrr.35.2021.01.25.09.42.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 09:42:37 -0800 (PST) Message-Id: In-Reply-To: References: Date: Mon, 25 Jan 2021 17:42:11 +0000 Subject: [PATCH 25/27] fsmonitor: disable if index is sparse Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, peff@peff.net, jrnieder@gmail.com, sunshine@sunshineco.com, pclouds@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The FS Monitor feature uses a bitmap over the index entries. This currently interacts poorly with a sparse index. We will revisit this interaction in the future, but for now protect the index by refusing to use the FS Monitor extension at all if the index is sparse. Signed-off-by: Derrick Stolee --- fsmonitor.c | 21 ++++++++++++++------- read-cache.c | 3 ++- t/t1092-sparse-checkout-compatibility.sh | 8 ++++---- 3 files changed, 20 insertions(+), 12 deletions(-) diff --git a/fsmonitor.c b/fsmonitor.c index 7b8cd3975b9..99b26576baa 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -58,6 +58,9 @@ int read_fsmonitor_extension(struct index_state *istate, const void *data, uint64_t timestamp; struct strbuf last_update = STRBUF_INIT; + if (istate->sparse_index) + return 0; + if (sz < sizeof(uint32_t) + 1 + sizeof(uint32_t)) return error("corrupt fsmonitor extension (too short)"); @@ -98,7 +101,8 @@ void fill_fsmonitor_bitmap(struct index_state *istate) { unsigned int i, skipped = 0; - ensure_full_index(istate); + if (istate->sparse_index) + return; istate->fsmonitor_dirty = ewah_new(); for (i = 0; i < istate->cache_nr; i++) { @@ -161,11 +165,7 @@ static int query_fsmonitor(int version, const char *last_update, struct strbuf * static void fsmonitor_refresh_callback(struct index_state *istate, const char *name) { - int pos; - - ensure_full_index(istate); - - pos = index_name_pos(istate, name, strlen(name)); + int pos = index_name_pos(istate, name, strlen(name)); if (pos >= 0) { struct cache_entry *ce = istate->cache[pos]; @@ -190,7 +190,8 @@ void refresh_fsmonitor(struct index_state *istate) char *buf; unsigned int i; - if (!core_fsmonitor || istate->fsmonitor_has_run_once) + if (!core_fsmonitor || istate->fsmonitor_has_run_once || + istate->sparse_index) return; hook_version = fsmonitor_hook_version(); @@ -300,6 +301,9 @@ void add_fsmonitor(struct index_state *istate) unsigned int i; struct strbuf last_update = STRBUF_INIT; + if (istate->sparse_index) + return; + if (!istate->fsmonitor_last_update) { trace_printf_key(&trace_fsmonitor, "add fsmonitor"); istate->cache_changed |= FSMONITOR_CHANGED; @@ -335,6 +339,9 @@ void tweak_fsmonitor(struct index_state *istate) unsigned int i; int fsmonitor_enabled = git_config_get_fsmonitor(); + if (istate->sparse_index) + fsmonitor_enabled = 0; + if (istate->fsmonitor_dirty) { if (fsmonitor_enabled) { ensure_full_index(istate); diff --git a/read-cache.c b/read-cache.c index 8c974829497..96d9b95128a 100644 --- a/read-cache.c +++ b/read-cache.c @@ -3017,7 +3017,8 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, if (err) return -1; } - if (!strip_extensions && istate->fsmonitor_last_update) { + if (!strip_extensions && istate->fsmonitor_last_update && + !istate->sparse_index) { struct strbuf sb = STRBUF_INIT; write_fsmonitor_extension(&sb, istate); diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 2e8efe6ab37..1cdf33a4025 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -388,7 +388,7 @@ test_expect_success 'sparse-index is expanded and converted back' ' init_repos && GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ - git -C sparse-index -c core.fsmonitor="" reset --hard && + git -C sparse-index reset --hard && test_region index convert_to_sparse trace2.txt && test_region index ensure_full_index trace2.txt ' @@ -398,19 +398,19 @@ test_expect_success 'sparse-index is not expanded' ' rm -f trace2.txt && GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ - git -C sparse-index -c core.fsmonitor="" status -uno && + git -C sparse-index status -uno && test_region ! index ensure_full_index trace2.txt && rm trace2.txt && echo >>sparse-index/README.md && GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ - git -C sparse-index -c core.fsmonitor="" add -A && + git -C sparse-index add -A && test_region ! index ensure_full_index trace2.txt && rm trace2.txt && echo >>sparse-index/extra.txt && GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ - git -C sparse-index -c core.fsmonitor="" add extra.txt && + git -C sparse-index add extra.txt && test_region ! index ensure_full_index trace2.txt ' From patchwork Mon Jan 25 17:42:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12043873 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 77C88C433DB for ; Mon, 25 Jan 2021 17:56:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 40C3722B3B for ; Mon, 25 Jan 2021 17:56:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731264AbhAYR4b (ORCPT ); Mon, 25 Jan 2021 12:56:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33052 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731325AbhAYRz2 (ORCPT ); Mon, 25 Jan 2021 12:55:28 -0500 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C6DF2C0611C3 for ; Mon, 25 Jan 2021 09:42:39 -0800 (PST) Received: by mail-wm1-x32f.google.com with SMTP id c127so49766wmf.5 for ; Mon, 25 Jan 2021 09:42:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=D9S1QwP7ENCfvG4pg2hrsdrzIOeCzOYkZSdS4yAEx3o=; b=N3uI11TZPM4pDDTjogH96xXG6shSh+Yv1YFvuZPq3AxTtfHdskHGUPWSIgybWkjnP+ veKQu28Ubkcy6nZpmqOhQm24WXlYH3Dvxu13J4hGqPhuh/duZqcVgeYGfywbFZw2Yk5+ +BxGGwn1s7qBqZvgb9unkBfrhPLEfBexWg8hpNZU075sR7FRenCgc/CKpvl6BUBs9spz IFW8cAH2PZnEobznHW25PiB/geZ4nbORWYsm7oB9m7Z75T8WBpRyCQrQFfulUuMS3hza rpuThE9gtrC3J4Uqj29KqWizMPJy6t/RhNiS5v4AGb530c6UttTreo4iRlo4xciccj1P gVQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=D9S1QwP7ENCfvG4pg2hrsdrzIOeCzOYkZSdS4yAEx3o=; b=C3J/iGetGR7Cge7JIzvLz+23XxdQjYeeCWoHYiYUq2UNI1+HOtTFv6XC9K9acOjjxj 8J04A0pDYFfry87V8zSDBtnYMTeERzzcsofJXKFYd8kEQoGNnFoSKPjDhuFMSFCB+lLw XV2HUXDHeckcveTnnLe66MHLYbzDtD6Z9SrgLCAKrVJJsPpaebpiJmRMjGsnqHk39tOE WrWrABelcQ0ZzbNxQybYTyppUTBWSf5SfS4Dm1WchHZqfc2Qd+jPEMe/Rdssm/eiM5zv uE++71S74dnF2UeNxJepS7veKDGw6/uyFOlIhpdzSYC6hl27qFTgLjth+T99udy1niZM Nsgw== X-Gm-Message-State: AOAM532joneUGpBMZbrgouq2lJ4GwvQD8DfOagGdSIqyiqAxft56AmdT kKIEQrzzt3MxqKPw6JGXn9RH//zFQA8= X-Google-Smtp-Source: ABdhPJy1mi2hDVsHRY9V1iy/03aryYYE7HP2gmU+CdO5mnwjOJnCOPLIPkWsy0NnNtRaL4p2MsWqOA== X-Received: by 2002:a1c:ed0b:: with SMTP id l11mr1153031wmh.47.1611596558430; Mon, 25 Jan 2021 09:42:38 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id h16sm23654785wrq.29.2021.01.25.09.42.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 09:42:37 -0800 (PST) Message-Id: <5f53b08225771adc0be12c39e7be169d8620f146.1611596534.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 25 Jan 2021 17:42:12 +0000 Subject: [PATCH 26/27] pathspec: stop calling ensure_full_index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, peff@peff.net, jrnieder@gmail.com, sunshine@sunshineco.com, pclouds@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The add_pathspec_matches_against_index() focuses on matching a pathspec to file entries in the index. It is possible that this already works correctly for its only use: checking if untracked files exist in the index. It is likely that this causes a behavior issue when adding a directory that exists at HEAD but is outside the sparse cone. I'm marking this as a place to pursue with future tests. Signed-off-by: Derrick Stolee --- pathspec.c | 1 - 1 file changed, 1 deletion(-) diff --git a/pathspec.c b/pathspec.c index 9b105855483..61dc771aa02 100644 --- a/pathspec.c +++ b/pathspec.c @@ -36,7 +36,6 @@ void add_pathspec_matches_against_index(const struct pathspec *pathspec, num_unmatched++; if (!num_unmatched) return; - ensure_full_index(istate); for (i = 0; i < istate->cache_nr; i++) { const struct cache_entry *ce = istate->cache[i]; ce_path_match(istate, ce, pathspec, seen); From patchwork Mon Jan 25 17:42:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12043885 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE7E4C433E0 for ; Mon, 25 Jan 2021 17:59:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7DEA222583 for ; Mon, 25 Jan 2021 17:59:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731219AbhAYR7X (ORCPT ); Mon, 25 Jan 2021 12:59:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33202 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731263AbhAYR4b (ORCPT ); Mon, 25 Jan 2021 12:56:31 -0500 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B9FB2C0611BC for ; Mon, 25 Jan 2021 09:42:40 -0800 (PST) Received: by mail-wm1-x334.google.com with SMTP id i9so98215wmq.1 for ; Mon, 25 Jan 2021 09:42:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:mime-version :content-transfer-encoding:fcc:to:cc; bh=fa2VGXxgeW6ECn5Gef1CpeYWB+LFL3W38AsWWJYtajw=; b=X3GoqgaJL7DdFAA+d2ZvnfXKfUaRCsdfxAUom4QLLtGLu9LY5BvIC5jmA4tWFV/pmE 5JbEwmMFA+uhGsaPCap4+MxSVJWPtmXQCYrykSj91230MRChD2n2J4FkFS/LrkSlYBpd smmvWDtpWN7FbiRM2Sce+SSzCqmYz7ixxvAF+8zHGXk/4s0ajDDXdlmk1NmDDOhdG+O5 bGfATGrys1VhT+qNNyDX662nNcY6Pwdiw5QUfSAq0dzShhh7U3zuS3RMPrMq/03bDVa5 OELQkqd3XyAE7W+5YXFCSofVCkIGa9P/C8i15Pmxnhkk1vqTyIVgBz2wO2YbLzFfdQii y4Jw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:mime-version:content-transfer-encoding:fcc:to:cc; bh=fa2VGXxgeW6ECn5Gef1CpeYWB+LFL3W38AsWWJYtajw=; b=a+oWG34cTbFCPTvYjxd/ndqDLaJJv3SblaKcgNlZUBgnp/2gDM3LK7OWWPtBoywDAw KF4QsKJnmQjVVzZkdPUTpZh/3UzZzvMu8dawzbLjqS2722kHmrRC+spInWes3cNMQBpe W4VWUcZ7s9MsWVbZAo7HHJX23qrT7fK/emR0rWRvzcDOj1nwG2OzYJ2NxqXj26OPxNvJ lHBum2xLL0QxacAn1LlLdpoHiw0CZnS9b16yraBKzVby5YlArJo5IxhnnhwyY/hoJxnO QbVlfLE2SzU/19x66j7oWTaVkzPp55k3qEa5BndyFC2p2puP0ifyM9SGQXSAjW67lggk S8uA== X-Gm-Message-State: AOAM530O9vI3240onYM2syHGjmTtZttOsqOfBIkf6P3oeE6/Qgt4e17H zPldPlYLJ+eae3CNyWyjEECakCHMeFg= X-Google-Smtp-Source: ABdhPJy0jX299CnDb5QM+itdl49FcmOU54UED7ZXM8+oT6PK8wTer3Bns6uPBQxshWSPO2DWcKH/0w== X-Received: by 2002:a7b:ca4d:: with SMTP id m13mr1250119wml.28.1611596559190; Mon, 25 Jan 2021 09:42:39 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id q6sm5270940wrw.43.2021.01.25.09.42.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 25 Jan 2021 09:42:38 -0800 (PST) Message-Id: <05e7548b780da6b2bf2342d91d8757568df0a6b8.1611596534.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 25 Jan 2021 17:42:13 +0000 Subject: [PATCH 27/27] cache-tree: integrate with sparse directory entries MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, peff@peff.net, jrnieder@gmail.com, sunshine@sunshineco.com, pclouds@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The cache-tree extension was previously disabled with sparse indexes. However, the cache-tree is an important performance feature for commands like 'git status' and 'git add'. Integrate it with sparse directory entries. When writing a sparse index, completely clear and recalculate the cache tree. By starting from scratch, the only integration necessary is to check if we hit a sparse directory entry and create a leaf of the cache-tree that has an entry_count of one and no subtrees. Once the cache-tree exists within a sparse index, we finally get improved performance. I test the sparse index performance using a private monorepo with over 2.1 million files at HEAD, but with a sparse-checkout definition that has only 68,000 paths in the populated cone. The sparse index has about 2,000 sparse directory entries. I compare three scenarios: 1. Use the full index. The index size is ~186 MB. 2. Use the sparse index. The index size is ~5.5 MB. 3. Use a commit where HEAD matches the populated set. The full index size is ~5.3MB. The third benchmark is included as a theoretical optimium for a repository of the same object database. First, a clean 'git status' improves from 3.1s to 240ms. Benchmark #1: full index (git status) Time (mean ± σ): 3.167 s ± 0.036 s [User: 2.006 s, System: 1.078 s] Range (min … max): 3.100 s … 3.208 s 10 runs Benchmark #2: sparse index (git status) Time (mean ± σ): 239.5 ms ± 8.1 ms [User: 189.4 ms, System: 226.8 ms] Range (min … max): 226.0 ms … 251.9 ms 13 runs Benchmark #3: small tree (git status) Time (mean ± σ): 195.3 ms ± 4.5 ms [User: 116.5 ms, System: 84.4 ms] Range (min … max): 188.8 ms … 202.8 ms 15 runs The optimimum is still 45ms faster. This is due in part to the 2,000+ sparse directory entries, but there might be other optimizations to make in the sparse-index case. In particular, I find that this performance difference disappears when I disable FS Monitor, which is somewhat disabled in the sparse-index case, but might still be adding overhead. The performance numbers for 'git add .' are much closer to optimal: Benchmark #1: full index (git add .) Time (mean ± σ): 3.076 s ± 0.022 s [User: 2.065 s, System: 0.943 s] Range (min … max): 3.044 s … 3.116 s 10 runs Benchmark #2: sparse index (git add .) Time (mean ± σ): 218.0 ms ± 6.6 ms [User: 195.7 ms, System: 206.6 ms] Range (min … max): 209.8 ms … 228.2 ms 13 runs Benchmark #3: small tree (git add .) Time (mean ± σ): 217.6 ms ± 5.4 ms [User: 131.9 ms, System: 86.7 ms] Range (min … max): 212.1 ms … 228.4 ms 14 runs In this test, I also used "echo >>README.md" to append a line to the README.md file, so the 'git add .' command is doing _something_ other than a no-op. Without this edit (and FS Monitor enabled) the small tree case again gains about 30ms on the sparse index case. Signed-off-by: Derrick Stolee --- cache-tree.c | 18 ++++++++++++++++++ sparse-index.c | 10 +++++++++- 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/cache-tree.c b/cache-tree.c index 5f07a39e501..9da6a4394e0 100644 --- a/cache-tree.c +++ b/cache-tree.c @@ -256,6 +256,24 @@ static int update_one(struct cache_tree *it, *skip_count = 0; + /* + * If the first entry of this region is a sparse directory + * entry corresponding exactly to 'base', then this cache_tree + * struct is a "leaf" in the data structure, pointing to the + * tree OID specified in the entry. + */ + if (entries > 0) { + const struct cache_entry *ce = cache[0]; + + if (S_ISSPARSEDIR(ce) && + ce->ce_namelen == baselen && + !strncmp(ce->name, base, baselen)) { + it->entry_count = 1; + oidcpy(&it->oid, &ce->oid); + return 1; + } + } + if (0 <= it->entry_count && has_object_file(&it->oid)) return it->entry_count; diff --git a/sparse-index.c b/sparse-index.c index a201f3b905c..9ea3b321400 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -181,7 +181,11 @@ int convert_to_sparse(struct index_state *istate) istate->cache_nr = convert_to_sparse_rec(istate, 0, 0, istate->cache_nr, "", 0, istate->cache_tree); - istate->drop_cache_tree = 1; + + /* Clear and recompute the cache-tree */ + cache_tree_free(&istate->cache_tree); + cache_tree_update(istate, 0); + istate->sparse_index = 1; trace2_region_leave("index", "convert_to_sparse", istate->repo); return 0; @@ -278,6 +282,10 @@ void ensure_full_index(struct index_state *istate) free(full); + /* Clear and recompute the cache-tree */ + cache_tree_free(&istate->cache_tree); + cache_tree_update(istate, 0); + trace2_region_leave("index", "ensure_full_index", istate->repo); }