From patchwork Mon Oct 21 13:56:10 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11202233 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 967E214ED for ; Mon, 21 Oct 2019 13:56:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 6A7242166E for ; Mon, 21 Oct 2019 13:56:36 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="oHfHvK2X" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729206AbfJUN4f (ORCPT ); Mon, 21 Oct 2019 09:56:35 -0400 Received: from mail-wr1-f68.google.com ([209.85.221.68]:41061 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727344AbfJUN4d (ORCPT ); Mon, 21 Oct 2019 09:56:33 -0400 Received: by mail-wr1-f68.google.com with SMTP id p4so14113205wrm.8 for ; Mon, 21 Oct 2019 06:56:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=yYZ77nJfh9SzjQaXN6Qh7Fg+eMDTGn0FlOJL2Gwsss0=; b=oHfHvK2X75z8xkGAssVLaS/HbaMh4JR/QT/YRaH6IKkiXmHlZfmogjZ/1SbS5Wd++x oGkY/HkGqaKXezh5bhCuRx70ATZ2GY1UggCqNNWlITdAiaPtUbAJWbzbhkFfOZMc+jl2 0o9MkYytAbV3GQV+NZC9qqR/FiqH9cj4I8MYSiMr2HLJCxUmuLKw2si1GYDMV+BJI8lG 6HAfVjDwXU4LcMe2HDODCU4/tQXTrRZBOxS4lOGBlaGKE2IXu7XbPKzha6ai6mmtxLeM qzdVoYWt7rZojyErGVt81BhqlJ+IbsxT2NV467pyqW0wvxspNhad0ai5dyYUrf9Fj8hx edzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=yYZ77nJfh9SzjQaXN6Qh7Fg+eMDTGn0FlOJL2Gwsss0=; b=LZIOJC4r6jJKBrnxrWNvfu4S9Kkgq3HBaDxFMcb/qZ018uXDii0/aohE2FGXC2WVa3 Z4dSRXpR4i4tg5TfPJd8ViBIXaMG6SttZwEOl9Ez+JQ3QTCr0H1kZ/A1U1gsTi/ItTF9 AJkw0/XO5jR2NtpnaUcJYRNrZdOwnwiRmdaShE8TiUNOqnrrYN+U+o7X4BFF4TjmbolK JMkmVvVugz7rjxs+DfKEeW0DNT8mbGiqJ2y7iLiGz2N2ZkyT8/ezoOtKv2x9F0FlxlYe yG91THgjt2sTg6zSUJIxAG6s294dtvE2WuBC2UJld1/2MqGNEStfYUFKkfAdOHBbGWDH xT+Q== X-Gm-Message-State: APjAAAWlfWBEFUtJi9jspSu5Gcvw8YGzMGMBCr2K1tfx9UGuIRtAwKMy AXcWbSIuLDKLaShecp5UjJH7xZVl X-Google-Smtp-Source: APXvYqzcRkYCaxaz1efVslwZ8w2ddtr6EampxA/4ee97zxediGJUlvPaOySPozsqclZmCBnHAbqZcg== X-Received: by 2002:adf:f192:: with SMTP id h18mr21323173wro.148.1571666189725; Mon, 21 Oct 2019 06:56:29 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id s10sm8068262wrn.46.2019.10.21.06.56.29 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 21 Oct 2019 06:56:29 -0700 (PDT) Message-Id: <55c306a73a5e100b56b184c1ce5428c1259cfc2a.1571666186.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Mon, 21 Oct 2019 13:56:10 +0000 Subject: [PATCH v5 01/17] sparse-checkout: create builtin with 'list' subcommand Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, jon@jonsimons.org, szeder.dev@gmail.com, Derrick Stolee , Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee The sparse-checkout feature is mostly hidden to users, as its only documentation is supplementary information in the docs for 'git read-tree'. In addition, users need to know how to edit the .git/info/sparse-checkout file with the right patterns, then run the appropriate 'git read-tree -mu HEAD' command. Keeping the working directory in sync with the sparse-checkout file requires care. Begin an effort to make the sparse-checkout feature a porcelain feature by creating a new 'git sparse-checkout' builtin. This builtin will be the preferred mechanism for manipulating the sparse-checkout file and syncing the working directory. The documentation provided is adapted from the "git read-tree" documentation with a few edits for clarity in the new context. Extra sections are added to hint toward a future change to a more restricted pattern set. Helped-by: Elijah Newren Signed-off-by: Derrick Stolee --- .gitignore | 1 + Documentation/git-read-tree.txt | 2 +- Documentation/git-sparse-checkout.txt | 89 +++++++++++++++++++++++++++ Makefile | 1 + builtin.h | 1 + builtin/sparse-checkout.c | 86 ++++++++++++++++++++++++++ command-list.txt | 1 + git.c | 1 + t/t1091-sparse-checkout-builtin.sh | 45 ++++++++++++++ 9 files changed, 226 insertions(+), 1 deletion(-) create mode 100644 Documentation/git-sparse-checkout.txt create mode 100644 builtin/sparse-checkout.c create mode 100755 t/t1091-sparse-checkout-builtin.sh diff --git a/.gitignore b/.gitignore index 89b3b79c1a..aebe7c0908 100644 --- a/.gitignore +++ b/.gitignore @@ -158,6 +158,7 @@ /git-show-branch /git-show-index /git-show-ref +/git-sparse-checkout /git-stage /git-stash /git-status diff --git a/Documentation/git-read-tree.txt b/Documentation/git-read-tree.txt index d271842608..da33f84f33 100644 --- a/Documentation/git-read-tree.txt +++ b/Documentation/git-read-tree.txt @@ -436,7 +436,7 @@ support. SEE ALSO -------- linkgit:git-write-tree[1]; linkgit:git-ls-files[1]; -linkgit:gitignore[5] +linkgit:gitignore[5]; linkgit:git-sparse-checkout[1]; GIT --- diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt new file mode 100644 index 0000000000..9d6ca22917 --- /dev/null +++ b/Documentation/git-sparse-checkout.txt @@ -0,0 +1,89 @@ +git-sparse-checkout(1) +====================== + +NAME +---- +git-sparse-checkout - Initialize and modify the sparse-checkout +configuration, which reduces the checkout to a set of directories +given by a list of prefixes. + + +SYNOPSIS +-------- +[verse] +'git sparse-checkout [options]' + + +DESCRIPTION +----------- + +Initialize and modify the sparse-checkout configuration, which reduces +the checkout to a set of directories given by a list of prefixes. + +THIS COMMAND IS EXPERIMENTAL. ITS BEHAVIOR, AND THE BEHAVIOR OF OTHER +COMMANDS IN THE PRESENCE OF SPARSE-CHECKOUTS, WILL LIKELY CHANGE IN +THE FUTURE. + + +COMMANDS +-------- +'list':: + Provide a list of the contents in the sparse-checkout file. + + +SPARSE CHECKOUT +--------------- + +"Sparse checkout" allows populating the working directory sparsely. +It uses the skip-worktree bit (see linkgit:git-update-index[1]) to tell +Git whether a file in the working directory is worth looking at. If +the skip-worktree bit is set, then the file is ignored in the working +directory. Git will not populate the contents of those files, which +makes a sparse checkout helpful when working in a repository with many +files, but only a few are important to the current user. + +The `$GIT_DIR/info/sparse-checkout` file is used to define the +skip-worktree reference bitmap. When Git updates the working +directory, it updates the skip-worktree bits in the index based +on this file. The files matching the patterns in the file will +appear in the working directory, and the rest will not. + +## FULL PATTERN SET + +By default, the sparse-checkout file uses the same syntax as `.gitignore` +files. + +While `$GIT_DIR/info/sparse-checkout` is usually used to specify what +files are included, you can also specify what files are _not_ included, +using negative patterns. For example, to remove the file `unwanted`: + +---------------- +/* +!unwanted +---------------- + +Another tricky thing is fully repopulating the working directory when you +no longer want sparse checkout. You cannot just disable "sparse +checkout" because skip-worktree bits are still in the index and your working +directory is still sparsely populated. You should re-populate the working +directory with the `$GIT_DIR/info/sparse-checkout` file content as +follows: + +---------------- +/* +---------------- + +Then you can disable sparse checkout. Sparse checkout support in 'git +checkout' and similar commands is disabled by default. You need to +set `core.sparseCheckout` to `true` in order to have sparse checkout +support. + +SEE ALSO +-------- + +linkgit:git-read-tree[1] +linkgit:gitignore[5] + +GIT +--- +Part of the linkgit:git[1] suite diff --git a/Makefile b/Makefile index de60c8e7aa..adefc229fe 100644 --- a/Makefile +++ b/Makefile @@ -1125,6 +1125,7 @@ BUILTIN_OBJS += builtin/shortlog.o BUILTIN_OBJS += builtin/show-branch.o BUILTIN_OBJS += builtin/show-index.o BUILTIN_OBJS += builtin/show-ref.o +BUILTIN_OBJS += builtin/sparse-checkout.o BUILTIN_OBJS += builtin/stash.o BUILTIN_OBJS += builtin/stripspace.o BUILTIN_OBJS += builtin/submodule--helper.o diff --git a/builtin.h b/builtin.h index 5cf5df69f7..2b25a80cde 100644 --- a/builtin.h +++ b/builtin.h @@ -225,6 +225,7 @@ int cmd_shortlog(int argc, const char **argv, const char *prefix); int cmd_show(int argc, const char **argv, const char *prefix); int cmd_show_branch(int argc, const char **argv, const char *prefix); int cmd_show_index(int argc, const char **argv, const char *prefix); +int cmd_sparse_checkout(int argc, const char **argv, const char *prefix); int cmd_status(int argc, const char **argv, const char *prefix); int cmd_stash(int argc, const char **argv, const char *prefix); int cmd_stripspace(int argc, const char **argv, const char *prefix); diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c new file mode 100644 index 0000000000..5717c9b2cb --- /dev/null +++ b/builtin/sparse-checkout.c @@ -0,0 +1,86 @@ +#include "builtin.h" +#include "config.h" +#include "dir.h" +#include "parse-options.h" +#include "pathspec.h" +#include "repository.h" +#include "run-command.h" +#include "strbuf.h" + +static char const * const builtin_sparse_checkout_usage[] = { + N_("git sparse-checkout list"), + NULL +}; + +static char *get_sparse_checkout_filename(void) +{ + return git_pathdup("info/sparse-checkout"); +} + +static void write_patterns_to_file(FILE *fp, struct pattern_list *pl) +{ + int i; + + for (i = 0; i < pl->nr; i++) { + struct path_pattern *p = pl->patterns[i]; + + if (p->flags & PATTERN_FLAG_NEGATIVE) + fprintf(fp, "!"); + + fprintf(fp, "%s", p->pattern); + + if (p->flags & PATTERN_FLAG_MUSTBEDIR) + fprintf(fp, "/"); + + fprintf(fp, "\n"); + } +} + +static int sparse_checkout_list(int argc, const char **argv) +{ + struct pattern_list pl; + char *sparse_filename; + int res; + + memset(&pl, 0, sizeof(pl)); + + sparse_filename = get_sparse_checkout_filename(); + res = add_patterns_from_file_to_list(sparse_filename, "", 0, &pl, NULL); + free(sparse_filename); + + if (res < 0) { + warning(_("this worktree is not sparse (sparse-checkout file may not exist)")); + return 0; + } + + write_patterns_to_file(stdout, &pl); + clear_pattern_list(&pl); + + return 0; +} + +int cmd_sparse_checkout(int argc, const char **argv, const char *prefix) +{ + static struct option builtin_sparse_checkout_options[] = { + OPT_END(), + }; + + if (argc == 2 && !strcmp(argv[1], "-h")) + usage_with_options(builtin_sparse_checkout_usage, + builtin_sparse_checkout_options); + + argc = parse_options(argc, argv, prefix, + builtin_sparse_checkout_options, + builtin_sparse_checkout_usage, + PARSE_OPT_STOP_AT_NON_OPTION); + + git_config(git_default_config, NULL); + + if (argc > 0) { + if (!strcmp(argv[0], "list")) + return sparse_checkout_list(argc, argv); + } + + usage_with_options(builtin_sparse_checkout_usage, + builtin_sparse_checkout_options); +} diff --git a/command-list.txt b/command-list.txt index a9ac72bef4..d3d28252b3 100644 --- a/command-list.txt +++ b/command-list.txt @@ -166,6 +166,7 @@ git-show-index plumbinginterrogators git-show-ref plumbinginterrogators git-sh-i18n purehelpers git-sh-setup purehelpers +git-sparse-checkout mainporcelain worktree git-stash mainporcelain git-stage complete git-status mainporcelain info diff --git a/git.c b/git.c index ce6ab0ece2..7be7ad34bd 100644 --- a/git.c +++ b/git.c @@ -572,6 +572,7 @@ static struct cmd_struct commands[] = { { "show-branch", cmd_show_branch, RUN_SETUP }, { "show-index", cmd_show_index }, { "show-ref", cmd_show_ref, RUN_SETUP }, + { "sparse-checkout", cmd_sparse_checkout, RUN_SETUP | NEED_WORK_TREE }, { "stage", cmd_add, RUN_SETUP | NEED_WORK_TREE }, /* * NEEDSWORK: Until the builtin stash is thoroughly robust and no diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh new file mode 100755 index 0000000000..9b73d44907 --- /dev/null +++ b/t/t1091-sparse-checkout-builtin.sh @@ -0,0 +1,45 @@ +#!/bin/sh + +test_description='sparse checkout builtin tests' + +. ./test-lib.sh + +test_expect_success 'setup' ' + git init repo && + ( + cd repo && + echo "initial" >a && + mkdir folder1 folder2 deep && + mkdir deep/deeper1 deep/deeper2 && + mkdir deep/deeper1/deepest && + cp a folder1 && + cp a folder2 && + cp a deep && + cp a deep/deeper1 && + cp a deep/deeper2 && + cp a deep/deeper1/deepest && + git add . && + git commit -m "initial commit" + ) +' + +test_expect_success 'git sparse-checkout list (empty)' ' + git -C repo sparse-checkout list >list 2>err && + test_must_be_empty list && + test_i18ngrep "this worktree is not sparse (sparse-checkout file may not exist)" err +' + +test_expect_success 'git sparse-checkout list (populated)' ' + test_when_finished rm -f repo/.git/info/sparse-checkout && + cat >repo/.git/info/sparse-checkout <<-EOF && + /folder1/* + /deep/ + **/a + !*bin* + EOF + cp repo/.git/info/sparse-checkout expect && + git -C repo sparse-checkout list >list && + test_cmp expect list +' + +test_done From patchwork Mon Oct 21 13:56:11 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11202227 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BA62F1390 for ; Mon, 21 Oct 2019 13:56:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 8F3982166E for ; Mon, 21 Oct 2019 13:56:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="aN8uiJFp" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729198AbfJUN4d (ORCPT ); Mon, 21 Oct 2019 09:56:33 -0400 Received: from mail-wr1-f68.google.com ([209.85.221.68]:41062 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729041AbfJUN4c (ORCPT ); Mon, 21 Oct 2019 09:56:32 -0400 Received: by mail-wr1-f68.google.com with SMTP id p4so14113229wrm.8 for ; Mon, 21 Oct 2019 06:56:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=MdttcHwYY98Nj6YWdfs9E5YtG7PCh3prnOQCnJGwcms=; b=aN8uiJFpAnxS9dtoA4ZHEiopTP0xljBn2Ie/VUPwUVFLOkZHvD+qVyKM5j/lp6VUS9 BDSqd+ENql4NUMJ+r/l/+3nO/ot+MlJT9xxai9kCZ4sXTTSJaqwD4JbewFqKdaFv3yVH qcaDx3kkmwu7Z52fDtrZ/ARUd25oOcqpWU6cdNIvzyXd0eMs9moOhmkqtik8kotqvsJc Wx6xWbbJF+tXIDp9K+5e3oq+ppVYjHI8FE2PAuM0NH9nnK1j2qxbuOYKTGM403K9FxEF NeEjKNqx9vZIbHS6z/7xg+UHjI2zfjLjqhjne86vPIn4LgEE6uPnAyUcL2I7hmzbTG4Q EuKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=MdttcHwYY98Nj6YWdfs9E5YtG7PCh3prnOQCnJGwcms=; b=huVinnFDfzEkSxd2TyE0d3QU1VI+P3kXrAvqUUoXQ9w259HhXFxlnA5+UQLzuc/bba 5XXpW9Whftj7VzW15BaCr3n9fMtqRS+AHumxmuErhUr+6KbjsshSv/R6oS/vP7+EeW2L 8BUz2eCZu+JFTE7axY+tyXej8AsiG/FNs9o+iTyPdcNKSoF9MblrzfaWJnn9DVMkD1Kp 6GxNxN6jPwEmEribURsF6YUdD771Ztg+OX10hsndSK9eVs6cpLI/HjH4BLRxN/La5KsZ dqztDsr56dLyma0VPwy1BZ8SkSuA4kKsu2AJ7jvLggJRARBOvu+sr2IjfabUW/9iEdhK 702Q== X-Gm-Message-State: APjAAAUxFeQONShCbt/eF/Pp0i8A6W5CLab2fPCJOaDnbqyo3fIiZ9RP wOUz/AgnPNj3b1fZmK8U04XiTJu5 X-Google-Smtp-Source: APXvYqy2Iop77NCzjI3NRLGOk0i1pm/qTXAAQ4lw3nGOqkI9Wg1YWKgvbPmnIutpDdgYHXuj2A/5Bg== X-Received: by 2002:adf:93e1:: with SMTP id 88mr10062764wrp.198.1571666190410; Mon, 21 Oct 2019 06:56:30 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 143sm24158728wmb.33.2019.10.21.06.56.29 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 21 Oct 2019 06:56:30 -0700 (PDT) Message-Id: In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Mon, 21 Oct 2019 13:56:11 +0000 Subject: [PATCH v5 02/17] sparse-checkout: create 'init' subcommand Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, jon@jonsimons.org, szeder.dev@gmail.com, Derrick Stolee , Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee Getting started with a sparse-checkout file can be daunting. Help users start their sparse enlistment using 'git sparse-checkout init'. This will set 'core.sparseCheckout=true' in their config, write an initial set of patterns to the sparse-checkout file, and update their working directory. Make sure to use the `extensions.worktreeConfig` setting and write the sparse checkout config to the worktree-specific config file. This avoids confusing interactions with other worktrees. The use of running another process for 'git read-tree' is sub- optimal. This will be removed in a later change. Signed-off-by: Derrick Stolee --- Documentation/git-sparse-checkout.txt | 11 ++++ builtin/sparse-checkout.c | 79 ++++++++++++++++++++++++++- t/t1091-sparse-checkout-builtin.sh | 41 ++++++++++++++ 3 files changed, 130 insertions(+), 1 deletion(-) diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt index 9d6ca22917..930a361567 100644 --- a/Documentation/git-sparse-checkout.txt +++ b/Documentation/git-sparse-checkout.txt @@ -30,6 +30,17 @@ COMMANDS 'list':: Provide a list of the contents in the sparse-checkout file. +'init':: + Enable the `core.sparseCheckout` setting. If the + sparse-checkout file does not exist, then populate it with + patterns that match every file in the root directory and + no other directories, then will remove all directories tracked + by Git. Add patterns to the sparse-checkout file to + repopulate the working directory. ++ +To avoid interfering with other worktrees, it first enables the +`extensions.worktreeConfig` setting and makes sure to set the +`core.sparseCheckout` setting in the worktree-specific config file. SPARSE CHECKOUT --------------- diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c index 5717c9b2cb..77aa52ca01 100644 --- a/builtin/sparse-checkout.c +++ b/builtin/sparse-checkout.c @@ -8,7 +8,7 @@ #include "strbuf.h" static char const * const builtin_sparse_checkout_usage[] = { - N_("git sparse-checkout list"), + N_("git sparse-checkout (init|list)"), NULL }; @@ -59,6 +59,81 @@ static int sparse_checkout_list(int argc, const char **argv) return 0; } +static int update_working_directory(void) +{ + struct argv_array argv = ARGV_ARRAY_INIT; + int result = 0; + argv_array_pushl(&argv, "read-tree", "-m", "-u", "HEAD", NULL); + + if (run_command_v_opt(argv.argv, RUN_GIT_CMD)) { + error(_("failed to update index with new sparse-checkout paths")); + result = 1; + } + + argv_array_clear(&argv); + return result; +} + +enum sparse_checkout_mode { + MODE_NO_PATTERNS = 0, + MODE_ALL_PATTERNS = 1, +}; + +static int sc_set_config(enum sparse_checkout_mode mode) +{ + struct argv_array argv = ARGV_ARRAY_INIT; + + if (git_config_set_gently("extensions.worktreeConfig", "true")) { + error(_("failed to set extensions.worktreeConfig setting")); + return 1; + } + + argv_array_pushl(&argv, "config", "--worktree", "core.sparseCheckout", NULL); + + if (mode) + argv_array_pushl(&argv, "true", NULL); + else + argv_array_pushl(&argv, "false", NULL); + + if (run_command_v_opt(argv.argv, RUN_GIT_CMD)) { + error(_("failed to enable core.sparseCheckout")); + return 1; + } + + return 0; +} + +static int sparse_checkout_init(int argc, const char **argv) +{ + struct pattern_list pl; + char *sparse_filename; + FILE *fp; + int res; + + if (sc_set_config(MODE_ALL_PATTERNS)) + return 1; + + memset(&pl, 0, sizeof(pl)); + + sparse_filename = get_sparse_checkout_filename(); + res = add_patterns_from_file_to_list(sparse_filename, "", 0, &pl, NULL); + + /* If we already have a sparse-checkout file, use it. */ + if (res >= 0) { + free(sparse_filename); + goto reset_dir; + } + + /* initial mode: all blobs at root */ + fp = xfopen(sparse_filename, "w"); + free(sparse_filename); + fprintf(fp, "/*\n!/*/\n"); + fclose(fp); + +reset_dir: + return update_working_directory(); +} + int cmd_sparse_checkout(int argc, const char **argv, const char *prefix) { static struct option builtin_sparse_checkout_options[] = { @@ -79,6 +154,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix) if (argc > 0) { if (!strcmp(argv[0], "list")) return sparse_checkout_list(argc, argv); + if (!strcmp(argv[0], "init")) + return sparse_checkout_init(argc, argv); } usage_with_options(builtin_sparse_checkout_usage, diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh index 9b73d44907..cd56cc384b 100755 --- a/t/t1091-sparse-checkout-builtin.sh +++ b/t/t1091-sparse-checkout-builtin.sh @@ -42,4 +42,45 @@ test_expect_success 'git sparse-checkout list (populated)' ' test_cmp expect list ' +test_expect_success 'git sparse-checkout init' ' + git -C repo sparse-checkout init && + cat >expect <<-EOF && + /* + !/*/ + EOF + test_cmp expect repo/.git/info/sparse-checkout && + git -C repo config --list >config && + test_i18ngrep "core.sparsecheckout=true" config && + ls repo >dir && + echo a >expect && + test_cmp expect dir +' + +test_expect_success 'git sparse-checkout list after init' ' + git -C repo sparse-checkout list >actual && + cat >expect <<-EOF && + /* + !/*/ + EOF + test_cmp expect actual +' + +test_expect_success 'init with existing sparse-checkout' ' + echo "*folder*" >> repo/.git/info/sparse-checkout && + git -C repo sparse-checkout init && + cat >expect <<-EOF && + /* + !/*/ + *folder* + EOF + test_cmp expect repo/.git/info/sparse-checkout && + ls repo >dir && + cat >expect <<-EOF && + a + folder1 + folder2 + EOF + test_cmp expect dir +' + test_done From patchwork Mon Oct 21 13:56:12 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11202231 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0ADDC139A for ; Mon, 21 Oct 2019 13:56:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id D405C2166E for ; Mon, 21 Oct 2019 13:56:35 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="YERgy5PD" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729197AbfJUN4d (ORCPT ); Mon, 21 Oct 2019 09:56:33 -0400 Received: from mail-wm1-f66.google.com ([209.85.128.66]:37455 "EHLO mail-wm1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729188AbfJUN4c (ORCPT ); Mon, 21 Oct 2019 09:56:32 -0400 Received: by mail-wm1-f66.google.com with SMTP id f22so12912550wmc.2 for ; Mon, 21 Oct 2019 06:56:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=+Dtd3kEyatD2qmW8hj5YNKVLqty3loyNp8hGKo37c+A=; b=YERgy5PD0yPaxirKuYDNqvi/SHwBlx5Y/Cfn39U91wpFLFAssXsPo6Ellm+u191YRc nvscjXeMY9kRxxvTA3z9wu6jxvaA33yebEcXBVC1COF1ejHcgGcH6ZmkIfic8SI+hmj6 Qgo8dyhVfftJcYLhiFdN364GnIuxsW+ucfkZsC5Lhqq704yxG3FVvnvKIFvEVgnJyjmG 3+ReDLJD4NmTsMby/rvu/T4jEjmqTbLTDrLciXRqQLOTsVMv1kgHS0gzrcuO/3p93PJq AKsrEHz2ouKTROq14+oGuFWg8jkrbqGLRCd5+6FIckWV5mGkPJv5aDU/eJA45XTk6c4H K0gw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=+Dtd3kEyatD2qmW8hj5YNKVLqty3loyNp8hGKo37c+A=; b=gLXQkrB624A4bCoL23nkxj8P2LtECVzdUCmo+reNFtmEsverk0dbZxF9qUsYpneNo0 Zd7TB7jYblLfR7S/FLBacswS7YIJnN7hlOa5S7VthO068GkpqN0RTlFIk1tZJ6iUBchw i2b7Dc04Vtp/9eWC2LVSG4vDs4ASred6sZWl0H3GLMbXAJpcOZKnOTSGF77QDgVYdvGh 8sXoE6wGif4/oeotd8Gb0GJXAul7BcyWVNW1EGLy3xi8u9cDE69QuaF7mqnfw6l1p4pl nA42u6nHJr3Vz4+iLTv8zSp2YwmyDpt0CI2HwDGlI+XNpuk04+xC9RcZmcDPqD/YP1Fq uLZQ== X-Gm-Message-State: APjAAAWewnrvA+COqJrSLHTFYo+8H282/Gc2u0l+aPAhpoN6ft0PG89l a66zD0GJAF7/QdWAXvWzt7A8si8X X-Google-Smtp-Source: APXvYqwaATFgIbw7UceScGMz49CUGDnBBJRcNUJ1DzS+XZYuLHwX+xVf8uIMbmalAfUTPy/ck49ILQ== X-Received: by 2002:a7b:cb0b:: with SMTP id u11mr135040wmj.125.1571666191146; Mon, 21 Oct 2019 06:56:31 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id b1sm7330874wru.83.2019.10.21.06.56.30 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 21 Oct 2019 06:56:30 -0700 (PDT) Message-Id: In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Mon, 21 Oct 2019 13:56:12 +0000 Subject: [PATCH v5 03/17] clone: add --sparse mode Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, jon@jonsimons.org, szeder.dev@gmail.com, Derrick Stolee , Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee When someone wants to clone a large repository, but plans to work using a sparse-checkout file, they either need to do a full checkout first and then reduce the patterns they included, or clone with --no-checkout, set up their patterns, and then run a checkout manually. This requires knowing a lot about the repo shape and how sparse-checkout works. Add a new '--sparse' option to 'git clone' that initializes the sparse-checkout file to include the following patterns: /* !/*/ These patterns include every file in the root directory, but no directories. This allows a repo to include files like a README or a bootstrapping script to grow enlistments from that point. During the 'git sparse-checkout init' call, we must first look to see if HEAD is valid, since 'git clone' does not have a valid HEAD at the point where it initializes the sparse-checkout. The following checkout within the clone command will create the HEAD ref and update the working directory correctly. Signed-off-by: Derrick Stolee --- Documentation/git-clone.txt | 8 +++++++- builtin/clone.c | 27 +++++++++++++++++++++++++++ builtin/sparse-checkout.c | 6 ++++++ t/t1091-sparse-checkout-builtin.sh | 13 +++++++++++++ 4 files changed, 53 insertions(+), 1 deletion(-) diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt index 34011c2940..0fe91d2f04 100644 --- a/Documentation/git-clone.txt +++ b/Documentation/git-clone.txt @@ -15,7 +15,7 @@ SYNOPSIS [--dissociate] [--separate-git-dir ] [--depth ] [--[no-]single-branch] [--no-tags] [--recurse-submodules[=]] [--[no-]shallow-submodules] - [--[no-]remote-submodules] [--jobs ] [--] + [--[no-]remote-submodules] [--jobs ] [--sparse] [--] [] DESCRIPTION @@ -156,6 +156,12 @@ objects from the source repository into a pack in the cloned repository. used, neither remote-tracking branches nor the related configuration variables are created. +--sparse:: + Initialize the sparse-checkout file so the working + directory starts with only the files in the root + of the repository. The sparse-checkout file can be + modified to grow the working directory as needed. + --mirror:: Set up a mirror of the source repository. This implies `--bare`. Compared to `--bare`, `--mirror` not only maps local branches of the diff --git a/builtin/clone.c b/builtin/clone.c index c46ee29f0a..4348d962c9 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -59,6 +59,7 @@ static const char *real_git_dir; static char *option_upload_pack = "git-upload-pack"; static int option_verbosity; static int option_progress = -1; +static int option_sparse_checkout; static enum transport_family family; static struct string_list option_config = STRING_LIST_INIT_NODUP; static struct string_list option_required_reference = STRING_LIST_INIT_NODUP; @@ -146,6 +147,8 @@ static struct option builtin_clone_options[] = { OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options), OPT_BOOL(0, "remote-submodules", &option_remote_submodules, N_("any cloned submodules will use their remote-tracking branch")), + OPT_BOOL(0, "sparse", &option_sparse_checkout, + N_("initialize sparse-checkout file to include only files at root")), OPT_END() }; @@ -733,6 +736,27 @@ static void update_head(const struct ref *our, const struct ref *remote, } } +static int git_sparse_checkout_init(const char *repo) +{ + struct argv_array argv = ARGV_ARRAY_INIT; + int result = 0; + argv_array_pushl(&argv, "-C", repo, "sparse-checkout", "init", NULL); + + /* + * We must apply the setting in the current process + * for the later checkout to use the sparse-checkout file. + */ + core_apply_sparse_checkout = 1; + + if (run_command_v_opt(argv.argv, RUN_GIT_CMD)) { + error(_("failed to initialize sparse-checkout")); + result = 1; + } + + argv_array_clear(&argv); + return result; +} + static int checkout(int submodule_progress) { struct object_id oid; @@ -1106,6 +1130,9 @@ int cmd_clone(int argc, const char **argv, const char *prefix) if (option_required_reference.nr || option_optional_reference.nr) setup_reference(); + if (option_sparse_checkout && git_sparse_checkout_init(repo)) + return 1; + remote = remote_get(option_origin); strbuf_addf(&default_refspec, "+%s*:%s*", src_ref_prefix, diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c index 77aa52ca01..6c336b7ab3 100644 --- a/builtin/sparse-checkout.c +++ b/builtin/sparse-checkout.c @@ -109,6 +109,7 @@ static int sparse_checkout_init(int argc, const char **argv) char *sparse_filename; FILE *fp; int res; + struct object_id oid; if (sc_set_config(MODE_ALL_PATTERNS)) return 1; @@ -130,6 +131,11 @@ static int sparse_checkout_init(int argc, const char **argv) fprintf(fp, "/*\n!/*/\n"); fclose(fp); + if (get_oid("HEAD", &oid)) { + /* assume we are in a fresh repo */ + return 0; + } + reset_dir: return update_working_directory(); } diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh index cd56cc384b..cb74715ca6 100755 --- a/t/t1091-sparse-checkout-builtin.sh +++ b/t/t1091-sparse-checkout-builtin.sh @@ -83,4 +83,17 @@ test_expect_success 'init with existing sparse-checkout' ' test_cmp expect dir ' +test_expect_success 'clone --sparse' ' + git clone --sparse repo clone && + git -C clone sparse-checkout list >actual && + cat >expect <<-EOF && + /* + !/*/ + EOF + test_cmp expect actual && + ls clone >dir && + echo a >expect && + test_cmp expect dir +' + test_done From patchwork Mon Oct 21 13:56:13 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11202229 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 051A014ED for ; Mon, 21 Oct 2019 13:56:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id D7849214B2 for ; Mon, 21 Oct 2019 13:56:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="nI7wlu0t" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729201AbfJUN4e (ORCPT ); Mon, 21 Oct 2019 09:56:34 -0400 Received: from mail-wr1-f66.google.com ([209.85.221.66]:45819 "EHLO mail-wr1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729189AbfJUN4d (ORCPT ); Mon, 21 Oct 2019 09:56:33 -0400 Received: by mail-wr1-f66.google.com with SMTP id q13so9141484wrs.12 for ; Mon, 21 Oct 2019 06:56:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=/ZKw8/obR5Ihbs8rwzjjHOhiY4JwaEHKEy7Urcr0ub0=; b=nI7wlu0tOUsHDSXZ0HcxjIroHu/pOXGD81cVeU3vZJe0IVKegp+1u+QUxNn/xcY08C pXulam3QHpptIiBB39OC267a92veymUabpTC3V/QxLcB2TP41PD1a58LpOEokw6ORp7S jsto5YF/LwZuKs2AMAesNgCYiyuhhR89kR97e7BAWZwRsW8JscLFC9gspvNZVao359pr fFwYGPDFTtvVFxDCl22wRwAz8v610hkJ4KvKlvxm/0BdVZKBWq+6eed5RntE9xma6IZW bFH6kQvkLc4VWrDEVyueDws67Kgv7LH6QOHCI5l9K48iQsv0402dTJ11tTlGNyoTA1gQ rZhQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=/ZKw8/obR5Ihbs8rwzjjHOhiY4JwaEHKEy7Urcr0ub0=; b=XZ5rye9NKt3f3XA0uYyqJyjxtGb/C8WHylpvb7LRVERdmLwzh4YDAwsLActr8eFOri TnWv2gLFXFVRP3ifOeOnsSpCsibhnKlI9rN6L86GPfABbUZZBAjQ6ywZofEajubgn7pD Arly5ek3mv3Af8pBA9W4I9E9G+9VK8BmdObXjdvMQca0JSiPD5B8iPqIARpfP/ETI6q7 immB9sBtheEFvr7UgWOtW1I4nMd9xO6xqwYER4MC+z9BMWXBYJKhKqcOp9oSjZ9nao36 eyoEzUC+ksND4Vj76Q5NUSfJMfG+Y1zUARBeVvRKin2LQXPxJ3NPls9JKim52nUHPzEb u22w== X-Gm-Message-State: APjAAAXz6SYOoxZ9CbD/6DCDbQBjm4y4tgj0FBQLlsZd6abyK6ZNtCS+ nxpSFsw91cdoDQaaHMmRVG8VDzZ7 X-Google-Smtp-Source: APXvYqxJn3Y+jAp6cJGeDh5LcNLzeZxqeRRqbYKugqGgOg99qHDi4L+NFeijssQAYF04I8n03vcKtw== X-Received: by 2002:a5d:6585:: with SMTP id q5mr19879289wru.74.1571666192124; Mon, 21 Oct 2019 06:56:32 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id q196sm10478742wme.23.2019.10.21.06.56.31 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 21 Oct 2019 06:56:31 -0700 (PDT) Message-Id: <7d9d66a89f473244af3601e13caa713d929a202d.1571666186.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Mon, 21 Oct 2019 13:56:13 +0000 Subject: [PATCH v5 04/17] sparse-checkout: 'set' subcommand Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, jon@jonsimons.org, szeder.dev@gmail.com, Derrick Stolee , Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee The 'git sparse-checkout set' subcommand takes a list of patterns as arguments and writes them to the sparse-checkout file. Then, it updates the working directory using 'git read-tree -mu HEAD'. The 'set' subcommand will replace the entire contents of the sparse-checkout file. The write_patterns_and_update() method is extracted from cmd_sparse_checkout() to make it easier to implement 'add' and/or 'remove' subcommands in the future. If the core.sparseCheckout config setting is disabled, then enable the config setting in the worktree config. If we set the config this way and the sparse-checkout fails, then re-disable the config setting. Signed-off-by: Derrick Stolee --- Documentation/git-sparse-checkout.txt | 6 ++++ builtin/sparse-checkout.c | 45 ++++++++++++++++++++++++++- t/t1091-sparse-checkout-builtin.sh | 32 +++++++++++++++++++ 3 files changed, 82 insertions(+), 1 deletion(-) diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt index 930a361567..b933043b3d 100644 --- a/Documentation/git-sparse-checkout.txt +++ b/Documentation/git-sparse-checkout.txt @@ -42,6 +42,12 @@ To avoid interfering with other worktrees, it first enables the `extensions.worktreeConfig` setting and makes sure to set the `core.sparseCheckout` setting in the worktree-specific config file. +'set':: + Write a set of patterns to the sparse-checkout file, as given as + a list of arguments following the 'set' subcommand. Update the + working directory to match the new patterns. Enable the + core.sparseCheckout config setting if it is not already enabled. + SPARSE CHECKOUT --------------- diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c index 6c336b7ab3..834ee421f0 100644 --- a/builtin/sparse-checkout.c +++ b/builtin/sparse-checkout.c @@ -8,7 +8,7 @@ #include "strbuf.h" static char const * const builtin_sparse_checkout_usage[] = { - N_("git sparse-checkout (init|list)"), + N_("git sparse-checkout (init|list|set) "), NULL }; @@ -140,6 +140,47 @@ static int sparse_checkout_init(int argc, const char **argv) return update_working_directory(); } +static int write_patterns_and_update(struct pattern_list *pl) +{ + char *sparse_filename; + FILE *fp; + + sparse_filename = get_sparse_checkout_filename(); + fp = fopen(sparse_filename, "w"); + write_patterns_to_file(fp, pl); + fclose(fp); + free(sparse_filename); + + return update_working_directory(); +} + +static int sparse_checkout_set(int argc, const char **argv, const char *prefix) +{ + static const char *empty_base = ""; + int i; + struct pattern_list pl; + int result; + int set_config = 0; + memset(&pl, 0, sizeof(pl)); + + for (i = 1; i < argc; i++) + add_pattern(argv[i], empty_base, 0, &pl, 0); + + if (!core_apply_sparse_checkout) { + sc_set_config(MODE_ALL_PATTERNS); + core_apply_sparse_checkout = 1; + set_config = 1; + } + + result = write_patterns_and_update(&pl); + + if (result && set_config) + sc_set_config(MODE_NO_PATTERNS); + + clear_pattern_list(&pl); + return result; +} + int cmd_sparse_checkout(int argc, const char **argv, const char *prefix) { static struct option builtin_sparse_checkout_options[] = { @@ -162,6 +203,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix) return sparse_checkout_list(argc, argv); if (!strcmp(argv[0], "init")) return sparse_checkout_init(argc, argv); + if (!strcmp(argv[0], "set")) + return sparse_checkout_set(argc, argv, prefix); } usage_with_options(builtin_sparse_checkout_usage, diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh index cb74715ca6..bf2dc55bb1 100755 --- a/t/t1091-sparse-checkout-builtin.sh +++ b/t/t1091-sparse-checkout-builtin.sh @@ -96,4 +96,36 @@ test_expect_success 'clone --sparse' ' test_cmp expect dir ' +test_expect_success 'set enables config' ' + git init empty-config && + ( + cd empty-config && + test_commit test file && + test_path_is_missing .git/config.worktree && + test_must_fail git sparse-checkout set nothing && + test_i18ngrep "sparseCheckout = false" .git/config.worktree && + git sparse-checkout set "/*" && + test_i18ngrep "sparseCheckout = true" .git/config.worktree + ) +' + +test_expect_success 'set sparse-checkout using builtin' ' + git -C repo sparse-checkout set "/*" "!/*/" "*folder*" && + cat >expect <<-EOF && + /* + !/*/ + *folder* + EOF + git -C repo sparse-checkout list >actual && + test_cmp expect actual && + test_cmp expect repo/.git/info/sparse-checkout && + ls repo >dir && + cat >expect <<-EOF && + a + folder1 + folder2 + EOF + test_cmp expect dir +' + test_done From patchwork Mon Oct 21 13:56:14 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11202235 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 82C5114ED for ; Mon, 21 Oct 2019 13:56:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 61134214B2 for ; Mon, 21 Oct 2019 13:56:39 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="EKXSoAjt" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729224AbfJUN4i (ORCPT ); Mon, 21 Oct 2019 09:56:38 -0400 Received: from mail-wr1-f66.google.com ([209.85.221.66]:45824 "EHLO mail-wr1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729203AbfJUN4g (ORCPT ); Mon, 21 Oct 2019 09:56:36 -0400 Received: by mail-wr1-f66.google.com with SMTP id q13so9141540wrs.12 for ; Mon, 21 Oct 2019 06:56:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=2sEVfkTntbqY+8J1j4mT+2x+4jgWJVhxafMhzxse4CY=; b=EKXSoAjtWyIaClzwZ1lAQrXILzq7VHvUQUuA60kiAG6740DVUpLKnke6IKVI8NL1Ba lXEy3plx/v7QxrBUSQXgq/p4uZEuiCcVu3DiRDPGINWbyfDi6whS4ttbeiDVveM2ekSu z6nwCJb6+f4JEa9Ou8wcU756kMg8PcaqOeDEHPN75guIt+KUJHS5NUyY2tAxrEbvxKNw QnY/0jIb/qNBYbv0MRtO27FM8E+xbWV5kY8sIUe/YuvMZ8DJowuU2fK79ErQ9/tQSxkr XlgOe+i8ADueHHdajci4bftV06VuiTt47LZCtDm+9sV0AzvDpdEcmYEgzMvY/r4RzGAi q4Bg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=2sEVfkTntbqY+8J1j4mT+2x+4jgWJVhxafMhzxse4CY=; b=TQcl2mKgwwmjB5CcfMNnVt+zBvLKj5JAKqqO4lvYO5WjRtt9pL95o1h3UGtVEeOEE5 IxgSWa4VxLSjcGthOjOmjyrDZNy8mnDjCh5YNI5w1UE9Lu19ciAMnWGBnrbx0lJIk8BQ 8NVVAlVO4/Y/zk8P5IEaxVM2r2dLYNfniMSo+itaWiySkmGXSpDtAP6+1w1x2CRwpg0A RSJo59jeI7CPoginm0D49A9t0DYiyaD/JG+fKlAQ++/0ZcRY3jaA42hzoNmSZo116RC3 Q/NVpzPP1d/YWUCzWf+b23OlNdZRb7KDezQPGtvK41rfgUtUsi4nKjSQmRqdk/iLTKFl OxLQ== X-Gm-Message-State: APjAAAWnfipBWYhNR9HJq2t/9QhnFrZx/PHbt70C9wt4QtPeqA/dpQHA IJGI64nP4LVRafWNdPro+cVL60QY X-Google-Smtp-Source: APXvYqzday6oS5zMeDKiv3tZcczN9e326v9FJyDiiZCTifjUj/ZIutRofcRDF7SiiYnnWZ0y223nTg== X-Received: by 2002:a5d:4f8b:: with SMTP id d11mr1421443wru.25.1571666193073; Mon, 21 Oct 2019 06:56:33 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id b1sm7330947wru.83.2019.10.21.06.56.32 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 21 Oct 2019 06:56:32 -0700 (PDT) Message-Id: <0e08898dcb42bd38ca3692b49a7e9f5763150c80.1571666186.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Mon, 21 Oct 2019 13:56:14 +0000 Subject: [PATCH v5 05/17] sparse-checkout: add '--stdin' option to set subcommand Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, jon@jonsimons.org, szeder.dev@gmail.com, Derrick Stolee , Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee The 'git sparse-checkout set' subcommand takes a list of patterns and places them in the sparse-checkout file. Then, it updates the working directory to match those patterns. For a large list of patterns, the command-line call can get very cumbersome. Add a '--stdin' option to instead read patterns over standard in. Signed-off-by: Derrick Stolee --- builtin/sparse-checkout.c | 35 ++++++++++++++++++++++++++++-- t/t1091-sparse-checkout-builtin.sh | 20 +++++++++++++++++ 2 files changed, 53 insertions(+), 2 deletions(-) diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c index 834ee421f0..f2e2bd772d 100644 --- a/builtin/sparse-checkout.c +++ b/builtin/sparse-checkout.c @@ -154,6 +154,15 @@ static int write_patterns_and_update(struct pattern_list *pl) return update_working_directory(); } +static char const * const builtin_sparse_checkout_set_usage[] = { + N_("git sparse-checkout set [--stdin|]"), + NULL +}; + +static struct sparse_checkout_set_opts { + int use_stdin; +} set_opts; + static int sparse_checkout_set(int argc, const char **argv, const char *prefix) { static const char *empty_base = ""; @@ -161,10 +170,32 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix) struct pattern_list pl; int result; int set_config = 0; + + static struct option builtin_sparse_checkout_set_options[] = { + OPT_BOOL(0, "stdin", &set_opts.use_stdin, + N_("read patterns from standard in")), + OPT_END(), + }; + memset(&pl, 0, sizeof(pl)); - for (i = 1; i < argc; i++) - add_pattern(argv[i], empty_base, 0, &pl, 0); + argc = parse_options(argc, argv, prefix, + builtin_sparse_checkout_set_options, + builtin_sparse_checkout_set_usage, + PARSE_OPT_KEEP_UNKNOWN); + + if (set_opts.use_stdin) { + struct strbuf line = STRBUF_INIT; + + while (!strbuf_getline(&line, stdin)) { + size_t len; + char *buf = strbuf_detach(&line, &len); + add_pattern(buf, empty_base, 0, &pl, 0); + } + } else { + for (i = 0; i < argc; i++) + add_pattern(argv[i], empty_base, 0, &pl, 0); + } if (!core_apply_sparse_checkout) { sc_set_config(MODE_ALL_PATTERNS); diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh index bf2dc55bb1..a9ff5eb9ec 100755 --- a/t/t1091-sparse-checkout-builtin.sh +++ b/t/t1091-sparse-checkout-builtin.sh @@ -128,4 +128,24 @@ test_expect_success 'set sparse-checkout using builtin' ' test_cmp expect dir ' +test_expect_success 'set sparse-checkout using --stdin' ' + cat >expect <<-EOF && + /* + !/*/ + /folder1/ + /folder2/ + EOF + git -C repo sparse-checkout set --stdin actual && + test_cmp expect actual && + test_cmp expect repo/.git/info/sparse-checkout && + ls repo >dir && + cat >expect <<-EOF && + a + folder1 + folder2 + EOF + test_cmp expect dir +' + test_done From patchwork Mon Oct 21 13:56:15 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11202259 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ECE5C139A for ; Mon, 21 Oct 2019 13:56:49 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id CA51A214B2 for ; Mon, 21 Oct 2019 13:56:49 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="BB6hmjmm" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729233AbfJUN4k (ORCPT ); Mon, 21 Oct 2019 09:56:40 -0400 Received: from mail-wr1-f68.google.com ([209.85.221.68]:44061 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729189AbfJUN4g (ORCPT ); Mon, 21 Oct 2019 09:56:36 -0400 Received: by mail-wr1-f68.google.com with SMTP id z9so14107988wrl.11 for ; Mon, 21 Oct 2019 06:56:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=HlIZIXXEGIo6p95Tf1uQT0iuIhvv91TiapywK9qe7Vw=; b=BB6hmjmmQB7Za6LpjR06iN/2ednnCeyv2AMuAxd18GnwjbDJYXuJTAy+Tu09wd+oGb PCWn8sLsQUcdinfnx5RcoC11yIF2hMUkB3bKil/+pib5cCF55n+7iHkuEIcsJkEQ5jsn FU+YlL1k0LfZs0/xpGla+Nsx7RuM9WwSzbOQ6T8GSUB8hbmhc5kDF0HKQxDuBpc6/oPx 9I72wT6IxRZfFWpF09LcYekEmXIGzPsBkimNGPIVwT3dAcwQNlvtKVxwGnyJfvwUIRT6 mU7dZLhMjH3Agr41hA9+f8hw6unbjtMKfeRIFjIcoPiHjvOexWfeLPT42O03X3M/4/i1 Ut2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=HlIZIXXEGIo6p95Tf1uQT0iuIhvv91TiapywK9qe7Vw=; b=NUl3id25FuQZC14ZOqU9v+nwz09FugXlq+cWqH0a/tlPkhpbKwX3STYleLb9JOlbdX sjPocfOUhOi6rwuszucO+Z6nkurDz7E+JCV3PdprVlGpI7sEQSWlPs4haz6qP5Z9l1i8 EzfAOCjZwLUg8e+kqFDxigqp/PezqTGei+YMGzP+7z6VtT5fxKpTNralTLVASamYq8an j3BlQTfVMdITVvm+jxGD4F6bsWwqFn/WTN85JwcrHgp5kNzG3JdUuC1+6vQSRjQNzic2 7niW2CXG220EIVpGFhWw5rpoUhzB1sNuxpxTv7hIbGwOGWVoEFoHYrAXNK/mOWM7olb8 rUSA== X-Gm-Message-State: APjAAAWqPb6XeG7CPMRJmHvpzRPooF+rTVpW09nqAqX1WMFrzIxd6BLW wubZ2epdIiggldYi6xvqGT5lwjL5 X-Google-Smtp-Source: APXvYqze5g4p3H+Me7aWSkWZR1ESwbyGLNaP29+Kdg4dSzvOg7JrSWAwhfBhhvOylRuGpA35zGetaA== X-Received: by 2002:adf:db0e:: with SMTP id s14mr16208321wri.341.1571666193894; Mon, 21 Oct 2019 06:56:33 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f17sm3191187wrs.66.2019.10.21.06.56.33 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 21 Oct 2019 06:56:33 -0700 (PDT) Message-Id: In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Mon, 21 Oct 2019 13:56:15 +0000 Subject: [PATCH v5 06/17] sparse-checkout: create 'disable' subcommand Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, jon@jonsimons.org, szeder.dev@gmail.com, Derrick Stolee , Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee The instructions for disabling a sparse-checkout to a full working directory are complicated and non-intuitive. Add a subcommand, 'git sparse-checkout disable', to perform those steps for the user. Signed-off-by: Derrick Stolee --- Documentation/git-sparse-checkout.txt | 27 ++++++++++++--------------- builtin/sparse-checkout.c | 26 +++++++++++++++++++++++++- t/t1091-sparse-checkout-builtin.sh | 15 +++++++++++++++ 3 files changed, 52 insertions(+), 16 deletions(-) diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt index b933043b3d..f794d4797a 100644 --- a/Documentation/git-sparse-checkout.txt +++ b/Documentation/git-sparse-checkout.txt @@ -48,6 +48,10 @@ To avoid interfering with other worktrees, it first enables the working directory to match the new patterns. Enable the core.sparseCheckout config setting if it is not already enabled. +'disable':: + Remove the sparse-checkout file, set `core.sparseCheckout` to + `false`, and restore the working directory to include all files. + SPARSE CHECKOUT --------------- @@ -65,6 +69,14 @@ directory, it updates the skip-worktree bits in the index based on this file. The files matching the patterns in the file will appear in the working directory, and the rest will not. +To enable the sparse-checkout feature, run `git sparse-checkout init` to +initialize a simple sparse-checkout file and enable the `core.sparseCheckout` +config setting. Then, run `git sparse-checkout set` to modify the patterns in +the sparse-checkout file. + +To repopulate the working directory with all files, use the +`git sparse-checkout disable` command. + ## FULL PATTERN SET By default, the sparse-checkout file uses the same syntax as `.gitignore` @@ -79,21 +91,6 @@ using negative patterns. For example, to remove the file `unwanted`: !unwanted ---------------- -Another tricky thing is fully repopulating the working directory when you -no longer want sparse checkout. You cannot just disable "sparse -checkout" because skip-worktree bits are still in the index and your working -directory is still sparsely populated. You should re-populate the working -directory with the `$GIT_DIR/info/sparse-checkout` file content as -follows: - ----------------- -/* ----------------- - -Then you can disable sparse checkout. Sparse checkout support in 'git -checkout' and similar commands is disabled by default. You need to -set `core.sparseCheckout` to `true` in order to have sparse checkout -support. SEE ALSO -------- diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c index f2e2bd772d..9fdcc6c4ef 100644 --- a/builtin/sparse-checkout.c +++ b/builtin/sparse-checkout.c @@ -8,7 +8,7 @@ #include "strbuf.h" static char const * const builtin_sparse_checkout_usage[] = { - N_("git sparse-checkout (init|list|set) "), + N_("git sparse-checkout (init|list|set|disable) "), NULL }; @@ -212,6 +212,28 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix) return result; } +static int sparse_checkout_disable(int argc, const char **argv) +{ + char *sparse_filename; + FILE *fp; + + if (sc_set_config(MODE_ALL_PATTERNS)) + die(_("failed to change config")); + + sparse_filename = get_sparse_checkout_filename(); + fp = xfopen(sparse_filename, "w"); + fprintf(fp, "/*\n"); + fclose(fp); + + if (update_working_directory()) + die(_("error while refreshing working directory")); + + unlink(sparse_filename); + free(sparse_filename); + + return sc_set_config(MODE_NO_PATTERNS); +} + int cmd_sparse_checkout(int argc, const char **argv, const char *prefix) { static struct option builtin_sparse_checkout_options[] = { @@ -236,6 +258,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix) return sparse_checkout_init(argc, argv); if (!strcmp(argv[0], "set")) return sparse_checkout_set(argc, argv, prefix); + if (!strcmp(argv[0], "disable")) + return sparse_checkout_disable(argc, argv); } usage_with_options(builtin_sparse_checkout_usage, diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh index a9ff5eb9ec..583c232e9e 100755 --- a/t/t1091-sparse-checkout-builtin.sh +++ b/t/t1091-sparse-checkout-builtin.sh @@ -148,4 +148,19 @@ test_expect_success 'set sparse-checkout using --stdin' ' test_cmp expect dir ' +test_expect_success 'sparse-checkout disable' ' + git -C repo sparse-checkout disable && + test_path_is_missing repo/.git/info/sparse-checkout && + git -C repo config --list >config && + test_i18ngrep "core.sparsecheckout=false" config && + ls repo >dir && + cat >expect <<-EOF && + a + deep + folder1 + folder2 + EOF + test_cmp expect dir +' + test_done From patchwork Mon Oct 21 13:56:16 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11202245 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D4C6D139A for ; Mon, 21 Oct 2019 13:56:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id B2A35214B2 for ; Mon, 21 Oct 2019 13:56:43 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="LIMogeRr" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729256AbfJUN4m (ORCPT ); Mon, 21 Oct 2019 09:56:42 -0400 Received: from mail-wr1-f67.google.com ([209.85.221.67]:45830 "EHLO mail-wr1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727344AbfJUN4h (ORCPT ); Mon, 21 Oct 2019 09:56:37 -0400 Received: by mail-wr1-f67.google.com with SMTP id q13so9141668wrs.12 for ; Mon, 21 Oct 2019 06:56:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=jp9fCPiVi8uv07s6DelseXp9idVSa4eBTh9PuKubi+E=; b=LIMogeRrJBStiml1i72o2K1ZI6soO3tSkstfApd+NCCFrUgLvuFIDeAvIn5DS6nCov HFMiQITa8lcqBaEk69WVoHlr2TIbAEhfjJuMl2Z4bcDfQokkmjLQ85eqw3d7cl4MFHtj LLjeWbDuKoBl/FJGdg68oXUuJn3xUnzG5x98yFgZ+njlWyWnsL2vexPZB+wZnMzk2dKk eY7wGhOcQeeYYiCoM6gV70xLd87vIWMSe3rDUlXsulH5xkXdSGsI5RZkMYWepKAAsdAT wJLhN3Fjh4vgK0qlMnWq8zc3PjrXvK5uM9CJaUAxzACQlV3Ty/aNo9pKppsRWcb16RXU qOkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=jp9fCPiVi8uv07s6DelseXp9idVSa4eBTh9PuKubi+E=; b=ipQH/CrhlsmQdEPrvKN8P6MOMF5icvKVQ9uegddJSCpaZKK9qUfzo++QrtyWC8xRVE ogWUugedZfS8wU3zYjAo91Kgu3YcacuxhPymYgF8/D2OGsEnL74RbUGC2Dsc4/qb5AHO tXVFCUxOdPdcFoOb5SdEbtWTHrGVmaJr0qxo7yIPRGF4BSqNrDk1A2NGrlUl0RZHQSXo PONN+9tQlgNgLSbOAtdHCNhjde9P99v1pB/JBKu/UJ2q4RfJ3aEOKrHZv87r+zL+cMSV QCTy3d1yeClqBzTeERLiG7MFonoVa1az1YI8l6cMnU/uzbDF2GlNjpSzse3USrhT2Sx4 ZoNw== X-Gm-Message-State: APjAAAXsJoGUbUStHmk1hknCqLnH7OqDLz+NLcTBBqik958RKkKNHAnK Nq3qTrG5ZuqTkhFo6u2BNYw8rRCf X-Google-Smtp-Source: APXvYqx7I2dHm0yvqyikfvZHiYrwXupLebr8Y+Xm30Yo6N5oHjJzzqlGwJzW4gPjFIXXPoo28iNrhw== X-Received: by 2002:adf:dbd2:: with SMTP id e18mr20145846wrj.268.1571666194613; Mon, 21 Oct 2019 06:56:34 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id e3sm13808906wme.39.2019.10.21.06.56.34 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 21 Oct 2019 06:56:34 -0700 (PDT) Message-Id: <44b0245e65f54cbcec743184886d64a6b5ad87cf.1571666187.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Jeff Hostetler via GitGitGadget" Date: Mon, 21 Oct 2019 13:56:16 +0000 Subject: [PATCH v5 07/17] trace2: add region in clear_ce_flags Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, jon@jonsimons.org, szeder.dev@gmail.com, Derrick Stolee , Junio C Hamano , Jeff Hostetler Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler When Git updates the working directory with the sparse-checkout feature enabled, the unpack_trees() method calls clear_ce_flags() to update the skip-wortree bits on the cache entries. This check can be expensive, depending on the patterns used. Add trace2 regions around the method, including some flag information, so we can get granular performance data during experiments. This data will be used to measure improvements to the pattern-matching algorithms for sparse-checkout. Signed-off-by: Jeff Hostetler Signed-off-by: Derrick Stolee --- unpack-trees.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/unpack-trees.c b/unpack-trees.c index 33ea7810d8..01a05ff66d 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -1407,15 +1407,23 @@ static int clear_ce_flags(struct index_state *istate, struct pattern_list *pl) { static struct strbuf prefix = STRBUF_INIT; + char label[100]; + int rval; strbuf_reset(&prefix); - return clear_ce_flags_1(istate, + xsnprintf(label, sizeof(label), "clear_ce_flags(0x%08lx,0x%08lx)", + (unsigned long)select_mask, (unsigned long)clear_mask); + trace2_region_enter("unpack_trees", label, the_repository); + rval = clear_ce_flags_1(istate, istate->cache, istate->cache_nr, &prefix, select_mask, clear_mask, pl, 0); + trace2_region_leave("unpack_trees", label, the_repository); + + return rval; } /* From patchwork Mon Oct 21 13:56:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11202243 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B93601390 for ; Mon, 21 Oct 2019 13:56:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 8DB8A2166E for ; Mon, 21 Oct 2019 13:56:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="aaBk+Sva" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729250AbfJUN4l (ORCPT ); Mon, 21 Oct 2019 09:56:41 -0400 Received: from mail-wm1-f65.google.com ([209.85.128.65]:38493 "EHLO mail-wm1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729216AbfJUN4j (ORCPT ); Mon, 21 Oct 2019 09:56:39 -0400 Received: by mail-wm1-f65.google.com with SMTP id 3so12920738wmi.3 for ; Mon, 21 Oct 2019 06:56:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=HPCoGDbDPcDqC/pbjT+30VDXyZIKnsXuCsp2/MBOYfs=; b=aaBk+Sva/wOHqr5lJv/j20KbP/kGVxqjLP4RMdPDMcsrhkg6a+YBSM//Z6Mm6tJW3r /pdNtmOwXf2jz8y1FWsjcm6PGup346cZawZvdgtiwI03IKBos7KhNBR39+lL6jM+D5Fz QrY1r3OT4JwjJEs8B81xJ00uX6zPKXO2BKIqkeiM2by8ixNyD+JEvAv8I7jnletBJu0B rM3yx46+AliNNTwmMLJUlagDnOWOEXfNaNh3Alv+m8J90LEiKva+1q/d7Js6qDGiwGeG eZlBxoHIsbOrKuMp2PFyRuk/sY0oMhjo/BuBVuCxWgn1zYaB+qL0YupaGvBwtbdcQkGC ut4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=HPCoGDbDPcDqC/pbjT+30VDXyZIKnsXuCsp2/MBOYfs=; b=B+Iu0o8fuKDocN43TkBeZDg4DvAAUpyjX3eKURrI6UyuT+MAxvHjIauSWscP5qUl7s Rzzd2qg0zZrL5Rlx5fFSUtpo3NGzEd9IOq2XZGZsxpi1t2ySdguEJltVYJ6zuEPPQedw OXScgQ7R9hpWqcbDnbKpzf2ES4+fkd8PADEhkCTnWaK/k/9fZwIYKZJjr8zxbbLcmYBE LoCsaoSwhK1wL3mSXu7Cc4zNTeHEk+qpCv605aK7TP4TEKxhyn0Bn5LOl4vNqoOs5vn+ y9P3oCI9Yp3nketL96qy4QMZeAhBJ7d26pIg8XkabhXdE0Xjbi5NCwTGaNDFcPu6/hgY kMXQ== X-Gm-Message-State: APjAAAXbO1tX3TJDGnIhE0uWbDG0yCfD63pPXgpHCsh6yJEcXjd+qdSg jth83xEQkCoEnJmeLm7eqhhAku0K X-Google-Smtp-Source: APXvYqxkwReR32ocqSqlMx7Z+GC+g4iRcQHlS20K8XXWnAg2E5HeFfaxKgOj6Rj8MS20Rc5/KRWKkA== X-Received: by 2002:a1c:c912:: with SMTP id f18mr7480422wmb.168.1571666195413; Mon, 21 Oct 2019 06:56:35 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id w17sm15998321wra.34.2019.10.21.06.56.34 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 21 Oct 2019 06:56:34 -0700 (PDT) Message-Id: <121d878882609907cc8d13f9d76f2414b15618be.1571666187.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Mon, 21 Oct 2019 13:56:17 +0000 Subject: [PATCH v5 08/17] sparse-checkout: add 'cone' mode Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, jon@jonsimons.org, szeder.dev@gmail.com, Derrick Stolee , Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee The sparse-checkout feature can have quadratic performance as the number of patterns and number of entries in the index grow. If there are 1,000 patterns and 1,000,000 entries, this time can be very significant. Create a new Boolean config option, core.sparseCheckoutCone, to indicate that we expect the sparse-checkout file to contain a more limited set of patterns. This is a separate config setting from core.sparseCheckout to avoid breaking older clients by introducing a tri-state option. The config option does nothing right now, but will be expanded upon in a later commit. Signed-off-by: Derrick Stolee --- Documentation/config/core.txt | 10 ++++-- Documentation/git-sparse-checkout.txt | 50 +++++++++++++++++++++++++++ cache.h | 4 ++- config.c | 5 +++ environment.c | 1 + t/t1091-sparse-checkout-builtin.sh | 14 ++++++++ 6 files changed, 81 insertions(+), 3 deletions(-) diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index 852d2ba37a..bdbbee58b9 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -593,8 +593,14 @@ core.multiPackIndex:: multi-pack-index design document]. core.sparseCheckout:: - Enable "sparse checkout" feature. See section "Sparse checkout" in - linkgit:git-read-tree[1] for more information. + Enable "sparse checkout" feature. See linkgit:git-sparse-checkout[1] + for more information. + +core.sparseCheckoutCone:: + Enables the "cone mode" of the sparse checkout feature. When the + sparse-checkout file contains a limited set of patterns, then this + mode provides significant performance advantages. See + linkgit:git-sparse-checkout[1] for more information. core.abbrev:: Set the length object names are abbreviated to. If diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt index f794d4797a..3931e4f2ad 100644 --- a/Documentation/git-sparse-checkout.txt +++ b/Documentation/git-sparse-checkout.txt @@ -92,6 +92,56 @@ using negative patterns. For example, to remove the file `unwanted`: ---------------- +## CONE PATTERN SET + +The full pattern set allows for arbitrary pattern matches and complicated +inclusion/exclusion rules. These can result in O(N*M) pattern matches when +updating the index, where N is the number of patterns and M is the number +of paths in the index. To combat this performance issue, a more restricted +pattern set is allowed when `core.spareCheckoutCone` is enabled. + +The accepted patterns in the cone pattern set are: + +1. *Recursive:* All paths inside a directory are included. + +2. *Parent:* All files immediately inside a directory are included. + +In addition to the above two patterns, we also expect that all files in the +root directory are included. If a recursive pattern is added, then all +leading directories are added as parent patterns. + +By default, when running `git sparse-checkout init`, the root directory is +added as a parent pattern. At this point, the sparse-checkout file contains +the following patterns: + +``` +/* +!/*/ +``` + +This says "include everything in root, but nothing two levels below root." +If we then add the folder `A/B/C` as a recursive pattern, the folders `A` and +`A/B` are added as parent patterns. The resulting sparse-checkout file is +now + +``` +/* +!/*/ +/A/ +!/A/*/ +/A/B/ +!/A/B/*/ +/A/B/C/ +``` + +Here, order matters, so the negative patterns are overridden by the positive +patterns that appear lower in the file. + +If `core.sparseCheckoutCone=true`, then Git will parse the sparse-checkout file +expecting patterns of these types. Git will warn if the patterns do not match. +If the patterns do match the expected format, then Git will use faster hash- +based algorithms to compute inclusion in the sparse-checkout. + SEE ALSO -------- diff --git a/cache.h b/cache.h index 04cabaac11..4980ee198e 100644 --- a/cache.h +++ b/cache.h @@ -918,12 +918,14 @@ extern char *git_replace_ref_base; extern int fsync_object_files; extern int core_preload_index; -extern int core_apply_sparse_checkout; extern int precomposed_unicode; extern int protect_hfs; extern int protect_ntfs; extern const char *core_fsmonitor; +int core_apply_sparse_checkout; +int core_sparse_checkout_cone; + /* * Include broken refs in all ref iterations, which will * generally choke dangerous operations rather than letting diff --git a/config.c b/config.c index e7052b3977..d75f88ca0c 100644 --- a/config.c +++ b/config.c @@ -1364,6 +1364,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb) return 0; } + if (!strcmp(var, "core.sparsecheckoutcone")) { + core_sparse_checkout_cone = git_config_bool(var, value); + return 0; + } + if (!strcmp(var, "core.precomposeunicode")) { precomposed_unicode = git_config_bool(var, value); return 0; diff --git a/environment.c b/environment.c index efa072680a..2a1a866659 100644 --- a/environment.c +++ b/environment.c @@ -67,6 +67,7 @@ enum object_creation_mode object_creation_mode = OBJECT_CREATION_MODE; char *notes_ref_name; int grafts_replace_parents = 1; int core_apply_sparse_checkout; +int core_sparse_checkout_cone; int merge_log_config = -1; int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */ unsigned long pack_size_limit_cfg; diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh index 583c232e9e..b3058d0e9b 100755 --- a/t/t1091-sparse-checkout-builtin.sh +++ b/t/t1091-sparse-checkout-builtin.sh @@ -148,6 +148,20 @@ test_expect_success 'set sparse-checkout using --stdin' ' test_cmp expect dir ' +test_expect_success 'cone mode: match patterns' ' + git -C repo config --worktree core.sparseCheckoutCone true && + rm -rf repo/a repo/folder1 repo/folder2 && + git -C repo read-tree -mu HEAD && + git -C repo reset --hard && + ls repo >dir && + cat >expect <<-EOF && + a + folder1 + folder2 + EOF + test_cmp expect dir +' + test_expect_success 'sparse-checkout disable' ' git -C repo sparse-checkout disable && test_path_is_missing repo/.git/info/sparse-checkout && From patchwork Mon Oct 21 13:56:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11202239 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B0A4014ED for ; Mon, 21 Oct 2019 13:56:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 85CD22166E for ; Mon, 21 Oct 2019 13:56:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Z2/ilT1C" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729240AbfJUN4k (ORCPT ); Mon, 21 Oct 2019 09:56:40 -0400 Received: from mail-wr1-f68.google.com ([209.85.221.68]:44066 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729207AbfJUN4i (ORCPT ); Mon, 21 Oct 2019 09:56:38 -0400 Received: by mail-wr1-f68.google.com with SMTP id z9so14108139wrl.11 for ; Mon, 21 Oct 2019 06:56:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=8OhQ9r8zuEoxDd6lz5n93F9RCFeHfZvMfPp1iLRHQs4=; b=Z2/ilT1C+XDTiU0xYqO6obWtr55w9hjNIDgPkF5EAGCy+LfneYFiqUN+5WBYl0qQZD V/47XHDjewAls/sz3h3SY8y2WS43WgCna5rXtgHeBB5XqMQV2UyZ4w/dUs4n3qSytPdH YhUajhoYq+OVnxDhcidUFSN81McwiZ/KWZm883/Wb3XL+Jcy8unYZikfc8vYyHsBKoqy kScXabHjJni7PbHXA7OJoKIIPSmXLrtorlpChPst60Q4ojKUJqa5AzdXS5u7vbq8vc6/ XLRWLKieLYE7A/zgKJsYDXQ67QXTOQmOWh4NErd/grzXmUzT8jx+Z+5J53ll6EXy84l1 Z26A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=8OhQ9r8zuEoxDd6lz5n93F9RCFeHfZvMfPp1iLRHQs4=; b=PR1XsdUu6s0YfTROZKi8WXSq/Rb8BnZobB3dizQkoQMETWKF1jtJuNWuwDoBAj8z6d OMhmeWDUuIthizAeiYfBy3n+o9g2tweHUGTn4UJvE94GRvMqbuUzB2r6hEXSoJK6+j9/ ImR3zALXBUn8QNh3k5OPxFm7ejrM/z4DWvWIzipCTHE4eQxbUPNbyO2RuYUfr12/AS7P mKAnOkGuzbIlRT7AWbk7Woimg9fQOGjiBoRXQ+8/eD+OpyBwlkLFpCsaT3BO8+HnVfS3 7bOKe0dCXCqG4/jSlW2ToHnzl/3vJ05yMv2OzDyrQ23U7Uf9A/hFruIY4Z6Uopx9wOFN UJbw== X-Gm-Message-State: APjAAAX8jhH+coDaJI0+48SxVkqf5Qf+GuTqVviLyNfBwzAdANXJSFpY NawL3oPrppj8cAsG9mSH2XUKFwgC X-Google-Smtp-Source: APXvYqyIAUokdKnHSLGpb+lXFnte8NZ7Z/DCgy9XKL05gMi3xX3BEMeIbOQzEZlVIz68RHhhZzFS2g== X-Received: by 2002:adf:8289:: with SMTP id 9mr21146007wrc.0.1571666196132; Mon, 21 Oct 2019 06:56:36 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 36sm17708333wrp.30.2019.10.21.06.56.35 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 21 Oct 2019 06:56:35 -0700 (PDT) Message-Id: In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Mon, 21 Oct 2019 13:56:18 +0000 Subject: [PATCH v5 09/17] sparse-checkout: use hashmaps for cone patterns Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, jon@jonsimons.org, szeder.dev@gmail.com, Derrick Stolee , Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee The parent and recursive patterns allowed by the "cone mode" option in sparse-checkout are restrictive enough that we can avoid using the regex parsing. Everything is based on prefix matches, so we can use hashsets to store the prefixes from the sparse-checkout file. When checking a path, we can strip path entries from the path and check the hashset for an exact match. As a test, I created a cone-mode sparse-checkout file for the Linux repository that actually includes every file. This was constructed by taking every folder in the Linux repo and creating the pattern pairs here: /$folder/ !/$folder/*/ This resulted in a sparse-checkout file sith 8,296 patterns. Running 'git read-tree -mu HEAD' on this file had the following performance: core.sparseCheckout=false: 0.21 s (0.00 s) core.sparseCheckout=true: 3.75 s (3.50 s) core.sparseCheckoutCone=true: 0.23 s (0.01 s) The times in parentheses above correspond to the time spent in the first clear_ce_flags() call, according to the trace2 performance traces. While this example is contrived, it demonstrates how these patterns can slow the sparse-checkout feature. Helped-by: Eric Wong Helped-by: Johannes Schindelin Signed-off-by: Derrick Stolee --- dir.c | 207 +++++++++++++++++++++++++++-- dir.h | 31 +++++ t/t1091-sparse-checkout-builtin.sh | 11 +- unpack-trees.c | 1 + 4 files changed, 241 insertions(+), 9 deletions(-) diff --git a/dir.c b/dir.c index 61f559f980..dfabf9982f 100644 --- a/dir.c +++ b/dir.c @@ -611,6 +611,150 @@ void parse_path_pattern(const char **pattern, *patternlen = len; } +static int pl_hashmap_cmp(const void *unused_cmp_data, + const struct hashmap_entry *a, + const struct hashmap_entry *b, + const void *key) +{ + const struct pattern_entry *ee1 = + container_of(a, struct pattern_entry, ent); + const struct pattern_entry *ee2 = + container_of(b, struct pattern_entry, ent); + + size_t min_len = ee1->patternlen <= ee2->patternlen + ? ee1->patternlen + : ee2->patternlen; + + return strncmp(ee1->pattern, ee2->pattern, min_len); +} + +static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern *given) +{ + struct pattern_entry *translated; + char *truncated; + char *data = NULL; + + if (!pl->use_cone_patterns) + return; + + if (given->flags & PATTERN_FLAG_NEGATIVE && + given->flags & PATTERN_FLAG_MUSTBEDIR && + !strcmp(given->pattern, "/*")) { + pl->full_cone = 0; + return; + } + + if (!given->flags && !strcmp(given->pattern, "/*")) { + pl->full_cone = 1; + return; + } + + if (given->patternlen > 2 && + !strcmp(given->pattern + given->patternlen - 2, "/*")) { + if (!(given->flags & PATTERN_FLAG_NEGATIVE)) { + /* Not a cone pattern. */ + pl->use_cone_patterns = 0; + warning(_("unrecognized pattern: '%s'"), given->pattern); + goto clear_hashmaps; + } + + truncated = xstrdup(given->pattern); + truncated[given->patternlen - 2] = 0; + + translated = xmalloc(sizeof(struct pattern_entry)); + translated->pattern = truncated; + translated->patternlen = given->patternlen - 2; + hashmap_entry_init(&translated->ent, + memhash(translated->pattern, translated->patternlen)); + + if (!hashmap_get_entry(&pl->recursive_hashmap, + translated, ent, NULL)) { + /* We did not see the "parent" included */ + warning(_("unrecognized negative pattern: '%s'"), + given->pattern); + free(truncated); + free(translated); + goto clear_hashmaps; + } + + hashmap_add(&pl->parent_hashmap, &translated->ent); + hashmap_remove(&pl->recursive_hashmap, &translated->ent, &data); + free(data); + return; + } + + if (given->flags & PATTERN_FLAG_NEGATIVE) { + warning(_("unrecognized negative pattern: '%s'"), + given->pattern); + goto clear_hashmaps; + } + + translated = xmalloc(sizeof(struct pattern_entry)); + + translated->pattern = xstrdup(given->pattern); + translated->patternlen = given->patternlen; + hashmap_entry_init(&translated->ent, + memhash(translated->pattern, translated->patternlen)); + + hashmap_add(&pl->recursive_hashmap, &translated->ent); + + if (hashmap_get_entry(&pl->parent_hashmap, translated, ent, NULL)) { + /* we already included this at the parent level */ + warning(_("your sparse-checkout file may have issues: pattern '%s' is repeated"), + given->pattern); + hashmap_remove(&pl->parent_hashmap, &translated->ent, &data); + free(data); + free(translated); + } + + return; + +clear_hashmaps: + warning(_("disabling cone pattern matching")); + hashmap_free_entries(&pl->parent_hashmap, struct pattern_entry, ent); + hashmap_free_entries(&pl->recursive_hashmap, struct pattern_entry, ent); + pl->use_cone_patterns = 0; +} + +static int hashmap_contains_path(struct hashmap *map, + struct strbuf *pattern) +{ + struct pattern_entry p; + + /* Check straight mapping */ + p.pattern = pattern->buf; + p.patternlen = pattern->len; + hashmap_entry_init(&p.ent, memhash(p.pattern, p.patternlen)); + return !!hashmap_get_entry(map, &p, ent, NULL); +} + +int hashmap_contains_parent(struct hashmap *map, + const char *path, + struct strbuf *buffer) +{ + char *slash_pos; + + strbuf_setlen(buffer, 0); + + if (path[0] != '/') + strbuf_addch(buffer, '/'); + + strbuf_addstr(buffer, path); + + slash_pos = strrchr(buffer->buf, '/'); + + while (slash_pos > buffer->buf) { + strbuf_setlen(buffer, slash_pos - buffer->buf); + + if (hashmap_contains_path(map, buffer)) + return 1; + + slash_pos = strrchr(buffer->buf, '/'); + } + + return 0; +} + void add_pattern(const char *string, const char *base, int baselen, struct pattern_list *pl, int srcpos) { @@ -635,6 +779,8 @@ void add_pattern(const char *string, const char *base, ALLOC_GROW(pl->patterns, pl->nr + 1, pl->alloc); pl->patterns[pl->nr++] = pattern; pattern->pl = pl; + + add_pattern_to_hashsets(pl, pattern); } static int read_skip_worktree_file_from_index(const struct index_state *istate, @@ -860,6 +1006,9 @@ static int add_patterns_from_buffer(char *buf, size_t size, int i, lineno = 1; char *entry; + hashmap_init(&pl->recursive_hashmap, pl_hashmap_cmp, NULL, 0); + hashmap_init(&pl->parent_hashmap, pl_hashmap_cmp, NULL, 0); + pl->filebuf = buf; if (skip_utf8_bom(&buf, size)) @@ -1096,16 +1245,58 @@ enum pattern_match_result path_matches_pattern_list( struct index_state *istate) { struct path_pattern *pattern; - pattern = last_matching_pattern_from_list(pathname, pathlen, basename, - dtype, pl, istate); - if (pattern) { - if (pattern->flags & PATTERN_FLAG_NEGATIVE) - return NOT_MATCHED; - else - return MATCHED; + struct strbuf parent_pathname = STRBUF_INIT; + int result = NOT_MATCHED; + const char *slash_pos; + + if (!pl->use_cone_patterns) { + pattern = last_matching_pattern_from_list(pathname, pathlen, basename, + dtype, pl, istate); + if (pattern) { + if (pattern->flags & PATTERN_FLAG_NEGATIVE) + return NOT_MATCHED; + else + return MATCHED; + } + + return UNDECIDED; + } + + if (pl->full_cone) + return MATCHED; + + strbuf_addch(&parent_pathname, '/'); + strbuf_add(&parent_pathname, pathname, pathlen); + + if (hashmap_contains_path(&pl->recursive_hashmap, + &parent_pathname)) { + result = MATCHED; + goto done; + } + + slash_pos = strrchr(parent_pathname.buf, '/'); + + if (slash_pos == parent_pathname.buf) { + /* include every file in root */ + result = MATCHED; + goto done; } - return UNDECIDED; + strbuf_setlen(&parent_pathname, slash_pos - parent_pathname.buf); + + if (hashmap_contains_path(&pl->parent_hashmap, &parent_pathname)) { + result = MATCHED; + goto done; + } + + if (hashmap_contains_parent(&pl->recursive_hashmap, + pathname, + &parent_pathname)) + result = MATCHED; + +done: + strbuf_release(&parent_pathname); + return result; } static struct path_pattern *last_matching_pattern_from_lists( diff --git a/dir.h b/dir.h index 2fbdef014f..f8edbca72b 100644 --- a/dir.h +++ b/dir.h @@ -4,6 +4,7 @@ /* See Documentation/technical/api-directory-listing.txt */ #include "cache.h" +#include "hashmap.h" #include "strbuf.h" struct dir_entry { @@ -37,6 +38,13 @@ struct path_pattern { int srcpos; }; +/* used for hashmaps for cone patterns */ +struct pattern_entry { + struct hashmap_entry ent; + char *pattern; + size_t patternlen; +}; + /* * Each excludes file will be parsed into a fresh exclude_list which * is appended to the relevant exclude_list_group (either EXC_DIRS or @@ -55,6 +63,26 @@ struct pattern_list { const char *src; struct path_pattern **patterns; + + /* + * While scanning the excludes, we attempt to match the patterns + * with a more restricted set that allows us to use hashsets for + * matching logic, which is faster than the linear lookup in the + * excludes array above. If non-zero, that check succeeded. + */ + unsigned use_cone_patterns; + unsigned full_cone; + + /* + * Stores paths where everything starting with those paths + * is included. + */ + struct hashmap recursive_hashmap; + + /* + * Used to check single-level parents of blobs. + */ + struct hashmap parent_hashmap; }; /* @@ -271,6 +299,9 @@ int is_excluded(struct dir_struct *dir, struct index_state *istate, const char *name, int *dtype); +int hashmap_contains_parent(struct hashmap *map, + const char *path, + struct strbuf *buffer); struct pattern_list *add_pattern_list(struct dir_struct *dir, int group_type, const char *src); int add_patterns_from_file_to_list(const char *fname, const char *base, int baselen, diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh index b3058d0e9b..9907278fc1 100755 --- a/t/t1091-sparse-checkout-builtin.sh +++ b/t/t1091-sparse-checkout-builtin.sh @@ -151,7 +151,8 @@ test_expect_success 'set sparse-checkout using --stdin' ' test_expect_success 'cone mode: match patterns' ' git -C repo config --worktree core.sparseCheckoutCone true && rm -rf repo/a repo/folder1 repo/folder2 && - git -C repo read-tree -mu HEAD && + git -C repo read-tree -mu HEAD 2>err && + test_i18ngrep ! "disabling cone patterns" err && git -C repo reset --hard && ls repo >dir && cat >expect <<-EOF && @@ -162,6 +163,14 @@ test_expect_success 'cone mode: match patterns' ' test_cmp expect dir ' +test_expect_success 'cone mode: warn on bad pattern' ' + test_when_finished mv sparse-checkout repo/.git/info/ && + cp repo/.git/info/sparse-checkout . && + echo "!/deep/deeper/*" >>repo/.git/info/sparse-checkout && + git -C repo read-tree -mu HEAD 2>err && + test_i18ngrep "unrecognized negative pattern" err +' + test_expect_success 'sparse-checkout disable' ' git -C repo sparse-checkout disable && test_path_is_missing repo/.git/info/sparse-checkout && diff --git a/unpack-trees.c b/unpack-trees.c index 01a05ff66d..a90d71845d 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -1482,6 +1482,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options o->skip_sparse_checkout = 1; if (!o->skip_sparse_checkout) { char *sparse = git_pathdup("info/sparse-checkout"); + pl.use_cone_patterns = core_sparse_checkout_cone; if (add_patterns_from_file_to_list(sparse, "", 0, &pl, NULL) < 0) o->skip_sparse_checkout = 1; else From patchwork Mon Oct 21 13:56:19 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11202247 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 45BCA14ED for ; Mon, 21 Oct 2019 13:56:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 197E4214B2 for ; Mon, 21 Oct 2019 13:56:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="k/R3ELZH" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729260AbfJUN4n (ORCPT ); Mon, 21 Oct 2019 09:56:43 -0400 Received: from mail-wr1-f65.google.com ([209.85.221.65]:38740 "EHLO mail-wr1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727152AbfJUN4k (ORCPT ); Mon, 21 Oct 2019 09:56:40 -0400 Received: by mail-wr1-f65.google.com with SMTP id v9so2863661wrq.5 for ; Mon, 21 Oct 2019 06:56:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=mO+or5JG+DSecj8vOEGlNCblM9TvT5SCVngeLbH9uOo=; b=k/R3ELZHa4emUUetMIo3JaSkUAuUdSV++2N2t/worEtWvK0laBXicbzFOe9DT4Ovu+ 1R0s+6ZSHOwwP1CK6U7Frmrsx9AZazag7HbsqzwnYHcnxGHGohTzuBX+1qEIAubv0vVd gio7AYx5Jjg/oR0edx+PTiKImqCuTlBVru+AAhqk1uiEBiDJASsk/q4hhxijlCwqHyFj 37iHzKccfNa7O9KjFyjc+WGW4VugY2qJyRepaKnCVBThkV4EkQn+TV//ht3Z+35pKlem n1sam/Z0OnIlYg/LhoSX3f5ni7bAQRjjNLAcV/oTJZsxx7HOra8Z7rC8ZTgz2ONvGFtP 3X8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=mO+or5JG+DSecj8vOEGlNCblM9TvT5SCVngeLbH9uOo=; b=hkJmC+kb1MbrnhW4ubeMN33aS9y6dWWU1K5h90c4O/OuQ5+UHlzj1pktIC0gdN56NP EEslNhz50t8PCtHOmiQEeP86J35P3GLrMo2VJF2n7sTsLcUdZEsPyIPFPcb1V6U3mheN tLz3Jamg5Nn4EFpfFdb0Z7tT+OMWhETpMRetKyW4EZFsZMqPUv7kr9Y58JrPRMfWX9Ys VpfGveGLqJF5Vp4wnkddTKPjtK3ww002z/BDhj+IIgegPSPH9N0H2ZZ5u5SyXYERa5il XM1PRzyC5dW8j9y89FXlOTf8YItGHvmxNLPFh6iRh38LpAELYZOdRF8VKs1ilwF5rv8R s+oA== X-Gm-Message-State: APjAAAVY825voFMo3BQzM5NoBjxQF9NiBLxcGvMprJZtZm6Xxo/HuiP4 gFXgvHv6UZRO4kWrfjTyHl0cjXWf X-Google-Smtp-Source: APXvYqylKRtp4aIF/cxXMAdp4midmFrJA6pS6HuyO2wfHyr0rlSQzexSpvfRH0GuyhASlwlazNabeA== X-Received: by 2002:adf:f004:: with SMTP id j4mr21589039wro.68.1571666196814; Mon, 21 Oct 2019 06:56:36 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id x21sm1440345wmj.42.2019.10.21.06.56.36 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 21 Oct 2019 06:56:36 -0700 (PDT) Message-Id: <0258ee80265f5f27a7de9b81eaf166648b4511d4.1571666187.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Mon, 21 Oct 2019 13:56:19 +0000 Subject: [PATCH v5 10/17] sparse-checkout: init and set in cone mode Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, jon@jonsimons.org, szeder.dev@gmail.com, Derrick Stolee , Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee To make the cone pattern set easy to use, update the behavior of 'git sparse-checkout [init|set]'. Add '--cone' flag to 'git sparse-checkout init' to set the config option 'core.sparseCheckoutCone=true'. When running 'git sparse-checkout set' in cone mode, a user only needs to supply a list of recursive folder matches. Git will automatically add the necessary parent matches for the leading directories. When testing 'git sparse-checkout set' in cone mode, check the error stream to ensure we do not see any errors. Specifically, we want to avoid the warning that the patterns do not match the cone-mode patterns. Helped-by: Eric Wong Helped-by: Johannes Schindelin Signed-off-by: Derrick Stolee --- builtin/sparse-checkout.c | 159 +++++++++++++++++++++++++++-- dir.c | 8 +- dir.h | 4 + t/t1091-sparse-checkout-builtin.sh | 51 +++++++++ 4 files changed, 208 insertions(+), 14 deletions(-) diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c index 9fdcc6c4ef..6c3bf68c89 100644 --- a/builtin/sparse-checkout.c +++ b/builtin/sparse-checkout.c @@ -6,6 +6,7 @@ #include "repository.h" #include "run-command.h" #include "strbuf.h" +#include "string-list.h" static char const * const builtin_sparse_checkout_usage[] = { N_("git sparse-checkout (init|list|set|disable) "), @@ -77,11 +78,13 @@ static int update_working_directory(void) enum sparse_checkout_mode { MODE_NO_PATTERNS = 0, MODE_ALL_PATTERNS = 1, + MODE_CONE_PATTERNS = 2, }; static int sc_set_config(enum sparse_checkout_mode mode) { struct argv_array argv = ARGV_ARRAY_INIT; + struct argv_array cone_argv = ARGV_ARRAY_INIT; if (git_config_set_gently("extensions.worktreeConfig", "true")) { error(_("failed to set extensions.worktreeConfig setting")); @@ -100,9 +103,31 @@ static int sc_set_config(enum sparse_checkout_mode mode) return 1; } + argv_array_pushl(&cone_argv, "config", "--worktree", + "core.sparseCheckoutCone", NULL); + + if (mode == MODE_CONE_PATTERNS) + argv_array_push(&cone_argv, "true"); + else + argv_array_push(&cone_argv, "false"); + + if (run_command_v_opt(cone_argv.argv, RUN_GIT_CMD)) { + error(_("failed to enable core.sparseCheckoutCone")); + return 1; + } + return 0; } +static char const * const builtin_sparse_checkout_init_usage[] = { + N_("git sparse-checkout init [--cone]"), + NULL +}; + +static struct sparse_checkout_init_opts { + int cone_mode; +} init_opts; + static int sparse_checkout_init(int argc, const char **argv) { struct pattern_list pl; @@ -110,8 +135,21 @@ static int sparse_checkout_init(int argc, const char **argv) FILE *fp; int res; struct object_id oid; + int mode; - if (sc_set_config(MODE_ALL_PATTERNS)) + static struct option builtin_sparse_checkout_init_options[] = { + OPT_BOOL(0, "cone", &init_opts.cone_mode, + N_("initialize the sparse-checkout in cone mode")), + OPT_END(), + }; + + argc = parse_options(argc, argv, NULL, + builtin_sparse_checkout_init_options, + builtin_sparse_checkout_init_usage, 0); + + mode = init_opts.cone_mode ? MODE_CONE_PATTERNS : MODE_ALL_PATTERNS; + + if (sc_set_config(mode)) return 1; memset(&pl, 0, sizeof(pl)); @@ -140,6 +178,70 @@ static int sparse_checkout_init(int argc, const char **argv) return update_working_directory(); } +static void insert_recursive_pattern(struct pattern_list *pl, struct strbuf *path) +{ + struct pattern_entry *e = xmalloc(sizeof(*e)); + e->patternlen = path->len; + e->pattern = strbuf_detach(path, NULL); + hashmap_entry_init(&e->ent, memhash(e->pattern, e->patternlen)); + + hashmap_add(&pl->recursive_hashmap, &e->ent); + + while (e->patternlen) { + char *slash = strrchr(e->pattern, '/'); + char *oldpattern = e->pattern; + size_t newlen; + + if (slash == e->pattern) + break; + + newlen = slash - e->pattern; + e = xmalloc(sizeof(struct pattern_entry)); + e->patternlen = newlen; + e->pattern = xstrndup(oldpattern, newlen); + hashmap_entry_init(&e->ent, memhash(e->pattern, e->patternlen)); + + if (!hashmap_get_entry(&pl->parent_hashmap, e, ent, NULL)) + hashmap_add(&pl->parent_hashmap, &e->ent); + } +} + +static void write_cone_to_file(FILE *fp, struct pattern_list *pl) +{ + int i; + struct pattern_entry *pe; + struct hashmap_iter iter; + struct string_list sl = STRING_LIST_INIT_DUP; + + hashmap_for_each_entry(&pl->parent_hashmap, &iter, pe, ent) + string_list_insert(&sl, pe->pattern); + + string_list_sort(&sl); + string_list_remove_duplicates(&sl, 0); + + fprintf(fp, "/*\n!/*/\n"); + + for (i = 0; i < sl.nr; i++) { + char *pattern = sl.items[i].string; + + if (strlen(pattern)) + fprintf(fp, "%s/\n!%s/*/\n", pattern, pattern); + } + + string_list_clear(&sl, 0); + + hashmap_for_each_entry(&pl->recursive_hashmap, &iter, pe, ent) + string_list_insert(&sl, pe->pattern); + + string_list_sort(&sl); + string_list_remove_duplicates(&sl, 0); + + for (i = 0; i < sl.nr; i++) { + char *pattern = sl.items[i].string; + fprintf(fp, "%s/\n", pattern); + } +} + static int write_patterns_and_update(struct pattern_list *pl) { char *sparse_filename; @@ -147,13 +249,33 @@ static int write_patterns_and_update(struct pattern_list *pl) sparse_filename = get_sparse_checkout_filename(); fp = fopen(sparse_filename, "w"); - write_patterns_to_file(fp, pl); + + if (core_sparse_checkout_cone) + write_cone_to_file(fp, pl); + else + write_patterns_to_file(fp, pl); + fclose(fp); free(sparse_filename); return update_working_directory(); } +static void strbuf_to_cone_pattern(struct strbuf *line, struct pattern_list *pl) +{ + strbuf_trim(line); + + strbuf_trim_trailing_dir_sep(line); + + if (!line->len) + return; + + if (line->buf[0] != '/') + strbuf_insert(line, 0, "/", 1); + + insert_recursive_pattern(pl, line); +} + static char const * const builtin_sparse_checkout_set_usage[] = { N_("git sparse-checkout set [--stdin|]"), NULL @@ -184,17 +306,34 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix) builtin_sparse_checkout_set_usage, PARSE_OPT_KEEP_UNKNOWN); - if (set_opts.use_stdin) { + if (core_sparse_checkout_cone) { struct strbuf line = STRBUF_INIT; - - while (!strbuf_getline(&line, stdin)) { - size_t len; - char *buf = strbuf_detach(&line, &len); - add_pattern(buf, empty_base, 0, &pl, 0); + hashmap_init(&pl.recursive_hashmap, pl_hashmap_cmp, NULL, 0); + hashmap_init(&pl.parent_hashmap, pl_hashmap_cmp, NULL, 0); + + if (set_opts.use_stdin) { + while (!strbuf_getline(&line, stdin)) + strbuf_to_cone_pattern(&line, &pl); + } else { + for (i = 0; i < argc; i++) { + strbuf_setlen(&line, 0); + strbuf_addstr(&line, argv[i]); + strbuf_to_cone_pattern(&line, &pl); + } } } else { - for (i = 0; i < argc; i++) - add_pattern(argv[i], empty_base, 0, &pl, 0); + if (set_opts.use_stdin) { + struct strbuf line = STRBUF_INIT; + + while (!strbuf_getline(&line, stdin)) { + size_t len; + char *buf = strbuf_detach(&line, &len); + add_pattern(buf, empty_base, 0, &pl, 0); + } + } else { + for (i = 0; i < argc; i++) + add_pattern(argv[i], empty_base, 0, &pl, 0); + } } if (!core_apply_sparse_checkout) { diff --git a/dir.c b/dir.c index dfabf9982f..35c1ca9e24 100644 --- a/dir.c +++ b/dir.c @@ -611,10 +611,10 @@ void parse_path_pattern(const char **pattern, *patternlen = len; } -static int pl_hashmap_cmp(const void *unused_cmp_data, - const struct hashmap_entry *a, - const struct hashmap_entry *b, - const void *key) +int pl_hashmap_cmp(const void *unused_cmp_data, + const struct hashmap_entry *a, + const struct hashmap_entry *b, + const void *key) { const struct pattern_entry *ee1 = container_of(a, struct pattern_entry, ent); diff --git a/dir.h b/dir.h index f8edbca72b..8e232085cd 100644 --- a/dir.h +++ b/dir.h @@ -299,6 +299,10 @@ int is_excluded(struct dir_struct *dir, struct index_state *istate, const char *name, int *dtype); +int pl_hashmap_cmp(const void *unused_cmp_data, + const struct hashmap_entry *a, + const struct hashmap_entry *b, + const void *key); int hashmap_contains_parent(struct hashmap *map, const char *path, struct strbuf *buffer); diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh index 9907278fc1..ae99803d40 100755 --- a/t/t1091-sparse-checkout-builtin.sh +++ b/t/t1091-sparse-checkout-builtin.sh @@ -186,4 +186,55 @@ test_expect_success 'sparse-checkout disable' ' test_cmp expect dir ' +test_expect_success 'cone mode: init and set' ' + git -C repo sparse-checkout init --cone && + git -C repo config --list >config && + test_i18ngrep "core.sparsecheckoutcone=true" config && + ls repo >dir && + echo a >expect && + test_cmp expect dir && + git -C repo sparse-checkout set deep/deeper1/deepest/ 2>err && + test_line_count = 0 err && + ls repo >dir && + cat >expect <<-EOF && + a + deep + EOF + test_cmp expect dir && + ls repo/deep >dir && + cat >expect <<-EOF && + a + deeper1 + EOF + test_cmp expect dir && + ls repo/deep/deeper1 >dir && + cat >expect <<-EOF && + a + deepest + EOF + test_cmp expect dir && + cat >expect <<-EOF && + /* + !/*/ + /deep/ + !/deep/*/ + /deep/deeper1/ + !/deep/deeper1/*/ + /deep/deeper1/deepest/ + EOF + test_cmp expect repo/.git/info/sparse-checkout && + git -C repo sparse-checkout set --stdin 2>err <<-EOF && + folder1 + folder2 + EOF + test_line_count = 0 err && + cat >expect <<-EOF && + a + folder1 + folder2 + EOF + ls repo >dir && + test_cmp expect dir +' + test_done From patchwork Mon Oct 21 13:56:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11202241 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0EA311390 for ; Mon, 21 Oct 2019 13:56:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id E09DB214B2 for ; Mon, 21 Oct 2019 13:56:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="lil45SBw" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729245AbfJUN4l (ORCPT ); Mon, 21 Oct 2019 09:56:41 -0400 Received: from mail-wr1-f68.google.com ([209.85.221.68]:39696 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729223AbfJUN4j (ORCPT ); Mon, 21 Oct 2019 09:56:39 -0400 Received: by mail-wr1-f68.google.com with SMTP id c6so1903183wrm.6 for ; Mon, 21 Oct 2019 06:56:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Ips6wWu7Xdah4SBzOLvtoLKr422mshCSp/ecYyPpC+c=; b=lil45SBwGHR/6ZNLZ/boqj11zNyVrEDbzwwghZdE/QqNxNttON0Y3bEx71vymtfCFw jYOFqz73WrLXyFBWRQQCLoH2SKH5eG4NBgZeBCKVgc2H5+JNEoVMNxGjD1XzQWXinuV3 7TvDQVdcfSoZGJEP0jL1qT8+rDjGjkvfcwrnlGuBcnJnJhiBFsDl02430xYYov38qUEk Tv+bUNDA+/2GS/xt/YnyASmCDGWlvG4jbPfxKiLCpbsN4bxaROuiD8NkOKpiyDxJmIsU sTnEV/tTH9yxD0qgVU4DR1hPvnsj/ngpXTebQFIg6cQDQYX0IgOQUw0P5Z98c30Lg43i sVBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Ips6wWu7Xdah4SBzOLvtoLKr422mshCSp/ecYyPpC+c=; b=M4XASQAoYvWqYryJK3bSO7fAjXWgwmpzCGk6bvj/rRWx4kspxR6u1/5PGhefaQsO5S ggI7bqOasXFcu8soK1jTsTVUHMm4adCNLWz9B8908jpjJPfwN4HxvG+yHWOsK3Wv4UOe z8oTtJ8td9/Jrr1/LGJlDnCVjoXg0dTQCKefzvn0GrO7Qmv0fdqDwgUmOdKGahzV7rIO lL/3V+uaX8xc44P5bC9LGKUK6L0d9GDNv4fjBm5RfbRu9K5+JAj/t/fEcURTMwJdoyve IlQ3djWglbGA4dT3FlxQm5mZkO5L5vwH0s+PUjJGDublobbcF8bCjNSsCSdJtn77akBl wUyQ== X-Gm-Message-State: APjAAAUgTInpxMVLKFM5Fa3pUkn/AhOrZZB2eorvrtReVixFCgGOE7R2 OhJxE5M9FNUT3nPklRQHKeCOYLVu X-Google-Smtp-Source: APXvYqy+0rBLL53CDkKyK8hLq57+etr9Uzgu6UgDSexPCI9PkOL6dauIs3n4hGjdYEy3b1upP9jtJw== X-Received: by 2002:a5d:50c9:: with SMTP id f9mr19259704wrt.36.1571666197456; Mon, 21 Oct 2019 06:56:37 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id w22sm1449966wmi.7.2019.10.21.06.56.36 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 21 Oct 2019 06:56:37 -0700 (PDT) Message-Id: <7b5c5bad2d58041f634690e2e5581feaa4d0fd02.1571666187.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Mon, 21 Oct 2019 13:56:20 +0000 Subject: [PATCH v5 11/17] unpack-trees: hash less in cone mode Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, jon@jonsimons.org, szeder.dev@gmail.com, Derrick Stolee , Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee The sparse-checkout feature in "cone mode" can use the fact that the recursive patterns are "connected" to the root via parent patterns to decide if a directory is entirely contained in the sparse-checkout or entirely removed. In these cases, we can skip hashing the paths within those directories and simply set the skipworktree bit to the correct value. Signed-off-by: Derrick Stolee --- dir.c | 4 ++-- dir.h | 1 + unpack-trees.c | 38 +++++++++++++++++++++++--------------- 3 files changed, 26 insertions(+), 17 deletions(-) diff --git a/dir.c b/dir.c index 35c1ca9e24..2ef92a50a0 100644 --- a/dir.c +++ b/dir.c @@ -1270,7 +1270,7 @@ enum pattern_match_result path_matches_pattern_list( if (hashmap_contains_path(&pl->recursive_hashmap, &parent_pathname)) { - result = MATCHED; + result = MATCHED_RECURSIVE; goto done; } @@ -1292,7 +1292,7 @@ enum pattern_match_result path_matches_pattern_list( if (hashmap_contains_parent(&pl->recursive_hashmap, pathname, &parent_pathname)) - result = MATCHED; + result = MATCHED_RECURSIVE; done: strbuf_release(&parent_pathname); diff --git a/dir.h b/dir.h index 8e232085cd..77a43dbf89 100644 --- a/dir.h +++ b/dir.h @@ -264,6 +264,7 @@ enum pattern_match_result { UNDECIDED = -1, NOT_MATCHED = 0, MATCHED = 1, + MATCHED_RECURSIVE = 2, }; /* diff --git a/unpack-trees.c b/unpack-trees.c index a90d71845d..c0dca20865 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -1283,15 +1283,17 @@ static int clear_ce_flags_dir(struct index_state *istate, struct cache_entry **cache_end; int dtype = DT_DIR; int rc; - enum pattern_match_result ret; - ret = path_matches_pattern_list(prefix->buf, prefix->len, - basename, &dtype, pl, istate); + enum pattern_match_result ret, orig_ret; + orig_ret = path_matches_pattern_list(prefix->buf, prefix->len, + basename, &dtype, pl, istate); strbuf_addch(prefix, '/'); /* If undecided, use matching result of parent dir in defval */ - if (ret == UNDECIDED) + if (orig_ret == UNDECIDED) ret = default_match; + else + ret = orig_ret; for (cache_end = cache; cache_end != cache + nr; cache_end++) { struct cache_entry *ce = *cache_end; @@ -1299,17 +1301,23 @@ static int clear_ce_flags_dir(struct index_state *istate, break; } - /* - * TODO: check pl, if there are no patterns that may conflict - * with ret (iow, we know in advance the incl/excl - * decision for the entire directory), clear flag here without - * calling clear_ce_flags_1(). That function will call - * the expensive path_matches_pattern_list() on every entry. - */ - rc = clear_ce_flags_1(istate, cache, cache_end - cache, - prefix, - select_mask, clear_mask, - pl, ret); + if (pl->use_cone_patterns && orig_ret == MATCHED_RECURSIVE) { + struct cache_entry **ce = cache; + rc = (cache_end - cache) / sizeof(struct cache_entry *); + + while (ce < cache_end) { + (*ce)->ce_flags &= ~clear_mask; + ce++; + } + } else if (pl->use_cone_patterns && orig_ret == NOT_MATCHED) { + rc = (cache_end - cache) / sizeof(struct cache_entry *); + } else { + rc = clear_ce_flags_1(istate, cache, cache_end - cache, + prefix, + select_mask, clear_mask, + pl, ret); + } + strbuf_setlen(prefix, prefix->len - 1); return rc; } From patchwork Mon Oct 21 13:56:21 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11202249 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A58931390 for ; Mon, 21 Oct 2019 13:56:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 7993B214B2 for ; Mon, 21 Oct 2019 13:56:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="CKhZ/5xb" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729263AbfJUN4n (ORCPT ); Mon, 21 Oct 2019 09:56:43 -0400 Received: from mail-wr1-f66.google.com ([209.85.221.66]:39695 "EHLO mail-wr1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729203AbfJUN4l (ORCPT ); Mon, 21 Oct 2019 09:56:41 -0400 Received: by mail-wr1-f66.google.com with SMTP id c6so1903228wrm.6 for ; Mon, 21 Oct 2019 06:56:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=T4b8TadolwlSOZvHpfmF35nObWmTrbh1mpJa/MQa9ds=; b=CKhZ/5xbU6hxSGJw55XhDmpAMkAiFr6eX6YdLo7yRhRxSIR1zdscCQo33OQIxTnFq3 de4KS1BPncnMdj21XS2Z0O6nC9yY9+rWx6mjHRgSQ5aCwyd9Z5mRm3ySk69n/PHmC1ST R5EKAC2tQyLZH50hbcUgarBJoREgZH3DnzEMsFtfnFG9At7EfTD85yg2r1y5YyohhzM7 1CquIRAWpki1D73glkoj4ogRUAzKj+9ya9v7mJWlrgO5IViWX4SbbYkEE/QvHkTi6iJP msr99k8g9ad5OVfDWRUIVzAGAXhlWtjTn2ZjHK5avV3Tdxl9zpcdoCxidDMllYDT+IQk 0V3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=T4b8TadolwlSOZvHpfmF35nObWmTrbh1mpJa/MQa9ds=; b=Q3cB3uAKtBn0yXdMgSw97M+EpPplJI2EhrRW3d1InDlWApMfyrkG6fD+5y0OyZxogg PGh6A41lmqXVWfpt8MwUxXcaReEaWVd1Xn+MbhbmsxrD2SE6U2ZWzZI2VsgNrvieeNSm FEIVu/mWtQfkukBH1/od/f2Xp0Sm5dnUx1JXQiUNVo5robs7hGXmXTCwS652Hm4/eOtF HVPetbs3Sd+sycC1N8rDxnj+V2euv1zqxH73BI1mkW4JLTEraq2H1SHyxiX99UTb/u49 BmV9f4zplFPBiSfeIP3hGWLtxZhYTT7pGZ46c6jMM27WhevyEkJfS33D8/zDPfYjfEd7 65WA== X-Gm-Message-State: APjAAAVJeXl/DDnmk0iCZlMw+XJUbrKbjlKCMtjF1uHX/s8zd5eT3UAJ 9F9xGQITWBSYCRQRK+3JB+UQAH7r X-Google-Smtp-Source: APXvYqz9y3/h/sDRNG2N5p1dSiN5T177vyShFGaxhXlPB4u9bKC+CHbilqDl6Wj8jFj6rzzPbplWzA== X-Received: by 2002:adf:d84c:: with SMTP id k12mr428635wrl.235.1571666198129; Mon, 21 Oct 2019 06:56:38 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id c16sm1363835wrw.32.2019.10.21.06.56.37 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 21 Oct 2019 06:56:37 -0700 (PDT) Message-Id: <386f4f8eb5d12d9d089684841d57ea66447cf675.1571666187.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Mon, 21 Oct 2019 13:56:21 +0000 Subject: [PATCH v5 12/17] unpack-trees: add progress to clear_ce_flags() Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, jon@jonsimons.org, szeder.dev@gmail.com, Derrick Stolee , Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee When a large repository has many sparse-checkout patterns, the process for updating the skip-worktree bits can take long enough that a user gets confused why nothing is happening. Update the clear_ce_flags() method to write progress. Signed-off-by: Derrick Stolee --- cache.h | 2 ++ unpack-trees.c | 56 ++++++++++++++++++++++++++++++++++++-------------- 2 files changed, 43 insertions(+), 15 deletions(-) diff --git a/cache.h b/cache.h index 4980ee198e..d3c89e7a53 100644 --- a/cache.h +++ b/cache.h @@ -304,6 +304,7 @@ static inline unsigned int canon_mode(unsigned int mode) struct split_index; struct untracked_cache; +struct progress; struct index_state { struct cache_entry **cache; @@ -326,6 +327,7 @@ struct index_state { uint64_t fsmonitor_last_update; struct ewah_bitmap *fsmonitor_dirty; struct mem_pool *ce_mem_pool; + struct progress *progress; }; /* Name hashing */ diff --git a/unpack-trees.c b/unpack-trees.c index c0dca20865..8bb684ad62 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -1269,7 +1269,8 @@ static int clear_ce_flags_1(struct index_state *istate, struct strbuf *prefix, int select_mask, int clear_mask, struct pattern_list *pl, - enum pattern_match_result default_match); + enum pattern_match_result default_match, + int progress_nr); /* Whole directory matching */ static int clear_ce_flags_dir(struct index_state *istate, @@ -1278,7 +1279,8 @@ static int clear_ce_flags_dir(struct index_state *istate, char *basename, int select_mask, int clear_mask, struct pattern_list *pl, - enum pattern_match_result default_match) + enum pattern_match_result default_match, + int progress_nr) { struct cache_entry **cache_end; int dtype = DT_DIR; @@ -1315,7 +1317,8 @@ static int clear_ce_flags_dir(struct index_state *istate, rc = clear_ce_flags_1(istate, cache, cache_end - cache, prefix, select_mask, clear_mask, - pl, ret); + pl, ret, + progress_nr); } strbuf_setlen(prefix, prefix->len - 1); @@ -1342,7 +1345,8 @@ static int clear_ce_flags_1(struct index_state *istate, struct strbuf *prefix, int select_mask, int clear_mask, struct pattern_list *pl, - enum pattern_match_result default_match) + enum pattern_match_result default_match, + int progress_nr) { struct cache_entry **cache_end = cache + nr; @@ -1356,8 +1360,11 @@ static int clear_ce_flags_1(struct index_state *istate, int len, dtype; enum pattern_match_result ret; + display_progress(istate->progress, progress_nr); + if (select_mask && !(ce->ce_flags & select_mask)) { cache++; + progress_nr++; continue; } @@ -1378,20 +1385,26 @@ static int clear_ce_flags_1(struct index_state *istate, prefix, prefix->buf + prefix->len - len, select_mask, clear_mask, - pl, default_match); + pl, default_match, + progress_nr); /* clear_c_f_dir eats a whole dir already? */ if (processed) { cache += processed; + progress_nr += processed; strbuf_setlen(prefix, prefix->len - len); continue; } strbuf_addch(prefix, '/'); - cache += clear_ce_flags_1(istate, cache, cache_end - cache, - prefix, - select_mask, clear_mask, pl, - default_match); + processed = clear_ce_flags_1(istate, cache, cache_end - cache, + prefix, + select_mask, clear_mask, pl, + default_match, progress_nr); + + cache += processed; + progress_nr += processed; + strbuf_setlen(prefix, prefix->len - len - 1); continue; } @@ -1406,19 +1419,27 @@ static int clear_ce_flags_1(struct index_state *istate, if (ret == MATCHED) ce->ce_flags &= ~clear_mask; cache++; + progress_nr++; } + + display_progress(istate->progress, progress_nr); return nr - (cache_end - cache); } static int clear_ce_flags(struct index_state *istate, int select_mask, int clear_mask, - struct pattern_list *pl) + struct pattern_list *pl, + int show_progress) { static struct strbuf prefix = STRBUF_INIT; char label[100]; int rval; strbuf_reset(&prefix); + if (show_progress) + istate->progress = start_delayed_progress( + _("Updating index flags"), + istate->cache_nr); xsnprintf(label, sizeof(label), "clear_ce_flags(0x%08lx,0x%08lx)", (unsigned long)select_mask, (unsigned long)clear_mask); @@ -1428,9 +1449,10 @@ static int clear_ce_flags(struct index_state *istate, istate->cache_nr, &prefix, select_mask, clear_mask, - pl, 0); + pl, 0, 0); trace2_region_leave("unpack_trees", label, the_repository); + stop_progress(&istate->progress); return rval; } @@ -1439,7 +1461,8 @@ static int clear_ce_flags(struct index_state *istate, */ static void mark_new_skip_worktree(struct pattern_list *pl, struct index_state *istate, - int select_flag, int skip_wt_flag) + int select_flag, int skip_wt_flag, + int show_progress) { int i; @@ -1463,7 +1486,7 @@ static void mark_new_skip_worktree(struct pattern_list *pl, * 2. Widen worktree according to sparse-checkout file. * Matched entries will have skip_wt_flag cleared (i.e. "in") */ - clear_ce_flags(istate, select_flag, skip_wt_flag, pl); + clear_ce_flags(istate, select_flag, skip_wt_flag, pl, show_progress); } static int verify_absent(const struct cache_entry *, @@ -1525,7 +1548,8 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options * Sparse checkout loop #1: set NEW_SKIP_WORKTREE on existing entries */ if (!o->skip_sparse_checkout) - mark_new_skip_worktree(o->pl, o->src_index, 0, CE_NEW_SKIP_WORKTREE); + mark_new_skip_worktree(o->pl, o->src_index, 0, + CE_NEW_SKIP_WORKTREE, o->verbose_update); if (!dfc) dfc = xcalloc(1, cache_entry_size(0)); @@ -1590,7 +1614,9 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options * If the will have NEW_SKIP_WORKTREE, also set CE_SKIP_WORKTREE * so apply_sparse_checkout() won't attempt to remove it from worktree */ - mark_new_skip_worktree(o->pl, &o->result, CE_ADDED, CE_SKIP_WORKTREE | CE_NEW_SKIP_WORKTREE); + mark_new_skip_worktree(o->pl, &o->result, + CE_ADDED, CE_SKIP_WORKTREE | CE_NEW_SKIP_WORKTREE, + o->verbose_update); ret = 0; for (i = 0; i < o->result.cache_nr; i++) { From patchwork Mon Oct 21 13:56:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11202261 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 003F8139A for ; Mon, 21 Oct 2019 13:56:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id D1CF6214B2 for ; Mon, 21 Oct 2019 13:56:50 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="oESJxDEP" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729272AbfJUN4t (ORCPT ); Mon, 21 Oct 2019 09:56:49 -0400 Received: from mail-wm1-f67.google.com ([209.85.128.67]:38941 "EHLO mail-wm1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729225AbfJUN4k (ORCPT ); Mon, 21 Oct 2019 09:56:40 -0400 Received: by mail-wm1-f67.google.com with SMTP id r141so3327463wme.4 for ; Mon, 21 Oct 2019 06:56:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=xCZIUJKlOnbwNhuxRGiI7T/gfJU3N5zEWpmYYb9368g=; b=oESJxDEPb4CG7bfP1yxGsQmbSoGjdkDn5rXDNL97dGNtBUZLYQylx+Q6jyh1j6ah89 r6pNvGmmQQXd6LoRlvZRj97EoNbPneoKnUzmstq7mHv7AKJVM8qECnl6TMZs7+AQ9Pyt 4REj+kXQLEsKFqBYJNzd4pRTj2SAu7XUAPQccF4IaWamArrux/bxJBeFoYg0a4hm2Tm7 CIhqY+FcO2GziW9pO/Qq38UmOGaVvjSxb2iZcpxzILusJGfNHYFqaSoD5usA5sknglcA rWQAZC10WMcmyTMYVTvRsAPRr9XtRSJlrGCOCxvDEBYMrnQKpdpmeJY8ZuXegtYawF4C U9ZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=xCZIUJKlOnbwNhuxRGiI7T/gfJU3N5zEWpmYYb9368g=; b=Kd8ClGmdS6EJQxREyHeOco2ux26hje/uvEBVBnO3yWm25yJWLB59CoeUT1bz6a8K9p lxBaiAE2p3nECXKJzjZ1th6xFmb+5POoteAJg+rUMnqlg+yUq+VhLCbUPDUx1kkL8Rd8 u/wVKKaPGDn983urb8zRzLipV7kI7uMZCxQPaZcOveJd/bI4VcBLEq0QuYKuz8b/6f4U sEhyFVG2uMx9uf16FxA9wHZlUiVmVcWJybYx1XCIC9xe2S1gTjxpeP7cqQxiR6yEp+N1 CfBM6FGDw9QbPJ9QZSNxckfZ4tkW+gtF7NuEvbfZK7nTV2fo9yWxMWMXWuvkqBieiUnw Zgxw== X-Gm-Message-State: APjAAAXvSRBt3pPlnSlf1X0XEDp4p9cxFEaUWX/AlxMcOzkviPbkWEkM W4AJQGZV+UdycRBf1rYi5rXUmXUa X-Google-Smtp-Source: APXvYqyq8W9JtPHMvWeDlY0X4ky1gY6FE9aLoJCTP/N42YnRE5vKm/eum+QV1W0HMF8DhgPcO5/IFQ== X-Received: by 2002:a7b:cc8c:: with SMTP id p12mr18859537wma.167.1571666198925; Mon, 21 Oct 2019 06:56:38 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id w7sm12134660wmd.22.2019.10.21.06.56.38 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 21 Oct 2019 06:56:38 -0700 (PDT) Message-Id: In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Mon, 21 Oct 2019 13:56:22 +0000 Subject: [PATCH v5 13/17] read-tree: show progress by default Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, jon@jonsimons.org, szeder.dev@gmail.com, Derrick Stolee , Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee The read-tree builtin has a --verbose option that signals to show progress and other data while updating the index. Update this to be on by default when stderr is a terminal window. This will help tools like 'git sparse-checkout' to automatically benefit from progress indicators when a user runs these commands. Signed-off-by: Derrick Stolee --- builtin/read-tree.c | 1 + 1 file changed, 1 insertion(+) diff --git a/builtin/read-tree.c b/builtin/read-tree.c index ca5e655d2f..69963d83dc 100644 --- a/builtin/read-tree.c +++ b/builtin/read-tree.c @@ -162,6 +162,7 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix) opts.head_idx = -1; opts.src_index = &the_index; opts.dst_index = &the_index; + opts.verbose_update = isatty(2); git_config(git_read_tree_config, NULL); From patchwork Mon Oct 21 13:56:23 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11202255 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 22400139A for ; Mon, 21 Oct 2019 13:56:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id F3F8A214B2 for ; Mon, 21 Oct 2019 13:56:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="J02uGhlG" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729268AbfJUN4q (ORCPT ); Mon, 21 Oct 2019 09:56:46 -0400 Received: from mail-wr1-f66.google.com ([209.85.221.66]:37656 "EHLO mail-wr1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729189AbfJUN4l (ORCPT ); Mon, 21 Oct 2019 09:56:41 -0400 Received: by mail-wr1-f66.google.com with SMTP id e11so5406259wrv.4 for ; Mon, 21 Oct 2019 06:56:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=zGyp46UQWdWdiI/hNLogBj37uH09PvgL7pOdg2I30Lo=; b=J02uGhlGqGObAx9bjy0oBbuC6oqyQk/E1jzF5edvMswZhfzsSMDRnU3PlwXRjXdZrh Ko4/3GcvWSkC85AS2VPfHK2nTtZY72+aw6u2VG4NguG8Ka65sVBR2XXyzgEKc+KowADl YNjMlz0TGvj0cu15ZGfSkVyWYbx5ba4IGIYVImZOQ5fq/Zb10eewQw4dEpK4ZChNU0+X rZnbRHGWC6/EG8TuJkphKODBHaHXyPdDcdEJ9DNunEGMXYAvOdDpnDOfRhoyiX6n+e5i 4+uSKpLKdnHRKiNoA0c0IXp42ozVBitEVuvvKviPstN1BIXL0IQ5q7Tmb9gIv1Ewz0br ablQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=zGyp46UQWdWdiI/hNLogBj37uH09PvgL7pOdg2I30Lo=; b=MXUWKzs8wlDKCAGBHl0YyTwILJWujUVD/jVydrGT/AyayJ2hPtGr/8xSTS3Z8bAEmA XnKH9QrjzJiXEqWUyRrQqZxLiMtxGm8EX05c8GO48515yAZJQHD2eA6HGMoFiKx4baUr zboClAgFZDrgoXGsIxXCd55wUtcB2Xfb86Qs6TsgaFcnbMkGndagoSteHzGwJ1vcjICf lSnLWMv2dA7snF+org34vb9khPi92ETsas2fpMUKOJTBkkOcAGDPeRq0TqE0nMEFRhEX 09CUS0bg83qvmzEc0AEOwD3KvdTPytfVtvwrjtgerVygHUbAAamEmNsREEhu9HtIZttR Lukw== X-Gm-Message-State: APjAAAWPkqzxBLq3Idsv1Bnd/7hKsEZKb3sf14/u6kXy6V+Kb1QsYOwk TjFGqaWPga2WmNPPt/ZC40tcB70d X-Google-Smtp-Source: APXvYqyuFB35Cz5GXSje3B7kFFz4fuh1qV+lkTbLM3O8G0unSohPnVmNmkmuf5B4tUGzN4cQkKCLLw== X-Received: by 2002:adf:8295:: with SMTP id 21mr19438479wrc.14.1571666199846; Mon, 21 Oct 2019 06:56:39 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id a9sm20577163wmf.14.2019.10.21.06.56.39 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 21 Oct 2019 06:56:39 -0700 (PDT) Message-Id: In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Mon, 21 Oct 2019 13:56:23 +0000 Subject: [PATCH v5 14/17] sparse-checkout: sanitize for nested folders Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, jon@jonsimons.org, szeder.dev@gmail.com, Derrick Stolee , Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee If a user provides folders A/ and A/B/ for inclusion in a cone-mode sparse-checkout file, the parsing logic will notice that A/ appears both as a "parent" type pattern and as a "recursive" type pattern. This is unexpected and hence will complain via a warning and revert to the old logic for checking sparse-checkout patterns. Prevent this from happening accidentally by sanitizing the folders for this type of inclusion in the 'git sparse-checkout' builtin. This happens in two ways: 1. Do not include any parent patterns that also appear as recursive patterns. 2. Do not include any recursive patterns deeper than other recursive patterns. In order to minimize duplicate code for scanning parents, create hashmap_contains_parent() method. It takes a strbuf buffer to avoid reallocating a buffer when calling in a tight loop. Helped-by: Eric Wong Helped-by: Johannes Schindelin Signed-off-by: Derrick Stolee --- builtin/sparse-checkout.c | 22 ++++++++++++++++++---- t/t1091-sparse-checkout-builtin.sh | 11 +++++++++++ 2 files changed, 29 insertions(+), 4 deletions(-) diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c index 6c3bf68c89..b02e085495 100644 --- a/builtin/sparse-checkout.c +++ b/builtin/sparse-checkout.c @@ -212,9 +212,17 @@ static void write_cone_to_file(FILE *fp, struct pattern_list *pl) struct pattern_entry *pe; struct hashmap_iter iter; struct string_list sl = STRING_LIST_INIT_DUP; + struct strbuf parent_pattern = STRBUF_INIT; - hashmap_for_each_entry(&pl->parent_hashmap, &iter, pe, ent) - string_list_insert(&sl, pe->pattern); + hashmap_for_each_entry(&pl->parent_hashmap, &iter, pe, ent) { + if (hashmap_get_entry(&pl->recursive_hashmap, pe, ent, NULL)) + continue; + + if (!hashmap_contains_parent(&pl->recursive_hashmap, + pe->pattern, + &parent_pattern)) + string_list_insert(&sl, pe->pattern); + } string_list_sort(&sl); string_list_remove_duplicates(&sl, 0); @@ -230,8 +238,14 @@ static void write_cone_to_file(FILE *fp, struct pattern_list *pl) string_list_clear(&sl, 0); - hashmap_for_each_entry(&pl->recursive_hashmap, &iter, pe, ent) - string_list_insert(&sl, pe->pattern); + hashmap_for_each_entry(&pl->recursive_hashmap, &iter, pe, ent) { + if (!hashmap_contains_parent(&pl->recursive_hashmap, + pe->pattern, + &parent_pattern)) + string_list_insert(&sl, pe->pattern); + } + + strbuf_release(&parent_pattern); string_list_sort(&sl); string_list_remove_duplicates(&sl, 0); diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh index ae99803d40..0792caeb7e 100755 --- a/t/t1091-sparse-checkout-builtin.sh +++ b/t/t1091-sparse-checkout-builtin.sh @@ -237,4 +237,15 @@ test_expect_success 'cone mode: init and set' ' test_cmp expect dir ' +test_expect_success 'cone mode: set with nested folders' ' + git -C repo sparse-checkout set deep deep/deeper1/deepest 2>err && + test_line_count = 0 err && + cat >expect <<-EOF && + /* + !/*/ + /deep/ + EOF + test_cmp repo/.git/info/sparse-checkout expect +' + test_done From patchwork Mon Oct 21 13:56:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11202253 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D8CD614ED for ; Mon, 21 Oct 2019 13:56:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id AF63C2166E for ; Mon, 21 Oct 2019 13:56:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="iOXvniG8" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729266AbfJUN4q (ORCPT ); Mon, 21 Oct 2019 09:56:46 -0400 Received: from mail-wm1-f65.google.com ([209.85.128.65]:51723 "EHLO mail-wm1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729251AbfJUN4n (ORCPT ); Mon, 21 Oct 2019 09:56:43 -0400 Received: by mail-wm1-f65.google.com with SMTP id q70so6276182wme.1 for ; Mon, 21 Oct 2019 06:56:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=DGEq05n6zWIGYB1Q74QUmDNQgYqoICLynjfwMW2q5Js=; b=iOXvniG8XIYOHUmYHDRZzwfz6k7WSFwEaPOVDyEgZJ28IMxyZOyXr0B+IPGx8Jd/1o K1mVIGbMeEzVCLkytF0PRPhRPf/3hl9Iwn3ZZgaY6igUAMxCXY16xwtyjFGiIrqjYdte dN5HNDIZWXel/aXusCRvpIzRIP1IqYSsOftGcxi0jnHEUAQSXU+mIQ0roI2PDVvJzQY8 L1y91JmnB5pHqW05YgKZMtM+D9axLbtIgV5mAtJHKqrtYALDQUSN40LOZ7skNv7NYlWk AZjlJ6ZCZ7kth73o/IpFPt6aXh05cKMFYW6rYyqnHoohLmmzGx8++asLeAFCPjEhJkox e6Vg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=DGEq05n6zWIGYB1Q74QUmDNQgYqoICLynjfwMW2q5Js=; b=F+A5f6Yl4uTY0Mg1pgaLPUnZ8jfDzCdnua5Egz+72xUWmFReg47LCiXOwOHI+LzFxN XyBydVMeB4CZYz66yZtbr7bB1CQj8hwJBRELoKb2H9tvAsTh2348Ku6uptgydL+ukSo3 JV0svMfXMuxGSLQe5znrFMFRrNbnZA3rxv3q7hyuFiu7fHskCylr1/KumjeWyTbHXVb2 MHKzjlgKPgfPm0fsmwKlY1O18kR5OVufkJfadxRgzleQv2LAbvw1CKUkL4JLZ1USM8UL Xm5HFrkyjjKCRtqlg7jYmKyR+zdoG8PFXSBOnxJO5U4HyJp79ja8XroKe/6/proTEWpm OquQ== X-Gm-Message-State: APjAAAUnldXDS8xbvux1bIv489QTMwkycRg2oYkJqzwQjoq95F6sR3kJ TCdo6FFXCU6/7vZ5jdVxW01O2eC4 X-Google-Smtp-Source: APXvYqxR7KIli9Mfu8t299udr50T7sPfp9jreNi9Pvvfq5rNgWcTMfrsJ7ucWjtvWb8MElDZ1BX/gQ== X-Received: by 2002:a7b:c94f:: with SMTP id i15mr20530263wml.31.1571666200662; Mon, 21 Oct 2019 06:56:40 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id w17sm15998581wra.34.2019.10.21.06.56.40 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 21 Oct 2019 06:56:40 -0700 (PDT) Message-Id: In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Mon, 21 Oct 2019 13:56:24 +0000 Subject: [PATCH v5 15/17] sparse-checkout: update working directory in-process Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, jon@jonsimons.org, szeder.dev@gmail.com, Derrick Stolee , Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee The sparse-checkout builtin used 'git read-tree -mu HEAD' to update the skip-worktree bits in the index and to update the working directory. This extra process is overly complex, and prone to failure. It also requires that we write our changes to the sparse-checkout file before trying to update the index. Remove this extra process call by creating a direct call to unpack_trees() in the same way 'git read-tree -mu HEAD' does. In addition, provide an in-memory list of patterns so we can avoid reading from the sparse-checkout file. This allows us to test a proposed change to the file before writing to it. An earlier version of this patch included a bug when the 'set' command failed due to the "Sparse checkout leaves no entry on working directory" error. It would not rollback the index.lock file, so the replay of the old sparse-checkout specification would fail. A test in t1091 now covers that scenario. Signed-off-by: Derrick Stolee --- builtin/read-tree.c | 2 +- builtin/sparse-checkout.c | 83 +++++++++++++++++++++++++----- t/t1091-sparse-checkout-builtin.sh | 28 ++++++++++ unpack-trees.c | 5 +- unpack-trees.h | 3 +- 5 files changed, 105 insertions(+), 16 deletions(-) diff --git a/builtin/read-tree.c b/builtin/read-tree.c index 69963d83dc..d7eeaa26ec 100644 --- a/builtin/read-tree.c +++ b/builtin/read-tree.c @@ -186,7 +186,7 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix) if (opts.reset || opts.merge || opts.prefix) { if (read_cache_unmerged() && (opts.prefix || opts.merge)) - die("You need to resolve your current index first"); + die(_("You need to resolve your current index first")); stage = opts.merge = 1; } resolve_undo_clear(); diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c index b02e085495..ea2e7ee51b 100644 --- a/builtin/sparse-checkout.c +++ b/builtin/sparse-checkout.c @@ -7,6 +7,11 @@ #include "run-command.h" #include "strbuf.h" #include "string-list.h" +#include "cache.h" +#include "cache-tree.h" +#include "lockfile.h" +#include "resolve-undo.h" +#include "unpack-trees.h" static char const * const builtin_sparse_checkout_usage[] = { N_("git sparse-checkout (init|list|set|disable) "), @@ -60,18 +65,54 @@ static int sparse_checkout_list(int argc, const char **argv) return 0; } -static int update_working_directory(void) +static int update_working_directory(struct pattern_list *pl) { - struct argv_array argv = ARGV_ARRAY_INIT; int result = 0; - argv_array_pushl(&argv, "read-tree", "-m", "-u", "HEAD", NULL); + struct unpack_trees_options o; + struct lock_file lock_file = LOCK_INIT; + struct object_id oid; + struct tree *tree; + struct tree_desc t; + struct repository *r = the_repository; - if (run_command_v_opt(argv.argv, RUN_GIT_CMD)) { - error(_("failed to update index with new sparse-checkout paths")); - result = 1; - } + if (repo_read_index_unmerged(r)) + die(_("You need to resolve your current index first")); + + if (get_oid("HEAD", &oid)) + return 0; + + tree = parse_tree_indirect(&oid); + parse_tree(tree); + init_tree_desc(&t, tree->buffer, tree->size); + + memset(&o, 0, sizeof(o)); + o.verbose_update = isatty(2); + o.merge = 1; + o.update = 1; + o.fn = oneway_merge; + o.head_idx = -1; + o.src_index = r->index; + o.dst_index = r->index; + o.skip_sparse_checkout = 0; + o.pl = pl; + o.keep_pattern_list = !!pl; + + resolve_undo_clear_index(r->index); + setup_work_tree(); + + cache_tree_free(&r->index->cache_tree); + + repo_hold_locked_index(r, &lock_file, LOCK_DIE_ON_ERROR); + + core_apply_sparse_checkout = 1; + result = unpack_trees(1, &t, &o); + + if (!result) { + prime_cache_tree(r, r->index, tree); + write_locked_index(r->index, &lock_file, COMMIT_LOCK); + } else + rollback_lock_file(&lock_file); - argv_array_clear(&argv); return result; } @@ -147,7 +188,11 @@ static int sparse_checkout_init(int argc, const char **argv) builtin_sparse_checkout_init_options, builtin_sparse_checkout_init_usage, 0); - mode = init_opts.cone_mode ? MODE_CONE_PATTERNS : MODE_ALL_PATTERNS; + if (init_opts.cone_mode) { + mode = MODE_CONE_PATTERNS; + core_sparse_checkout_cone = 1; + } else + mode = MODE_ALL_PATTERNS; if (sc_set_config(mode)) return 1; @@ -175,7 +220,8 @@ static int sparse_checkout_init(int argc, const char **argv) } reset_dir: - return update_working_directory(); + core_apply_sparse_checkout = 1; + return update_working_directory(NULL); } static void insert_recursive_pattern(struct pattern_list *pl, struct strbuf *path) @@ -260,6 +306,15 @@ static int write_patterns_and_update(struct pattern_list *pl) { char *sparse_filename; FILE *fp; + int result; + + result = update_working_directory(pl); + + if (result) { + clear_pattern_list(pl); + update_working_directory(NULL); + return result; + } sparse_filename = get_sparse_checkout_filename(); fp = fopen(sparse_filename, "w"); @@ -270,9 +325,11 @@ static int write_patterns_and_update(struct pattern_list *pl) write_patterns_to_file(fp, pl); fclose(fp); + free(sparse_filename); + clear_pattern_list(pl); - return update_working_directory(); + return 0; } static void strbuf_to_cone_pattern(struct strbuf *line, struct pattern_list *pl) @@ -324,6 +381,7 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix) struct strbuf line = STRBUF_INIT; hashmap_init(&pl.recursive_hashmap, pl_hashmap_cmp, NULL, 0); hashmap_init(&pl.parent_hashmap, pl_hashmap_cmp, NULL, 0); + pl.use_cone_patterns = 1; if (set_opts.use_stdin) { while (!strbuf_getline(&line, stdin)) @@ -378,7 +436,8 @@ static int sparse_checkout_disable(int argc, const char **argv) fprintf(fp, "/*\n"); fclose(fp); - if (update_working_directory()) + core_apply_sparse_checkout = 1; + if (update_working_directory(NULL)) die(_("error while refreshing working directory")); unlink(sparse_filename); diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh index 0792caeb7e..a580b79595 100755 --- a/t/t1091-sparse-checkout-builtin.sh +++ b/t/t1091-sparse-checkout-builtin.sh @@ -248,4 +248,32 @@ test_expect_success 'cone mode: set with nested folders' ' test_cmp repo/.git/info/sparse-checkout expect ' +test_expect_success 'revert to old sparse-checkout on bad update' ' + echo update >repo/deep/deeper2/a && + cp repo/.git/info/sparse-checkout expect && + test_must_fail git -C repo sparse-checkout set deep/deeper1 2>err && + test_i18ngrep "Cannot update sparse checkout" err && + test_cmp repo/.git/info/sparse-checkout expect && + ls repo/deep >dir && + cat >expect <<-EOF && + a + deeper1 + deeper2 + EOF + test_cmp dir expect +' + +test_expect_success 'revert to old sparse-checkout on empty update' ' + git init empty-test && + ( + echo >file && + git add file && + git commit -m "test" && + test_must_fail git sparse-checkout set nothing 2>err && + test_i18ngrep "Sparse checkout leaves no entry on working directory" err && + test_i18ngrep ! ".git/index.lock" err && + git sparse-checkout set file + ) +' + test_done diff --git a/unpack-trees.c b/unpack-trees.c index 8bb684ad62..3789a22cf0 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -1511,7 +1511,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options memset(&pl, 0, sizeof(pl)); if (!core_apply_sparse_checkout || !o->update) o->skip_sparse_checkout = 1; - if (!o->skip_sparse_checkout) { + if (!o->skip_sparse_checkout && !o->pl) { char *sparse = git_pathdup("info/sparse-checkout"); pl.use_cone_patterns = core_sparse_checkout_cone; if (add_patterns_from_file_to_list(sparse, "", 0, &pl, NULL) < 0) @@ -1684,7 +1684,8 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options done: trace_performance_leave("unpack_trees"); - clear_pattern_list(&pl); + if (!o->keep_pattern_list) + clear_pattern_list(&pl); return ret; return_failed: diff --git a/unpack-trees.h b/unpack-trees.h index f2eee0c7c5..ca94a421a5 100644 --- a/unpack-trees.h +++ b/unpack-trees.h @@ -59,7 +59,8 @@ struct unpack_trees_options { quiet, exiting_early, show_all_errors, - dry_run; + dry_run, + keep_pattern_list; const char *prefix; int cache_bottom; struct dir_struct *dir; From patchwork Mon Oct 21 13:56:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11202251 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9A9D21390 for ; Mon, 21 Oct 2019 13:56:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 78DAA2166E for ; Mon, 21 Oct 2019 13:56:45 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="doTK1PP1" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729264AbfJUN4o (ORCPT ); Mon, 21 Oct 2019 09:56:44 -0400 Received: from mail-wm1-f66.google.com ([209.85.128.66]:38944 "EHLO mail-wm1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729255AbfJUN4n (ORCPT ); Mon, 21 Oct 2019 09:56:43 -0400 Received: by mail-wm1-f66.google.com with SMTP id r141so3327606wme.4 for ; Mon, 21 Oct 2019 06:56:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=ODUGrtgS3butVwd0ur5USwIQx48hIHS2pkvBQPyGujg=; b=doTK1PP1ZxDcSYkHbINb2C7r/Bth6xwBYcVFCBTH+aDZypzFbvMCAa128cVdkmG7Wc To6Y+L02O8E7UXCML9wPfMH4qrV2ope+pmwcZ/ZWyBOf4F8rElWTXwKQKp2mdGh4ojrp GrG74eifaZV1warjlA8XcqRcnmqscjRlj1LtFxka+JTeyBhdgrOR5+I25cUA6EIUh6zo Bulk5f+gl3iJZ3A1IvR/8OtV87YrpHds3I8t+TFsUBi1Y0eqHqLJRoSkax6hAHizN7t5 CZzkrFb8jZE8jmPDG6IvrjiOxWnzdVoL1OY1V470/E9OJJk0s6FKN4nEyGW1n3EJWk8K tvjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=ODUGrtgS3butVwd0ur5USwIQx48hIHS2pkvBQPyGujg=; b=GjDeopLM/jEHKwKBS/DJcd/3PfcdR8GZbNCXpG7NwQzG0IA1AVkpKvgqR8tPkhGHtC phJbjTVKkVQTEUgm8KzUaeNPWaJn8WwAfrieyXRmvItUmpbRFnYDu65W8Dzd85U+V5m3 2G7cR40kZhmW8jRiY5449P1Aaswwy2u6oI4LPH1DzAX4fpmw4dhAi1jwCpKkeB3A6pXx MOI2xxwjuOER2BcsfBO/2TUFUAAtE07XY0LwTHEsh1DbnFlZKhX4WgNJYFPqUwlwjq+a jdGSMO5HMpoDx1X/b+Kz8BEnE/60NE2QSSJXRoY90l+nyfrIbbUWMy17V7drMuD7JRnL OpTw== X-Gm-Message-State: APjAAAWGkzLMYvtB8r2Zs1Pr6bK5InhLoCK8/By8thkXaeGMgVuz4Z81 MVYNtUgiJJReJzc/EBBcJVRcKRrn X-Google-Smtp-Source: APXvYqxRl+1oh7Wo61iiAaMxLiVKPmmjemcAIMs6AwIMgp1ak+GQVzi5NyjRDl5OoZ30qyg/WKnKWA== X-Received: by 2002:a7b:c24a:: with SMTP id b10mr6023399wmj.124.1571666201296; Mon, 21 Oct 2019 06:56:41 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id d199sm5649557wmd.35.2019.10.21.06.56.40 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 21 Oct 2019 06:56:40 -0700 (PDT) Message-Id: <88eff318e04165e23406098bfdd61d52bdc604cb.1571666187.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Mon, 21 Oct 2019 13:56:25 +0000 Subject: [PATCH v5 16/17] sparse-checkout: write using lockfile Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, jon@jonsimons.org, szeder.dev@gmail.com, Derrick Stolee , Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee If two 'git sparse-checkout set' subcommands are launched at the same time, the behavior can be unexpected as they compete to write the sparse-checkout file and update the working directory. Take a lockfile around the writes to the sparse-checkout file. In addition, acquire this lock around the working directory update to avoid two commands updating the working directory in different ways. Signed-off-by: Derrick Stolee --- builtin/sparse-checkout.c | 15 ++++++++++++--- t/t1091-sparse-checkout-builtin.sh | 7 +++++++ 2 files changed, 19 insertions(+), 3 deletions(-) diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c index ea2e7ee51b..76f65d8edd 100644 --- a/builtin/sparse-checkout.c +++ b/builtin/sparse-checkout.c @@ -306,25 +306,34 @@ static int write_patterns_and_update(struct pattern_list *pl) { char *sparse_filename; FILE *fp; + int fd; + struct lock_file lk = LOCK_INIT; int result; result = update_working_directory(pl); + sparse_filename = get_sparse_checkout_filename(); + fd = hold_lock_file_for_update(&lk, sparse_filename, + LOCK_DIE_ON_ERROR); + + result = update_working_directory(pl); if (result) { + rollback_lock_file(&lk); + free(sparse_filename); clear_pattern_list(pl); update_working_directory(NULL); return result; } - sparse_filename = get_sparse_checkout_filename(); - fp = fopen(sparse_filename, "w"); + fp = xfdopen(fd, "w"); if (core_sparse_checkout_cone) write_cone_to_file(fp, pl); else write_patterns_to_file(fp, pl); - fclose(fp); + fflush(fp); + commit_lock_file(&lk); free(sparse_filename); clear_pattern_list(pl); diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh index a580b79595..2db706a9e4 100755 --- a/t/t1091-sparse-checkout-builtin.sh +++ b/t/t1091-sparse-checkout-builtin.sh @@ -276,4 +276,11 @@ test_expect_success 'revert to old sparse-checkout on empty update' ' ) ' +test_expect_success 'fail when lock is taken' ' + test_when_finished rm -rf repo/.git/info/sparse-checkout.lock && + touch repo/.git/info/sparse-checkout.lock && + test_must_fail git -C repo sparse-checkout set deep 2>err && + test_i18ngrep "File exists" err +' + test_done From patchwork Mon Oct 21 13:56:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11202257 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D1E9E1390 for ; Mon, 21 Oct 2019 13:56:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id B11D72166E for ; Mon, 21 Oct 2019 13:56:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="p61dO9mB" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729269AbfJUN4s (ORCPT ); Mon, 21 Oct 2019 09:56:48 -0400 Received: from mail-wr1-f65.google.com ([209.85.221.65]:36149 "EHLO mail-wr1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729257AbfJUN4o (ORCPT ); Mon, 21 Oct 2019 09:56:44 -0400 Received: by mail-wr1-f65.google.com with SMTP id w18so13574822wrt.3 for ; Mon, 21 Oct 2019 06:56:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=2OSiW1tIz6I5ZXpwz9+l07iHgNNKzetcZNlUCdKH8oM=; b=p61dO9mBxgWdq6aQQ7Ouf9Ss7o8V0ocZtXXLX90U8cWyveGF6T8kyIT3M+sP91B/09 GHvX32uhLvGqrIDtDgI3vAg1kUMsFELnnovPBAPh2C72TUPIxk0R5RiRERH4mWHgR16Z /KLC9yj96T4iWul58rOw3xVW2e8zjwr+vTw44CmgpUZaIaLJFElHGy0oEwe+L3ynobbL yRCGB9fKN2kIzko9T7ER5hBI8XsB4D5GbacenwHIHs1z7GOPkUOrOcX1y4uO+KVNkSwb glh73otvxNKY+Yj633mkmA+cRiCsrKwX9KM5AYPi3I9pCx3l4B7Uh1S9VFHo4p8T9MJ7 nFJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=2OSiW1tIz6I5ZXpwz9+l07iHgNNKzetcZNlUCdKH8oM=; b=QZJ4lhpj4Rs/ckcQdHUl0a3UoeVQpySPRZY/siLsf1dnatNM/A+jWCTAKmUFIxwUrJ 1sL8/DaDmtkY0wS837B7CCUhzSZs1x2NFNGy8P0o0q5pZ7C9cCCM1o/oEIvdwOitZl9+ qNTd/n3mIh12Y8KcdLcDjR3fAymHOfGW1oVg4vh7ZmUQSR2mFKxhhgwR1Wdh8FKCXUjz CrnFklKZVydWxKP1AxKmO+ytJ67HC3HPYNQWVMXXmL6G63NZbVgb7RqCDcAvK4Z9b9G4 5+7ToI3k2osLgO4jGP5l/98RjTWprRH/Zn2xjoUii8T7sMZ5UFxlo3viIhT6YszTWpaP +jVg== X-Gm-Message-State: APjAAAVPpveGbxJq8eHK+y6aoh143nOhZWAxKxqBa/KQhPFiyVE4OzbQ 9hnv5r0zOd2EwThtfAi5+XaX3Jix X-Google-Smtp-Source: APXvYqy1s115QDglSeNbtYvKwyob3szIL//xWa2z4o43Tcmz8OXU/AufMCrHfj0MZR9TNhZ1R8Z+/A== X-Received: by 2002:adf:b21a:: with SMTP id u26mr20896633wra.119.1571666202149; Mon, 21 Oct 2019 06:56:42 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o70sm21075723wme.29.2019.10.21.06.56.41 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 21 Oct 2019 06:56:41 -0700 (PDT) Message-Id: In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Mon, 21 Oct 2019 13:56:26 +0000 Subject: [PATCH v5 17/17] sparse-checkout: cone mode should not interact with .gitignore Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, jon@jonsimons.org, szeder.dev@gmail.com, Derrick Stolee , Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee During the development of the sparse-checkout "cone mode" feature, an incorrect placement of the initializer for "use_cone_patterns = 1" caused warnings to show up when a .gitignore file was present with non-cone-mode patterns. This was fixed in the original commit introducing the cone mode, but now we should add a test to avoid hitting this problem again in the future. Signed-off-by: Derrick Stolee --- t/t1091-sparse-checkout-builtin.sh | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh index 2db706a9e4..4205e03449 100755 --- a/t/t1091-sparse-checkout-builtin.sh +++ b/t/t1091-sparse-checkout-builtin.sh @@ -283,4 +283,11 @@ test_expect_success 'fail when lock is taken' ' test_i18ngrep "File exists" err ' +test_expect_success '.gitignore should not warn about cone mode' ' + git -C repo config --worktree core.sparseCheckoutCone true && + echo "**/bin/*" >repo/.gitignore && + git -C repo reset --hard 2>err && + test_i18ngrep ! "disabling cone patterns" err +' + test_done