From patchwork Thu Nov 21 22:04:33 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11256945 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E6D4913A4 for ; Thu, 21 Nov 2019 22:05:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id ADF602068E for ; Thu, 21 Nov 2019 22:05:06 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="OcSXf3DB" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726541AbfKUWE7 (ORCPT ); Thu, 21 Nov 2019 17:04:59 -0500 Received: from mail-wr1-f66.google.com ([209.85.221.66]:45721 "EHLO mail-wr1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726329AbfKUWE7 (ORCPT ); Thu, 21 Nov 2019 17:04:59 -0500 Received: by mail-wr1-f66.google.com with SMTP id z10so6259113wrs.12 for ; Thu, 21 Nov 2019 14:04:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=zuxJvocOqPdij+qW4nZ8mgj+xzeMp+CZFQRyarg28SA=; b=OcSXf3DBLhZvURTAnYaUDwJStYzWcZ0xoGL2izUGVpv9wURMMoeRFTGL7aofbsL/ZB FXMCLlax3/siRWYWfGFR5bN8WhrLiMRyouRf9/OkiIisVIF6CdLM046+JQJK40vck7cw 4O5FLoz+O7LL4uRQqr1d5AxLWO8Lydk32EpNVadgoWIua6650d5hSIHT6fEBF4toWlIj 0PaY1KPVlRmcnBlrai7aF1mh6AAJzdlwr5v3uiHowGR5hTvAfL5wBnIVNdAzYlnesph7 Taes0YMSpeEOGQA2/fr1bpd7/tbPZLcszVFFeqXIzk5xUReXJEoIksX1uoyn/O2Xscxk tRUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=zuxJvocOqPdij+qW4nZ8mgj+xzeMp+CZFQRyarg28SA=; b=T6ijRkVYQ3dJLFQwwv63ca1zLQuQiHO87WtxIYgMrI/t9v3RhftOwTj63FdU4SI2oS NR6pM60RDtXq6QyDi+315LfsARBu6VmalK6Ju7SOyMvP4bHBc3R2rPKBV33JwOQQk1dr uF+Yq5t/0N4Xd1nCTV+a/PatWCMtiO/YLrqKtADGHO8Or1hoTpfP4hM2aq5CCPoQyuxP jv4BxiGnFlb2vIi08TZiE5brydqtZ+evJZ0wCUKVhfZsoAJ2l5h4EXAvfA57H9oLcudy 8UURu1LLVR8htXGOUGYwSj16yb5KbLDqcePzClo2CBnyek3Kyyat1/RYR2Fk1nV8IAs6 TiHQ== X-Gm-Message-State: APjAAAVDXgzmHgE83mimr+Tif5ukInfYwy4sZUj4/YjB1ba14MT0FZSv eHEe/D17NuTnRGPoIbZugUh6uk3z X-Google-Smtp-Source: APXvYqxgbkNwA48p4rR7MqK43KEe8StOnTozUBDZyO72IrxfHkUm29hSA2qSGI76vrZ56gdZmMMQxw== X-Received: by 2002:adf:fc0a:: with SMTP id i10mr14445011wrr.105.1574373894902; Thu, 21 Nov 2019 14:04:54 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id y78sm1111825wmd.32.2019.11.21.14.04.54 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 21 Nov 2019 14:04:54 -0800 (PST) Message-Id: <81a15260695f83cf46185ed5357dd4f0e5dae384.1574373892.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 21 Nov 2019 22:04:33 +0000 Subject: [PATCH v6 01/19] sparse-checkout: create builtin with 'list' subcommand Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, jon@jonsimons.org, szeder.dev@gmail.com, Derrick Stolee , Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee The sparse-checkout feature is mostly hidden to users, as its only documentation is supplementary information in the docs for 'git read-tree'. In addition, users need to know how to edit the .git/info/sparse-checkout file with the right patterns, then run the appropriate 'git read-tree -mu HEAD' command. Keeping the working directory in sync with the sparse-checkout file requires care. Begin an effort to make the sparse-checkout feature a porcelain feature by creating a new 'git sparse-checkout' builtin. This builtin will be the preferred mechanism for manipulating the sparse-checkout file and syncing the working directory. The documentation provided is adapted from the "git read-tree" documentation with a few edits for clarity in the new context. Extra sections are added to hint toward a future change to a more restricted pattern set. Helped-by: Elijah Newren Signed-off-by: Derrick Stolee --- .gitignore | 1 + Documentation/git-read-tree.txt | 2 +- Documentation/git-sparse-checkout.txt | 89 +++++++++++++++++++++++++++ Makefile | 1 + builtin.h | 1 + builtin/sparse-checkout.c | 86 ++++++++++++++++++++++++++ command-list.txt | 1 + git.c | 1 + t/t1091-sparse-checkout-builtin.sh | 45 ++++++++++++++ 9 files changed, 226 insertions(+), 1 deletion(-) create mode 100644 Documentation/git-sparse-checkout.txt create mode 100644 builtin/sparse-checkout.c create mode 100755 t/t1091-sparse-checkout-builtin.sh diff --git a/.gitignore b/.gitignore index 89b3b79c1a..aebe7c0908 100644 --- a/.gitignore +++ b/.gitignore @@ -158,6 +158,7 @@ /git-show-branch /git-show-index /git-show-ref +/git-sparse-checkout /git-stage /git-stash /git-status diff --git a/Documentation/git-read-tree.txt b/Documentation/git-read-tree.txt index d271842608..da33f84f33 100644 --- a/Documentation/git-read-tree.txt +++ b/Documentation/git-read-tree.txt @@ -436,7 +436,7 @@ support. SEE ALSO -------- linkgit:git-write-tree[1]; linkgit:git-ls-files[1]; -linkgit:gitignore[5] +linkgit:gitignore[5]; linkgit:git-sparse-checkout[1]; GIT --- diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt new file mode 100644 index 0000000000..87ffcbbcb0 --- /dev/null +++ b/Documentation/git-sparse-checkout.txt @@ -0,0 +1,89 @@ +git-sparse-checkout(1) +====================== + +NAME +---- +git-sparse-checkout - Initialize and modify the sparse-checkout +configuration, which reduces the checkout to a set of paths +given by a list of atterns. + + +SYNOPSIS +-------- +[verse] +'git sparse-checkout [options]' + + +DESCRIPTION +----------- + +Initialize and modify the sparse-checkout configuration, which reduces +the checkout to a set of paths given by a list of patterns. + +THIS COMMAND IS EXPERIMENTAL. ITS BEHAVIOR, AND THE BEHAVIOR OF OTHER +COMMANDS IN THE PRESENCE OF SPARSE-CHECKOUTS, WILL LIKELY CHANGE IN +THE FUTURE. + + +COMMANDS +-------- +'list':: + Provide a list of the contents in the sparse-checkout file. + + +SPARSE CHECKOUT +--------------- + +"Sparse checkout" allows populating the working directory sparsely. +It uses the skip-worktree bit (see linkgit:git-update-index[1]) to tell +Git whether a file in the working directory is worth looking at. If +the skip-worktree bit is set, then the file is ignored in the working +directory. Git will not populate the contents of those files, which +makes a sparse checkout helpful when working in a repository with many +files, but only a few are important to the current user. + +The `$GIT_DIR/info/sparse-checkout` file is used to define the +skip-worktree reference bitmap. When Git updates the working +directory, it updates the skip-worktree bits in the index based +on this file. The files matching the patterns in the file will +appear in the working directory, and the rest will not. + +## FULL PATTERN SET + +By default, the sparse-checkout file uses the same syntax as `.gitignore` +files. + +While `$GIT_DIR/info/sparse-checkout` is usually used to specify what +files are included, you can also specify what files are _not_ included, +using negative patterns. For example, to remove the file `unwanted`: + +---------------- +/* +!unwanted +---------------- + +Another tricky thing is fully repopulating the working directory when you +no longer want sparse checkout. You cannot just disable "sparse +checkout" because skip-worktree bits are still in the index and your working +directory is still sparsely populated. You should re-populate the working +directory with the `$GIT_DIR/info/sparse-checkout` file content as +follows: + +---------------- +/* +---------------- + +Then you can disable sparse checkout. Sparse checkout support in 'git +checkout' and similar commands is disabled by default. You need to +set `core.sparseCheckout` to `true` in order to have sparse checkout +support. + +SEE ALSO +-------- + +linkgit:git-read-tree[1] +linkgit:gitignore[5] + +GIT +--- +Part of the linkgit:git[1] suite diff --git a/Makefile b/Makefile index de60c8e7aa..adefc229fe 100644 --- a/Makefile +++ b/Makefile @@ -1125,6 +1125,7 @@ BUILTIN_OBJS += builtin/shortlog.o BUILTIN_OBJS += builtin/show-branch.o BUILTIN_OBJS += builtin/show-index.o BUILTIN_OBJS += builtin/show-ref.o +BUILTIN_OBJS += builtin/sparse-checkout.o BUILTIN_OBJS += builtin/stash.o BUILTIN_OBJS += builtin/stripspace.o BUILTIN_OBJS += builtin/submodule--helper.o diff --git a/builtin.h b/builtin.h index 5cf5df69f7..2b25a80cde 100644 --- a/builtin.h +++ b/builtin.h @@ -225,6 +225,7 @@ int cmd_shortlog(int argc, const char **argv, const char *prefix); int cmd_show(int argc, const char **argv, const char *prefix); int cmd_show_branch(int argc, const char **argv, const char *prefix); int cmd_show_index(int argc, const char **argv, const char *prefix); +int cmd_sparse_checkout(int argc, const char **argv, const char *prefix); int cmd_status(int argc, const char **argv, const char *prefix); int cmd_stash(int argc, const char **argv, const char *prefix); int cmd_stripspace(int argc, const char **argv, const char *prefix); diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c new file mode 100644 index 0000000000..5717c9b2cb --- /dev/null +++ b/builtin/sparse-checkout.c @@ -0,0 +1,86 @@ +#include "builtin.h" +#include "config.h" +#include "dir.h" +#include "parse-options.h" +#include "pathspec.h" +#include "repository.h" +#include "run-command.h" +#include "strbuf.h" + +static char const * const builtin_sparse_checkout_usage[] = { + N_("git sparse-checkout list"), + NULL +}; + +static char *get_sparse_checkout_filename(void) +{ + return git_pathdup("info/sparse-checkout"); +} + +static void write_patterns_to_file(FILE *fp, struct pattern_list *pl) +{ + int i; + + for (i = 0; i < pl->nr; i++) { + struct path_pattern *p = pl->patterns[i]; + + if (p->flags & PATTERN_FLAG_NEGATIVE) + fprintf(fp, "!"); + + fprintf(fp, "%s", p->pattern); + + if (p->flags & PATTERN_FLAG_MUSTBEDIR) + fprintf(fp, "/"); + + fprintf(fp, "\n"); + } +} + +static int sparse_checkout_list(int argc, const char **argv) +{ + struct pattern_list pl; + char *sparse_filename; + int res; + + memset(&pl, 0, sizeof(pl)); + + sparse_filename = get_sparse_checkout_filename(); + res = add_patterns_from_file_to_list(sparse_filename, "", 0, &pl, NULL); + free(sparse_filename); + + if (res < 0) { + warning(_("this worktree is not sparse (sparse-checkout file may not exist)")); + return 0; + } + + write_patterns_to_file(stdout, &pl); + clear_pattern_list(&pl); + + return 0; +} + +int cmd_sparse_checkout(int argc, const char **argv, const char *prefix) +{ + static struct option builtin_sparse_checkout_options[] = { + OPT_END(), + }; + + if (argc == 2 && !strcmp(argv[1], "-h")) + usage_with_options(builtin_sparse_checkout_usage, + builtin_sparse_checkout_options); + + argc = parse_options(argc, argv, prefix, + builtin_sparse_checkout_options, + builtin_sparse_checkout_usage, + PARSE_OPT_STOP_AT_NON_OPTION); + + git_config(git_default_config, NULL); + + if (argc > 0) { + if (!strcmp(argv[0], "list")) + return sparse_checkout_list(argc, argv); + } + + usage_with_options(builtin_sparse_checkout_usage, + builtin_sparse_checkout_options); +} diff --git a/command-list.txt b/command-list.txt index a9ac72bef4..d3d28252b3 100644 --- a/command-list.txt +++ b/command-list.txt @@ -166,6 +166,7 @@ git-show-index plumbinginterrogators git-show-ref plumbinginterrogators git-sh-i18n purehelpers git-sh-setup purehelpers +git-sparse-checkout mainporcelain worktree git-stash mainporcelain git-stage complete git-status mainporcelain info diff --git a/git.c b/git.c index ce6ab0ece2..7be7ad34bd 100644 --- a/git.c +++ b/git.c @@ -572,6 +572,7 @@ static struct cmd_struct commands[] = { { "show-branch", cmd_show_branch, RUN_SETUP }, { "show-index", cmd_show_index }, { "show-ref", cmd_show_ref, RUN_SETUP }, + { "sparse-checkout", cmd_sparse_checkout, RUN_SETUP | NEED_WORK_TREE }, { "stage", cmd_add, RUN_SETUP | NEED_WORK_TREE }, /* * NEEDSWORK: Until the builtin stash is thoroughly robust and no diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh new file mode 100755 index 0000000000..9b73d44907 --- /dev/null +++ b/t/t1091-sparse-checkout-builtin.sh @@ -0,0 +1,45 @@ +#!/bin/sh + +test_description='sparse checkout builtin tests' + +. ./test-lib.sh + +test_expect_success 'setup' ' + git init repo && + ( + cd repo && + echo "initial" >a && + mkdir folder1 folder2 deep && + mkdir deep/deeper1 deep/deeper2 && + mkdir deep/deeper1/deepest && + cp a folder1 && + cp a folder2 && + cp a deep && + cp a deep/deeper1 && + cp a deep/deeper2 && + cp a deep/deeper1/deepest && + git add . && + git commit -m "initial commit" + ) +' + +test_expect_success 'git sparse-checkout list (empty)' ' + git -C repo sparse-checkout list >list 2>err && + test_must_be_empty list && + test_i18ngrep "this worktree is not sparse (sparse-checkout file may not exist)" err +' + +test_expect_success 'git sparse-checkout list (populated)' ' + test_when_finished rm -f repo/.git/info/sparse-checkout && + cat >repo/.git/info/sparse-checkout <<-EOF && + /folder1/* + /deep/ + **/a + !*bin* + EOF + cp repo/.git/info/sparse-checkout expect && + git -C repo sparse-checkout list >list && + test_cmp expect list +' + +test_done From patchwork Thu Nov 21 22:04:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11256949 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4809F1871 for ; Thu, 21 Nov 2019 22:05:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 1E5612068E for ; Thu, 21 Nov 2019 22:05:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="rAOUnSH9" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726634AbfKUWFA (ORCPT ); Thu, 21 Nov 2019 17:05:00 -0500 Received: from mail-wm1-f66.google.com ([209.85.128.66]:52329 "EHLO mail-wm1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726380AbfKUWFA (ORCPT ); Thu, 21 Nov 2019 17:05:00 -0500 Received: by mail-wm1-f66.google.com with SMTP id l1so5424877wme.2 for ; Thu, 21 Nov 2019 14:04:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=suGgwvchBkO3S3lXVcP+pqRoztUaFjvojE6ACC2GhY8=; b=rAOUnSH9w1T/U13S7LyfsEjxSiPTPjhFTdUJQLglHd0Y356KymGyiMrE0HIy03C1F3 t0BPmZhdT8hE+CQ357iI550nBYb0rnlbCvD/uega3UJuSWahM6GIeYlSDujvwpxI2mHx ZSs4G+RA6vjoz/5yl4nwOjmkvHeO3uTVMnA6JaelJT2acQtECsyfeqZwBrVeq0bYcWsq SKTkQLjyTmbP8ys9XXR2iYZwjbGA4h3wf1x3DzJDs0J8W+U8+icJ5J7EhltCOlJEIDl/ k0V18A9CzrtBm7T2VMIFyL+i9aLD7j98SBUqWKICvOheUrTo1JOFL1CnD1SUpLZyAuUO Ce3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=suGgwvchBkO3S3lXVcP+pqRoztUaFjvojE6ACC2GhY8=; b=kZnzx6uWj8Uvhijy1YPAc6g6wM+ha7fwhZf2fblwKjMcJn1ffL2MTPJZNK4QMuw6Eo mud7+6Cyg/RnHvKwb2JbuDugxHBZgcE7ywN4C+7vLLOdF0UXPgPsWHOGb/iiiw0jjk7O fg6NRUMYPPsnsKbwAbQMZ90hxB+69018hTFAQkYPm9UO2gxawjcDy2rj+LvpI6JgLKD2 etvuIYaRWUQMBr1pPFPIcpczQafV3R6YWgHqRTtC3mcNVDtz0/Qn1jHsSlft/YtgdAVJ m5kOl5cZxtGzK0aXU/vMTFosdQSxTd0BakVGOc5s1oAmzW56TR+W4MmUyoBaOiXqa1HH zsxQ== X-Gm-Message-State: APjAAAUkfnaCrCsfSi724mybAm3nqVBoyoxUd9nB6bUjah3g+1dYCxGF 1iHVfLzBBDWX8nGbQkeb8nz+BrpN X-Google-Smtp-Source: APXvYqw5dzPbHmIxvaESnrqsAcjX4QOBAPQvwgGbg7lSC7wvv8kQishk9/cK7nWkGbGN/mJWN484lw== X-Received: by 2002:a1c:f415:: with SMTP id z21mr13119853wma.140.1574373896903; Thu, 21 Nov 2019 14:04:56 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id v128sm1247932wmb.14.2019.11.21.14.04.56 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 21 Nov 2019 14:04:56 -0800 (PST) Message-Id: <7ef5f2ef693153a0f786e1d913d4258059cd587a.1574373892.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 21 Nov 2019 22:04:35 +0000 Subject: [PATCH v6 03/19] clone: add --sparse mode Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, jon@jonsimons.org, szeder.dev@gmail.com, Derrick Stolee , Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee When someone wants to clone a large repository, but plans to work using a sparse-checkout file, they either need to do a full checkout first and then reduce the patterns they included, or clone with --no-checkout, set up their patterns, and then run a checkout manually. This requires knowing a lot about the repo shape and how sparse-checkout works. Add a new '--sparse' option to 'git clone' that initializes the sparse-checkout file to include the following patterns: /* !/*/ These patterns include every file in the root directory, but no directories. This allows a repo to include files like a README or a bootstrapping script to grow enlistments from that point. During the 'git sparse-checkout init' call, we must first look to see if HEAD is valid, since 'git clone' does not have a valid HEAD at the point where it initializes the sparse-checkout. The following checkout within the clone command will create the HEAD ref and update the working directory correctly. Signed-off-by: Derrick Stolee --- Documentation/git-clone.txt | 8 +++++++- builtin/clone.c | 27 +++++++++++++++++++++++++++ builtin/sparse-checkout.c | 6 ++++++ t/t1091-sparse-checkout-builtin.sh | 13 +++++++++++++ 4 files changed, 53 insertions(+), 1 deletion(-) diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt index 34011c2940..0fe91d2f04 100644 --- a/Documentation/git-clone.txt +++ b/Documentation/git-clone.txt @@ -15,7 +15,7 @@ SYNOPSIS [--dissociate] [--separate-git-dir ] [--depth ] [--[no-]single-branch] [--no-tags] [--recurse-submodules[=]] [--[no-]shallow-submodules] - [--[no-]remote-submodules] [--jobs ] [--] + [--[no-]remote-submodules] [--jobs ] [--sparse] [--] [] DESCRIPTION @@ -156,6 +156,12 @@ objects from the source repository into a pack in the cloned repository. used, neither remote-tracking branches nor the related configuration variables are created. +--sparse:: + Initialize the sparse-checkout file so the working + directory starts with only the files in the root + of the repository. The sparse-checkout file can be + modified to grow the working directory as needed. + --mirror:: Set up a mirror of the source repository. This implies `--bare`. Compared to `--bare`, `--mirror` not only maps local branches of the diff --git a/builtin/clone.c b/builtin/clone.c index c46ee29f0a..4348d962c9 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -59,6 +59,7 @@ static const char *real_git_dir; static char *option_upload_pack = "git-upload-pack"; static int option_verbosity; static int option_progress = -1; +static int option_sparse_checkout; static enum transport_family family; static struct string_list option_config = STRING_LIST_INIT_NODUP; static struct string_list option_required_reference = STRING_LIST_INIT_NODUP; @@ -146,6 +147,8 @@ static struct option builtin_clone_options[] = { OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options), OPT_BOOL(0, "remote-submodules", &option_remote_submodules, N_("any cloned submodules will use their remote-tracking branch")), + OPT_BOOL(0, "sparse", &option_sparse_checkout, + N_("initialize sparse-checkout file to include only files at root")), OPT_END() }; @@ -733,6 +736,27 @@ static void update_head(const struct ref *our, const struct ref *remote, } } +static int git_sparse_checkout_init(const char *repo) +{ + struct argv_array argv = ARGV_ARRAY_INIT; + int result = 0; + argv_array_pushl(&argv, "-C", repo, "sparse-checkout", "init", NULL); + + /* + * We must apply the setting in the current process + * for the later checkout to use the sparse-checkout file. + */ + core_apply_sparse_checkout = 1; + + if (run_command_v_opt(argv.argv, RUN_GIT_CMD)) { + error(_("failed to initialize sparse-checkout")); + result = 1; + } + + argv_array_clear(&argv); + return result; +} + static int checkout(int submodule_progress) { struct object_id oid; @@ -1106,6 +1130,9 @@ int cmd_clone(int argc, const char **argv, const char *prefix) if (option_required_reference.nr || option_optional_reference.nr) setup_reference(); + if (option_sparse_checkout && git_sparse_checkout_init(repo)) + return 1; + remote = remote_get(option_origin); strbuf_addf(&default_refspec, "+%s*:%s*", src_ref_prefix, diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c index fcf97e9df8..e3418fbe2a 100644 --- a/builtin/sparse-checkout.c +++ b/builtin/sparse-checkout.c @@ -102,6 +102,7 @@ static int sparse_checkout_init(int argc, const char **argv) char *sparse_filename; FILE *fp; int res; + struct object_id oid; if (set_config(MODE_ALL_PATTERNS)) return 1; @@ -126,6 +127,11 @@ static int sparse_checkout_init(int argc, const char **argv) fprintf(fp, "/*\n!/*/\n"); fclose(fp); + if (get_oid("HEAD", &oid)) { + /* assume we are in a fresh repo */ + return 0; + } + reset_dir: return update_working_directory(); } diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh index 21143c529c..78c20cb7e3 100755 --- a/t/t1091-sparse-checkout-builtin.sh +++ b/t/t1091-sparse-checkout-builtin.sh @@ -82,4 +82,17 @@ test_expect_success 'init with existing sparse-checkout' ' test_cmp expect dir ' +test_expect_success 'clone --sparse' ' + git clone --sparse repo clone && + git -C clone sparse-checkout list >actual && + cat >expect <<-EOF && + /* + !/*/ + EOF + test_cmp expect actual && + ls clone >dir && + echo a >expect && + test_cmp expect dir +' + test_done From patchwork Thu Nov 21 22:04:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11256951 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7AD4314DB for ; Thu, 21 Nov 2019 22:05:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 516582068E for ; Thu, 21 Nov 2019 22:05:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Aya0eymv" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726714AbfKUWFD (ORCPT ); Thu, 21 Nov 2019 17:05:03 -0500 Received: from mail-wm1-f66.google.com ([209.85.128.66]:37753 "EHLO mail-wm1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726510AbfKUWFB (ORCPT ); Thu, 21 Nov 2019 17:05:01 -0500 Received: by mail-wm1-f66.google.com with SMTP id f129so4113312wmf.2 for ; Thu, 21 Nov 2019 14:04:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=mqWQvZmcTJamruP6gGMqpeVPBu5fmRJkGWAyy4kBscc=; b=Aya0eymvgvbiytn915bdn8Xim/Tz//AGsqA2++gYkqZppbdU5k3kizoT5IJUI3pMXE ajoUquHDkxLXhy2WrDlL4qoU/+gK466ShXaBLuw7vdYuCrtqWH+aB+DUIT1ZQvOaqbKs h1x7IqbJFkAhB/aKJnZqOeONIyFsQEpJCgQnmnX4A8G5b9Hd2aUUD98xM7svZOmwuvPq erhcIbqGM0oW7rP3WvOrBUPABLDQRzd9ueWjuiYlIg/2cHVbj2++bamhesPyWNLUf33l BqHpYgzF+4yw94x71sQh8LAMHue7lluw8wXlFu0GhqQlT+GpnDnkwDOE84WGtwRVZFS7 qokg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=mqWQvZmcTJamruP6gGMqpeVPBu5fmRJkGWAyy4kBscc=; b=lP3fwA2GPA+SkT2Y58ond/LAY1uaZ3SwKKyh7s+ZrCU+T8eb3W7HJ7yZP4dEXuub+6 URnl037PWFWulZG+AsmGXQZisDaK607XKmjf5m8OPuj46Pd1fgnQ5R+Pz6wo1iywREjr NdO+h+CgJ4JPCII7cMK3RN3myuMluDdJs9xdNq6vDzESETEsd5q5PyegRgsAyGczY2sC 1ziCRSJa+xtuhXsAZ4HIrxPFtwNm8uxTFWVlzeF87r4AYD8JEQmywvhpNsbe9YH80bBf U+TfLJLXGfQb5pKJ7c69q5kYYH655IgbgVeWHSKJERs+US0zKLp1NJ2JSIeXd5jJZMO0 wvSQ== X-Gm-Message-State: APjAAAW3kkYiYb9KCE+mXQZYJB0eoIvupiKB7JMP8hMC8Ix+x45S7R6X 32svE+79pgZeKcJk2dq1gTUya5ZH X-Google-Smtp-Source: APXvYqzjCTTv2ujDkeMmxJS19beMAsPWO2r8avUuGxQZVHe/nNYuSErI5eFI8N3LsN66Y6YyOi1m1g== X-Received: by 2002:a05:600c:2389:: with SMTP id m9mr13194700wma.65.1574373897958; Thu, 21 Nov 2019 14:04:57 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id b3sm1029474wmj.44.2019.11.21.14.04.57 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 21 Nov 2019 14:04:57 -0800 (PST) Message-Id: <248fc172acc1cb3d0fdb5c3cf4ac91a51d0c85d9.1574373892.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 21 Nov 2019 22:04:36 +0000 Subject: [PATCH v6 04/19] sparse-checkout: 'set' subcommand Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, jon@jonsimons.org, szeder.dev@gmail.com, Derrick Stolee , Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee The 'git sparse-checkout set' subcommand takes a list of patterns as arguments and writes them to the sparse-checkout file. Then, it updates the working directory using 'git read-tree -mu HEAD'. The 'set' subcommand will replace the entire contents of the sparse-checkout file. The write_patterns_and_update() method is extracted from cmd_sparse_checkout() to make it easier to implement 'add' and/or 'remove' subcommands in the future. If the core.sparseCheckout config setting is disabled, then enable the config setting in the worktree config. If we set the config this way and the sparse-checkout fails, then re-disable the config setting. Signed-off-by: Derrick Stolee --- Documentation/git-sparse-checkout.txt | 6 ++++ builtin/sparse-checkout.c | 47 +++++++++++++++++++++++++-- t/t1091-sparse-checkout-builtin.sh | 33 +++++++++++++++++++ 3 files changed, 84 insertions(+), 2 deletions(-) diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt index 491be1345f..ca62669b8c 100644 --- a/Documentation/git-sparse-checkout.txt +++ b/Documentation/git-sparse-checkout.txt @@ -42,6 +42,12 @@ To avoid interfering with other worktrees, it first enables the `extensions.worktreeConfig` setting and makes sure to set the `core.sparseCheckout` setting in the worktree-specific config file. +'set':: + Write a set of patterns to the sparse-checkout file, as given as + a list of arguments following the 'set' subcommand. Update the + working directory to match the new patterns. Enable the + core.sparseCheckout config setting if it is not already enabled. + SPARSE CHECKOUT --------------- diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c index e3418fbe2a..95cbd0a42c 100644 --- a/builtin/sparse-checkout.c +++ b/builtin/sparse-checkout.c @@ -8,7 +8,7 @@ #include "strbuf.h" static char const * const builtin_sparse_checkout_usage[] = { - N_("git sparse-checkout (init|list)"), + N_("git sparse-checkout (init|list|set) "), NULL }; @@ -66,7 +66,7 @@ static int update_working_directory(void) argv_array_pushl(&argv, "read-tree", "-m", "-u", "HEAD", NULL); if (run_command_v_opt(argv.argv, RUN_GIT_CMD)) { - error(_("failed to update index with new sparse-checkout paths")); + error(_("failed to update index with new sparse-checkout patterns")); result = 1; } @@ -136,6 +136,47 @@ static int sparse_checkout_init(int argc, const char **argv) return update_working_directory(); } +static int write_patterns_and_update(struct pattern_list *pl) +{ + char *sparse_filename; + FILE *fp; + + sparse_filename = get_sparse_checkout_filename(); + fp = fopen(sparse_filename, "w"); + write_patterns_to_file(fp, pl); + fclose(fp); + free(sparse_filename); + + return update_working_directory(); +} + +static int sparse_checkout_set(int argc, const char **argv, const char *prefix) +{ + static const char *empty_base = ""; + int i; + struct pattern_list pl; + int result; + int changed_config = 0; + memset(&pl, 0, sizeof(pl)); + + for (i = 1; i < argc; i++) + add_pattern(argv[i], empty_base, 0, &pl, 0); + + if (!core_apply_sparse_checkout) { + set_config(MODE_ALL_PATTERNS); + core_apply_sparse_checkout = 1; + changed_config = 1; + } + + result = write_patterns_and_update(&pl); + + if (result && changed_config) + set_config(MODE_NO_PATTERNS); + + clear_pattern_list(&pl); + return result; +} + int cmd_sparse_checkout(int argc, const char **argv, const char *prefix) { static struct option builtin_sparse_checkout_options[] = { @@ -158,6 +199,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix) return sparse_checkout_list(argc, argv); if (!strcmp(argv[0], "init")) return sparse_checkout_init(argc, argv); + if (!strcmp(argv[0], "set")) + return sparse_checkout_set(argc, argv, prefix); } usage_with_options(builtin_sparse_checkout_usage, diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh index 78c20cb7e3..72d8bc5c25 100755 --- a/t/t1091-sparse-checkout-builtin.sh +++ b/t/t1091-sparse-checkout-builtin.sh @@ -95,4 +95,37 @@ test_expect_success 'clone --sparse' ' test_cmp expect dir ' +test_expect_success 'set enables config' ' + git init empty-config && + ( + cd empty-config && + test_commit test file && + test_path_is_missing .git/config.worktree && + test_must_fail git sparse-checkout set nothing && + test_path_is_file .git/config.worktree && + test_must_fail git config core.sparseCheckout && + git sparse-checkout set "/*" && + test_cmp_config true core.sparseCheckout + ) +' + +test_expect_success 'set sparse-checkout using builtin' ' + git -C repo sparse-checkout set "/*" "!/*/" "*folder*" && + cat >expect <<-EOF && + /* + !/*/ + *folder* + EOF + git -C repo sparse-checkout list >actual && + test_cmp expect actual && + test_cmp expect repo/.git/info/sparse-checkout && + ls repo >dir && + cat >expect <<-EOF && + a + folder1 + folder2 + EOF + test_cmp expect dir +' + test_done From patchwork Thu Nov 21 22:04:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11256957 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B9722930 for ; Thu, 21 Nov 2019 22:05:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 9644F2068E for ; Thu, 21 Nov 2019 22:05:09 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="IekN0+i9" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726704AbfKUWFD (ORCPT ); Thu, 21 Nov 2019 17:05:03 -0500 Received: from mail-wm1-f67.google.com ([209.85.128.67]:40671 "EHLO mail-wm1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726329AbfKUWFB (ORCPT ); Thu, 21 Nov 2019 17:05:01 -0500 Received: by mail-wm1-f67.google.com with SMTP id y5so5443359wmi.5 for ; Thu, 21 Nov 2019 14:04:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Hu0t5CgsSWWSJsNVgcqafIWPol6C25UdVdqh+XEAbMM=; b=IekN0+i9/FtvUVSM4zX2c+j+Ga0bO0RLKPUSTycD7PkV4e6voK/iLB9ESy+/NALQNt jyfPrYMVv8B8EMvpr34CaB3HIvrfzOxqBGq0+yVmXCbphWvvLks8PIcU0CL9LnLVwh7y mXSZPVA6aandkSUhw9EISv06D/wwHjFl1NTfnqGobyJIXJG8ZfZM5oQRaJRUO3zn25hh kVbNEVwp+dVGKIbDUdLXKcB/qBILWMaxB+2srDXvVZINLw9lfGskVRrEr9h7LyxpQAy8 WcOJ4KuNrCticTMYuTPT64tEcp1j4XNPR8dcIUclcE3JRzzw12wxze9kUweJqVgOQupQ O62w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Hu0t5CgsSWWSJsNVgcqafIWPol6C25UdVdqh+XEAbMM=; b=DLPmNJIaw0FPhEyAgqkmkqBWb70L5yAGA2mMDrFctTQcHTAgrd/FXbterN5sGpHfmb n59/KaZLKYPTYLFL7BtYJRyHE3S5nSpaDPYxyYlBoShLbpvIykOVTUC2Ogk0gM1u5XhQ EfoOm17/sUerYHDnkkoipK1k5U9fjeISIIxiEi5Wd59YLcxH3SvFVqM0t8cKHEjqxTOm sFt77tL36CgOHsVyhbm/ff2BQv9QLmDqycmuw3m0uNPK7OZd98F4lhSsPSAxXlYQf30O 5o/IHsNPUQGJEWw44nDgLi7LnQkyi115Sis/gUsaMqKlyI3tXhbBXp7pnZViIYOPP9Uc imnQ== X-Gm-Message-State: APjAAAWJx9rWuZAC09XOc+g/Cy9/2HAIQjYbleuyqxHO5lkbYesCAqq/ t1UqoxuiAvm1Ijt0EjvS8rowJynK X-Google-Smtp-Source: APXvYqwwreV+F4BZIwSJM7zIdTzGZw/26PWVBYFVEe0UmpnSSW6vYQvFNuIKAeTLmFUV7JO437rPQg== X-Received: by 2002:a1c:e08a:: with SMTP id x132mr13026833wmg.146.1574373898706; Thu, 21 Nov 2019 14:04:58 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id d7sm5091711wrx.11.2019.11.21.14.04.58 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 21 Nov 2019 14:04:58 -0800 (PST) Message-Id: <751798216f35a4d26563898b639d0757eca9c5ff.1574373892.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 21 Nov 2019 22:04:37 +0000 Subject: [PATCH v6 05/19] sparse-checkout: add '--stdin' option to set subcommand Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, jon@jonsimons.org, szeder.dev@gmail.com, Derrick Stolee , Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee The 'git sparse-checkout set' subcommand takes a list of patterns and places them in the sparse-checkout file. Then, it updates the working directory to match those patterns. For a large list of patterns, the command-line call can get very cumbersome. Add a '--stdin' option to instead read patterns over standard in. Signed-off-by: Derrick Stolee --- Documentation/git-sparse-checkout.txt | 3 +++ builtin/sparse-checkout.c | 34 +++++++++++++++++++++++++-- t/t1091-sparse-checkout-builtin.sh | 20 ++++++++++++++++ 3 files changed, 55 insertions(+), 2 deletions(-) diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt index ca62669b8c..a724eae09c 100644 --- a/Documentation/git-sparse-checkout.txt +++ b/Documentation/git-sparse-checkout.txt @@ -47,6 +47,9 @@ To avoid interfering with other worktrees, it first enables the a list of arguments following the 'set' subcommand. Update the working directory to match the new patterns. Enable the core.sparseCheckout config setting if it is not already enabled. ++ +When the `--stdin` option is provided, the patterns are read from +standard in as a newline-delimited list instead of from the arguments. SPARSE CHECKOUT --------------- diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c index 95cbd0a42c..82bff0020d 100644 --- a/builtin/sparse-checkout.c +++ b/builtin/sparse-checkout.c @@ -150,6 +150,15 @@ static int write_patterns_and_update(struct pattern_list *pl) return update_working_directory(); } +static char const * const builtin_sparse_checkout_set_usage[] = { + N_("git sparse-checkout set (--stdin | )"), + NULL +}; + +static struct sparse_checkout_set_opts { + int use_stdin; +} set_opts; + static int sparse_checkout_set(int argc, const char **argv, const char *prefix) { static const char *empty_base = ""; @@ -157,10 +166,31 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix) struct pattern_list pl; int result; int changed_config = 0; + + static struct option builtin_sparse_checkout_set_options[] = { + OPT_BOOL(0, "stdin", &set_opts.use_stdin, + N_("read patterns from standard in")), + OPT_END(), + }; + memset(&pl, 0, sizeof(pl)); - for (i = 1; i < argc; i++) - add_pattern(argv[i], empty_base, 0, &pl, 0); + argc = parse_options(argc, argv, prefix, + builtin_sparse_checkout_set_options, + builtin_sparse_checkout_set_usage, + PARSE_OPT_KEEP_UNKNOWN); + + if (set_opts.use_stdin) { + struct strbuf line = STRBUF_INIT; + + while (!strbuf_getline(&line, stdin)) { + char *buf = strbuf_detach(&line, NULL); + add_pattern(buf, empty_base, 0, &pl, 0); + } + } else { + for (i = 0; i < argc; i++) + add_pattern(argv[i], empty_base, 0, &pl, 0); + } if (!core_apply_sparse_checkout) { set_config(MODE_ALL_PATTERNS); diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh index 72d8bc5c25..07e73b4674 100755 --- a/t/t1091-sparse-checkout-builtin.sh +++ b/t/t1091-sparse-checkout-builtin.sh @@ -128,4 +128,24 @@ test_expect_success 'set sparse-checkout using builtin' ' test_cmp expect dir ' +test_expect_success 'set sparse-checkout using --stdin' ' + cat >expect <<-EOF && + /* + !/*/ + /folder1/ + /folder2/ + EOF + git -C repo sparse-checkout set --stdin actual && + test_cmp expect actual && + test_cmp expect repo/.git/info/sparse-checkout && + ls repo >dir && + cat >expect <<-EOF && + a + folder1 + folder2 + EOF + test_cmp expect dir +' + test_done From patchwork Thu Nov 21 22:04:38 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11256959 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DD7E114DB for ; Thu, 21 Nov 2019 22:05:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id BE6892068E for ; Thu, 21 Nov 2019 22:05:09 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="BM/AdlbE" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726762AbfKUWFI (ORCPT ); Thu, 21 Nov 2019 17:05:08 -0500 Received: from mail-wm1-f65.google.com ([209.85.128.65]:55202 "EHLO mail-wm1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726380AbfKUWFD (ORCPT ); Thu, 21 Nov 2019 17:05:03 -0500 Received: by mail-wm1-f65.google.com with SMTP id x26so5129324wmk.4 for ; Thu, 21 Nov 2019 14:05:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=BLwAFH7+821xmGEVunmv2X4aeXkGjSS/WiZj+vxjF2o=; b=BM/AdlbE9MDVloBjbsnuaV+fcOMYOAMVNExvuxsU6DwWpEvQ7KL+THgELPdlDTOGj8 /JrNb8zxDiCqTsrkIdmC1q4HYC2dXFhMgBGSa9LE6e1rrO5gFfrdoaWG5XuC2rATeoDe UibYC3Qkw40HKnmG342PNjF5x3rko7+onqq4lsBaPDrzKOlP354xT9KgXdhsfhDDOiW3 zJKhYCf8xEMGse9/D2Ukw6tUFnlGHQUs//yFnWkqgla7r76JEdjVsL72RJnmjHpx+Ooi m6/uQ7fl6j2ZD6x3dGM+3yktD77buDY+YMX/H5aJevofLqvxYyn4VDhUAAce9fXjYwuL MAwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=BLwAFH7+821xmGEVunmv2X4aeXkGjSS/WiZj+vxjF2o=; b=SPqunbDl6nejH70KIMy4X5fiM9pAKBKa/DZI6VQoaMoEOCVPZKV53mmFzV5dvKGbm3 HC3AvmzDMp8H7yrsTOsy1z739q/LVyWwMP2oKMWpcVIWfODmF5B0SWnrkAZ3yGhYgTdQ wH+iuAxYGfunjm1G1x1x6pwRtST3V1J/LwiUng/p52ia3gzwdCxZ0rZxjlz9fX7GpH6C vNDrDr9v3YNW2bOaiKZMAmO4D3zAZoiQC7UmE/f/oWHsISwB8JT7xCKRY6HYdIS8C1LK 9iIaeXOYwQFu+EWJajISEZQ+AnxUO61MuQHg187P+9ubSbmLjOX/KxJpXr9Gq3jfk7o6 hpCg== X-Gm-Message-State: APjAAAXibNSFij9yckR5DfrxnNMP752phyz2FuaxAkjaAohXxfKB3QUY I+Wn+13TGQFBi7Qff6hf3jcxLD59 X-Google-Smtp-Source: APXvYqw1N+XTKZT3r2QZ99aosDLWQtrE/umZAExRI0k02x4jm/W6zZ5D/3UT7pu01E2hBFt+lkxPdw== X-Received: by 2002:a05:600c:2102:: with SMTP id u2mr12791134wml.49.1574373899511; Thu, 21 Nov 2019 14:04:59 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id v9sm4601654wrs.95.2019.11.21.14.04.58 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 21 Nov 2019 14:04:59 -0800 (PST) Message-Id: <6431141e0385c2d700fd213d9ad8238f75f1cab3.1574373892.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 21 Nov 2019 22:04:38 +0000 Subject: [PATCH v6 06/19] sparse-checkout: create 'disable' subcommand Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, jon@jonsimons.org, szeder.dev@gmail.com, Derrick Stolee , Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee The instructions for disabling a sparse-checkout to a full working directory are complicated and non-intuitive. Add a subcommand, 'git sparse-checkout disable', to perform those steps for the user. Signed-off-by: Derrick Stolee --- Documentation/git-sparse-checkout.txt | 27 ++++++++++++--------------- builtin/sparse-checkout.c | 26 +++++++++++++++++++++++++- t/t1091-sparse-checkout-builtin.sh | 15 +++++++++++++++ 3 files changed, 52 insertions(+), 16 deletions(-) diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt index a724eae09c..c2cb19f80d 100644 --- a/Documentation/git-sparse-checkout.txt +++ b/Documentation/git-sparse-checkout.txt @@ -51,6 +51,10 @@ To avoid interfering with other worktrees, it first enables the When the `--stdin` option is provided, the patterns are read from standard in as a newline-delimited list instead of from the arguments. +'disable':: + Remove the sparse-checkout file, set `core.sparseCheckout` to + `false`, and restore the working directory to include all files. + SPARSE CHECKOUT --------------- @@ -68,6 +72,14 @@ directory, it updates the skip-worktree bits in the index based on this file. The files matching the patterns in the file will appear in the working directory, and the rest will not. +To enable the sparse-checkout feature, run `git sparse-checkout init` to +initialize a simple sparse-checkout file and enable the `core.sparseCheckout` +config setting. Then, run `git sparse-checkout set` to modify the patterns in +the sparse-checkout file. + +To repopulate the working directory with all files, use the +`git sparse-checkout disable` command. + ## FULL PATTERN SET By default, the sparse-checkout file uses the same syntax as `.gitignore` @@ -82,21 +94,6 @@ using negative patterns. For example, to remove the file `unwanted`: !unwanted ---------------- -Another tricky thing is fully repopulating the working directory when you -no longer want sparse checkout. You cannot just disable "sparse -checkout" because skip-worktree bits are still in the index and your working -directory is still sparsely populated. You should re-populate the working -directory with the `$GIT_DIR/info/sparse-checkout` file content as -follows: - ----------------- -/* ----------------- - -Then you can disable sparse checkout. Sparse checkout support in 'git -checkout' and similar commands is disabled by default. You need to -set `core.sparseCheckout` to `true` in order to have sparse checkout -support. SEE ALSO -------- diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c index 82bff0020d..e3a8d3460a 100644 --- a/builtin/sparse-checkout.c +++ b/builtin/sparse-checkout.c @@ -8,7 +8,7 @@ #include "strbuf.h" static char const * const builtin_sparse_checkout_usage[] = { - N_("git sparse-checkout (init|list|set) "), + N_("git sparse-checkout (init|list|set|disable) "), NULL }; @@ -207,6 +207,28 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix) return result; } +static int sparse_checkout_disable(int argc, const char **argv) +{ + char *sparse_filename; + FILE *fp; + + if (set_config(MODE_ALL_PATTERNS)) + die(_("failed to change config")); + + sparse_filename = get_sparse_checkout_filename(); + fp = xfopen(sparse_filename, "w"); + fprintf(fp, "/*\n"); + fclose(fp); + + if (update_working_directory()) + die(_("error while refreshing working directory")); + + unlink(sparse_filename); + free(sparse_filename); + + return set_config(MODE_NO_PATTERNS); +} + int cmd_sparse_checkout(int argc, const char **argv, const char *prefix) { static struct option builtin_sparse_checkout_options[] = { @@ -231,6 +253,8 @@ int cmd_sparse_checkout(int argc, const char **argv, const char *prefix) return sparse_checkout_init(argc, argv); if (!strcmp(argv[0], "set")) return sparse_checkout_set(argc, argv, prefix); + if (!strcmp(argv[0], "disable")) + return sparse_checkout_disable(argc, argv); } usage_with_options(builtin_sparse_checkout_usage, diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh index 07e73b4674..c385c62c92 100755 --- a/t/t1091-sparse-checkout-builtin.sh +++ b/t/t1091-sparse-checkout-builtin.sh @@ -148,4 +148,19 @@ test_expect_success 'set sparse-checkout using --stdin' ' test_cmp expect dir ' +test_expect_success 'sparse-checkout disable' ' + git -C repo sparse-checkout disable && + test_path_is_missing repo/.git/info/sparse-checkout && + git -C repo config --list >config && + test_must_fail git config core.sparseCheckout && + ls repo >dir && + cat >expect <<-EOF && + a + deep + folder1 + folder2 + EOF + test_cmp expect dir +' + test_done From patchwork Thu Nov 21 22:04:39 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11256961 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 117171871 for ; Thu, 21 Nov 2019 22:05:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id E63E8206D8 for ; Thu, 21 Nov 2019 22:05:09 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="k2nzkk/G" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726767AbfKUWFJ (ORCPT ); Thu, 21 Nov 2019 17:05:09 -0500 Received: from mail-wr1-f67.google.com ([209.85.221.67]:43598 "EHLO mail-wr1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726666AbfKUWFE (ORCPT ); Thu, 21 Nov 2019 17:05:04 -0500 Received: by mail-wr1-f67.google.com with SMTP id n1so6270976wra.10 for ; Thu, 21 Nov 2019 14:05:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=jp9fCPiVi8uv07s6DelseXp9idVSa4eBTh9PuKubi+E=; b=k2nzkk/G3edVtdwPWaMBa2iMgwUiy4x4HaObAlxhFlOdSIZDkzef2VR0S8K4lxjZC1 JOdj1u5J8iV7HUI6VMb/McOYXDWpNOMMPHsHQH49xuCG3GkwEzviqpIlgoRmbK6lTvkW exyJnxjniI+X4ybcdoW3nKAsqGhAeYpQugBaP/TKXMBukmBSPdJbtQthjFxln5gV7F1M VqtW+ngj/Gk3v2ERqSZfUysOEQ9yYIlN4t3jBCRNfohOp7/IWGNfKcVoidKPobTgq1QD 8px+9Hxk/c/PyMkbGF6bzD99AnqiEm+sqR2SfiHLyUS6cziOxUUw3QL3nX5FpwK3SO6d Ca/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=jp9fCPiVi8uv07s6DelseXp9idVSa4eBTh9PuKubi+E=; b=ahY+E6HZVxAS1hCVOkymNBdAhMqH4+RqkA7JlGowh9ahAeYOAvD07//YCAYsHCAzMd I4xKBvWnc9/XZvCGbfVa9DsxMVJpnKkDmOchDpEPIHJvbu09eMhkjbWoBSnPj0qnLcDc I1StMeancGSq+J0s+v45rCnfGFax9pXw+Q4yOKY6fo9ZDb8GelL1Y2pvTJmGIDnBtVTI Wzt9/lNg19BN7qvv2CHhbM8zZYnfxB2Oailu3p7CH5K4iYg/ZVjqAstcu/gITI6Ot4xJ /yvHU639wyuZuO0lHk2IDZkzFb9hWbm1Qp0AB6pgmoalwrubVZM7yxX7hNkPa75BVMON 9dsQ== X-Gm-Message-State: APjAAAWaVFAEFs1st4bO3U69c2BCTuhBJIogjyjP/RcOh76bUwtxSo5b 09TUZzcEWBC1DfGBa2JqrwJwSp9F X-Google-Smtp-Source: APXvYqyLihRzApO5OE1K126Z785ENDIehVI23aHsHfNdTBaYytu9z5FOhzXoZIj9As5MSUdIaKpgWQ== X-Received: by 2002:a5d:4946:: with SMTP id r6mr6281050wrs.155.1574373900217; Thu, 21 Nov 2019 14:05:00 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m15sm5024989wrq.97.2019.11.21.14.04.59 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 21 Nov 2019 14:04:59 -0800 (PST) Message-Id: <0bc87c1a88f5419a9e11681e1d3c079b2426def2.1574373892.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Jeff Hostetler via GitGitGadget" Date: Thu, 21 Nov 2019 22:04:39 +0000 Subject: [PATCH v6 07/19] trace2: add region in clear_ce_flags Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, jon@jonsimons.org, szeder.dev@gmail.com, Derrick Stolee , Junio C Hamano , Jeff Hostetler Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler When Git updates the working directory with the sparse-checkout feature enabled, the unpack_trees() method calls clear_ce_flags() to update the skip-wortree bits on the cache entries. This check can be expensive, depending on the patterns used. Add trace2 regions around the method, including some flag information, so we can get granular performance data during experiments. This data will be used to measure improvements to the pattern-matching algorithms for sparse-checkout. Signed-off-by: Jeff Hostetler Signed-off-by: Derrick Stolee --- unpack-trees.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/unpack-trees.c b/unpack-trees.c index 33ea7810d8..01a05ff66d 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -1407,15 +1407,23 @@ static int clear_ce_flags(struct index_state *istate, struct pattern_list *pl) { static struct strbuf prefix = STRBUF_INIT; + char label[100]; + int rval; strbuf_reset(&prefix); - return clear_ce_flags_1(istate, + xsnprintf(label, sizeof(label), "clear_ce_flags(0x%08lx,0x%08lx)", + (unsigned long)select_mask, (unsigned long)clear_mask); + trace2_region_enter("unpack_trees", label, the_repository); + rval = clear_ce_flags_1(istate, istate->cache, istate->cache_nr, &prefix, select_mask, clear_mask, pl, 0); + trace2_region_leave("unpack_trees", label, the_repository); + + return rval; } /* From patchwork Thu Nov 21 22:04:40 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11256955 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1E2C41871 for ; Thu, 21 Nov 2019 22:05:08 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id F13EB2068E for ; Thu, 21 Nov 2019 22:05:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="YHUzAmhi" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726729AbfKUWFF (ORCPT ); Thu, 21 Nov 2019 17:05:05 -0500 Received: from mail-wr1-f66.google.com ([209.85.221.66]:36719 "EHLO mail-wr1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726655AbfKUWFE (ORCPT ); Thu, 21 Nov 2019 17:05:04 -0500 Received: by mail-wr1-f66.google.com with SMTP id z3so6321814wru.3 for ; Thu, 21 Nov 2019 14:05:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=l4KS8xjCkHfu4yECfI0p/qY1/ieLMG0/1p+g8ObGUGU=; b=YHUzAmhis/8tfFLB5zH4kWsqL+PIPvF2VkzScn5/DxqZ1Sox0ocVaSJickkP+YDb4p UClHnYqord4NDLvQ0fHuoVz80PImRumOx56Ryl6ZIJWRRQU7o9RLqg8lte2xOvd2jM+L 2q4Jg6i7UMs7kcythwNWksyW6ZebJDdkS86D0Y0SX/sQFogfY3MG3PKeiP7n/RDdjD5h X1XCtmKSLkJB6DgkPusyuwknadmy3APoZuPtiOnUOUpbNqfxXFGOtad/MfPthF8NLedD V1J7E0zKBqW/aHfOlUyyUlI76ZJHVU4KqiKwrdzbgW8OMvj6DvF5m428/qUm3Qxwhmwg HUBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=l4KS8xjCkHfu4yECfI0p/qY1/ieLMG0/1p+g8ObGUGU=; b=HnniT18bEclSBqcL0CgVKHCEkdkYHhE6+p2ZwubCL+c7a/s9Cs0L4Na0wDiFqsYQ04 yr+JuIFPzXHel0AVbwuG22AfckP6/oaerabG4ykis6Idb19KauYJPVWXymsGEIMxpHMK R34X7qx50eNk7QYCPDeH+3N+J8eaffYJNPtuZJnneZplzFWOpceG/xqAe7wV6dxR7wXY yWb3Hvo/JxQQPkAMcZ++VuANyfoWQVlfjMLb3W5DAQZc2DNXdPHKe9vXQgRa0fdtTu85 uQxB/ltSb/hUZzSXr7v725nmPFcMxYMhxUyuZySChIMNqad02/b37vsHXBd54mQBCkIE Etmg== X-Gm-Message-State: APjAAAWEeT9AqhlLkn1K8y9WbMlJi0KgpLogDHdPMu2Se3ZvcNSVHo7K hpS88mtXjcuFA8Pmdac615KmA2Na X-Google-Smtp-Source: APXvYqx+OJxwrJHVGIlFmHlfCNv3LdTQ0eLnCP4lj4P8GU80RAOMbFJImxEy7yg3O0MwHleGzOgPPw== X-Received: by 2002:a05:6000:1204:: with SMTP id e4mr6325957wrx.35.1574373900817; Thu, 21 Nov 2019 14:05:00 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id h16sm4814843wrs.48.2019.11.21.14.05.00 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 21 Nov 2019 14:05:00 -0800 (PST) Message-Id: <3b9e11d7f2a7ec7b9e0250e603e5de42f10efb92.1574373892.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 21 Nov 2019 22:04:40 +0000 Subject: [PATCH v6 08/19] sparse-checkout: add 'cone' mode Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, jon@jonsimons.org, szeder.dev@gmail.com, Derrick Stolee , Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee The sparse-checkout feature can have quadratic performance as the number of patterns and number of entries in the index grow. If there are 1,000 patterns and 1,000,000 entries, this time can be very significant. Create a new Boolean config option, core.sparseCheckoutCone, to indicate that we expect the sparse-checkout file to contain a more limited set of patterns. This is a separate config setting from core.sparseCheckout to avoid breaking older clients by introducing a tri-state option. The config option does nothing right now, but will be expanded upon in a later commit. Signed-off-by: Derrick Stolee --- Documentation/config/core.txt | 10 ++++- Documentation/git-sparse-checkout.txt | 55 ++++++++++++++++++++++++++- cache.h | 4 +- config.c | 5 +++ environment.c | 1 + t/t1091-sparse-checkout-builtin.sh | 14 +++++++ 6 files changed, 85 insertions(+), 4 deletions(-) diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index 852d2ba37a..bdbbee58b9 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -593,8 +593,14 @@ core.multiPackIndex:: multi-pack-index design document]. core.sparseCheckout:: - Enable "sparse checkout" feature. See section "Sparse checkout" in - linkgit:git-read-tree[1] for more information. + Enable "sparse checkout" feature. See linkgit:git-sparse-checkout[1] + for more information. + +core.sparseCheckoutCone:: + Enables the "cone mode" of the sparse checkout feature. When the + sparse-checkout file contains a limited set of patterns, then this + mode provides significant performance advantages. See + linkgit:git-sparse-checkout[1] for more information. core.abbrev:: Set the length object names are abbreviated to. If diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt index c2cb19f80d..8535f0cf40 100644 --- a/Documentation/git-sparse-checkout.txt +++ b/Documentation/git-sparse-checkout.txt @@ -80,7 +80,9 @@ the sparse-checkout file. To repopulate the working directory with all files, use the `git sparse-checkout disable` command. -## FULL PATTERN SET + +FULL PATTERN SET +---------------- By default, the sparse-checkout file uses the same syntax as `.gitignore` files. @@ -95,6 +97,57 @@ using negative patterns. For example, to remove the file `unwanted`: ---------------- +CONE PATTERN SET +---------------- + +The full pattern set allows for arbitrary pattern matches and complicated +inclusion/exclusion rules. These can result in O(N*M) pattern matches when +updating the index, where N is the number of patterns and M is the number +of paths in the index. To combat this performance issue, a more restricted +pattern set is allowed when `core.spareCheckoutCone` is enabled. + +The accepted patterns in the cone pattern set are: + +1. *Recursive:* All paths inside a directory are included. + +2. *Parent:* All files immediately inside a directory are included. + +In addition to the above two patterns, we also expect that all files in the +root directory are included. If a recursive pattern is added, then all +leading directories are added as parent patterns. + +By default, when running `git sparse-checkout init`, the root directory is +added as a parent pattern. At this point, the sparse-checkout file contains +the following patterns: + +---------------- +/* +!/*/ +---------------- + +This says "include everything in root, but nothing two levels below root." +If we then add the folder `A/B/C` as a recursive pattern, the folders `A` and +`A/B` are added as parent patterns. The resulting sparse-checkout file is +now + +---------------- +/* +!/*/ +/A/ +!/A/*/ +/A/B/ +!/A/B/*/ +/A/B/C/ +---------------- + +Here, order matters, so the negative patterns are overridden by the positive +patterns that appear lower in the file. + +If `core.sparseCheckoutCone=true`, then Git will parse the sparse-checkout file +expecting patterns of these types. Git will warn if the patterns do not match. +If the patterns do match the expected format, then Git will use faster hash- +based algorithms to compute inclusion in the sparse-checkout. + SEE ALSO -------- diff --git a/cache.h b/cache.h index 04cabaac11..4980ee198e 100644 --- a/cache.h +++ b/cache.h @@ -918,12 +918,14 @@ extern char *git_replace_ref_base; extern int fsync_object_files; extern int core_preload_index; -extern int core_apply_sparse_checkout; extern int precomposed_unicode; extern int protect_hfs; extern int protect_ntfs; extern const char *core_fsmonitor; +int core_apply_sparse_checkout; +int core_sparse_checkout_cone; + /* * Include broken refs in all ref iterations, which will * generally choke dangerous operations rather than letting diff --git a/config.c b/config.c index e7052b3977..d75f88ca0c 100644 --- a/config.c +++ b/config.c @@ -1364,6 +1364,11 @@ static int git_default_core_config(const char *var, const char *value, void *cb) return 0; } + if (!strcmp(var, "core.sparsecheckoutcone")) { + core_sparse_checkout_cone = git_config_bool(var, value); + return 0; + } + if (!strcmp(var, "core.precomposeunicode")) { precomposed_unicode = git_config_bool(var, value); return 0; diff --git a/environment.c b/environment.c index efa072680a..2a1a866659 100644 --- a/environment.c +++ b/environment.c @@ -67,6 +67,7 @@ enum object_creation_mode object_creation_mode = OBJECT_CREATION_MODE; char *notes_ref_name; int grafts_replace_parents = 1; int core_apply_sparse_checkout; +int core_sparse_checkout_cone; int merge_log_config = -1; int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */ unsigned long pack_size_limit_cfg; diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh index c385c62c92..0b2715db52 100755 --- a/t/t1091-sparse-checkout-builtin.sh +++ b/t/t1091-sparse-checkout-builtin.sh @@ -148,6 +148,20 @@ test_expect_success 'set sparse-checkout using --stdin' ' test_cmp expect dir ' +test_expect_success 'cone mode: match patterns' ' + git -C repo config --worktree core.sparseCheckoutCone true && + rm -rf repo/a repo/folder1 repo/folder2 && + git -C repo read-tree -mu HEAD && + git -C repo reset --hard && + ls repo >dir && + cat >expect <<-EOF && + a + folder1 + folder2 + EOF + test_cmp expect dir +' + test_expect_success 'sparse-checkout disable' ' git -C repo sparse-checkout disable && test_path_is_missing repo/.git/info/sparse-checkout && From patchwork Thu Nov 21 22:04:41 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11256953 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E84F4930 for ; Thu, 21 Nov 2019 22:05:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id BECA02068E for ; Thu, 21 Nov 2019 22:05:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="UuKxgmrJ" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726744AbfKUWFF (ORCPT ); Thu, 21 Nov 2019 17:05:05 -0500 Received: from mail-wm1-f66.google.com ([209.85.128.66]:37763 "EHLO mail-wm1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726658AbfKUWFF (ORCPT ); Thu, 21 Nov 2019 17:05:05 -0500 Received: by mail-wm1-f66.google.com with SMTP id f129so4113476wmf.2 for ; Thu, 21 Nov 2019 14:05:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=v7BNALQq5QzaAlOjzfT78+LGH1XjX6y+F3G1lsXugk4=; b=UuKxgmrJg/Zqs0aSN5+JvkvqRWSgWu4/xCBQwXLs4WxcHNa5KVVCO8EBzg4wNmf7eI GXRINXVxCTYK845yR6U3wWBVyiKmDTT4cJ7GOP2bxh6DI1JeaCJ4n2Qzqr90Kp2Wi9gQ G2I+nLS37CZFPEyZD4w2+2eJiK6KYTtLow9xX4in4naPbzZaOeFYNH86HBENbws59NpC kD0Vd3diJEGgC9IQCKoiSpuZvtN+rsBKiWGpBjVTXmqhBBxGHl1Ou/qNj8OWnN4leUCx BalaAzkvZVjpuWPQcKJe7inaRgxO/60sIdLAgMdKyvJl2tCEEcGoIdOTWOloTZeorskx qoZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=v7BNALQq5QzaAlOjzfT78+LGH1XjX6y+F3G1lsXugk4=; b=uHoUTD0ykIcjfExzgq8nxkIE1r5E3xNauVcWkRGsjqWfngMwzfOQlrJJBvPNpCdahN JUDOz1Fl9tFK108JeV38NAFLiKosutMpM1b1m/Vped04uOrgIWS/hbGxGCrk2apjD4Kh uWK3bd2yUpvzWbvAHT0Y6h059bGbmQNslS6Hm5LLQMPJx1Y0qU2pqUYci+XXLH5VYLfH fCYzWRwtmZg0vVB7dPQfTmVO+y4zPFRisUpzT2955tL1qHBYPI0DlYrLtt9A02iKz8Qs jJmqAaHNbRZ8yH9UFPjDGszCMs1zHi/kSpWtuPaF4DIiOXlBRPLgmQKqF4BNLEVHnBbH ptNg== X-Gm-Message-State: APjAAAWwFgbilhFhcsjGJRRvhcOYSL522Ns2F44n6tpqtHOlL4GaNuwh J9GjUOtmaWkhXPpTmzbSytnZnrmM X-Google-Smtp-Source: APXvYqy50kij9e2zQxhv4YEYW/cbgowLhwFTIqLyQJybhb9CuiHip9tsd5odPjGyePmnj848EezICw== X-Received: by 2002:a1c:2048:: with SMTP id g69mr12914591wmg.121.1574373901671; Thu, 21 Nov 2019 14:05:01 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 76sm1203462wma.0.2019.11.21.14.05.00 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 21 Nov 2019 14:05:01 -0800 (PST) Message-Id: In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 21 Nov 2019 22:04:41 +0000 Subject: [PATCH v6 09/19] sparse-checkout: use hashmaps for cone patterns Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, jon@jonsimons.org, szeder.dev@gmail.com, Derrick Stolee , Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee The parent and recursive patterns allowed by the "cone mode" option in sparse-checkout are restrictive enough that we can avoid using the regex parsing. Everything is based on prefix matches, so we can use hashsets to store the prefixes from the sparse-checkout file. When checking a path, we can strip path entries from the path and check the hashset for an exact match. As a test, I created a cone-mode sparse-checkout file for the Linux repository that actually includes every file. This was constructed by taking every folder in the Linux repo and creating the pattern pairs here: /$folder/ !/$folder/*/ This resulted in a sparse-checkout file sith 8,296 patterns. Running 'git read-tree -mu HEAD' on this file had the following performance: core.sparseCheckout=false: 0.21 s (0.00 s) core.sparseCheckout=true: 3.75 s (3.50 s) core.sparseCheckoutCone=true: 0.23 s (0.01 s) The times in parentheses above correspond to the time spent in the first clear_ce_flags() call, according to the trace2 performance traces. While this example is contrived, it demonstrates how these patterns can slow the sparse-checkout feature. Helped-by: Eric Wong Helped-by: Johannes Schindelin Signed-off-by: Derrick Stolee --- dir.c | 207 +++++++++++++++++++++++++++-- dir.h | 31 +++++ t/t1091-sparse-checkout-builtin.sh | 11 +- unpack-trees.c | 1 + 4 files changed, 241 insertions(+), 9 deletions(-) diff --git a/dir.c b/dir.c index 61f559f980..dfabf9982f 100644 --- a/dir.c +++ b/dir.c @@ -611,6 +611,150 @@ void parse_path_pattern(const char **pattern, *patternlen = len; } +static int pl_hashmap_cmp(const void *unused_cmp_data, + const struct hashmap_entry *a, + const struct hashmap_entry *b, + const void *key) +{ + const struct pattern_entry *ee1 = + container_of(a, struct pattern_entry, ent); + const struct pattern_entry *ee2 = + container_of(b, struct pattern_entry, ent); + + size_t min_len = ee1->patternlen <= ee2->patternlen + ? ee1->patternlen + : ee2->patternlen; + + return strncmp(ee1->pattern, ee2->pattern, min_len); +} + +static void add_pattern_to_hashsets(struct pattern_list *pl, struct path_pattern *given) +{ + struct pattern_entry *translated; + char *truncated; + char *data = NULL; + + if (!pl->use_cone_patterns) + return; + + if (given->flags & PATTERN_FLAG_NEGATIVE && + given->flags & PATTERN_FLAG_MUSTBEDIR && + !strcmp(given->pattern, "/*")) { + pl->full_cone = 0; + return; + } + + if (!given->flags && !strcmp(given->pattern, "/*")) { + pl->full_cone = 1; + return; + } + + if (given->patternlen > 2 && + !strcmp(given->pattern + given->patternlen - 2, "/*")) { + if (!(given->flags & PATTERN_FLAG_NEGATIVE)) { + /* Not a cone pattern. */ + pl->use_cone_patterns = 0; + warning(_("unrecognized pattern: '%s'"), given->pattern); + goto clear_hashmaps; + } + + truncated = xstrdup(given->pattern); + truncated[given->patternlen - 2] = 0; + + translated = xmalloc(sizeof(struct pattern_entry)); + translated->pattern = truncated; + translated->patternlen = given->patternlen - 2; + hashmap_entry_init(&translated->ent, + memhash(translated->pattern, translated->patternlen)); + + if (!hashmap_get_entry(&pl->recursive_hashmap, + translated, ent, NULL)) { + /* We did not see the "parent" included */ + warning(_("unrecognized negative pattern: '%s'"), + given->pattern); + free(truncated); + free(translated); + goto clear_hashmaps; + } + + hashmap_add(&pl->parent_hashmap, &translated->ent); + hashmap_remove(&pl->recursive_hashmap, &translated->ent, &data); + free(data); + return; + } + + if (given->flags & PATTERN_FLAG_NEGATIVE) { + warning(_("unrecognized negative pattern: '%s'"), + given->pattern); + goto clear_hashmaps; + } + + translated = xmalloc(sizeof(struct pattern_entry)); + + translated->pattern = xstrdup(given->pattern); + translated->patternlen = given->patternlen; + hashmap_entry_init(&translated->ent, + memhash(translated->pattern, translated->patternlen)); + + hashmap_add(&pl->recursive_hashmap, &translated->ent); + + if (hashmap_get_entry(&pl->parent_hashmap, translated, ent, NULL)) { + /* we already included this at the parent level */ + warning(_("your sparse-checkout file may have issues: pattern '%s' is repeated"), + given->pattern); + hashmap_remove(&pl->parent_hashmap, &translated->ent, &data); + free(data); + free(translated); + } + + return; + +clear_hashmaps: + warning(_("disabling cone pattern matching")); + hashmap_free_entries(&pl->parent_hashmap, struct pattern_entry, ent); + hashmap_free_entries(&pl->recursive_hashmap, struct pattern_entry, ent); + pl->use_cone_patterns = 0; +} + +static int hashmap_contains_path(struct hashmap *map, + struct strbuf *pattern) +{ + struct pattern_entry p; + + /* Check straight mapping */ + p.pattern = pattern->buf; + p.patternlen = pattern->len; + hashmap_entry_init(&p.ent, memhash(p.pattern, p.patternlen)); + return !!hashmap_get_entry(map, &p, ent, NULL); +} + +int hashmap_contains_parent(struct hashmap *map, + const char *path, + struct strbuf *buffer) +{ + char *slash_pos; + + strbuf_setlen(buffer, 0); + + if (path[0] != '/') + strbuf_addch(buffer, '/'); + + strbuf_addstr(buffer, path); + + slash_pos = strrchr(buffer->buf, '/'); + + while (slash_pos > buffer->buf) { + strbuf_setlen(buffer, slash_pos - buffer->buf); + + if (hashmap_contains_path(map, buffer)) + return 1; + + slash_pos = strrchr(buffer->buf, '/'); + } + + return 0; +} + void add_pattern(const char *string, const char *base, int baselen, struct pattern_list *pl, int srcpos) { @@ -635,6 +779,8 @@ void add_pattern(const char *string, const char *base, ALLOC_GROW(pl->patterns, pl->nr + 1, pl->alloc); pl->patterns[pl->nr++] = pattern; pattern->pl = pl; + + add_pattern_to_hashsets(pl, pattern); } static int read_skip_worktree_file_from_index(const struct index_state *istate, @@ -860,6 +1006,9 @@ static int add_patterns_from_buffer(char *buf, size_t size, int i, lineno = 1; char *entry; + hashmap_init(&pl->recursive_hashmap, pl_hashmap_cmp, NULL, 0); + hashmap_init(&pl->parent_hashmap, pl_hashmap_cmp, NULL, 0); + pl->filebuf = buf; if (skip_utf8_bom(&buf, size)) @@ -1096,16 +1245,58 @@ enum pattern_match_result path_matches_pattern_list( struct index_state *istate) { struct path_pattern *pattern; - pattern = last_matching_pattern_from_list(pathname, pathlen, basename, - dtype, pl, istate); - if (pattern) { - if (pattern->flags & PATTERN_FLAG_NEGATIVE) - return NOT_MATCHED; - else - return MATCHED; + struct strbuf parent_pathname = STRBUF_INIT; + int result = NOT_MATCHED; + const char *slash_pos; + + if (!pl->use_cone_patterns) { + pattern = last_matching_pattern_from_list(pathname, pathlen, basename, + dtype, pl, istate); + if (pattern) { + if (pattern->flags & PATTERN_FLAG_NEGATIVE) + return NOT_MATCHED; + else + return MATCHED; + } + + return UNDECIDED; + } + + if (pl->full_cone) + return MATCHED; + + strbuf_addch(&parent_pathname, '/'); + strbuf_add(&parent_pathname, pathname, pathlen); + + if (hashmap_contains_path(&pl->recursive_hashmap, + &parent_pathname)) { + result = MATCHED; + goto done; + } + + slash_pos = strrchr(parent_pathname.buf, '/'); + + if (slash_pos == parent_pathname.buf) { + /* include every file in root */ + result = MATCHED; + goto done; } - return UNDECIDED; + strbuf_setlen(&parent_pathname, slash_pos - parent_pathname.buf); + + if (hashmap_contains_path(&pl->parent_hashmap, &parent_pathname)) { + result = MATCHED; + goto done; + } + + if (hashmap_contains_parent(&pl->recursive_hashmap, + pathname, + &parent_pathname)) + result = MATCHED; + +done: + strbuf_release(&parent_pathname); + return result; } static struct path_pattern *last_matching_pattern_from_lists( diff --git a/dir.h b/dir.h index 2fbdef014f..f8edbca72b 100644 --- a/dir.h +++ b/dir.h @@ -4,6 +4,7 @@ /* See Documentation/technical/api-directory-listing.txt */ #include "cache.h" +#include "hashmap.h" #include "strbuf.h" struct dir_entry { @@ -37,6 +38,13 @@ struct path_pattern { int srcpos; }; +/* used for hashmaps for cone patterns */ +struct pattern_entry { + struct hashmap_entry ent; + char *pattern; + size_t patternlen; +}; + /* * Each excludes file will be parsed into a fresh exclude_list which * is appended to the relevant exclude_list_group (either EXC_DIRS or @@ -55,6 +63,26 @@ struct pattern_list { const char *src; struct path_pattern **patterns; + + /* + * While scanning the excludes, we attempt to match the patterns + * with a more restricted set that allows us to use hashsets for + * matching logic, which is faster than the linear lookup in the + * excludes array above. If non-zero, that check succeeded. + */ + unsigned use_cone_patterns; + unsigned full_cone; + + /* + * Stores paths where everything starting with those paths + * is included. + */ + struct hashmap recursive_hashmap; + + /* + * Used to check single-level parents of blobs. + */ + struct hashmap parent_hashmap; }; /* @@ -271,6 +299,9 @@ int is_excluded(struct dir_struct *dir, struct index_state *istate, const char *name, int *dtype); +int hashmap_contains_parent(struct hashmap *map, + const char *path, + struct strbuf *buffer); struct pattern_list *add_pattern_list(struct dir_struct *dir, int group_type, const char *src); int add_patterns_from_file_to_list(const char *fname, const char *base, int baselen, diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh index 0b2715db52..1ed003ac1d 100755 --- a/t/t1091-sparse-checkout-builtin.sh +++ b/t/t1091-sparse-checkout-builtin.sh @@ -151,7 +151,8 @@ test_expect_success 'set sparse-checkout using --stdin' ' test_expect_success 'cone mode: match patterns' ' git -C repo config --worktree core.sparseCheckoutCone true && rm -rf repo/a repo/folder1 repo/folder2 && - git -C repo read-tree -mu HEAD && + git -C repo read-tree -mu HEAD 2>err && + test_i18ngrep ! "disabling cone patterns" err && git -C repo reset --hard && ls repo >dir && cat >expect <<-EOF && @@ -162,6 +163,14 @@ test_expect_success 'cone mode: match patterns' ' test_cmp expect dir ' +test_expect_success 'cone mode: warn on bad pattern' ' + test_when_finished mv sparse-checkout repo/.git/info/ && + cp repo/.git/info/sparse-checkout . && + echo "!/deep/deeper/*" >>repo/.git/info/sparse-checkout && + git -C repo read-tree -mu HEAD 2>err && + test_i18ngrep "unrecognized negative pattern" err +' + test_expect_success 'sparse-checkout disable' ' git -C repo sparse-checkout disable && test_path_is_missing repo/.git/info/sparse-checkout && diff --git a/unpack-trees.c b/unpack-trees.c index 01a05ff66d..a90d71845d 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -1482,6 +1482,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options o->skip_sparse_checkout = 1; if (!o->skip_sparse_checkout) { char *sparse = git_pathdup("info/sparse-checkout"); + pl.use_cone_patterns = core_sparse_checkout_cone; if (add_patterns_from_file_to_list(sparse, "", 0, &pl, NULL) < 0) o->skip_sparse_checkout = 1; else From patchwork Thu Nov 21 22:04:42 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11256965 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7C5EB930 for ; Thu, 21 Nov 2019 22:05:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 48C5E206D8 for ; Thu, 21 Nov 2019 22:05:13 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="E1zwsNM4" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726776AbfKUWFM (ORCPT ); Thu, 21 Nov 2019 17:05:12 -0500 Received: from mail-wr1-f67.google.com ([209.85.221.67]:45744 "EHLO mail-wr1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726658AbfKUWFI (ORCPT ); Thu, 21 Nov 2019 17:05:08 -0500 Received: by mail-wr1-f67.google.com with SMTP id z10so6259539wrs.12 for ; Thu, 21 Nov 2019 14:05:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=CeQpqzkkcpx3RAGRByIDR8QVywUcDDYDsIGH+weGgSw=; b=E1zwsNM4Qa5ZKk7mMIXrFsAVf1gv/kIOIo133YSiJuRqLMtCzUwzwUCmjTCgYzfYk1 qiubJkg2S8ewyMMVYZDo6ToBZt1ATSWjx+r0s9+h/HvbNaRSRg4yfU9Z14vmxVp+R3/5 qQTklthIPM3AoyTnCggCj9uyVSx7gqqMgCGPPBzyXbBwCKctUciKX3BOT/vVCAoi1ZNA E2NO3iP8mHED/NIeS8qMTOtEziSBH6wKu1KlhAe+FTS7m6Agby/WKQK3mFHt+PvDMp6i weVOPiU04e+LYLvOhL4tlWNSTh7VMMH8I+yhLtGGdSQAf/LTxaWU8SGrkCaj9J/CUeF1 qZQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=CeQpqzkkcpx3RAGRByIDR8QVywUcDDYDsIGH+weGgSw=; b=Pu8JKIbxxrMPj95AMbOcCP3TK7CNtzqSSjVxq30bhZ0AfmjFHMveBYBMcNKvkF650t e5zAW3DTENbDSybzBJnMarqeAb6TVhfLC+NHd8Cr3Th+hIDV6nc88VrrkF60QxH+VhOb WNMWDHCKhUC6YCvMzyxYFhi7UPhl1QorPYlRmdDQpPs9A28P6y9AO0lm3uhPyNFYpN/t 5xMnNUyrE6cSn7f+X8V10dRdbicBl87TOfmV609wWOW4n/B7RnvYh/+eQkAa9lUf3ZGy VX7ai2jU1Inf5QeZz93BnVC5yM7N7UrETYYtBmvPBIq4OYGRffp7EwB7IxUI6SDdHRht K0Kg== X-Gm-Message-State: APjAAAVsQpPx1ZtJhTOCv4mwvUXsDgyYDVntlGwvu0ro6N3GJio1qfyR ZsqGXVG0hlSQC3KuXl6QV5cSZmoG X-Google-Smtp-Source: APXvYqz1be/liLqyKRx5bnSbB5+NmsfLJXqkSXjEuJeaLXv33Ze3NPxWtkV18slbeE6JRuPHzgMjEA== X-Received: by 2002:a5d:6351:: with SMTP id b17mr14392977wrw.126.1574373905167; Thu, 21 Nov 2019 14:05:05 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id k125sm1208232wmf.2.2019.11.21.14.05.04 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 21 Nov 2019 14:05:04 -0800 (PST) Message-Id: <951e622aacdb2069177038bff6eb1a0330d46f34.1574373892.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 21 Nov 2019 22:04:42 +0000 Subject: [PATCH v6 10/19] sparse-checkout: init and set in cone mode Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, jon@jonsimons.org, szeder.dev@gmail.com, Derrick Stolee , Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee To make the cone pattern set easy to use, update the behavior of 'git sparse-checkout (init|set)'. Add '--cone' flag to 'git sparse-checkout init' to set the config option 'core.sparseCheckoutCone=true'. When running 'git sparse-checkout set' in cone mode, a user only needs to supply a list of recursive folder matches. Git will automatically add the necessary parent matches for the leading directories. When testing 'git sparse-checkout set' in cone mode, check the error stream to ensure we do not see any errors. Specifically, we want to avoid the warning that the patterns do not match the cone-mode patterns. Helped-by: Eric Wong Helped-by: Johannes Schindelin Signed-off-by: Derrick Stolee --- builtin/sparse-checkout.c | 163 ++++++++++++++++++++++++++--- dir.c | 8 +- dir.h | 4 + t/t1091-sparse-checkout-builtin.sh | 51 +++++++++ 4 files changed, 206 insertions(+), 20 deletions(-) diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c index e3a8d3460a..85cc801f03 100644 --- a/builtin/sparse-checkout.c +++ b/builtin/sparse-checkout.c @@ -6,6 +6,7 @@ #include "repository.h" #include "run-command.h" #include "strbuf.h" +#include "string-list.h" static char const * const builtin_sparse_checkout_usage[] = { N_("git sparse-checkout (init|list|set|disable) "), @@ -74,9 +75,65 @@ static int update_working_directory(void) return result; } +static void write_cone_to_file(FILE *fp, struct pattern_list *pl) +{ + int i; + struct pattern_entry *pe; + struct hashmap_iter iter; + struct string_list sl = STRING_LIST_INIT_DUP; + + hashmap_for_each_entry(&pl->parent_hashmap, &iter, pe, ent) + string_list_insert(&sl, pe->pattern); + + string_list_sort(&sl); + string_list_remove_duplicates(&sl, 0); + + fprintf(fp, "/*\n!/*/\n"); + + for (i = 0; i < sl.nr; i++) { + char *pattern = sl.items[i].string; + + if (strlen(pattern)) + fprintf(fp, "%s/\n!%s/*/\n", pattern, pattern); + } + + string_list_clear(&sl, 0); + + hashmap_for_each_entry(&pl->recursive_hashmap, &iter, pe, ent) + string_list_insert(&sl, pe->pattern); + + string_list_sort(&sl); + string_list_remove_duplicates(&sl, 0); + + for (i = 0; i < sl.nr; i++) { + char *pattern = sl.items[i].string; + fprintf(fp, "%s/\n", pattern); + } +} + +static int write_patterns_and_update(struct pattern_list *pl) +{ + char *sparse_filename; + FILE *fp; + + sparse_filename = get_sparse_checkout_filename(); + fp = fopen(sparse_filename, "w"); + + if (core_sparse_checkout_cone) + write_cone_to_file(fp, pl); + else + write_patterns_to_file(fp, pl); + + fclose(fp); + free(sparse_filename); + + return update_working_directory(); +} + enum sparse_checkout_mode { MODE_NO_PATTERNS = 0, MODE_ALL_PATTERNS = 1, + MODE_CONE_PATTERNS = 2, }; static int set_config(enum sparse_checkout_mode mode) @@ -93,9 +150,22 @@ static int set_config(enum sparse_checkout_mode mode) "core.sparseCheckout", mode ? "true" : NULL); + git_config_set_in_file_gently(config_path, + "core.sparseCheckoutCone", + mode == MODE_CONE_PATTERNS ? "true" : NULL); + return 0; } +static char const * const builtin_sparse_checkout_init_usage[] = { + N_("git sparse-checkout init [--cone]"), + NULL +}; + +static struct sparse_checkout_init_opts { + int cone_mode; +} init_opts; + static int sparse_checkout_init(int argc, const char **argv) { struct pattern_list pl; @@ -103,8 +173,21 @@ static int sparse_checkout_init(int argc, const char **argv) FILE *fp; int res; struct object_id oid; + int mode; - if (set_config(MODE_ALL_PATTERNS)) + static struct option builtin_sparse_checkout_init_options[] = { + OPT_BOOL(0, "cone", &init_opts.cone_mode, + N_("initialize the sparse-checkout in cone mode")), + OPT_END(), + }; + + argc = parse_options(argc, argv, NULL, + builtin_sparse_checkout_init_options, + builtin_sparse_checkout_init_usage, 0); + + mode = init_opts.cone_mode ? MODE_CONE_PATTERNS : MODE_ALL_PATTERNS; + + if (set_config(mode)) return 1; memset(&pl, 0, sizeof(pl)); @@ -136,18 +219,47 @@ static int sparse_checkout_init(int argc, const char **argv) return update_working_directory(); } -static int write_patterns_and_update(struct pattern_list *pl) +static void insert_recursive_pattern(struct pattern_list *pl, struct strbuf *path) { - char *sparse_filename; - FILE *fp; + struct pattern_entry *e = xmalloc(sizeof(*e)); + e->patternlen = path->len; + e->pattern = strbuf_detach(path, NULL); + hashmap_entry_init(&e->ent, memhash(e->pattern, e->patternlen)); - sparse_filename = get_sparse_checkout_filename(); - fp = fopen(sparse_filename, "w"); - write_patterns_to_file(fp, pl); - fclose(fp); - free(sparse_filename); + hashmap_add(&pl->recursive_hashmap, &e->ent); - return update_working_directory(); + while (e->patternlen) { + char *slash = strrchr(e->pattern, '/'); + char *oldpattern = e->pattern; + size_t newlen; + + if (slash == e->pattern) + break; + + newlen = slash - e->pattern; + e = xmalloc(sizeof(struct pattern_entry)); + e->patternlen = newlen; + e->pattern = xstrndup(oldpattern, newlen); + hashmap_entry_init(&e->ent, memhash(e->pattern, e->patternlen)); + + if (!hashmap_get_entry(&pl->parent_hashmap, e, ent, NULL)) + hashmap_add(&pl->parent_hashmap, &e->ent); + } +} + +static void strbuf_to_cone_pattern(struct strbuf *line, struct pattern_list *pl) +{ + strbuf_trim(line); + + strbuf_trim_trailing_dir_sep(line); + + if (!line->len) + return; + + if (line->buf[0] != '/') + strbuf_insert(line, 0, "/", 1); + + insert_recursive_pattern(pl, line); } static char const * const builtin_sparse_checkout_set_usage[] = { @@ -180,16 +292,35 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix) builtin_sparse_checkout_set_usage, PARSE_OPT_KEEP_UNKNOWN); - if (set_opts.use_stdin) { + if (core_sparse_checkout_cone) { struct strbuf line = STRBUF_INIT; - while (!strbuf_getline(&line, stdin)) { - char *buf = strbuf_detach(&line, NULL); - add_pattern(buf, empty_base, 0, &pl, 0); + hashmap_init(&pl.recursive_hashmap, pl_hashmap_cmp, NULL, 0); + hashmap_init(&pl.parent_hashmap, pl_hashmap_cmp, NULL, 0); + + if (set_opts.use_stdin) { + while (!strbuf_getline(&line, stdin)) + strbuf_to_cone_pattern(&line, &pl); + } else { + for (i = 0; i < argc; i++) { + strbuf_setlen(&line, 0); + strbuf_addstr(&line, argv[i]); + strbuf_to_cone_pattern(&line, &pl); + } } } else { - for (i = 0; i < argc; i++) - add_pattern(argv[i], empty_base, 0, &pl, 0); + if (set_opts.use_stdin) { + struct strbuf line = STRBUF_INIT; + + while (!strbuf_getline(&line, stdin)) { + size_t len; + char *buf = strbuf_detach(&line, &len); + add_pattern(buf, empty_base, 0, &pl, 0); + } + } else { + for (i = 0; i < argc; i++) + add_pattern(argv[i], empty_base, 0, &pl, 0); + } } if (!core_apply_sparse_checkout) { diff --git a/dir.c b/dir.c index dfabf9982f..35c1ca9e24 100644 --- a/dir.c +++ b/dir.c @@ -611,10 +611,10 @@ void parse_path_pattern(const char **pattern, *patternlen = len; } -static int pl_hashmap_cmp(const void *unused_cmp_data, - const struct hashmap_entry *a, - const struct hashmap_entry *b, - const void *key) +int pl_hashmap_cmp(const void *unused_cmp_data, + const struct hashmap_entry *a, + const struct hashmap_entry *b, + const void *key) { const struct pattern_entry *ee1 = container_of(a, struct pattern_entry, ent); diff --git a/dir.h b/dir.h index f8edbca72b..8e232085cd 100644 --- a/dir.h +++ b/dir.h @@ -299,6 +299,10 @@ int is_excluded(struct dir_struct *dir, struct index_state *istate, const char *name, int *dtype); +int pl_hashmap_cmp(const void *unused_cmp_data, + const struct hashmap_entry *a, + const struct hashmap_entry *b, + const void *key); int hashmap_contains_parent(struct hashmap *map, const char *path, struct strbuf *buffer); diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh index 1ed003ac1d..fbd46c3f61 100755 --- a/t/t1091-sparse-checkout-builtin.sh +++ b/t/t1091-sparse-checkout-builtin.sh @@ -186,4 +186,55 @@ test_expect_success 'sparse-checkout disable' ' test_cmp expect dir ' +test_expect_success 'cone mode: init and set' ' + git -C repo sparse-checkout init --cone && + git -C repo config --list >config && + test_i18ngrep "core.sparsecheckoutcone=true" config && + ls repo >dir && + echo a >expect && + test_cmp expect dir && + git -C repo sparse-checkout set deep/deeper1/deepest/ 2>err && + test_must_be_empty err && + ls repo >dir && + cat >expect <<-EOF && + a + deep + EOF + test_cmp expect dir && + ls repo/deep >dir && + cat >expect <<-EOF && + a + deeper1 + EOF + test_cmp expect dir && + ls repo/deep/deeper1 >dir && + cat >expect <<-EOF && + a + deepest + EOF + test_cmp expect dir && + cat >expect <<-EOF && + /* + !/*/ + /deep/ + !/deep/*/ + /deep/deeper1/ + !/deep/deeper1/*/ + /deep/deeper1/deepest/ + EOF + test_cmp expect repo/.git/info/sparse-checkout && + git -C repo sparse-checkout set --stdin 2>err <<-EOF && + folder1 + folder2 + EOF + test_must_be_empty err && + cat >expect <<-EOF && + a + folder1 + folder2 + EOF + ls repo >dir && + test_cmp expect dir +' + test_done From patchwork Thu Nov 21 22:04:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11256967 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5680A930 for ; Thu, 21 Nov 2019 22:05:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 3696D2068E for ; Thu, 21 Nov 2019 22:05:16 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="jKa0AEkr" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726825AbfKUWFP (ORCPT ); Thu, 21 Nov 2019 17:05:15 -0500 Received: from mail-wr1-f66.google.com ([209.85.221.66]:45758 "EHLO mail-wr1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726774AbfKUWFN (ORCPT ); Thu, 21 Nov 2019 17:05:13 -0500 Received: by mail-wr1-f66.google.com with SMTP id z10so6259805wrs.12 for ; Thu, 21 Nov 2019 14:05:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Ips6wWu7Xdah4SBzOLvtoLKr422mshCSp/ecYyPpC+c=; b=jKa0AEkr6p8g2Ac4R0qn21U2vqvxaOjPpZMnhMcAYgR3Y/6DeH4B0GbUbxNRPGS+pr 0zHf0LFhq8HBaUBKPUXvpeueBPQxIx9GVLk6Nh9s+TXxbPcJD3Blr13AAP4j8Q6RyB0p c4okumknjEkvqeY/7xjpWzUIVysQFjY9dLBWv7UflcQsExdqQ4QSYcpqpSmTb2tFCDZt 6Bdb/nG1dunS7pypsh2E0i0g/gQTnE/0tImDGiel798wIrZWf8vRzjmkLg9mIoqTu6Yr zEYjiiwSedA20tk5f1VEntJ12zB2AXjw2l2fvxASNmGFUbLIreBQrnqB5Kx27zudT8Sw oM9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Ips6wWu7Xdah4SBzOLvtoLKr422mshCSp/ecYyPpC+c=; b=l4CInu9XmiIAvd9qbMstTpuLb4L1hUPmvVy35ESS8wyvQzWtaP+I1FFpNCk418qkzE 2i8Oi0nz1vs7ipB1JWymO8ioMIkEqW3/WV6dJ7M+1W56k8UFfoYNCorMaeuaEqH59VYh XvYY/fUuXKyuojkE0tpAKfvHSkWwlb6QJVH6ouF4D1ldpdMxXVR+vYUgEhAC3g4h5OBX ca+uKwhP4S3w0bmWCWWXgXpUliZig72JNCKR6q2fVolhBU2xCRK3tLp4Zj8c2cxtOlYf aDy3Yh+G91UrhNgtJx/BtqxEe54WfulOA9uSbD07p1++upXVEjPsdiHVq1ELiCCWg6iS IaWQ== X-Gm-Message-State: APjAAAUzcec9/xGJapo1sY7pEXjvRq80W6nPLOAS6vI9jYzcYzXc1mHv /1cKkntuC5UcLFExSzEVs5UqcjmU X-Google-Smtp-Source: APXvYqzFDU4++SliBwzdy8NUh6p0omvv8HdjO6FA/WZPwGc/xXbOwU0RUK+Qlv/zUS/P90L0Jy4/Nw== X-Received: by 2002:adf:f5cf:: with SMTP id k15mr9528634wrp.265.1574373910818; Thu, 21 Nov 2019 14:05:10 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id g207sm611793wmg.40.2019.11.21.14.05.10 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 21 Nov 2019 14:05:10 -0800 (PST) Message-Id: <745910bb8e7afe9cf0c66e0d8f873a00b506fdfa.1574373892.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 21 Nov 2019 22:04:43 +0000 Subject: [PATCH v6 11/19] unpack-trees: hash less in cone mode Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, jon@jonsimons.org, szeder.dev@gmail.com, Derrick Stolee , Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee The sparse-checkout feature in "cone mode" can use the fact that the recursive patterns are "connected" to the root via parent patterns to decide if a directory is entirely contained in the sparse-checkout or entirely removed. In these cases, we can skip hashing the paths within those directories and simply set the skipworktree bit to the correct value. Signed-off-by: Derrick Stolee --- dir.c | 4 ++-- dir.h | 1 + unpack-trees.c | 38 +++++++++++++++++++++++--------------- 3 files changed, 26 insertions(+), 17 deletions(-) diff --git a/dir.c b/dir.c index 35c1ca9e24..2ef92a50a0 100644 --- a/dir.c +++ b/dir.c @@ -1270,7 +1270,7 @@ enum pattern_match_result path_matches_pattern_list( if (hashmap_contains_path(&pl->recursive_hashmap, &parent_pathname)) { - result = MATCHED; + result = MATCHED_RECURSIVE; goto done; } @@ -1292,7 +1292,7 @@ enum pattern_match_result path_matches_pattern_list( if (hashmap_contains_parent(&pl->recursive_hashmap, pathname, &parent_pathname)) - result = MATCHED; + result = MATCHED_RECURSIVE; done: strbuf_release(&parent_pathname); diff --git a/dir.h b/dir.h index 8e232085cd..77a43dbf89 100644 --- a/dir.h +++ b/dir.h @@ -264,6 +264,7 @@ enum pattern_match_result { UNDECIDED = -1, NOT_MATCHED = 0, MATCHED = 1, + MATCHED_RECURSIVE = 2, }; /* diff --git a/unpack-trees.c b/unpack-trees.c index a90d71845d..c0dca20865 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -1283,15 +1283,17 @@ static int clear_ce_flags_dir(struct index_state *istate, struct cache_entry **cache_end; int dtype = DT_DIR; int rc; - enum pattern_match_result ret; - ret = path_matches_pattern_list(prefix->buf, prefix->len, - basename, &dtype, pl, istate); + enum pattern_match_result ret, orig_ret; + orig_ret = path_matches_pattern_list(prefix->buf, prefix->len, + basename, &dtype, pl, istate); strbuf_addch(prefix, '/'); /* If undecided, use matching result of parent dir in defval */ - if (ret == UNDECIDED) + if (orig_ret == UNDECIDED) ret = default_match; + else + ret = orig_ret; for (cache_end = cache; cache_end != cache + nr; cache_end++) { struct cache_entry *ce = *cache_end; @@ -1299,17 +1301,23 @@ static int clear_ce_flags_dir(struct index_state *istate, break; } - /* - * TODO: check pl, if there are no patterns that may conflict - * with ret (iow, we know in advance the incl/excl - * decision for the entire directory), clear flag here without - * calling clear_ce_flags_1(). That function will call - * the expensive path_matches_pattern_list() on every entry. - */ - rc = clear_ce_flags_1(istate, cache, cache_end - cache, - prefix, - select_mask, clear_mask, - pl, ret); + if (pl->use_cone_patterns && orig_ret == MATCHED_RECURSIVE) { + struct cache_entry **ce = cache; + rc = (cache_end - cache) / sizeof(struct cache_entry *); + + while (ce < cache_end) { + (*ce)->ce_flags &= ~clear_mask; + ce++; + } + } else if (pl->use_cone_patterns && orig_ret == NOT_MATCHED) { + rc = (cache_end - cache) / sizeof(struct cache_entry *); + } else { + rc = clear_ce_flags_1(istate, cache, cache_end - cache, + prefix, + select_mask, clear_mask, + pl, ret); + } + strbuf_setlen(prefix, prefix->len - 1); return rc; } From patchwork Thu Nov 21 22:04:44 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11256971 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D5B6213A4 for ; Thu, 21 Nov 2019 22:05:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id AF494206DA for ; Thu, 21 Nov 2019 22:05:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="IWJrw6i8" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726861AbfKUWFR (ORCPT ); Thu, 21 Nov 2019 17:05:17 -0500 Received: from mail-wm1-f67.google.com ([209.85.128.67]:39060 "EHLO mail-wm1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726784AbfKUWFP (ORCPT ); Thu, 21 Nov 2019 17:05:15 -0500 Received: by mail-wm1-f67.google.com with SMTP id t26so5449622wmi.4 for ; Thu, 21 Nov 2019 14:05:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=T4b8TadolwlSOZvHpfmF35nObWmTrbh1mpJa/MQa9ds=; b=IWJrw6i8U9WHUSqG+FmENh/HacPXBRf+fSQRt+XL9Yr8cWnmsAFgllTP3qcXsImgN+ 81csQJUPXg8Dq3I2TyyIAH/nTY7HZudr0Wx54EmLKchgt0CKMKkDfTP5f3UkIItFaagi P/1WXVU+uUU+hMNwgoXoUeKySt8HZXU+rJGOUSRVOHW6h2XC++qfOID/Em3VXCOJ9+er AK2i12vdtFMMEL345KMnif6xFGCN1diYdnEi8O2sPfJfIvcHHHJkfdHdh2GKt8gTv9Jv mNipd1Uu5Gsz32bzKzEzxMOqn9LBNJ6nGFllrSe0UChaxWJtHsem8OXmqUXwYBk7brx7 SQxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=T4b8TadolwlSOZvHpfmF35nObWmTrbh1mpJa/MQa9ds=; b=MBzkJZTxkV/Sul/kwY3wXdklL6CKjHvF+v+1XvSZ0233T14n6xoNSZb1Qr/jlhsKdE noiQjw2A9Upl9/vYs9DqSgv7df8TMohP8nEyEVkMEsJ4oX7NdKr8/yxXRFrPKm/bLaB5 fdJBUH3AVrFaY4gw6VU8IcnKWMaMBv1Y5v8FAJtIc/MWHNFFyMgL483matVXLV0jBjJr niBKH8dtk5SchyixczIl6O4xKa5v57aMe7Vgcs/ao3sXDo7RYn+nMCEhGn0P4BkeLkEl 45lAwQlV1A4zYPgNxGmCWy+l44t8+rjRFysY0gn9OmWgTB137czWMz2NiUgLVRWUaN6R KG6Q== X-Gm-Message-State: APjAAAXrxjYxb7zmFmYMcodSuOhzGyO0BC+IwGantBy4g5n6f5tL8Mhj qHKAcVJ9CmPotIGXFINarGL7Vf5N X-Google-Smtp-Source: APXvYqxLsFEGa0+io2gNQeFitGEuyG8bFXPioaoPop6NySWW9O9MiaHArNsoVF0yCPB9baP3IvkG2g== X-Received: by 2002:a7b:c5d9:: with SMTP id n25mr113184wmk.8.1574373911617; Thu, 21 Nov 2019 14:05:11 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id k1sm5097677wrp.29.2019.11.21.14.05.10 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 21 Nov 2019 14:05:11 -0800 (PST) Message-Id: <3d0f951d33a64dffa2f09651eca6416ee883acab.1574373892.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 21 Nov 2019 22:04:44 +0000 Subject: [PATCH v6 12/19] unpack-trees: add progress to clear_ce_flags() Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, jon@jonsimons.org, szeder.dev@gmail.com, Derrick Stolee , Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee When a large repository has many sparse-checkout patterns, the process for updating the skip-worktree bits can take long enough that a user gets confused why nothing is happening. Update the clear_ce_flags() method to write progress. Signed-off-by: Derrick Stolee --- cache.h | 2 ++ unpack-trees.c | 56 ++++++++++++++++++++++++++++++++++++-------------- 2 files changed, 43 insertions(+), 15 deletions(-) diff --git a/cache.h b/cache.h index 4980ee198e..d3c89e7a53 100644 --- a/cache.h +++ b/cache.h @@ -304,6 +304,7 @@ static inline unsigned int canon_mode(unsigned int mode) struct split_index; struct untracked_cache; +struct progress; struct index_state { struct cache_entry **cache; @@ -326,6 +327,7 @@ struct index_state { uint64_t fsmonitor_last_update; struct ewah_bitmap *fsmonitor_dirty; struct mem_pool *ce_mem_pool; + struct progress *progress; }; /* Name hashing */ diff --git a/unpack-trees.c b/unpack-trees.c index c0dca20865..8bb684ad62 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -1269,7 +1269,8 @@ static int clear_ce_flags_1(struct index_state *istate, struct strbuf *prefix, int select_mask, int clear_mask, struct pattern_list *pl, - enum pattern_match_result default_match); + enum pattern_match_result default_match, + int progress_nr); /* Whole directory matching */ static int clear_ce_flags_dir(struct index_state *istate, @@ -1278,7 +1279,8 @@ static int clear_ce_flags_dir(struct index_state *istate, char *basename, int select_mask, int clear_mask, struct pattern_list *pl, - enum pattern_match_result default_match) + enum pattern_match_result default_match, + int progress_nr) { struct cache_entry **cache_end; int dtype = DT_DIR; @@ -1315,7 +1317,8 @@ static int clear_ce_flags_dir(struct index_state *istate, rc = clear_ce_flags_1(istate, cache, cache_end - cache, prefix, select_mask, clear_mask, - pl, ret); + pl, ret, + progress_nr); } strbuf_setlen(prefix, prefix->len - 1); @@ -1342,7 +1345,8 @@ static int clear_ce_flags_1(struct index_state *istate, struct strbuf *prefix, int select_mask, int clear_mask, struct pattern_list *pl, - enum pattern_match_result default_match) + enum pattern_match_result default_match, + int progress_nr) { struct cache_entry **cache_end = cache + nr; @@ -1356,8 +1360,11 @@ static int clear_ce_flags_1(struct index_state *istate, int len, dtype; enum pattern_match_result ret; + display_progress(istate->progress, progress_nr); + if (select_mask && !(ce->ce_flags & select_mask)) { cache++; + progress_nr++; continue; } @@ -1378,20 +1385,26 @@ static int clear_ce_flags_1(struct index_state *istate, prefix, prefix->buf + prefix->len - len, select_mask, clear_mask, - pl, default_match); + pl, default_match, + progress_nr); /* clear_c_f_dir eats a whole dir already? */ if (processed) { cache += processed; + progress_nr += processed; strbuf_setlen(prefix, prefix->len - len); continue; } strbuf_addch(prefix, '/'); - cache += clear_ce_flags_1(istate, cache, cache_end - cache, - prefix, - select_mask, clear_mask, pl, - default_match); + processed = clear_ce_flags_1(istate, cache, cache_end - cache, + prefix, + select_mask, clear_mask, pl, + default_match, progress_nr); + + cache += processed; + progress_nr += processed; + strbuf_setlen(prefix, prefix->len - len - 1); continue; } @@ -1406,19 +1419,27 @@ static int clear_ce_flags_1(struct index_state *istate, if (ret == MATCHED) ce->ce_flags &= ~clear_mask; cache++; + progress_nr++; } + + display_progress(istate->progress, progress_nr); return nr - (cache_end - cache); } static int clear_ce_flags(struct index_state *istate, int select_mask, int clear_mask, - struct pattern_list *pl) + struct pattern_list *pl, + int show_progress) { static struct strbuf prefix = STRBUF_INIT; char label[100]; int rval; strbuf_reset(&prefix); + if (show_progress) + istate->progress = start_delayed_progress( + _("Updating index flags"), + istate->cache_nr); xsnprintf(label, sizeof(label), "clear_ce_flags(0x%08lx,0x%08lx)", (unsigned long)select_mask, (unsigned long)clear_mask); @@ -1428,9 +1449,10 @@ static int clear_ce_flags(struct index_state *istate, istate->cache_nr, &prefix, select_mask, clear_mask, - pl, 0); + pl, 0, 0); trace2_region_leave("unpack_trees", label, the_repository); + stop_progress(&istate->progress); return rval; } @@ -1439,7 +1461,8 @@ static int clear_ce_flags(struct index_state *istate, */ static void mark_new_skip_worktree(struct pattern_list *pl, struct index_state *istate, - int select_flag, int skip_wt_flag) + int select_flag, int skip_wt_flag, + int show_progress) { int i; @@ -1463,7 +1486,7 @@ static void mark_new_skip_worktree(struct pattern_list *pl, * 2. Widen worktree according to sparse-checkout file. * Matched entries will have skip_wt_flag cleared (i.e. "in") */ - clear_ce_flags(istate, select_flag, skip_wt_flag, pl); + clear_ce_flags(istate, select_flag, skip_wt_flag, pl, show_progress); } static int verify_absent(const struct cache_entry *, @@ -1525,7 +1548,8 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options * Sparse checkout loop #1: set NEW_SKIP_WORKTREE on existing entries */ if (!o->skip_sparse_checkout) - mark_new_skip_worktree(o->pl, o->src_index, 0, CE_NEW_SKIP_WORKTREE); + mark_new_skip_worktree(o->pl, o->src_index, 0, + CE_NEW_SKIP_WORKTREE, o->verbose_update); if (!dfc) dfc = xcalloc(1, cache_entry_size(0)); @@ -1590,7 +1614,9 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options * If the will have NEW_SKIP_WORKTREE, also set CE_SKIP_WORKTREE * so apply_sparse_checkout() won't attempt to remove it from worktree */ - mark_new_skip_worktree(o->pl, &o->result, CE_ADDED, CE_SKIP_WORKTREE | CE_NEW_SKIP_WORKTREE); + mark_new_skip_worktree(o->pl, &o->result, + CE_ADDED, CE_SKIP_WORKTREE | CE_NEW_SKIP_WORKTREE, + o->verbose_update); ret = 0; for (i = 0; i < o->result.cache_nr; i++) { From patchwork Thu Nov 21 22:04:45 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11256969 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9DFB113A4 for ; Thu, 21 Nov 2019 22:05:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 7E89D206EC for ; Thu, 21 Nov 2019 22:05:16 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="FrSlHRWI" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726836AbfKUWFQ (ORCPT ); Thu, 21 Nov 2019 17:05:16 -0500 Received: from mail-wr1-f67.google.com ([209.85.221.67]:45763 "EHLO mail-wr1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726658AbfKUWFO (ORCPT ); Thu, 21 Nov 2019 17:05:14 -0500 Received: by mail-wr1-f67.google.com with SMTP id z10so6259880wrs.12 for ; Thu, 21 Nov 2019 14:05:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=PnSPKH4Kk5DgwiPdWmgPEHDSjBOF7JG5Q8SvcQS2vJo=; b=FrSlHRWIyTVl/bQDObBFW5KMweBKWYakyuzje9wBKsz0I4xrvL8BgnOS/sApQ5FxNm 7/3UfYtthSsG+6XCX6fdgadr4NdkXurwT4dEr0XsORoylKE9kRAxvd70qkprFU/x8Ae1 fLrPksfVIMurABqi1i9I2yZTDMNaaKLujjjk1eUyn6tdyP3zMVEGATY40sjq+ghjCE68 T7+a3OCpyM115XdXqNPKvQERmXJZp2zWXwr07nABRYM+n0TiAOb6t+daHBkxToZSsrM/ P4pFTfuW6SlC1j9qAeSSukLB0DtRibVx5SFiCzOF4DkmyapAqsYOKb7XSbm1bknBMzt6 i7Nw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=PnSPKH4Kk5DgwiPdWmgPEHDSjBOF7JG5Q8SvcQS2vJo=; b=rIy0o6Z4a6hmiwIla2Hh/LnnEltUP5odHmXRPFIaH1cCbL8bLVxfjOcqVgEf/UOGv8 fdwXFDNi87F2BwKoib782UEsrhRYzymL+R59NCFedTLNbcHk7eswONj4cJZurAlxIKmb 4U4AdUrz37On9hFC1bSaGvxtsmDh7MMv9/Yoaw64yBn5YerDrJjN9tQy8H1EMYQq2756 fK7Glm/xy9rg9KYIpTIptCRczwDjLNCTe7kqXRFSKrxPVl1ct4LQHIcrsFyMo2vjgREp cxhT5Lzlp6sJSn73L12mZbL40BwC2i2lXNbHqRvYai7U9Wi3Z5qMn0JZvP/TWzMSU33r X01Q== X-Gm-Message-State: APjAAAUJH3DTXB4PlGps9jm1Nmc8osyhiHRI5yVxRSAnrI7pohs4pHag GVg0tM8CCg6NYtfHxaaXHwL6sGxd X-Google-Smtp-Source: APXvYqy/Gb8Z3QrxbEvxZfjfcACtrsGelroUUeswqS04c76xyFnTURM5F3STQmu6W4ZoPCnz5KXzYw== X-Received: by 2002:adf:c00a:: with SMTP id z10mr13747667wre.81.1574373912324; Thu, 21 Nov 2019 14:05:12 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id w7sm4903102wru.62.2019.11.21.14.05.11 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 21 Nov 2019 14:05:11 -0800 (PST) Message-Id: <2703b344f6f7195ec309454cda46f642df31a3d9.1574373892.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 21 Nov 2019 22:04:45 +0000 Subject: [PATCH v6 13/19] sparse-checkout: sanitize for nested folders Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, jon@jonsimons.org, szeder.dev@gmail.com, Derrick Stolee , Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee If a user provides folders A/ and A/B/ for inclusion in a cone-mode sparse-checkout file, the parsing logic will notice that A/ appears both as a "parent" type pattern and as a "recursive" type pattern. This is unexpected and hence will complain via a warning and revert to the old logic for checking sparse-checkout patterns. Prevent this from happening accidentally by sanitizing the folders for this type of inclusion in the 'git sparse-checkout' builtin. This happens in two ways: 1. Do not include any parent patterns that also appear as recursive patterns. 2. Do not include any recursive patterns deeper than other recursive patterns. In order to minimize duplicate code for scanning parents, create hashmap_contains_parent() method. It takes a strbuf buffer to avoid reallocating a buffer when calling in a tight loop. Helped-by: Eric Wong Helped-by: Johannes Schindelin Signed-off-by: Derrick Stolee --- builtin/sparse-checkout.c | 22 ++++++++++++++++++---- t/t1091-sparse-checkout-builtin.sh | 11 +++++++++++ 2 files changed, 29 insertions(+), 4 deletions(-) diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c index 85cc801f03..55b337ad8e 100644 --- a/builtin/sparse-checkout.c +++ b/builtin/sparse-checkout.c @@ -81,9 +81,17 @@ static void write_cone_to_file(FILE *fp, struct pattern_list *pl) struct pattern_entry *pe; struct hashmap_iter iter; struct string_list sl = STRING_LIST_INIT_DUP; + struct strbuf parent_pattern = STRBUF_INIT; - hashmap_for_each_entry(&pl->parent_hashmap, &iter, pe, ent) - string_list_insert(&sl, pe->pattern); + hashmap_for_each_entry(&pl->parent_hashmap, &iter, pe, ent) { + if (hashmap_get_entry(&pl->recursive_hashmap, pe, ent, NULL)) + continue; + + if (!hashmap_contains_parent(&pl->recursive_hashmap, + pe->pattern, + &parent_pattern)) + string_list_insert(&sl, pe->pattern); + } string_list_sort(&sl); string_list_remove_duplicates(&sl, 0); @@ -99,8 +107,14 @@ static void write_cone_to_file(FILE *fp, struct pattern_list *pl) string_list_clear(&sl, 0); - hashmap_for_each_entry(&pl->recursive_hashmap, &iter, pe, ent) - string_list_insert(&sl, pe->pattern); + hashmap_for_each_entry(&pl->recursive_hashmap, &iter, pe, ent) { + if (!hashmap_contains_parent(&pl->recursive_hashmap, + pe->pattern, + &parent_pattern)) + string_list_insert(&sl, pe->pattern); + } + + strbuf_release(&parent_pattern); string_list_sort(&sl); string_list_remove_duplicates(&sl, 0); diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh index fbd46c3f61..b88d08da98 100755 --- a/t/t1091-sparse-checkout-builtin.sh +++ b/t/t1091-sparse-checkout-builtin.sh @@ -237,4 +237,15 @@ test_expect_success 'cone mode: init and set' ' test_cmp expect dir ' +test_expect_success 'cone mode: set with nested folders' ' + git -C repo sparse-checkout set deep deep/deeper1/deepest 2>err && + test_line_count = 0 err && + cat >expect <<-EOF && + /* + !/*/ + /deep/ + EOF + test_cmp repo/.git/info/sparse-checkout expect +' + test_done From patchwork Thu Nov 21 22:04:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11256975 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E51DA930 for ; Thu, 21 Nov 2019 22:05:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id BC19B2068E for ; Thu, 21 Nov 2019 22:05:22 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="X1AJzlLD" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726887AbfKUWFV (ORCPT ); Thu, 21 Nov 2019 17:05:21 -0500 Received: from mail-wm1-f67.google.com ([209.85.128.67]:35826 "EHLO mail-wm1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726792AbfKUWFQ (ORCPT ); Thu, 21 Nov 2019 17:05:16 -0500 Received: by mail-wm1-f67.google.com with SMTP id 8so5496840wmo.0 for ; Thu, 21 Nov 2019 14:05:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=qvP7nsMmPV7mOGwJ1al5h9COXtfc1yRa2KCmKVgUN6M=; b=X1AJzlLDGL1gv5p2WbfI3EOXi+434MiGt0FjYF0GoLRevtLdOkhIBxojdmeq6LEEDN 60VMqFA9WtjWAOOhYRsgXCMQpQDcrI8izF13bdaGRupq8ggziJ2PMgyFyfDVtGF+5qyq vllAroQnIk7djIroRvH7IXFa1eC8htdpXnEPksJAe6hUQd6DBNexKgayyjs35YU3A6pM Or6gYtlbndEzTyKNsibkKzDdqTDO/HTPtYpCTCHgs1Qd0oViLI7luM7sn72b/PcfWcVR 5tDtDfbPnWaQhpILvPeVDkbQh8YTLHtcaFTmd6QMyvRdLZgQ2q3Qp10vwpdfYwfW1tjF /PIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=qvP7nsMmPV7mOGwJ1al5h9COXtfc1yRa2KCmKVgUN6M=; b=A3VXUMh6aN4jWGAlvrE3qg4zZUO7XhwPw975AUy5jUz84lp632KbEFMQStxEB3bC1d 0xVgm6PTsEFh3nQrUnygf3F8xyO7rX1oscTWmJ+5Bxkn0LXna/YH0j7BMfqpNiTIGEJV 60Su/6E3h46d9e3x7XEhlkxCsaQ799R+Wl2PuOIZorn1Oz8q4lQn0lqA7WV2Ul+xocgI eIB1hEbFmveW4Pws1BY6DMRCDcLN5MsYnpQya1WwmqFHYx2LW8+Le91uUEhH2iJGkjxq lcj6+TcdEqmau1je/FfJL1BMwWrD7wJS01hgmjsUzzlIomEG8NcNqnGn4HemDI1CXA8B A1RQ== X-Gm-Message-State: APjAAAVJ8sge6m0TYm2q5PHWh0a1gZGYIYmQ0RX0kaPfipb85FWY/X7l aG/OB0y9Uusccpr7h7rhs7di0si1 X-Google-Smtp-Source: APXvYqwRZu1Nq/ohX98bNvqutEPPfqzdK/A26E3RObWPbTI95BlOC/ciymKNb/3urm7gf2TxHI4R8w== X-Received: by 2002:a1c:e308:: with SMTP id a8mr12946844wmh.55.1574373913056; Thu, 21 Nov 2019 14:05:13 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id x9sm4706982wru.32.2019.11.21.14.05.12 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 21 Nov 2019 14:05:12 -0800 (PST) Message-Id: <63e01c202162689e5b8c9adff8ce140219a0bebe.1574373892.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 21 Nov 2019 22:04:46 +0000 Subject: [PATCH v6 14/19] sparse-checkout: update working directory in-process Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, jon@jonsimons.org, szeder.dev@gmail.com, Derrick Stolee , Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee The sparse-checkout builtin used 'git read-tree -mu HEAD' to update the skip-worktree bits in the index and to update the working directory. This extra process is overly complex, and prone to failure. It also requires that we write our changes to the sparse-checkout file before trying to update the index. Remove this extra process call by creating a direct call to unpack_trees() in the same way 'git read-tree -mu HEAD' does. In addition, provide an in-memory list of patterns so we can avoid reading from the sparse-checkout file. This allows us to test a proposed change to the file before writing to it. An earlier version of this patch included a bug when the 'set' command failed due to the "Sparse checkout leaves no entry on working directory" error. It would not rollback the index.lock file, so the replay of the old sparse-checkout specification would fail. A test in t1091 now covers that scenario. Signed-off-by: Derrick Stolee --- builtin/read-tree.c | 2 +- builtin/sparse-checkout.c | 83 +++++++++++++++++++++++++----- t/t1091-sparse-checkout-builtin.sh | 28 ++++++++++ unpack-trees.c | 5 +- unpack-trees.h | 3 +- 5 files changed, 105 insertions(+), 16 deletions(-) diff --git a/builtin/read-tree.c b/builtin/read-tree.c index ca5e655d2f..af7424b94c 100644 --- a/builtin/read-tree.c +++ b/builtin/read-tree.c @@ -185,7 +185,7 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix) if (opts.reset || opts.merge || opts.prefix) { if (read_cache_unmerged() && (opts.prefix || opts.merge)) - die("You need to resolve your current index first"); + die(_("You need to resolve your current index first")); stage = opts.merge = 1; } resolve_undo_clear(); diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c index 55b337ad8e..a5d32e4702 100644 --- a/builtin/sparse-checkout.c +++ b/builtin/sparse-checkout.c @@ -7,6 +7,11 @@ #include "run-command.h" #include "strbuf.h" #include "string-list.h" +#include "cache.h" +#include "cache-tree.h" +#include "lockfile.h" +#include "resolve-undo.h" +#include "unpack-trees.h" static char const * const builtin_sparse_checkout_usage[] = { N_("git sparse-checkout (init|list|set|disable) "), @@ -60,18 +65,54 @@ static int sparse_checkout_list(int argc, const char **argv) return 0; } -static int update_working_directory(void) +static int update_working_directory(struct pattern_list *pl) { - struct argv_array argv = ARGV_ARRAY_INIT; int result = 0; - argv_array_pushl(&argv, "read-tree", "-m", "-u", "HEAD", NULL); + struct unpack_trees_options o; + struct lock_file lock_file = LOCK_INIT; + struct object_id oid; + struct tree *tree; + struct tree_desc t; + struct repository *r = the_repository; - if (run_command_v_opt(argv.argv, RUN_GIT_CMD)) { - error(_("failed to update index with new sparse-checkout patterns")); - result = 1; - } + if (repo_read_index_unmerged(r)) + die(_("you need to resolve your current index first")); + + if (get_oid("HEAD", &oid)) + return 0; + + tree = parse_tree_indirect(&oid); + parse_tree(tree); + init_tree_desc(&t, tree->buffer, tree->size); + + memset(&o, 0, sizeof(o)); + o.verbose_update = isatty(2); + o.merge = 1; + o.update = 1; + o.fn = oneway_merge; + o.head_idx = -1; + o.src_index = r->index; + o.dst_index = r->index; + o.skip_sparse_checkout = 0; + o.pl = pl; + o.keep_pattern_list = !!pl; + + resolve_undo_clear_index(r->index); + setup_work_tree(); + + cache_tree_free(&r->index->cache_tree); + + repo_hold_locked_index(r, &lock_file, LOCK_DIE_ON_ERROR); + + core_apply_sparse_checkout = 1; + result = unpack_trees(1, &t, &o); + + if (!result) { + prime_cache_tree(r, r->index, tree); + write_locked_index(r->index, &lock_file, COMMIT_LOCK); + } else + rollback_lock_file(&lock_file); - argv_array_clear(&argv); return result; } @@ -129,6 +170,15 @@ static int write_patterns_and_update(struct pattern_list *pl) { char *sparse_filename; FILE *fp; + int result; + + result = update_working_directory(pl); + + if (result) { + clear_pattern_list(pl); + update_working_directory(NULL); + return result; + } sparse_filename = get_sparse_checkout_filename(); fp = fopen(sparse_filename, "w"); @@ -139,9 +189,11 @@ static int write_patterns_and_update(struct pattern_list *pl) write_patterns_to_file(fp, pl); fclose(fp); + free(sparse_filename); + clear_pattern_list(pl); - return update_working_directory(); + return 0; } enum sparse_checkout_mode { @@ -199,7 +251,11 @@ static int sparse_checkout_init(int argc, const char **argv) builtin_sparse_checkout_init_options, builtin_sparse_checkout_init_usage, 0); - mode = init_opts.cone_mode ? MODE_CONE_PATTERNS : MODE_ALL_PATTERNS; + if (init_opts.cone_mode) { + mode = MODE_CONE_PATTERNS; + core_sparse_checkout_cone = 1; + } else + mode = MODE_ALL_PATTERNS; if (set_config(mode)) return 1; @@ -230,7 +286,8 @@ static int sparse_checkout_init(int argc, const char **argv) } reset_dir: - return update_working_directory(); + core_apply_sparse_checkout = 1; + return update_working_directory(NULL); } static void insert_recursive_pattern(struct pattern_list *pl, struct strbuf *path) @@ -311,6 +368,7 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix) hashmap_init(&pl.recursive_hashmap, pl_hashmap_cmp, NULL, 0); hashmap_init(&pl.parent_hashmap, pl_hashmap_cmp, NULL, 0); + pl.use_cone_patterns = 1; if (set_opts.use_stdin) { while (!strbuf_getline(&line, stdin)) @@ -365,7 +423,8 @@ static int sparse_checkout_disable(int argc, const char **argv) fprintf(fp, "/*\n"); fclose(fp); - if (update_working_directory()) + core_apply_sparse_checkout = 1; + if (update_working_directory(NULL)) die(_("error while refreshing working directory")); unlink(sparse_filename); diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh index b88d08da98..53aeb5980f 100755 --- a/t/t1091-sparse-checkout-builtin.sh +++ b/t/t1091-sparse-checkout-builtin.sh @@ -248,4 +248,32 @@ test_expect_success 'cone mode: set with nested folders' ' test_cmp repo/.git/info/sparse-checkout expect ' +test_expect_success 'revert to old sparse-checkout on bad update' ' + echo update >repo/deep/deeper2/a && + cp repo/.git/info/sparse-checkout expect && + test_must_fail git -C repo sparse-checkout set deep/deeper1 2>err && + test_i18ngrep "Cannot update sparse checkout" err && + test_cmp repo/.git/info/sparse-checkout expect && + ls repo/deep >dir && + cat >expect <<-EOF && + a + deeper1 + deeper2 + EOF + test_cmp dir expect +' + +test_expect_success 'revert to old sparse-checkout on empty update' ' + git init empty-test && + ( + echo >file && + git add file && + git commit -m "test" && + test_must_fail git sparse-checkout set nothing 2>err && + test_i18ngrep "Sparse checkout leaves no entry on working directory" err && + test_i18ngrep ! ".git/index.lock" err && + git sparse-checkout set file + ) +' + test_done diff --git a/unpack-trees.c b/unpack-trees.c index 8bb684ad62..3789a22cf0 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -1511,7 +1511,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options memset(&pl, 0, sizeof(pl)); if (!core_apply_sparse_checkout || !o->update) o->skip_sparse_checkout = 1; - if (!o->skip_sparse_checkout) { + if (!o->skip_sparse_checkout && !o->pl) { char *sparse = git_pathdup("info/sparse-checkout"); pl.use_cone_patterns = core_sparse_checkout_cone; if (add_patterns_from_file_to_list(sparse, "", 0, &pl, NULL) < 0) @@ -1684,7 +1684,8 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options done: trace_performance_leave("unpack_trees"); - clear_pattern_list(&pl); + if (!o->keep_pattern_list) + clear_pattern_list(&pl); return ret; return_failed: diff --git a/unpack-trees.h b/unpack-trees.h index f2eee0c7c5..ca94a421a5 100644 --- a/unpack-trees.h +++ b/unpack-trees.h @@ -59,7 +59,8 @@ struct unpack_trees_options { quiet, exiting_early, show_all_errors, - dry_run; + dry_run, + keep_pattern_list; const char *prefix; int cache_bottom; struct dir_struct *dir; From patchwork Thu Nov 21 22:04:47 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11256979 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7DCDC14DB for ; Thu, 21 Nov 2019 22:05:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 6048D206DA for ; Thu, 21 Nov 2019 22:05:24 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="J7+TbP/H" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726909AbfKUWFW (ORCPT ); Thu, 21 Nov 2019 17:05:22 -0500 Received: from mail-wr1-f65.google.com ([209.85.221.65]:38535 "EHLO mail-wr1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726802AbfKUWFQ (ORCPT ); Thu, 21 Nov 2019 17:05:16 -0500 Received: by mail-wr1-f65.google.com with SMTP id i12so6313023wro.5 for ; Thu, 21 Nov 2019 14:05:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:mime-version :content-transfer-encoding:fcc:to:cc; bh=WrXdb3eI+X2vlTbxqFu8O3zgpIiouKGMmlNb0VYchmk=; b=J7+TbP/H8XKhr6lr0Fcdorq87OsGMKunA0Zei0i0QHGx1C9v7uIWOywrCGYxW9AoiQ lQs5HjNitaT9J6gHcOH3x0ys9+MKyT5LOpo06aCCa8ES0ZeWN7jJSoQZt3uyxUllRF2q D1AEOlE6cy1/CPa7yQ8W8Id/UMGTdZx0FgjT3jUK2iXSX141iACyqiwxm+hYJ+p8YCkQ pzxiweHQK3akJ3faZFhbDCIJHpFKlDNaLgx27hp+T7FiFqu1iX2gJPhz+1z8XLcxtS0c 3r86/hmYhmOwqtEXQos/ABDypXLXVGy74z44EYANVkpF9dGpR9WY1R8irXp0HAd4KNGQ Kg/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:mime-version:content-transfer-encoding:fcc:to:cc; bh=WrXdb3eI+X2vlTbxqFu8O3zgpIiouKGMmlNb0VYchmk=; b=NYpJQKajX6Kz2CO1esU4N4nWsMaYBsft1eI7DEHgVdTuEqt1GFyjXs4yne9f0Sfvkn 5ywanYS8vwGyJc7rcL6MXdzD6YpEDQ0ybg89nbb0y+kpfh9LCbGb9bM8BER2kx3kwXq2 LBI7HlG/st9IYoCb/EqQBTCIM+fgsd837dd0FOlbcPt2k++wjKZw0CTAKCrLwMDXobYB BFhFPoXIvkXZOqzHg5onr9ZhmGCcUbZebYUNBTv7fxnS4Rf5z5Apj8cEEKNYf+oC2H49 Uu/D/KjA1uWbumO/8QlLR1nS7j6U6nD2EPTQts/mkgsb4BA6znXMbcC2rt6qI8tW/9o3 WDSg== X-Gm-Message-State: APjAAAVMD5aFgiD8WKXdkL67V0mpZlcY/zb94oyoZCLA+Zixea3Q8ztR i+zIQV/Cfvl1O+diezy+wwYYoFZf X-Google-Smtp-Source: APXvYqyxMxAMw8ZJ109fmbIHEg/w7oMOhOy0pD82W0cYITsdbNbCwv5cgS+GIjGMdkmUOJ5+cOgHOg== X-Received: by 2002:a5d:6706:: with SMTP id o6mr13845545wru.54.1574373913817; Thu, 21 Nov 2019 14:05:13 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id g8sm1056006wmk.23.2019.11.21.14.05.13 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 21 Nov 2019 14:05:13 -0800 (PST) Message-Id: In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 21 Nov 2019 22:04:47 +0000 Subject: [PATCH v6 15/19] sparse-checkout: use in-process update for disable subcommand MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: newren@gmail.com, jon@jonsimons.org, szeder.dev@gmail.com, Derrick Stolee , Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee The 'git sparse-checkout disable' subcommand returns a user to a full working directory. The old process for doing this required updating the sparse-checkout file with the "/*" pattern and then updating the working directory with core.sparseCheckout enabled. Finally, the sparse-checkout file could be removed and the config setting disabled. However, it is valuable to keep a user's sparse-checkout file intact so they can re-enable the sparse-checkout they previously used with 'git sparse-checkout init'. This is now possible with the in-process mechanism for updating the working directory. Reported-by: Szeder Gábor Signed-off-by: Derrick Stolee --- Documentation/git-sparse-checkout.txt | 6 ++++-- builtin/sparse-checkout.c | 25 ++++++++++++------------- t/t1091-sparse-checkout-builtin.sh | 3 ++- 3 files changed, 18 insertions(+), 16 deletions(-) diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt index 8535f0cf40..b975285673 100644 --- a/Documentation/git-sparse-checkout.txt +++ b/Documentation/git-sparse-checkout.txt @@ -52,8 +52,10 @@ When the `--stdin` option is provided, the patterns are read from standard in as a newline-delimited list instead of from the arguments. 'disable':: - Remove the sparse-checkout file, set `core.sparseCheckout` to - `false`, and restore the working directory to include all files. + Disable the `core.sparseCheckout` config setting, and restore the + working directory to include all files. Leaves the sparse-checkout + file intact so a later 'git sparse-checkout init' command may + return the working directory to the same state. SPARSE CHECKOUT --------------- diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c index a5d32e4702..a11ea65599 100644 --- a/builtin/sparse-checkout.c +++ b/builtin/sparse-checkout.c @@ -412,24 +412,23 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix) static int sparse_checkout_disable(int argc, const char **argv) { - char *sparse_filename; - FILE *fp; + static const char *empty_base = ""; + struct pattern_list pl; + struct strbuf match_all = STRBUF_INIT; - if (set_config(MODE_ALL_PATTERNS)) - die(_("failed to change config")); + memset(&pl, 0, sizeof(pl)); + hashmap_init(&pl.recursive_hashmap, pl_hashmap_cmp, NULL, 0); + hashmap_init(&pl.parent_hashmap, pl_hashmap_cmp, NULL, 0); + pl.use_cone_patterns = 0; + core_apply_sparse_checkout = 1; - sparse_filename = get_sparse_checkout_filename(); - fp = xfopen(sparse_filename, "w"); - fprintf(fp, "/*\n"); - fclose(fp); + strbuf_addstr(&match_all, "/*"); + add_pattern(strbuf_detach(&match_all, NULL), empty_base, 0, &pl, 0); - core_apply_sparse_checkout = 1; - if (update_working_directory(NULL)) + if (update_working_directory(&pl)) die(_("error while refreshing working directory")); - unlink(sparse_filename); - free(sparse_filename); - + clear_pattern_list(&pl); return set_config(MODE_NO_PATTERNS); } diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh index 53aeb5980f..b8f18e2a09 100755 --- a/t/t1091-sparse-checkout-builtin.sh +++ b/t/t1091-sparse-checkout-builtin.sh @@ -172,8 +172,9 @@ test_expect_success 'cone mode: warn on bad pattern' ' ' test_expect_success 'sparse-checkout disable' ' + test_when_finished rm -rf repo/.git/info/sparse-checkout && git -C repo sparse-checkout disable && - test_path_is_missing repo/.git/info/sparse-checkout && + test_path_is_file repo/.git/info/sparse-checkout && git -C repo config --list >config && test_must_fail git config core.sparseCheckout && ls repo >dir && From patchwork Thu Nov 21 22:04:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11256977 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1D8E313A4 for ; Thu, 21 Nov 2019 22:05:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id F1B3D206D8 for ; Thu, 21 Nov 2019 22:05:23 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="uDm4oAor" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726912AbfKUWFX (ORCPT ); Thu, 21 Nov 2019 17:05:23 -0500 Received: from mail-wm1-f68.google.com ([209.85.128.68]:40714 "EHLO mail-wm1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726813AbfKUWFQ (ORCPT ); Thu, 21 Nov 2019 17:05:16 -0500 Received: by mail-wm1-f68.google.com with SMTP id y5so5444025wmi.5 for ; Thu, 21 Nov 2019 14:05:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=EX5YDmS2oMRkwwKDr9GF8nMuKZ8EWcHKff5rZlGTEzI=; b=uDm4oAorcreqhBKRhMdEToobSfwHECmXEwwk74jI/2Tb/qChWGidoRsT3ElKrd+v3G SFpVwa/uGX2DIyVS3j1MdTSqAxw29IEdBShJWxQaP4WvkFDzY2MhQd4g7aj7bX5HBEmR YBgbsrOBlc1/3VMp3q6Iq1MMKNEl0M4/UlBie1Ax5xf7oWehac5Ntgy7aOEj9Qq0+Mqm ksbCZgSRHsT4x73wqja36PrZnmDb0g35IRypraTV5Mv3pNQbawLHRewQ3qTKQfRmkwDy kiW0FqF6kOqZihprSn3EYw8JO+xv+vb1YoFlikofkw2EDJVMN0S3bcI71JrsumRFn/jV ZSow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=EX5YDmS2oMRkwwKDr9GF8nMuKZ8EWcHKff5rZlGTEzI=; b=o8kIB9YezxZ7GqDeD/6UZe/QVr94h1Q7ePZ5PUZpjOxtoQwN3srpm0drf2TvrvClut ABPu9RkWQf2jwk2Dm+dz4kqx8PGkMhzQEwsbTnDqK2UljIcc+tNR7DR2zEGBVNPCU7yC kZ1pgR5u8jJ0XaFeJLYYTiL2RhPR2s51rSiZyXYmBcEfvpQHlUCChC464qOFbADRJt/Q xpJE/K3N6iKpPT4JnyUshe4DDspKqwKK4bHkMLOVwDTrPxOXYXqEi5x5F2fj9SO5sIwK jVLHgJG7qKGDRfOzf7DhWU1DF73Rykx98J6puQOFUaOXm5NdLI1D/JMvdApSAvd6YTl+ nNEA== X-Gm-Message-State: APjAAAXp/ly9gvfYGe4BgmDAbfHU+QTVQ2cqxueeZTTAk3rx+HZ2M7Us 6irtLx/qr1TS8cZ3l8so/cpd6bVJ X-Google-Smtp-Source: APXvYqwGW1CNs4JaVCn6uAO35nTuT1+MmBr+YND5IKUKFW6d2TCXwODI0Ho9UziTTXU3axoWbBLhfQ== X-Received: by 2002:a05:600c:21c9:: with SMTP id x9mr12746549wmj.54.1574373914537; Thu, 21 Nov 2019 14:05:14 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id t185sm1171896wmf.45.2019.11.21.14.05.13 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 21 Nov 2019 14:05:14 -0800 (PST) Message-Id: In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 21 Nov 2019 22:04:48 +0000 Subject: [PATCH v6 16/19] sparse-checkout: write using lockfile Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, jon@jonsimons.org, szeder.dev@gmail.com, Derrick Stolee , Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee If two 'git sparse-checkout set' subcommands are launched at the same time, the behavior can be unexpected as they compete to write the sparse-checkout file and update the working directory. Take a lockfile around the writes to the sparse-checkout file. In addition, acquire this lock around the working directory update to avoid two commands updating the working directory in different ways. Signed-off-by: Derrick Stolee --- builtin/sparse-checkout.c | 15 +++++++++++---- t/t1091-sparse-checkout-builtin.sh | 7 +++++++ 2 files changed, 18 insertions(+), 4 deletions(-) diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c index a11ea65599..9a620ff014 100644 --- a/builtin/sparse-checkout.c +++ b/builtin/sparse-checkout.c @@ -170,25 +170,32 @@ static int write_patterns_and_update(struct pattern_list *pl) { char *sparse_filename; FILE *fp; + int fd; + struct lock_file lk = LOCK_INIT; int result; - result = update_working_directory(pl); + sparse_filename = get_sparse_checkout_filename(); + fd = hold_lock_file_for_update(&lk, sparse_filename, + LOCK_DIE_ON_ERROR); + result = update_working_directory(pl); if (result) { + rollback_lock_file(&lk); + free(sparse_filename); clear_pattern_list(pl); update_working_directory(NULL); return result; } - sparse_filename = get_sparse_checkout_filename(); - fp = fopen(sparse_filename, "w"); + fp = xfdopen(fd, "w"); if (core_sparse_checkout_cone) write_cone_to_file(fp, pl); else write_patterns_to_file(fp, pl); - fclose(fp); + fflush(fp); + commit_lock_file(&lk); free(sparse_filename); clear_pattern_list(pl); diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh index b8f18e2a09..f074b7f3be 100755 --- a/t/t1091-sparse-checkout-builtin.sh +++ b/t/t1091-sparse-checkout-builtin.sh @@ -277,4 +277,11 @@ test_expect_success 'revert to old sparse-checkout on empty update' ' ) ' +test_expect_success 'fail when lock is taken' ' + test_when_finished rm -rf repo/.git/info/sparse-checkout.lock && + touch repo/.git/info/sparse-checkout.lock && + test_must_fail git -C repo sparse-checkout set deep 2>err && + test_i18ngrep "File exists" err +' + test_done From patchwork Thu Nov 21 22:04:49 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11256973 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 359C413A4 for ; Thu, 21 Nov 2019 22:05:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 1554B2068E for ; Thu, 21 Nov 2019 22:05:22 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="R1Pu9fJw" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726880AbfKUWFV (ORCPT ); Thu, 21 Nov 2019 17:05:21 -0500 Received: from mail-wr1-f68.google.com ([209.85.221.68]:38540 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726855AbfKUWFS (ORCPT ); Thu, 21 Nov 2019 17:05:18 -0500 Received: by mail-wr1-f68.google.com with SMTP id i12so6313084wro.5 for ; Thu, 21 Nov 2019 14:05:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=WCHjYPVvLSfbWOovV0VZCb0RhxBQMPojNdRCPNPSy+g=; b=R1Pu9fJwY7sTdL7luCb4Jni8kVPl0FAwWQXZDi7s76og37JnrajEveao0EucYXTi9M yh4bPDbT9Nh/RfDBN/CMuLrvIwceNZrQD54FEndap8xAHOReoxVlVMcysmS1audFjG8z YqnJPDd0mlhoorehuqPYMR8JPASLA8yif9fXzqp4PXakaQSZz2rUklhI+GtAC8yjeWcD koz1S0n/9a2vYqyJ6UkhLsSHJPQKLiqWKlm1bVwV7os0YxYiQ8TSDNAYflPUnBvigd4h hGI8KpsX5pS6n7FTazLNswnIOwAYHPI1YfFxwF/lwC/eK53RAe7ysbtU7qSOt7+NUoq3 ibpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=WCHjYPVvLSfbWOovV0VZCb0RhxBQMPojNdRCPNPSy+g=; b=f54u4az/6iyxidMjDwHEpzZB6b/ojEgrdbVracXHqLOUUrDDAc1bZgwWrS8oh5vy7D 3NqgHO+VdeimIVUcVjTb+mhPG9rl5yGqElmXM0qd8DUG4ppjvkmnuCUmrZeVq1IcgsP0 2gWWkyis1KRlYedB/fBk8vEOE4N/+acL6DnzhWFtb5romXWL9wfCLjQ9wDxRnJr96EER WmfljE4UUa6H4M4lUMuZ0FcfbIdXJagS3/H1xO0L1fyHAVlYVmXzL8nZESqZ2/Qs+gFO KzYF+mEhyQ9Sj4rBFr43zElSRvxRIajRW0EylLqavbjFd1Nadi0/NAwDH+9KGZGHCVW3 f1Mg== X-Gm-Message-State: APjAAAX335VfzdnJJNATppak0UxqPe5AFxjyTMxCnv+Tch8RzeAIYTfF HvAhy2QG4KN5rKUDpqHrOXu+3QJ1 X-Google-Smtp-Source: APXvYqzXlYfxMg4sMaist9vv5l/mFdZp8Xri5RtAGqRON+/B9SbTF+WDhnDAHUwsod0DIqO1lSld9g== X-Received: by 2002:adf:82cc:: with SMTP id 70mr12236540wrc.231.1574373915247; Thu, 21 Nov 2019 14:05:15 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id y15sm4729843wrh.94.2019.11.21.14.05.14 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 21 Nov 2019 14:05:14 -0800 (PST) Message-Id: <4097d8f6dec0cc25e356c17a8cda73d3c32a4894.1574373892.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 21 Nov 2019 22:04:49 +0000 Subject: [PATCH v6 17/19] sparse-checkout: cone mode should not interact with .gitignore Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, jon@jonsimons.org, szeder.dev@gmail.com, Derrick Stolee , Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee During the development of the sparse-checkout "cone mode" feature, an incorrect placement of the initializer for "use_cone_patterns = 1" caused warnings to show up when a .gitignore file was present with non-cone-mode patterns. This was fixed in the original commit introducing the cone mode, but now we should add a test to avoid hitting this problem again in the future. Signed-off-by: Derrick Stolee --- t/t1091-sparse-checkout-builtin.sh | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh index f074b7f3be..e61ddb4ad5 100755 --- a/t/t1091-sparse-checkout-builtin.sh +++ b/t/t1091-sparse-checkout-builtin.sh @@ -284,4 +284,11 @@ test_expect_success 'fail when lock is taken' ' test_i18ngrep "File exists" err ' +test_expect_success '.gitignore should not warn about cone mode' ' + git -C repo config --worktree core.sparseCheckoutCone true && + echo "**/bin/*" >repo/.gitignore && + git -C repo reset --hard 2>err && + test_i18ngrep ! "disabling cone patterns" err +' + test_done From patchwork Thu Nov 21 22:04:50 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11256981 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E7DA5930 for ; Thu, 21 Nov 2019 22:05:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id C9C60206DA for ; Thu, 21 Nov 2019 22:05:24 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="HiQEHq8O" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726905AbfKUWFW (ORCPT ); Thu, 21 Nov 2019 17:05:22 -0500 Received: from mail-wr1-f68.google.com ([209.85.221.68]:42933 "EHLO mail-wr1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726846AbfKUWFR (ORCPT ); Thu, 21 Nov 2019 17:05:17 -0500 Received: by mail-wr1-f68.google.com with SMTP id a15so6286151wrf.9 for ; Thu, 21 Nov 2019 14:05:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=n1aeEF8voJJ5tk7dX1MuJjkfozlm4w+6/6kB4OTgu2M=; b=HiQEHq8OjGUl+5B0mrR89d6HnEjfhcJirEY1N+JyPAZxOWDTol2hQyLZEo1jRE6YKV Iy++Ioi5Yra1UrS/fhKq/fvQi8MqvxEcWPigcALF+ZOmMOTSnezVhdZd42613cpOY42m agXV0dwHo5EKwYmkuGcBmeA/MCKDvSaW5fTATD0uWxiC26JlJ8L2oSpCrX8Tbsz+kBDi wq+8hYBBq7hw/dSdMby/CIeyZG8R316pJjgmNgG+qCRrZ7/MtJSlccgz8APsyz2bRwp9 ycEB2MVDc1Xqk5hgdFAMjvVkqVRdtcMPTM3Xj9TSS3+l2kz2p8OTJU3Lw8HtONDTpSCX xF2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=n1aeEF8voJJ5tk7dX1MuJjkfozlm4w+6/6kB4OTgu2M=; b=qXyQXyLq6hHfwQfKDDEQpviMPzzm87C2DrQLlmIA4RFhQr5qjV+sIVLe63Gwc7d9Qw JSUya60mQW5nwYarSrJzi0sKt+LK/sjAKaKIXkRuup8KfgXiXTNYiaHyPJo4LNPbuVhR Fc/Z4bSrFmHuJVs7UePsf2ZRndUm851Pj69cQ6WkOA/jioAm2MOTDBIOP2Kd4hrdGqPm 1nlLtmdU8ehdtkHWVKY4kFFj0gg4HPPJ5GWzwlcv8sW9U7dX1p9HOZMLhh9fmPb4SB3m 8ucVH2NdxRjtj+n7kyfsMraNQsbFBHl/DM775tSUWDkl9QO3H6ctyiqFM83ANYund0Bj IGig== X-Gm-Message-State: APjAAAXWS6wKu2TyA6Bzt7uq8mB2L9LKR1OPJt7rHJKA9TshQwd1WVKk 08/CA1TJlrtkDpUDPryljTUDgkLf X-Google-Smtp-Source: APXvYqzq+b0wKzuzFVmWeMe649gPHKGceu05UMoTXwe1xsgQngXKQYoMRc9a56sxtMOZkkf8l241aQ== X-Received: by 2002:a5d:526f:: with SMTP id l15mr13616272wrc.169.1574373915901; Thu, 21 Nov 2019 14:05:15 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id d67sm1079640wmd.13.2019.11.21.14.05.15 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 21 Nov 2019 14:05:15 -0800 (PST) Message-Id: In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 21 Nov 2019 22:04:50 +0000 Subject: [PATCH v6 18/19] sparse-checkout: update working directory in-process for 'init' Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, jon@jonsimons.org, szeder.dev@gmail.com, Derrick Stolee , Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee The 'git sparse-checkout init' subcommand previously wrote directly to the sparse-checkout file and then updated the working directory. This may fail if there are modified files not included in the initial pattern set. However, that left a populated sparse-checkout file. Use the in-process working directory update to guarantee that the init subcommand only changes the sparse-checkout file if the working directory update succeeds. Signed-off-by: Derrick Stolee --- builtin/sparse-checkout.c | 36 ++++++++++++++++++++---------------- 1 file changed, 20 insertions(+), 16 deletions(-) diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c index 9a620ff014..5dbfb2a7e0 100644 --- a/builtin/sparse-checkout.c +++ b/builtin/sparse-checkout.c @@ -13,6 +13,8 @@ #include "resolve-undo.h" #include "unpack-trees.h" +static const char *empty_base = ""; + static char const * const builtin_sparse_checkout_usage[] = { N_("git sparse-checkout (init|list|set|disable) "), NULL @@ -243,10 +245,10 @@ static int sparse_checkout_init(int argc, const char **argv) { struct pattern_list pl; char *sparse_filename; - FILE *fp; int res; struct object_id oid; int mode; + struct strbuf pattern = STRBUF_INIT; static struct option builtin_sparse_checkout_init_options[] = { OPT_BOOL(0, "cone", &init_opts.cone_mode, @@ -275,26 +277,30 @@ static int sparse_checkout_init(int argc, const char **argv) /* If we already have a sparse-checkout file, use it. */ if (res >= 0) { free(sparse_filename); - goto reset_dir; + core_apply_sparse_checkout = 1; + return update_working_directory(NULL); } - /* initial mode: all blobs at root */ - fp = xfopen(sparse_filename, "w"); - if (!fp) - die(_("failed to open '%s'"), sparse_filename); + if (get_oid("HEAD", &oid)) { + FILE *fp; - free(sparse_filename); - fprintf(fp, "/*\n!/*/\n"); - fclose(fp); + /* assume we are in a fresh repo, but update the sparse-checkout file */ + fp = xfopen(sparse_filename, "w"); + if (!fp) + die(_("failed to open '%s'"), sparse_filename); - if (get_oid("HEAD", &oid)) { - /* assume we are in a fresh repo */ + free(sparse_filename); + fprintf(fp, "/*\n!/*/\n"); + fclose(fp); return 0; } -reset_dir: - core_apply_sparse_checkout = 1; - return update_working_directory(NULL); + strbuf_addstr(&pattern, "/*"); + add_pattern(strbuf_detach(&pattern, NULL), empty_base, 0, &pl, 0); + strbuf_addstr(&pattern, "!/*/"); + add_pattern(strbuf_detach(&pattern, NULL), empty_base, 0, &pl, 0); + + return write_patterns_and_update(&pl); } static void insert_recursive_pattern(struct pattern_list *pl, struct strbuf *path) @@ -351,7 +357,6 @@ static struct sparse_checkout_set_opts { static int sparse_checkout_set(int argc, const char **argv, const char *prefix) { - static const char *empty_base = ""; int i; struct pattern_list pl; int result; @@ -419,7 +424,6 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix) static int sparse_checkout_disable(int argc, const char **argv) { - static const char *empty_base = ""; struct pattern_list pl; struct strbuf match_all = STRBUF_INIT; From patchwork Thu Nov 21 22:04:51 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11256983 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DD966930 for ; Thu, 21 Nov 2019 22:05:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id BE3662068E for ; Thu, 21 Nov 2019 22:05:26 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="GL4gip38" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726869AbfKUWFV (ORCPT ); Thu, 21 Nov 2019 17:05:21 -0500 Received: from mail-wm1-f67.google.com ([209.85.128.67]:39069 "EHLO mail-wm1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726858AbfKUWFS (ORCPT ); Thu, 21 Nov 2019 17:05:18 -0500 Received: by mail-wm1-f67.google.com with SMTP id t26so5449810wmi.4 for ; Thu, 21 Nov 2019 14:05:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:mime-version :content-transfer-encoding:fcc:to:cc; bh=kuc6H/vkYHxrEsGSG2uEwneropQ9h+GKXu7xTBk9qmg=; b=GL4gip38cSqYk55TaJ4fI6TuIFyQq4nlkYe/w43gBkqWGjUJPGaZFpqyPO2+liiP2k ELnwrsKJNKCOqqGTmgTWpxZ1CTyaVM5esaQQ1mQSdhVcC4A+y0zW5hlqRKXpClrZN3sa /mgTJJThNriFQkhZXLiCvQFrEmB9qqusHjmHrcEpcNNoyKH7fxlOFck4SrSvUyHupfVd YRqHvvSXcxZPhNy84V8Qwx438bNpoFd99uFwt3L3r72paUeDaScI10bHPJuD419ORwgV 4mYe0IspMM+gPf9j5bryF6kkvpcgrAKYQqOt34vVY7bSzNpmp5HRUtiS+2Rd5b/rQ934 qV8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:mime-version:content-transfer-encoding:fcc:to:cc; bh=kuc6H/vkYHxrEsGSG2uEwneropQ9h+GKXu7xTBk9qmg=; b=O7Q7g+OiON88UvmTkLr6PzZU2ib4glXIow12UYqtyLMllI/uOoQCwPjLYeegRCCdlM 38C4jIBMyGMP023Gc/95i0yOfm0TWrN3GQvyVSZgUl8pei0/Rn7IWRb44LJ1XUwp62XI Jj7p64gZSRznHluIP4F0Ckd5y3ruZZmwPaeMEh+ZqO8N4OdYdRek+ZcJQt9WHTiFmwPA uS1rVZcr9hhrt15mqZyWmcFfxbkCJl0t47xvIpmlhfeFO07osiB7bCX254fvhIgkYAWB 1WOLZckmK66TdB5a1u42+rICbPEbF167Z6Q3oQ4uXSkND6CVE5QwBu+vJiNAv7fojj7T Y8gA== X-Gm-Message-State: APjAAAXVyHxBUfrxwATj+vb+X+I0uOSJfrZXwXLU5EdSmVdnHGbfwnxU 6JHObTGe+yILfnsw74eiuiIlGZS2 X-Google-Smtp-Source: APXvYqxMoiC8WTeQGTMo49NcQZMTuKHc7AoXlULyoM4hSaBf/hHk6sN4xHgPr0lbu0E4NF3Q77i59g== X-Received: by 2002:a7b:ca57:: with SMTP id m23mr1026239wml.65.1574373916516; Thu, 21 Nov 2019 14:05:16 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 4sm1158743wmd.33.2019.11.21.14.05.16 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 21 Nov 2019 14:05:16 -0800 (PST) Message-Id: <7577ffc034be90bd60da17e8f3189d999e7f8149.1574373892.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 21 Nov 2019 22:04:51 +0000 Subject: [PATCH v6 19/19] sparse-checkout: check for dirty status MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: newren@gmail.com, jon@jonsimons.org, szeder.dev@gmail.com, Derrick Stolee , Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee The index-merge performed by 'git sparse-checkout' will erase any staged changes, which can lead to data loss. Prevent these attempts by requiring a clean 'git status' output. Helped-by: Szeder Gábor Signed-off-by: Derrick Stolee --- builtin/sparse-checkout.c | 13 +++++++++++++ t/t1091-sparse-checkout-builtin.sh | 15 ++++++++++++++- 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c index 5dbfb2a7e0..a542d617a5 100644 --- a/builtin/sparse-checkout.c +++ b/builtin/sparse-checkout.c @@ -12,6 +12,7 @@ #include "lockfile.h" #include "resolve-undo.h" #include "unpack-trees.h" +#include "wt-status.h" static const char *empty_base = ""; @@ -256,6 +257,10 @@ static int sparse_checkout_init(int argc, const char **argv) OPT_END(), }; + repo_read_index(the_repository); + require_clean_work_tree(the_repository, + N_("initialize sparse-checkout"), NULL, 1, 0); + argc = parse_options(argc, argv, NULL, builtin_sparse_checkout_init_options, builtin_sparse_checkout_init_usage, 0); @@ -368,6 +373,10 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix) OPT_END(), }; + repo_read_index(the_repository); + require_clean_work_tree(the_repository, + N_("set sparse-checkout patterns"), NULL, 1, 0); + memset(&pl, 0, sizeof(pl)); argc = parse_options(argc, argv, prefix, @@ -427,6 +436,10 @@ static int sparse_checkout_disable(int argc, const char **argv) struct pattern_list pl; struct strbuf match_all = STRBUF_INIT; + repo_read_index(the_repository); + require_clean_work_tree(the_repository, + N_("disable sparse-checkout"), NULL, 1, 0); + memset(&pl, 0, sizeof(pl)); hashmap_init(&pl.recursive_hashmap, pl_hashmap_cmp, NULL, 0); hashmap_init(&pl.parent_hashmap, pl_hashmap_cmp, NULL, 0); diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh index e61ddb4ad5..d5e2892526 100755 --- a/t/t1091-sparse-checkout-builtin.sh +++ b/t/t1091-sparse-checkout-builtin.sh @@ -250,10 +250,11 @@ test_expect_success 'cone mode: set with nested folders' ' ' test_expect_success 'revert to old sparse-checkout on bad update' ' + test_when_finished git -C repo reset --hard && echo update >repo/deep/deeper2/a && cp repo/.git/info/sparse-checkout expect && test_must_fail git -C repo sparse-checkout set deep/deeper1 2>err && - test_i18ngrep "Cannot update sparse checkout" err && + test_i18ngrep "cannot set sparse-checkout patterns" err && test_cmp repo/.git/info/sparse-checkout expect && ls repo/deep >dir && cat >expect <<-EOF && @@ -291,4 +292,16 @@ test_expect_success '.gitignore should not warn about cone mode' ' test_i18ngrep ! "disabling cone patterns" err ' +test_expect_success 'sparse-checkout (init|set|disable) fails with dirty status' ' + git clone repo dirty && + echo dirty >dirty/folder1/a && + test_must_fail git -C dirty sparse-checkout init && + test_must_fail git -C dirty sparse-checkout set /folder2/* /deep/deeper1/* && + test_must_fail git -C dirty sparse-checkout disable && + git -C dirty reset --hard && + git -C dirty sparse-checkout init && + git -C dirty sparse-checkout set /folder2/* /deep/deeper1/* && + git -C dirty sparse-checkout disable +' + test_done