From patchwork Thu May 7 13:17:33 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11533603 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4CC0481 for ; Thu, 7 May 2020 13:17:49 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3496720838 for ; Thu, 7 May 2020 13:17:49 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="OKwC2+KG" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726797AbgEGNRs (ORCPT ); Thu, 7 May 2020 09:17:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37970 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-FAIL-OK-FAIL) by vger.kernel.org with ESMTP id S1725879AbgEGNRr (ORCPT ); Thu, 7 May 2020 09:17:47 -0400 Received: from mail-wm1-x344.google.com (mail-wm1-x344.google.com [IPv6:2a00:1450:4864:20::344]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A79C5C05BD09 for ; Thu, 7 May 2020 06:17:45 -0700 (PDT) Received: by mail-wm1-x344.google.com with SMTP id u127so6780532wmg.1 for ; Thu, 07 May 2020 06:17:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=weKXCsP2M9DFkclCwIaqBScvevm4IdrU27uUPWoGnEw=; b=OKwC2+KGjgvy2WEoPmMeEMOsQBu/yhJSAfpnAWyPBoE0zNNnxBHHRpfNGphWgJgHm6 25bP//1eRQgDqbN8lCUK5C2wn3HbJ+Y7q6BOAieiyhGU9RL3GP3cHHFRxZmI0ujMAaOG ZTZbQTs+o/gp7qdxh5MK/DQdaO8/X5KMpeLLSyvzviICsZnmk6f9/J2v1F9bU/j8ycsi 8CsTzX41H9wsp33LEnj4BHrRIt1/bT9OCjoxVYT/OC9EccjW/gwP8R7XOWoqSbEu1aya wwau7K5xaatWMAeuWInuRBWKzhlrHoLEsDiIjJ3MTuBbQGJOVan8FKkw4xO2kAiws/Pi DacQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=weKXCsP2M9DFkclCwIaqBScvevm4IdrU27uUPWoGnEw=; b=E4izbsJHBbzO5LPTn+1/UrxCg2izQbL7nGLeF4L53PFHf3hATw0tX94THYDbkfCN2M xwlPR1AieDlgXd78a8DsSETkCPhK6FUbPS/ATKRN2PjimVLJQ2uHAqekwQOi5Xv6OQL2 vdn8cd85zxpmyBn5FXk9aXukArUarYuOYoaqeS9BELTBSThE2aNaQAhYr5n66WybAkjB wcP64qlx9WqPpZzENKplEdhCXsBMgqd2bYn7Nb44Nrhc52yZ34rtS/A8G22CyO+Z3nig 0mWPerBfyPorx2e/T/DSgRWLyRrh65tUN4haNP+2qgZdnnZy8fvlx9PFQl+hq3SweXnm rlFg== X-Gm-Message-State: AGi0PuZ7tAcyAuesZ2sPOJuWR10qgfIw54/t+LnCQUfxs2ylykxGaDZI mkNnGlls51HfQo7m0nKD6mpV+ew2 X-Google-Smtp-Source: APiQypK4SjdHehH0BguXaZiFLkyn35GQCQ66eCi32nlJ6QrbygPMbJbYLCxQKZWdKZagYf6NM+uZbw== X-Received: by 2002:a1c:6a17:: with SMTP id f23mr9838212wmc.136.1588857464294; Thu, 07 May 2020 06:17:44 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 1sm8134268wmz.13.2020.05.07.06.17.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 May 2020 06:17:43 -0700 (PDT) Message-Id: <5bfe3f3fc8a99b3d4fdd4286da17cd935090c614.1588857462.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 07 May 2020 13:17:33 +0000 Subject: [PATCH 01/10] unpack-trees: avoid array out-of-bounds error Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmaill.com, peff@peff.net, me@ttaylorr.com, jrnieder@gmail.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee The loop in warn_conflicted_path() that checks for the count of entries with the same path uses "i+count" for the array entry. However, the loop only verifies that the value of count is below the array size. Fix this by adding i to the condition. I hit this condition during a test of the in-tree sparse-checkout feature, so it is exercised by the end of the series. Signed-off-by: Derrick Stolee --- unpack-trees.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/unpack-trees.c b/unpack-trees.c index 9a3ccd9d083..4f880f2da90 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -563,10 +563,11 @@ static int warn_conflicted_path(struct index_state *istate, add_rejected_path(o, WARNING_SPARSE_UNMERGED_FILE, conflicting_path); /* Find out how many higher stage entries at same path */ - while (++count < istate->cache_nr && + while (i + ++count < istate->cache_nr && !strcmp(conflicting_path, istate->cache[i+count]->name)) /* do nothing */; + return count; } From patchwork Thu May 7 13:17:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11533605 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D00FF1668 for ; Thu, 7 May 2020 13:17:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id ADCAA20870 for ; Thu, 7 May 2020 13:17:51 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Bbapj/pe" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726815AbgEGNRt (ORCPT ); Thu, 7 May 2020 09:17:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37978 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-FAIL-OK-FAIL) by vger.kernel.org with ESMTP id S1726776AbgEGNRr (ORCPT ); Thu, 7 May 2020 09:17:47 -0400 Received: from mail-wm1-x344.google.com (mail-wm1-x344.google.com [IPv6:2a00:1450:4864:20::344]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 11D2CC05BD43 for ; Thu, 7 May 2020 06:17:47 -0700 (PDT) Received: by mail-wm1-x344.google.com with SMTP id k12so6467917wmj.3 for ; Thu, 07 May 2020 06:17:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=WrrjA/SgYyXFHBt9gbXWdPSiiDxOhuTGU9CTQ9nrm/w=; b=Bbapj/pebRfOpjuVp3h86M5vqU8oEJ1ZSyHeJgAOUMdh4pfvhMwT6g/f5vispwX7pH 0Czy2/AXCGp+2SkjJ9LgUEeT9xkz7o71K5lTrnOhufp4hLL1tooo4Iq+cr/vevllP9sj lqHAcVf+qinBtahQO1FQbG5vSferg958T2vjbmjagoERcTdHciiqGOegvARy4/wi/vK3 mSFKSBC1bis79xrLnHaopcCujxcyO5m3/BZ5DOY4UjA9p2Dy5IAA3wV13d3OhZnnKl2d JnC4zEIDMi7ZX5gkwy14oOcA/XClZGG1DjzcSugjfLbggxPUlz9+8nYu1+iXEU37KYSG eIEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=WrrjA/SgYyXFHBt9gbXWdPSiiDxOhuTGU9CTQ9nrm/w=; b=KflBCtIxP97wJkcaFgZAaCj8w9x5MvV/h5daQezXn6twU0APSnB4FsloN17ZSBfCUD ZWn8+sb/7y2PQrf0kAORLFGSvuly2hE00bGB3SmeaJCsGncq3x7Cv3Tcov8txVIgs1F5 AkPXFPX7SPoxixH3Gay2fyTmEeiMhvGl0asRQ7P9ZVqwFUQiASN8uawptuQNZ6ZkK6k9 ZVbj3R4DtGYV9JDrL9/+2Rlhc5y091TBZmO5h6nRpaJ0x6/ujVAIHk5hFv2AsAEEh1RU vsJjkdWfiA/R+NXNnNr1hIY5beRf8IY2kbZRuo/fleSrLqkRYWTRjAbslIWSjV4GBZGb XAdw== X-Gm-Message-State: AGi0PuZxt/7t7hUwY3mevnafhxe4+pyIpf/aCTtdv2GEAIPG5mS9C38y JOckz7w7J3hYZZYUskD3OPCrO10/ X-Google-Smtp-Source: APiQypKqNE0AuKnAI/K0HfS4YvIZsWbGsSTuNA2I8FZYPUc0ggBEBpBjEG1/orLjWK0n8m+zc4jogw== X-Received: by 2002:a7b:ce89:: with SMTP id q9mr11052500wmj.185.1588857465152; Thu, 07 May 2020 06:17:45 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id b82sm8447026wmh.1.2020.05.07.06.17.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 May 2020 06:17:44 -0700 (PDT) Message-Id: <0181a134bfb6986dc0e54ae624c478446a1324a9.1588857462.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 07 May 2020 13:17:34 +0000 Subject: [PATCH 02/10] sparse-checkout: move code from builtin Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmaill.com, peff@peff.net, me@ttaylorr.com, jrnieder@gmail.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee In anticipation of some new features in the sparse-checkout space, create a new sparse-checkout.c file at root to store the implementation of these features. Several helper methods from builtin/sparse-checkout.c are moved to this file. Signed-off-by: Derrick Stolee --- Makefile | 1 + builtin/sparse-checkout.c | 225 +----------------------------------- sparse-checkout.c | 233 ++++++++++++++++++++++++++++++++++++++ sparse-checkout.h | 16 +++ 4 files changed, 251 insertions(+), 224 deletions(-) create mode 100644 sparse-checkout.c create mode 100644 sparse-checkout.h diff --git a/Makefile b/Makefile index 9804a0758b2..2e8029b8026 100644 --- a/Makefile +++ b/Makefile @@ -985,6 +985,7 @@ LIB_OBJS += sha1-name.o LIB_OBJS += shallow.o LIB_OBJS += sideband.o LIB_OBJS += sigchain.o +LIB_OBJS += sparse-checkout.o LIB_OBJS += split-index.o LIB_OBJS += stable-qsort.o LIB_OBJS += strbuf.o diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c index 95d08824172..fd247e428e4 100644 --- a/builtin/sparse-checkout.c +++ b/builtin/sparse-checkout.c @@ -9,11 +9,11 @@ #include "string-list.h" #include "cache.h" #include "cache-tree.h" -#include "lockfile.h" #include "resolve-undo.h" #include "unpack-trees.h" #include "wt-status.h" #include "quote.h" +#include "sparse-checkout.h" static const char *empty_base = ""; @@ -22,30 +22,6 @@ static char const * const builtin_sparse_checkout_usage[] = { NULL }; -static char *get_sparse_checkout_filename(void) -{ - return git_pathdup("info/sparse-checkout"); -} - -static void write_patterns_to_file(FILE *fp, struct pattern_list *pl) -{ - int i; - - for (i = 0; i < pl->nr; i++) { - struct path_pattern *p = pl->patterns[i]; - - if (p->flags & PATTERN_FLAG_NEGATIVE) - fprintf(fp, "!"); - - fprintf(fp, "%s", p->pattern); - - if (p->flags & PATTERN_FLAG_MUSTBEDIR) - fprintf(fp, "/"); - - fprintf(fp, "\n"); - } -} - static int sparse_checkout_list(int argc, const char **argv) { struct pattern_list pl; @@ -92,153 +68,6 @@ static int sparse_checkout_list(int argc, const char **argv) return 0; } -static int update_working_directory(struct pattern_list *pl) -{ - enum update_sparsity_result result; - struct unpack_trees_options o; - struct lock_file lock_file = LOCK_INIT; - struct repository *r = the_repository; - - memset(&o, 0, sizeof(o)); - o.verbose_update = isatty(2); - o.update = 1; - o.head_idx = -1; - o.src_index = r->index; - o.dst_index = r->index; - o.skip_sparse_checkout = 0; - o.pl = pl; - - setup_work_tree(); - - repo_hold_locked_index(r, &lock_file, LOCK_DIE_ON_ERROR); - - setup_unpack_trees_porcelain(&o, "sparse-checkout"); - result = update_sparsity(&o); - clear_unpack_trees_porcelain(&o); - - if (result == UPDATE_SPARSITY_WARNINGS) - /* - * We don't do any special handling of warnings from untracked - * files in the way or dirty entries that can't be removed. - */ - result = UPDATE_SPARSITY_SUCCESS; - if (result == UPDATE_SPARSITY_SUCCESS) - write_locked_index(r->index, &lock_file, COMMIT_LOCK); - else - rollback_lock_file(&lock_file); - - return result; -} - -static char *escaped_pattern(char *pattern) -{ - char *p = pattern; - struct strbuf final = STRBUF_INIT; - - while (*p) { - if (is_glob_special(*p)) - strbuf_addch(&final, '\\'); - - strbuf_addch(&final, *p); - p++; - } - - return strbuf_detach(&final, NULL); -} - -static void write_cone_to_file(FILE *fp, struct pattern_list *pl) -{ - int i; - struct pattern_entry *pe; - struct hashmap_iter iter; - struct string_list sl = STRING_LIST_INIT_DUP; - struct strbuf parent_pattern = STRBUF_INIT; - - hashmap_for_each_entry(&pl->parent_hashmap, &iter, pe, ent) { - if (hashmap_get_entry(&pl->recursive_hashmap, pe, ent, NULL)) - continue; - - if (!hashmap_contains_parent(&pl->recursive_hashmap, - pe->pattern, - &parent_pattern)) - string_list_insert(&sl, pe->pattern); - } - - string_list_sort(&sl); - string_list_remove_duplicates(&sl, 0); - - fprintf(fp, "/*\n!/*/\n"); - - for (i = 0; i < sl.nr; i++) { - char *pattern = escaped_pattern(sl.items[i].string); - - if (strlen(pattern)) - fprintf(fp, "%s/\n!%s/*/\n", pattern, pattern); - free(pattern); - } - - string_list_clear(&sl, 0); - - hashmap_for_each_entry(&pl->recursive_hashmap, &iter, pe, ent) { - if (!hashmap_contains_parent(&pl->recursive_hashmap, - pe->pattern, - &parent_pattern)) - string_list_insert(&sl, pe->pattern); - } - - strbuf_release(&parent_pattern); - - string_list_sort(&sl); - string_list_remove_duplicates(&sl, 0); - - for (i = 0; i < sl.nr; i++) { - char *pattern = escaped_pattern(sl.items[i].string); - fprintf(fp, "%s/\n", pattern); - free(pattern); - } -} - -static int write_patterns_and_update(struct pattern_list *pl) -{ - char *sparse_filename; - FILE *fp; - int fd; - struct lock_file lk = LOCK_INIT; - int result; - - sparse_filename = get_sparse_checkout_filename(); - - if (safe_create_leading_directories(sparse_filename)) - die(_("failed to create directory for sparse-checkout file")); - - fd = hold_lock_file_for_update(&lk, sparse_filename, - LOCK_DIE_ON_ERROR); - - result = update_working_directory(pl); - if (result) { - rollback_lock_file(&lk); - free(sparse_filename); - clear_pattern_list(pl); - update_working_directory(NULL); - return result; - } - - fp = xfdopen(fd, "w"); - - if (core_sparse_checkout_cone) - write_cone_to_file(fp, pl); - else - write_patterns_to_file(fp, pl); - - fflush(fp); - commit_lock_file(&lk); - - free(sparse_filename); - clear_pattern_list(pl); - - return 0; -} - enum sparse_checkout_mode { MODE_NO_PATTERNS = 0, MODE_ALL_PATTERNS = 1, @@ -339,58 +168,6 @@ static int sparse_checkout_init(int argc, const char **argv) return write_patterns_and_update(&pl); } -static void insert_recursive_pattern(struct pattern_list *pl, struct strbuf *path) -{ - struct pattern_entry *e = xmalloc(sizeof(*e)); - e->patternlen = path->len; - e->pattern = strbuf_detach(path, NULL); - hashmap_entry_init(&e->ent, - ignore_case ? - strihash(e->pattern) : - strhash(e->pattern)); - - hashmap_add(&pl->recursive_hashmap, &e->ent); - - while (e->patternlen) { - char *slash = strrchr(e->pattern, '/'); - char *oldpattern = e->pattern; - size_t newlen; - - if (slash == e->pattern) - break; - - newlen = slash - e->pattern; - e = xmalloc(sizeof(struct pattern_entry)); - e->patternlen = newlen; - e->pattern = xstrndup(oldpattern, newlen); - hashmap_entry_init(&e->ent, - ignore_case ? - strihash(e->pattern) : - strhash(e->pattern)); - - if (!hashmap_get_entry(&pl->parent_hashmap, e, ent, NULL)) - hashmap_add(&pl->parent_hashmap, &e->ent); - } -} - -static void strbuf_to_cone_pattern(struct strbuf *line, struct pattern_list *pl) -{ - strbuf_trim(line); - - strbuf_trim_trailing_dir_sep(line); - - if (strbuf_normalize_path(line)) - die(_("could not normalize path %s"), line->buf); - - if (!line->len) - return; - - if (line->buf[0] != '/') - strbuf_insertstr(line, 0, "/"); - - insert_recursive_pattern(pl, line); -} - static char const * const builtin_sparse_checkout_set_usage[] = { N_("git sparse-checkout (set|add) (--stdin | )"), NULL diff --git a/sparse-checkout.c b/sparse-checkout.c new file mode 100644 index 00000000000..5ada307b42c --- /dev/null +++ b/sparse-checkout.c @@ -0,0 +1,233 @@ +#include "cache.h" +#include "config.h" +#include "dir.h" +#include "lockfile.h" +#include "quote.h" +#include "repository.h" +#include "sparse-checkout.h" +#include "strbuf.h" +#include "string-list.h" +#include "unpack-trees.h" + +char *get_sparse_checkout_filename(void) +{ + return git_pathdup("info/sparse-checkout"); +} + +void write_patterns_to_file(FILE *fp, struct pattern_list *pl) +{ + int i; + + for (i = 0; i < pl->nr; i++) { + struct path_pattern *p = pl->patterns[i]; + + if (p->flags & PATTERN_FLAG_NEGATIVE) + fprintf(fp, "!"); + + fprintf(fp, "%s", p->pattern); + + if (p->flags & PATTERN_FLAG_MUSTBEDIR) + fprintf(fp, "/"); + + fprintf(fp, "\n"); + } +} + +int update_working_directory(struct pattern_list *pl) +{ + enum update_sparsity_result result; + struct unpack_trees_options o; + struct lock_file lock_file = LOCK_INIT; + struct repository *r = the_repository; + + memset(&o, 0, sizeof(o)); + o.verbose_update = isatty(2); + o.update = 1; + o.head_idx = -1; + o.src_index = r->index; + o.dst_index = r->index; + o.skip_sparse_checkout = 0; + o.pl = pl; + + setup_work_tree(); + + repo_hold_locked_index(r, &lock_file, LOCK_DIE_ON_ERROR); + + setup_unpack_trees_porcelain(&o, "sparse-checkout"); + result = update_sparsity(&o); + clear_unpack_trees_porcelain(&o); + + if (result == UPDATE_SPARSITY_WARNINGS) + /* + * We don't do any special handling of warnings from untracked + * files in the way or dirty entries that can't be removed. + */ + result = UPDATE_SPARSITY_SUCCESS; + if (result == UPDATE_SPARSITY_SUCCESS) + write_locked_index(r->index, &lock_file, COMMIT_LOCK); + else + rollback_lock_file(&lock_file); + + return result; +} + +static char *escaped_pattern(char *pattern) +{ + char *p = pattern; + struct strbuf final = STRBUF_INIT; + + while (*p) { + if (is_glob_special(*p)) + strbuf_addch(&final, '\\'); + + strbuf_addch(&final, *p); + p++; + } + + return strbuf_detach(&final, NULL); +} + +static void write_cone_to_file(FILE *fp, struct pattern_list *pl) +{ + int i; + struct pattern_entry *pe; + struct hashmap_iter iter; + struct string_list sl = STRING_LIST_INIT_DUP; + struct strbuf parent_pattern = STRBUF_INIT; + + hashmap_for_each_entry(&pl->parent_hashmap, &iter, pe, ent) { + if (hashmap_get_entry(&pl->recursive_hashmap, pe, ent, NULL)) + continue; + + if (!hashmap_contains_parent(&pl->recursive_hashmap, + pe->pattern, + &parent_pattern)) + string_list_insert(&sl, pe->pattern); + } + + string_list_sort(&sl); + string_list_remove_duplicates(&sl, 0); + + fprintf(fp, "/*\n!/*/\n"); + + for (i = 0; i < sl.nr; i++) { + char *pattern = escaped_pattern(sl.items[i].string); + + if (strlen(pattern)) + fprintf(fp, "%s/\n!%s/*/\n", pattern, pattern); + free(pattern); + } + + string_list_clear(&sl, 0); + + hashmap_for_each_entry(&pl->recursive_hashmap, &iter, pe, ent) { + if (!hashmap_contains_parent(&pl->recursive_hashmap, + pe->pattern, + &parent_pattern)) + string_list_insert(&sl, pe->pattern); + } + + strbuf_release(&parent_pattern); + + string_list_sort(&sl); + string_list_remove_duplicates(&sl, 0); + + for (i = 0; i < sl.nr; i++) { + char *pattern = escaped_pattern(sl.items[i].string); + fprintf(fp, "%s/\n", pattern); + free(pattern); + } +} + +int write_patterns_and_update(struct pattern_list *pl) +{ + char *sparse_filename; + FILE *fp; + int fd; + struct lock_file lk = LOCK_INIT; + int result; + + sparse_filename = get_sparse_checkout_filename(); + + if (safe_create_leading_directories(sparse_filename)) + die(_("failed to create directory for sparse-checkout file")); + + fd = hold_lock_file_for_update(&lk, sparse_filename, + LOCK_DIE_ON_ERROR); + + result = update_working_directory(pl); + if (result) { + rollback_lock_file(&lk); + free(sparse_filename); + clear_pattern_list(pl); + update_working_directory(NULL); + return result; + } + + fp = xfdopen(fd, "w"); + + if (core_sparse_checkout_cone) + write_cone_to_file(fp, pl); + else + write_patterns_to_file(fp, pl); + + fflush(fp); + commit_lock_file(&lk); + + free(sparse_filename); + clear_pattern_list(pl); + + return 0; +} + +void insert_recursive_pattern(struct pattern_list *pl, struct strbuf *path) +{ + struct pattern_entry *e = xmalloc(sizeof(*e)); + e->patternlen = path->len; + e->pattern = strbuf_detach(path, NULL); + hashmap_entry_init(&e->ent, + ignore_case ? + strihash(e->pattern) : + strhash(e->pattern)); + + hashmap_add(&pl->recursive_hashmap, &e->ent); + + while (e->patternlen) { + char *slash = strrchr(e->pattern, '/'); + char *oldpattern = e->pattern; + size_t newlen; + + if (slash == e->pattern) + break; + + newlen = slash - e->pattern; + e = xmalloc(sizeof(struct pattern_entry)); + e->patternlen = newlen; + e->pattern = xstrndup(oldpattern, newlen); + hashmap_entry_init(&e->ent, + ignore_case ? + strihash(e->pattern) : + strhash(e->pattern)); + + if (!hashmap_get_entry(&pl->parent_hashmap, e, ent, NULL)) + hashmap_add(&pl->parent_hashmap, &e->ent); + } +} + +void strbuf_to_cone_pattern(struct strbuf *line, struct pattern_list *pl) +{ + strbuf_trim(line); + + strbuf_trim_trailing_dir_sep(line); + + if (strbuf_normalize_path(line)) + die(_("could not normalize path %s"), line->buf); + + if (!line->len) + return; + + if (line->buf[0] != '/') + strbuf_insertstr(line, 0, "/"); + + insert_recursive_pattern(pl, line); +} diff --git a/sparse-checkout.h b/sparse-checkout.h new file mode 100644 index 00000000000..4148832760d --- /dev/null +++ b/sparse-checkout.h @@ -0,0 +1,16 @@ +#ifndef SPARSE_CHECKOUT_H +#define SPARSE_CHECKOUT_H + +#include "cache.h" +#include "repository.h" + +struct pattern_list; + +char *get_sparse_checkout_filename(void); +void write_patterns_to_file(FILE *fp, struct pattern_list *pl); +int update_working_directory(struct pattern_list *pl); +int write_patterns_and_update(struct pattern_list *pl); +void insert_recursive_pattern(struct pattern_list *pl, struct strbuf *path); +void strbuf_to_cone_pattern(struct strbuf *line, struct pattern_list *pl); + +#endif From patchwork Thu May 7 13:17:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11533615 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1AAB081 for ; Thu, 7 May 2020 13:17:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0326120870 for ; Thu, 7 May 2020 13:17:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="dD40MJMn" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726884AbgEGNRv (ORCPT ); Thu, 7 May 2020 09:17:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37980 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-FAIL-OK-FAIL) by vger.kernel.org with ESMTP id S1725879AbgEGNRs (ORCPT ); Thu, 7 May 2020 09:17:48 -0400 Received: from mail-wr1-x442.google.com (mail-wr1-x442.google.com [IPv6:2a00:1450:4864:20::442]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 681AFC05BD09 for ; Thu, 7 May 2020 06:17:47 -0700 (PDT) Received: by mail-wr1-x442.google.com with SMTP id h9so6373419wrt.0 for ; Thu, 07 May 2020 06:17:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=pDfgA3QsHCBOCopOt25hTXa0JXEJs5+1SFn2nKXdcE0=; b=dD40MJMnfEbbRTTEWOs2cAVJ8ff5mHC2FAVq85/RATiSp4lxDR0YuJQWpsvQQSB+sW YAjvh8G8q1yKkbDuK2ikeBg7NLp0HP+ePkPF2XZ1PA97zL8fqRec2JVufUx5+h9nxy+U 8Xqu9TkzOUcILkQEiYcbVuB0TcZL4A5sgjrU4PztautR6u98rX+JWT8QsPae+c96Qd4X 4YOmhlc8GgHhyfECZM7sNZsC0AA7ENZeVrJi/gnznZHpoPJParMDGhJWbfEG/S18tP+r wwDwlzjOo+AY9UMt/u+4J9dgdAkxZde7mOaLwayOV8bzBxHUPJyZSxBd9QTjyhItdiwd aFCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=pDfgA3QsHCBOCopOt25hTXa0JXEJs5+1SFn2nKXdcE0=; b=RXUaDd5vi89f4QR2aTZGt3pgPAADklpZ2G621Brb6bCQPGF27BN/y1uC7HD6SPInX/ c2SWENXCq/B9WfF0m0oJRIJVGbK94vKpi1X2BMvqzYHKIQbdT0qrTNjuDzm0egujpuHg N3e0FBraZClU1F4dsc8xn9IjO+jB6xPkWfhJm/K+BfF9x/IrEKJXFcccflwtE4sUQYKX XXjCN/lQD/AHzsbl8Roi/hDoCiGx0JRoIFXXTvvxvbN/b6J9cVz6mHmYwxLSsnidRH5R WssSjQ+rdJ7/lPzDA8F3WEywx8O1jXdzmKC+bJBrPcYnkj1jvzmJrwdOHfEVOx2/3pLu YA7g== X-Gm-Message-State: AGi0Pua7fF19MVP9OYlua4bDcIrY42sxhyGkDCuSXPo6IoCIppDVGC7Q qRPj/RFuTFOH3RaaRTRrFfHVzlaJ X-Google-Smtp-Source: APiQypJu3BSHcXtvQe+HcCOVvmGpeGmgDQWaKaZQekhKQPg2jzjJezdBfw5DhI0YTN+imgHJuWjcew== X-Received: by 2002:adf:80a3:: with SMTP id 32mr15106951wrl.199.1588857466079; Thu, 07 May 2020 06:17:46 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id c16sm5144608wrv.62.2020.05.07.06.17.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 May 2020 06:17:45 -0700 (PDT) Message-Id: <444a6b5f894f28e96f713e5caccba18e1ea3b3eb.1588857462.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 07 May 2020 13:17:35 +0000 Subject: [PATCH 03/10] sparse-checkout: move code from unpack-trees.c Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmaill.com, peff@peff.net, me@ttaylorr.com, jrnieder@gmail.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee There is no need to have two ways to load the sparse-checkout patterns, especially duplicating the info/sparse-checkout filename. Move this code out of unpack-trees.c and into sparse-checkout.c. Signed-off-by: Derrick Stolee --- sparse-checkout.c | 12 ++++++++++++ sparse-checkout.h | 1 + unpack-trees.c | 7 ++----- 3 files changed, 15 insertions(+), 5 deletions(-) diff --git a/sparse-checkout.c b/sparse-checkout.c index 5ada307b42c..875b620568d 100644 --- a/sparse-checkout.c +++ b/sparse-checkout.c @@ -33,6 +33,18 @@ void write_patterns_to_file(FILE *fp, struct pattern_list *pl) } } +int populate_sparse_checkout_patterns(struct pattern_list *pl) +{ + int result; + char *sparse = get_sparse_checkout_filename(); + + pl->use_cone_patterns = core_sparse_checkout_cone; + result = add_patterns_from_file_to_list(sparse, "", 0, pl, NULL); + free(sparse); + + return result; +} + int update_working_directory(struct pattern_list *pl) { enum update_sparsity_result result; diff --git a/sparse-checkout.h b/sparse-checkout.h index 4148832760d..e0c840f07f9 100644 --- a/sparse-checkout.h +++ b/sparse-checkout.h @@ -7,6 +7,7 @@ struct pattern_list; char *get_sparse_checkout_filename(void); +int populate_sparse_checkout_patterns(struct pattern_list *pl); void write_patterns_to_file(FILE *fp, struct pattern_list *pl); int update_working_directory(struct pattern_list *pl); int write_patterns_and_update(struct pattern_list *pl); diff --git a/unpack-trees.c b/unpack-trees.c index 4f880f2da90..6f9bdeb57a0 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -16,6 +16,7 @@ #include "fsmonitor.h" #include "object-store.h" #include "promisor-remote.h" +#include "sparse-checkout.h" /* * Error messages expected by scripts out of plumbing commands such as @@ -1550,14 +1551,10 @@ static void mark_new_skip_worktree(struct pattern_list *pl, static void populate_from_existing_patterns(struct unpack_trees_options *o, struct pattern_list *pl) { - char *sparse = git_pathdup("info/sparse-checkout"); - - pl->use_cone_patterns = core_sparse_checkout_cone; - if (add_patterns_from_file_to_list(sparse, "", 0, pl, NULL) < 0) + if (populate_sparse_checkout_patterns(pl) < 0) o->skip_sparse_checkout = 1; else o->pl = pl; - free(sparse); } From patchwork Thu May 7 13:17:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11533611 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8757C1668 for ; Thu, 7 May 2020 13:17:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 64F8320870 for ; Thu, 7 May 2020 13:17:53 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="rCXQm030" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726891AbgEGNRw (ORCPT ); Thu, 7 May 2020 09:17:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37986 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-FAIL-OK-FAIL) by vger.kernel.org with ESMTP id S1725900AbgEGNRt (ORCPT ); Thu, 7 May 2020 09:17:49 -0400 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E3479C05BD43 for ; Thu, 7 May 2020 06:17:48 -0700 (PDT) Received: by mail-wm1-x32b.google.com with SMTP id 188so6481733wmc.2 for ; Thu, 07 May 2020 06:17:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=rWPjZP74fDB5dDeAKF6w5JQg0EJvFWS7OvDjbaqb0zc=; b=rCXQm030sDmSmJVzJ9Nj49dbCDRnixcbb7MzgdUQiV4vz1qhZZagdmhYa/2oQlzxvw t+doUlYVlvkiHt+49RHjG+JpGfUjiMXicoAj+mHcOFtssZJlFQQdf0IYW9uoo+x+ARvY uGLtGXinqPRY30fVanLb02qDy8+mATojUFv6WQuY+co3D1d7sH8M/GKVv5BUC/ScpmUH whgJzwGo8NSJrTtokPPayvq8w5x1wcfXFk4G1lNY1KAVfFVmvED+cquNdRZLMLNJSPCH yTXznB1GNeyAzbB6WSw1C+oJytvqgwKRaXT4yrrtAR8LCEggJUgYW9EgK1a39GX4fxT0 PxPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=rWPjZP74fDB5dDeAKF6w5JQg0EJvFWS7OvDjbaqb0zc=; b=LHC5rZGxS3ds6ORl1+8PIPFIbcDXX/NRL9z61bYS46VOL3SsalnhjqcVtRVRjSy4VE Exsz7m71sbxN4+4A8f4uiVc1z2WXN1XX5TjyU9L7OJwADrO8WrEmop8JdR6PrkStfCYk l0gWeX2cUhBQTZgzNz9pLSgPJ6j2QdGIINngEFlZSCU4bJ1bmtRyYOFfXcdj2UJ1vWvA xlsRhjErOm8uKOCyVWtM2TE0OBGI4uW0D192wJnTe9jz5kA0Dy63SFLNuapTMJImN/yE NSrXjyYSxAndpZ5qh/RwxloAXBx8isr34DpH7JagLuoKSmq/9Ab3qjoDQTUvNfKTKLE4 YUdQ== X-Gm-Message-State: AGi0PuYVu87370zze1ZEApMGYLtWtrZsrAbA4X2yqdqbGKBXoRpr7dKy N+MQW3n0caENxRBWrAE5g7dAyw1y X-Google-Smtp-Source: APiQypIT/pG+Y0xSmblHqz3dmoLnmB6jPrK4pkiT2Z5KP1hMDQzMA6SGYiVYpfsPQ/TqBTq+S3vTWg== X-Received: by 2002:a1c:e087:: with SMTP id x129mr10752564wmg.127.1588857466846; Thu, 07 May 2020 06:17:46 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id a187sm8098478wmh.40.2020.05.07.06.17.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 May 2020 06:17:46 -0700 (PDT) Message-Id: <2188577cd848d7cee77f06f1ad2b181864e5e36d.1588857462.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 07 May 2020 13:17:36 +0000 Subject: [PATCH 04/10] sparse-checkout: allow in-tree definitions Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmaill.com, peff@peff.net, me@ttaylorr.com, jrnieder@gmail.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee One of the difficulties of using the sparse-checkout feature is not knowing which directories are absolutely needed for working in a portion of the repository. Some of this can be documented in README files or included in a bootstrapping tool along with the repository. This is done in an ad-hoc way by every project that wants to use it. Let's make this process easier for users by creating a way to define a useful sparse-checkout definition inside the Git tree data. This has several benefits. In particular, the data is available to anyone who has a copy of the repository without needing a different data source. Second, the needs of the repository can change over time and Git can present a way to automatically update the working directory as these sparse-checkout definitions change over time. When sitting down to design this feature, there were several possible options. The first option is to literally include files in the repository that could replace the sparse-checkout file. This presents full generality for the repository, but the sparse-checkout patterns may look strange to a non-expert. While the general case could be useful to some, we are actively working to make "cone mode" the generally accepted way to work with sparse-checkout. In cone mode, the user selects a set of directories to include and all matches are directory-based prefix matches. This is much faster but also much easier to understand. Another option would be to interpret a file as a list of necessary directories. This would match the input to 'git sparse-checkout set' and output of 'git sparse-checkout list' when in cone mode. However, there are subtleties around custom data parsers that may not make this the safest approach. Both of the above options also suffer from a drawback that the file format is too simple to allow for extensions in the future. For example, in a repository with many dependent parts, changing a necessary directory in a core component would require updating every in-tree sparse-checkout definition. Since those files are likely very long, this presents a lot of noise for users to handle as they update these dependencies. Instead, let's create a format that can easily be extended to satisfy this need and any other needs we may want to add in the future. For this reason, I selected the config file format for these in-tree sparse-checkout definitions. To use this feature, add the "--in-tree" option when setting or adding directories to the sparse-checkout definition. For example: $ git sparse-checkout set --in-tree .sparse/base $ git sparse-checkout add --in-tree .sparse/extra These commands add values to the multi-valued config setting "sparse.inTree". When updating the sparse-checkout definition, these values describe paths in the repository to find the sparse-checkout data. After the commands listed earlier, we expect to see the following in .git/config.worktree: [sparse] intree = .sparse/base intree = .sparse/extra When applying the sparse-checkout definitions from this config, the blobs at HEAD:.sparse/base and HEAD:.sparse/extra are loaded. In those files, the multi-valued config values "sparse.dir" are considered as the directories to construct a cone mode sparse-checkout file. The end result is as if these paths were provided to "git sparse-checkout set" in cone mode. For example, suppose .sparse/base had the following content: [sparse] dir = A dir = B/C dir = D/E/F and .sparse/extra had the following content: [sparse] dir = D dir = X Then, the output of "git sparse-checkout list" would be A B/C D X Note that since "D" contains "D/E/F", that directory replaces the position of "D/E/F" in the list. Since these are parsed using the config library, the parser is robust enough to understand comments and complicated string values. The key benefit to this approach is that it can be extended by defining new config values. In a later change, we will introduce "sparse.inherit" to point to another file in the tree. This will solve the problem of editing many files when core dependencies change. Signed-off-by: Derrick Stolee --- Documentation/config.txt | 2 + Documentation/config/sparse.txt | 15 +++ Documentation/git-sparse-checkout.txt | 43 ++++++++- builtin/sparse-checkout.c | 43 +++++++++ sparse-checkout.c | 128 +++++++++++++++++++++++++- sparse-checkout.h | 11 +++ t/t1091-sparse-checkout-builtin.sh | 101 ++++++++++++++++++++ 7 files changed, 337 insertions(+), 6 deletions(-) create mode 100644 Documentation/config/sparse.txt diff --git a/Documentation/config.txt b/Documentation/config.txt index 08b13ba72be..40f44948229 100644 --- a/Documentation/config.txt +++ b/Documentation/config.txt @@ -435,6 +435,8 @@ include::config/sequencer.txt[] include::config/showbranch.txt[] +include::config/sparse.txt[] + include::config/splitindex.txt[] include::config/ssh.txt[] diff --git a/Documentation/config/sparse.txt b/Documentation/config/sparse.txt new file mode 100644 index 00000000000..c1fce87cd33 --- /dev/null +++ b/Documentation/config/sparse.txt @@ -0,0 +1,15 @@ +sparse.inTree:: + The `core.sparseCheckout` config option enables the `sparse-checkout` + feature, but if there are any values for the multi-valued + `sparse.inTree` config option, then the sparse-checkout patterns are + defined by parsing the files listed in these values. See + linkgit:git-sparse-checkout[1] for more information. + +sparse.dir:: + This config setting is ignored if present in the repository config. + Instead, this multi-valued option is present in the files listed by + `sparse.inTree` and specifies the directories needed in the + working directory. The union of all `sparse.dir` values across all + `sparse.inTree` files forms the input for `git sparse-checkout set` + in cone mode. See linkgit:git-sparse-checkout[1] for more + information. diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt index 1a3ace60820..da9322c5e41 100644 --- a/Documentation/git-sparse-checkout.txt +++ b/Documentation/git-sparse-checkout.txt @@ -62,13 +62,20 @@ directories (recursively) as well as files that are siblings of ancestor directories. The input format matches the output of `git ls-tree --name-only`. This includes interpreting pathnames that begin with a double quote (") as C-style quoted strings. ++ +When the `--in-tree` option is provided, the paths provided are interpreted +as files within the working directory that are used to construct the +`sparse-checkout` patterns. See 'IN-TREE PATTERN SET' below. 'add':: Update the sparse-checkout file to include additional patterns. By default, these patterns are read from the command-line arguments, but they can be read from stdin using the `--stdin` option. When `core.sparseCheckoutCone` is enabled, the given patterns are interpreted - as directory names as in the 'set' subcommand. + as directory names as in the 'set' subcommand. When the `--in-tree` + option is provided, the input is interpreted as locations of files + describing a sparse-checkout definition as in the 'set' subcommand + and the 'IN-TREE PATTERN SET' section below. 'reapply:: Reapply the sparsity pattern rules to paths in the working tree. @@ -197,6 +204,40 @@ case-insensitive check. This corrects for case mismatched filenames in the directory. +IN-TREE PATTERN SET +------------------- + +As your project changes, your sparse-checkout pattern sets may also change. +It is important to be able to construct a valid sparse-checkout pattern set +when switching between points in history. The in-tree pattern sets allow +versioning cone-mode sparse-checkout patterns next to your other artifacts. + +To enable the feature, create a sparse-checkout definition using the Git +config format. The file should specify the multi-valued config variable +`sparse.dir` to a list of directories to include in the sparse-checkout +definition. If multiple files are specified, the resulting sparse-checkout +definition is the union of all directories from all such files. For +example, the following file contains a list of three directories, `A`, +`B/C`, and `D/E/F`: + +---------------------------------- +[sparse] + dir = A + dir = B/C +# Comments are allowed to describe +# why a directory is necessary + dir = D/E/F +---------------------------------- + +Use `git sparse-checkout set --in-tree ` to initialize the patterns +to those included in the file at ``. This will override any existing +patterns you have in your sparse-checkout file. + +After switching between commits with different versions of this file, run +`git sparse-checkout reapply` to adjust the sparse-checkout patterns to +the new definition. + + SUBMODULES ---------- diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c index fd247e428e4..621f1801c03 100644 --- a/builtin/sparse-checkout.c +++ b/builtin/sparse-checkout.c @@ -175,6 +175,7 @@ static char const * const builtin_sparse_checkout_set_usage[] = { static struct sparse_checkout_set_opts { int use_stdin; + int in_tree; } set_opts; static void add_patterns_from_input(struct pattern_list *pl, @@ -312,12 +313,52 @@ static int modify_pattern_list(int argc, const char **argv, enum modify_type m) return result; } +static int modify_in_tree_list(int argc, const char **argv, enum modify_type m) +{ + int result = 0; + int i; + struct string_list sl = STRING_LIST_INIT_DUP; + struct pattern_list pl; + + memset(&pl, 0, sizeof(pl)); + + switch(m) { + case ADD: + if (load_in_tree_list_from_config(the_repository, &sl)) + return 1; + if (!sl.nr) + warning(_("the existing in-tree config has no entries; this overwrites the existing sparse-checkout definition.")); + populate_sparse_checkout_patterns(&pl); + break; + + case REPLACE: + hashmap_init(&pl.recursive_hashmap, pl_hashmap_cmp, NULL, 0); + hashmap_init(&pl.parent_hashmap, pl_hashmap_cmp, NULL, 0); + break; + } + + for (i = 0; i < argc; i++) + string_list_insert(&sl, argv[i]); + + if (load_in_tree_pattern_list(the_repository, &sl, &pl) || + set_sparse_in_tree_config(the_repository, &sl) || + write_patterns_and_update(&pl)) + result = 1; + + string_list_clear(&sl, 0); + clear_pattern_list(&pl); + + return result; +} + static int sparse_checkout_set(int argc, const char **argv, const char *prefix, enum modify_type m) { static struct option builtin_sparse_checkout_set_options[] = { OPT_BOOL(0, "stdin", &set_opts.use_stdin, N_("read patterns from standard in")), + OPT_BOOL(0, "in-tree", &set_opts.in_tree, + N_("define the sparse-checkout from files in the tree")), OPT_END(), }; @@ -328,6 +369,8 @@ static int sparse_checkout_set(int argc, const char **argv, const char *prefix, builtin_sparse_checkout_set_usage, PARSE_OPT_KEEP_UNKNOWN); + if (set_opts.in_tree) + return modify_in_tree_list(argc, argv, m); return modify_pattern_list(argc, argv, m); } diff --git a/sparse-checkout.c b/sparse-checkout.c index 875b620568d..d6c27ca19c4 100644 --- a/sparse-checkout.c +++ b/sparse-checkout.c @@ -8,6 +8,7 @@ #include "strbuf.h" #include "string-list.h" #include "unpack-trees.h" +#include "object-store.h" char *get_sparse_checkout_filename(void) { @@ -33,14 +34,113 @@ void write_patterns_to_file(FILE *fp, struct pattern_list *pl) } } +int load_in_tree_list_from_config(struct repository *r, + struct string_list *sl) +{ + struct string_list_item *item; + const struct string_list *cl; + + cl = repo_config_get_value_multi(r, SPARSE_CHECKOUT_IN_TREE); + + if (!cl) + return 1; + + for_each_string_list_item(item, cl) + string_list_insert(sl, item->string); + + return 0; +} + +static int sparse_dir_cb(const char *var, const char *value, void *data) +{ + struct strbuf path = STRBUF_INIT; + struct pattern_list *pl = (struct pattern_list *)data; + + if (!strcmp(var, SPARSE_CHECKOUT_DIR)) { + strbuf_addstr(&path, value); + strbuf_to_cone_pattern(&path, pl); + strbuf_release(&path); + } + + return 0; +} + +static int load_in_tree_from_blob(struct pattern_list *pl, + struct object_id *oid) +{ + return git_config_from_blob_oid(sparse_dir_cb, + SPARSE_CHECKOUT_DIR, + oid, pl); +} + +int load_in_tree_pattern_list(struct repository *r, + struct string_list *sl, + struct pattern_list *pl) +{ + struct index_state *istate = r->index; + struct string_list_item *item; + struct strbuf path = STRBUF_INIT; + + pl->use_cone_patterns = 1; + + for_each_string_list_item(item, sl) { + struct object_id *oid; + enum object_type type; + int pos = index_name_pos(istate, item->string, strlen(item->string)); + + /* + * Exit silently, as this is likely the case where Git + * changed branches to a location where the inherit file + * does not exist. Do not update the sparse-checkout. + */ + if (pos < 0) + return 1; + + oid = &istate->cache[pos]->oid; + type = oid_object_info(r, oid, NULL); + + if (type != OBJ_BLOB) { + warning(_("expected a file at '%s'; not updating sparse-checkout"), + oid_to_hex(oid)); + return 1; + } + + load_in_tree_from_blob(pl, oid); + } + + strbuf_release(&path); + + return 0; +} + int populate_sparse_checkout_patterns(struct pattern_list *pl) { int result; - char *sparse = get_sparse_checkout_filename(); - - pl->use_cone_patterns = core_sparse_checkout_cone; - result = add_patterns_from_file_to_list(sparse, "", 0, pl, NULL); - free(sparse); + const char *in_tree; + + if (!git_config_get_value(SPARSE_CHECKOUT_IN_TREE, &in_tree) && + in_tree) { + struct string_list paths = STRING_LIST_INIT_DUP; + /* If we do not have this config, skip this step! */ + if (load_in_tree_list_from_config(the_repository, &paths) || + !paths.nr) + return 1; + + /* Check diff for paths over from/to. If any changed, reload. */ + /* or for now, reload always! */ + hashmap_init(&pl->recursive_hashmap, pl_hashmap_cmp, NULL, 0); + hashmap_init(&pl->parent_hashmap, pl_hashmap_cmp, NULL, 0); + pl->use_cone_patterns = 1; + + result = load_in_tree_pattern_list(the_repository, &paths, pl); + string_list_clear(&paths, 0); + } else { + char *sparse = get_sparse_checkout_filename(); + + pl->use_cone_patterns = core_sparse_checkout_cone; + result = add_patterns_from_file_to_list(sparse, "", 0, pl, NULL); + free(sparse); + } return result; } @@ -243,3 +343,21 @@ void strbuf_to_cone_pattern(struct strbuf *line, struct pattern_list *pl) insert_recursive_pattern(pl, line); } + +int set_sparse_in_tree_config(struct repository *r, struct string_list *sl) +{ + struct string_list_item *item; + const char *config_path = git_path("config.worktree"); + + /* clear existing values */ + git_config_set_multivar_in_file_gently(config_path, + SPARSE_CHECKOUT_IN_TREE, + NULL, NULL, 1); + + for_each_string_list_item(item, sl) + git_config_set_multivar_in_file_gently( + config_path, SPARSE_CHECKOUT_IN_TREE, + item->string, CONFIG_REGEX_NONE, 0); + + return 0; +} diff --git a/sparse-checkout.h b/sparse-checkout.h index e0c840f07f9..993a5701a60 100644 --- a/sparse-checkout.h +++ b/sparse-checkout.h @@ -4,14 +4,25 @@ #include "cache.h" #include "repository.h" +#define SPARSE_CHECKOUT_DIR "sparse.dir" +#define SPARSE_CHECKOUT_IN_TREE "sparse.intree" + struct pattern_list; char *get_sparse_checkout_filename(void); int populate_sparse_checkout_patterns(struct pattern_list *pl); void write_patterns_to_file(FILE *fp, struct pattern_list *pl); int update_working_directory(struct pattern_list *pl); +int write_patterns(struct pattern_list *pl, int and_update); int write_patterns_and_update(struct pattern_list *pl); void insert_recursive_pattern(struct pattern_list *pl, struct strbuf *path); void strbuf_to_cone_pattern(struct strbuf *line, struct pattern_list *pl); +int load_in_tree_list_from_config(struct repository *r, + struct string_list *sl); +int load_in_tree_pattern_list(struct repository *r, + struct string_list *sl, + struct pattern_list *pl); +int set_sparse_in_tree_config(struct repository *r, struct string_list *sl); + #endif diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh index 88cdde255cd..1040bf9c261 100755 --- a/t/t1091-sparse-checkout-builtin.sh +++ b/t/t1091-sparse-checkout-builtin.sh @@ -604,4 +604,105 @@ test_expect_success MINGW 'cone mode replaces backslashes with slashes' ' check_files repo/deep a deeper1 ' +test_expect_success 'basis of --in-tree' ' + git -C repo config auto.crlf false && + cat >folder1 <<-\EOF && + [sparse] + dir = folder1 + EOF + cat >folder2 <<-\EOF && + [sparse] + dir = folder2 + EOF + cat >deep <<-\EOF && + [sparse] + dir = deep + EOF + cat >deeper1 <<-\EOF && + [sparse] + dir = deep/deeper1 + EOF + cat >sparse <<-\EOF && + [sparse] + dir = .sparse + EOF + mkdir repo/.sparse && + for file in folder1 folder2 deep deeper1 sparse + do + cp $file repo/.sparse/ || return 1 + done && + git -C repo add .sparse && + git -C repo commit -m "Add sparse specifications" && + + git -C repo sparse-checkout set --in-tree .sparse/folder1 && + check_files repo a folder1 && + git -C repo config --get-all sparse.inTree >actual-config && + echo .sparse/folder1 >expect-config && + test_cmp expect-config actual-config && + check_files repo a folder1 && + + git -C repo sparse-checkout set --in-tree .sparse/folder2 && + git -C repo config --get-all sparse.inTree >actual-config && + echo .sparse/folder2 >expect-config && + test_cmp expect-config actual-config && + check_files repo a folder2 && + + git -C repo sparse-checkout set --in-tree .sparse/deeper1 && + git -C repo config --get-all sparse.inTree >actual-config && + echo .sparse/deeper1 >expect-config && + test_cmp expect-config actual-config && + check_files repo a deep && + check_files repo/deep a deeper1 && + + git -C repo sparse-checkout set --in-tree .sparse/deeper1 .sparse/deep .sparse/folder1 && + check_files repo a deep folder1 && + check_files repo/deep a deeper1 deeper2 && + cat >expect-list <<-EOF && + deep + folder1 + EOF + git -C repo sparse-checkout list >actual-list && + test_cmp expect-list actual-list && + + git -C repo sparse-checkout set --in-tree .sparse/folder1 .sparse/deeper1 && + git -C repo config --get-all sparse.inTree >actual-config && + cat >expect-config <<-\EOF && + .sparse/deeper1 + .sparse/folder1 + EOF + test_cmp expect-config actual-config && + check_files repo a deep folder1 +' + +test_expect_success '"add" with --in-tree' ' + git -C repo sparse-checkout set --in-tree .sparse/folder1 && + git -C repo config --get-all sparse.inTree >actual-config && + echo .sparse/folder1 >expect-config && + test_cmp expect-config actual-config && + check_files repo a folder1 && + git -C repo sparse-checkout add --in-tree .sparse/deeper1 && + git -C repo config --get-all sparse.inTree >actual-config && + cat >expect-config <<-\EOF && + .sparse/deeper1 + .sparse/folder1 + EOF + test_cmp expect-config actual-config && + check_files repo a deep folder1 +' + +test_expect_success 'reapply after updating in-tree file' ' + git -C repo sparse-checkout set --in-tree .sparse/sparse && + check_files repo a && + test_path_is_dir repo/.sparse && + echo "\tdir = folder1" >>repo/.sparse/sparse && + git -C repo commit -a -m "Update sparse file" && + git -C repo sparse-checkout reapply && + check_files repo a folder1 && + test_path_is_dir repo/.sparse && + git -C repo checkout HEAD~1 && + git -C repo sparse-checkout reapply && + check_files repo a && + test_path_is_dir repo/.sparse +' + test_done From patchwork Thu May 7 13:17:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11533627 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BE7001668 for ; Thu, 7 May 2020 13:18:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A0B0E20838 for ; Thu, 7 May 2020 13:18:04 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="EiV7RrR0" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726948AbgEGNSC (ORCPT ); Thu, 7 May 2020 09:18:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37988 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-FAIL-OK-FAIL) by vger.kernel.org with ESMTP id S1726825AbgEGNRt (ORCPT ); Thu, 7 May 2020 09:17:49 -0400 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6F1E3C05BD09 for ; Thu, 7 May 2020 06:17:49 -0700 (PDT) Received: by mail-wm1-x32d.google.com with SMTP id u16so6768658wmc.5 for ; Thu, 07 May 2020 06:17:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=XniF/noAQTD6rKie5zR2SyZD9WSAcjZrMTxOpDFZj/0=; b=EiV7RrR0edv7UKc5yTB2O8sICPzMp1ntgoM9HY3Ssx684weMzqb5hAUi+HvSI5DKbw LKIocFy8yfoIFivgavke8u/IUAyEh38LEHjIonCumHFbiU7gD+YTFlscJrj3+WzY6rxg kvtkTPxWMhh/6ND5QOqvNSdxvtrJDRU5Hi+3zuCsZ6PmGnl6LkGMQR51PkyteSU+KwuR VMiP+EHSMxcL88PMJBUjDqx34JL+CAFsIJpWF6vYwXTZzrQSVHEIwzDOEZEf0bNOxFv/ 764tYf38A/pwin7XTZLg5dSVlCG9LbXzYNecdjKo+zbCHS9nLeu5UeYS37iJsyJlfJd0 iZKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=XniF/noAQTD6rKie5zR2SyZD9WSAcjZrMTxOpDFZj/0=; b=Kp1URwgv6y2ocPZ5pmKWkXs0xMfY94Y+vOIC+Ith4wjlIYBZKsZrXZYhSGVsVx5bpn PGl+pw1zy4v2FDFPEgrjKrupSsZndHAXG6gl2TG8dD04kj17trYEKmxOxnbyGmSPK6yD 8Ca1zjEkYkvK/otwe4F6+gL12BZ9yp9yC89F677DxPyFDEFZKBLR0X31lKS3b/rUA/K9 qsgHZqbwD5ZwBjPrmxKZjh7CKPZoN5/pt0RlUxQniGFQkJJEHztFTqC+5vA2Zip2i/qu Q5R6DHkOJW1g3vxrrBUV8IE17yRMQm7w74lZPZBO1w+iIEwQ6cN3cI5SO58NF2bc8Hx9 cK1w== X-Gm-Message-State: AGi0PuYWXl5Kaf/ih6/kH4Q0DsQ8+IR0awvYmKERysBPHZYcSA8yUh63 zaUIyJPnMUiFuov1oy/FGfIZVN6O X-Google-Smtp-Source: APiQypISGYcIq+c9DAWFvDnf0NAGFeLb12xRn1OX6Dr4YBGoTFsVYVdjvxGqzU8ycjWRCV4IUuSifg== X-Received: by 2002:a7b:c181:: with SMTP id y1mr11024115wmi.83.1588857467721; Thu, 07 May 2020 06:17:47 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id v7sm15067793wmg.3.2020.05.07.06.17.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 May 2020 06:17:47 -0700 (PDT) Message-Id: <9a2cb7bb5ed7a5dc039d4b47bfee83c589252a45.1588857462.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 07 May 2020 13:17:37 +0000 Subject: [PATCH 05/10] sparse-checkout: automatically update in-tree definition Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmaill.com, peff@peff.net, me@ttaylorr.com, jrnieder@gmail.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee A benefit of having sparse-checkouts defined in the tree data is that the sparse-checkout definition can change over time. To take advantage of that data, a user could run "git sparse-checkout reapply" whenever they thought the sparse-checkout definition was out of date. That is likely too manual of a process, since a user would probably only discover this after a failure to accomplish their goal, such as a build failure. Prevent user frustration by automatically updating the sparse-checkout definition after changing the index when an in-tree definition is provided by config. This will happen during every index change, including every step of a rebase, including conflict states. Special care was needed around the --no-sparse-checkout option in "git read-tree". This previously relied on changing the skip_sparse_checkout option in "struct unpack_trees_options" to prevent applying the skip-worktree bits. However, now that we make a second update to the index focusing on the skip-worktree bits, this needs to be prevented in another way. The simplest thing to do was disable the feature through the core_apply_sparse_checkout global variable. Signed-off-by: Derrick Stolee --- Documentation/git-sparse-checkout.txt | 7 ++++--- builtin/commit.c | 4 +++- builtin/read-tree.c | 4 ++++ read-cache.c | 8 +++++--- sparse-checkout.c | 26 +++++++++++++++++++++++++- sparse-checkout.h | 1 + t/t1091-sparse-checkout-builtin.sh | 19 +++++++++++++++---- 7 files changed, 57 insertions(+), 12 deletions(-) diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt index da9322c5e41..c1713ebb1d2 100644 --- a/Documentation/git-sparse-checkout.txt +++ b/Documentation/git-sparse-checkout.txt @@ -233,9 +233,10 @@ Use `git sparse-checkout set --in-tree ` to initialize the patterns to those included in the file at ``. This will override any existing patterns you have in your sparse-checkout file. -After switching between commits with different versions of this file, run -`git sparse-checkout reapply` to adjust the sparse-checkout patterns to -the new definition. +As Git switches between commits, it will update the in-tree sparse-checkout +definition according to the files available at the new commit. If any of +the specified files do not exist at the new commit, then the sparse-checkout +definition will not change. SUBMODULES diff --git a/builtin/commit.c b/builtin/commit.c index 7ba33a3bec4..0eab8d74469 100644 --- a/builtin/commit.c +++ b/builtin/commit.c @@ -36,6 +36,7 @@ #include "help.h" #include "commit-reach.h" #include "commit-graph.h" +#include "sparse-checkout.h" static const char * const builtin_commit_usage[] = { N_("git commit [] [--] ..."), @@ -222,7 +223,8 @@ static int commit_index_files(void) case COMMIT_AS_IS: break; /* nothing to do */ case COMMIT_NORMAL: - err = commit_lock_file(&index_lock); + err = commit_lock_file(&index_lock) || + update_in_tree_sparse_checkout(); break; case COMMIT_PARTIAL: err = commit_lock_file(&index_lock); diff --git a/builtin/read-tree.c b/builtin/read-tree.c index af7424b94c8..9ae81ffffa1 100644 --- a/builtin/read-tree.c +++ b/builtin/read-tree.c @@ -247,6 +247,10 @@ int cmd_read_tree(int argc, const char **argv, const char *cmd_prefix) parse_tree(tree); init_tree_desc(t+i, tree->buffer, tree->size); } + + if (opts.skip_sparse_checkout) + core_apply_sparse_checkout = 0; + if (unpack_trees(nr_trees, t, &opts)) return 128; diff --git a/read-cache.c b/read-cache.c index aa427c5c170..150e73feb0d 100644 --- a/read-cache.c +++ b/read-cache.c @@ -25,6 +25,7 @@ #include "fsmonitor.h" #include "thread-utils.h" #include "progress.h" +#include "sparse-checkout.h" /* Mask for the name length in ce_flags in the on-disk index */ @@ -3074,9 +3075,10 @@ static int do_write_locked_index(struct index_state *istate, struct lock_file *l if (ret) return ret; - if (flags & COMMIT_LOCK) - ret = commit_locked_index(lock); - else + if (flags & COMMIT_LOCK) { + ret = commit_locked_index(lock) || + update_in_tree_sparse_checkout(); + } else ret = close_lock_file_gently(lock); run_hook_le(NULL, "post-index-change", diff --git a/sparse-checkout.c b/sparse-checkout.c index d6c27ca19c4..6c58fda9722 100644 --- a/sparse-checkout.c +++ b/sparse-checkout.c @@ -92,9 +92,12 @@ int load_in_tree_pattern_list(struct repository *r, * Exit silently, as this is likely the case where Git * changed branches to a location where the inherit file * does not exist. Do not update the sparse-checkout. + * + * Use -1 return to ensure populate_from_existing_patterns() + * skips the sparse-checkout updates. */ if (pos < 0) - return 1; + return -1; oid = &istate->cache[pos]->oid; type = oid_object_info(r, oid, NULL); @@ -145,6 +148,7 @@ int populate_sparse_checkout_patterns(struct pattern_list *pl) return result; } +static int updating_sparse_checkout = 0; int update_working_directory(struct pattern_list *pl) { enum update_sparsity_result result; @@ -152,6 +156,10 @@ int update_working_directory(struct pattern_list *pl) struct lock_file lock_file = LOCK_INIT; struct repository *r = the_repository; + if (updating_sparse_checkout) + return 0; + updating_sparse_checkout = 1; + memset(&o, 0, sizeof(o)); o.verbose_update = isatty(2); o.update = 1; @@ -180,9 +188,24 @@ int update_working_directory(struct pattern_list *pl) else rollback_lock_file(&lock_file); + updating_sparse_checkout = 0; return result; } +int update_in_tree_sparse_checkout(void) +{ + const char *first_value; + + if (!core_apply_sparse_checkout) + return 0; + + /* only update if doing so due to sparse.inTree. */ + if (!git_config_get_value(SPARSE_CHECKOUT_IN_TREE, &first_value) && + first_value) + return update_working_directory(NULL); + return 0; +} + static char *escaped_pattern(char *pattern) { char *p = pattern; @@ -273,6 +296,7 @@ int write_patterns_and_update(struct pattern_list *pl) free(sparse_filename); clear_pattern_list(pl); update_working_directory(NULL); + updating_sparse_checkout = 0; return result; } diff --git a/sparse-checkout.h b/sparse-checkout.h index 993a5701a60..fb0ba48524a 100644 --- a/sparse-checkout.h +++ b/sparse-checkout.h @@ -13,6 +13,7 @@ char *get_sparse_checkout_filename(void); int populate_sparse_checkout_patterns(struct pattern_list *pl); void write_patterns_to_file(FILE *fp, struct pattern_list *pl); int update_working_directory(struct pattern_list *pl); +int update_in_tree_sparse_checkout(void); int write_patterns(struct pattern_list *pl, int and_update); int write_patterns_and_update(struct pattern_list *pl); void insert_recursive_pattern(struct pattern_list *pl, struct strbuf *path); diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh index 1040bf9c261..fdaafba5377 100755 --- a/t/t1091-sparse-checkout-builtin.sh +++ b/t/t1091-sparse-checkout-builtin.sh @@ -605,6 +605,7 @@ test_expect_success MINGW 'cone mode replaces backslashes with slashes' ' ' test_expect_success 'basis of --in-tree' ' + git -C repo branch no-in-tree && git -C repo config auto.crlf false && cat >folder1 <<-\EOF && [sparse] @@ -690,18 +691,28 @@ test_expect_success '"add" with --in-tree' ' check_files repo a deep folder1 ' -test_expect_success 'reapply after updating in-tree file' ' +test_expect_success 'automatically change after updating in-tree file' ' git -C repo sparse-checkout set --in-tree .sparse/sparse && check_files repo a && test_path_is_dir repo/.sparse && - echo "\tdir = folder1" >>repo/.sparse/sparse && + printf "\tdir = folder1\n" >>repo/.sparse/sparse && git -C repo commit -a -m "Update sparse file" && - git -C repo sparse-checkout reapply && check_files repo a folder1 && test_path_is_dir repo/.sparse && git -C repo checkout HEAD~1 && - git -C repo sparse-checkout reapply && check_files repo a && + test_path_is_dir repo/.sparse && + git -C repo checkout - && + check_files repo a folder1 && + test_path_is_dir repo/.sparse +' + +test_expect_success 'keep definition when in-tree file is missing' ' + git -C repo checkout no-in-tree && + check_files repo a folder1 && + test_path_is_missing repo/.sparse && + git -C repo checkout - && + check_files repo a folder1 && test_path_is_dir repo/.sparse ' From patchwork Thu May 7 13:17:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11533625 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A5EB31668 for ; Thu, 7 May 2020 13:18:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8CC5720838 for ; Thu, 7 May 2020 13:18:02 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="SjQf+YYH" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726942AbgEGNSA (ORCPT ); Thu, 7 May 2020 09:18:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37990 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-FAIL-OK-FAIL) by vger.kernel.org with ESMTP id S1726843AbgEGNRu (ORCPT ); Thu, 7 May 2020 09:17:50 -0400 Received: from mail-wr1-x444.google.com (mail-wr1-x444.google.com [IPv6:2a00:1450:4864:20::444]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C464BC05BD0A for ; Thu, 7 May 2020 06:17:49 -0700 (PDT) Received: by mail-wr1-x444.google.com with SMTP id z8so6375080wrw.3 for ; Thu, 07 May 2020 06:17:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=IPf8rjZxrrHEPWul7GBKVWG0iLlgOhqVnBivx2n2ugU=; b=SjQf+YYHhbpdslSPBZnS3/lZ4Be5pE/+q1+g01w9VOYQVCKsv4LbEL/UD6ukRyqHFa 58wmlzGF7YSKSRNAOHvGOTK+KPpwyGmW19J7xDjKuz+nPvJqHB6AHILMHX+ewK9CAiID a1e8m/L5rOfWAmSiHgYZRe4ExZcaHSSci4LZ6edd02bJTamfbqxQkbck5CmISHGnkSu2 VrlvvYjul8pHYRSi31FUipRdnXSU6/AYTCfQxOR+X4p36chqeC6j/FjnBN+v+eLbtEnf dOjiFpL2Jn7JwD5VHwvB3STthE1VoV68iqSpvtym4Kl5cdVvP9n9y+bl+79vvo2/8Xmg 0pCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=IPf8rjZxrrHEPWul7GBKVWG0iLlgOhqVnBivx2n2ugU=; b=oMWbe2hsibSgBAJTDuA2h3J0bfK2QKVSGKDjiI8Y0BCED0azQUo7n3smW1JaUEchtl rI2MkW3i78JxFcUZ7fOC8JOpFDXmttnsFTNQV4S8AV1E+aiYy0s2POsG2xXjfZ3E9V7a YxYxI8fnVCKGWLpCyuXztil9uDzxQRGWVGqhpN6Ld8hKr4kriskbsq9oWrdtLMtjAb50 OUvqr8/BCWvk8OmPczfYSwI4GEaTACent3xR4c3j6WvIlPY8H6EZbks9hCi3YsMLG8gr yg1goGyRVoH+uUioVMK/QF/0N0YNwtwbBuTBW+0DiPvVnJkfWmf8GnQsRSwcH+950ugz qP8Q== X-Gm-Message-State: AGi0Pua7ongMUCP1mH6O43M9L820UU98ouifbxUQuE4usmw7IK0Q6E1m LJLkgFL4Ww1ZSLMbEwTeW+oSGg3f X-Google-Smtp-Source: APiQypIVcFqtsARLiBBbHu0VVMsL/7jjo1s/zm55D/wO1DPhRbDQwBDmFeabg7p6zDjHqrZldOHaPA== X-Received: by 2002:a5d:4704:: with SMTP id y4mr16172777wrq.96.1588857468362; Thu, 07 May 2020 06:17:48 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id p6sm7825533wrt.3.2020.05.07.06.17.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 May 2020 06:17:47 -0700 (PDT) Message-Id: In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 07 May 2020 13:17:38 +0000 Subject: [PATCH 06/10] sparse-checkout: use oidset to prevent repeat blobs Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmaill.com, peff@peff.net, me@ttaylorr.com, jrnieder@gmail.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee As we parse the in-tree config files that store the sparse.dir values used to create an in-tree sparse-checkout definition, we can easily avoid parsing the same file multiple times by using an oidset on those blobs. We only parse if the oid is new to the oidset. This is unlikely to have a major performance benefit right now, but will be extremely important when we introduce the sparse.inherit options to link multiple files in a directed graph. This oidset will prevent infinite loops when cycles exist in that digraph, or exponential blowups even in the case of a directed acyclic graph. Signed-off-by: Derrick Stolee --- sparse-checkout.c | 24 +++++++++++++++++++----- 1 file changed, 19 insertions(+), 5 deletions(-) diff --git a/sparse-checkout.c b/sparse-checkout.c index 6c58fda9722..d01f4d7b525 100644 --- a/sparse-checkout.c +++ b/sparse-checkout.c @@ -9,6 +9,7 @@ #include "string-list.h" #include "unpack-trees.h" #include "object-store.h" +#include "oidset.h" char *get_sparse_checkout_filename(void) { @@ -77,9 +78,12 @@ int load_in_tree_pattern_list(struct repository *r, struct string_list *sl, struct pattern_list *pl) { + int result = 0; struct index_state *istate = r->index; struct string_list_item *item; struct strbuf path = STRBUF_INIT; + struct oidset set; + oidset_init(&set, 16); pl->use_cone_patterns = 1; @@ -96,24 +100,34 @@ int load_in_tree_pattern_list(struct repository *r, * Use -1 return to ensure populate_from_existing_patterns() * skips the sparse-checkout updates. */ - if (pos < 0) - return -1; + if (pos < 0) { + result = -1; + goto cleanup; + } oid = &istate->cache[pos]->oid; + + if (oidset_contains(&set, oid)) + continue; + + oidset_insert(&set, oid); + type = oid_object_info(r, oid, NULL); if (type != OBJ_BLOB) { warning(_("expected a file at '%s'; not updating sparse-checkout"), oid_to_hex(oid)); - return 1; + result = 1; + goto cleanup; } load_in_tree_from_blob(pl, oid); } +cleanup: strbuf_release(&path); - - return 0; + oidset_clear(&set); + return result; } int populate_sparse_checkout_patterns(struct pattern_list *pl) From patchwork Thu May 7 13:17:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11533613 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 246D717EF for ; Thu, 7 May 2020 13:17:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 06DE520870 for ; Thu, 7 May 2020 13:17:54 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="JmOl7qbv" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726900AbgEGNRx (ORCPT ); Thu, 7 May 2020 09:17:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37992 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-FAIL-OK-FAIL) by vger.kernel.org with ESMTP id S1726776AbgEGNRv (ORCPT ); Thu, 7 May 2020 09:17:51 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CBCB9C05BD0B for ; Thu, 7 May 2020 06:17:50 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id v12so5028186wrp.12 for ; Thu, 07 May 2020 06:17:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=vBH8GHoad1DqQohZ1/l1O0nAKoJlvUeNDjY0kA+rhC4=; b=JmOl7qbvunF4WLEC+LZC2+O6BsF7Ke3/E5w+svMZx18uHj6b41+L0vijz6XEmFCocQ 1Mz+gARViKvU7H4raQ0nl8+y9Kykem/yWk5hihRXqTAby7/6+H/PsUNw1TJ5/w/fRAAs UW4TBNL3MJ5yjOaUcpkSKCl38uhO3cYEmLfX97aiLcZEloy3hvKldrEDjK9UVAHGhDgq XQ6Q0INVVsuVdkQnK2SL32T1nw+g1B+fXQteG5+Jsa1kYAUAJ8YfhdHLTNCEEi+cNX9a n5/r9c9NiZ7oeAWseZNXPXt9EdR33BYHgMz1bnGPNHwtM1jm7fS6cURavgWsNdp4Uoc3 nqsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=vBH8GHoad1DqQohZ1/l1O0nAKoJlvUeNDjY0kA+rhC4=; b=MAwg/Lmq3ynUvicEYtnXQ08UTRCxOA8FxKJQ9+wzSgqnYnWfyC8ixYPftkfKqJdqZF 5Hh6H8IP7DBSCc145ZMVQ4bKOKGe2ekohcFrZwgW31gsU1WNsAmzVsus0PQ8EHPcRg88 ikys232ZUEVIlhSkQu5POQKcAZw3AY79EP2icdpNPDD91LNoeVgLV+e8x6vod9hEWtLE UumEi2QZHxzzaRBdAO8RiZ+PvZ4ZrrUZSc2Eo5vk5qN4FqpskixogaZBVtPhlkkH9sUP RS/3DgsRCOjQ8/e8Ubz7KS/VotFM7uqQLQ9ddWWnNlfkJ/NwnQ7HG7nWTgitD2+5oqSx C7xQ== X-Gm-Message-State: AGi0PuZmqITFX7QYZeoPHALheN0FN7kjuEZL2avSQyS3d6O2b2C5RfOP dauanjcpwaMNxZqKsdpycf1GoEgj X-Google-Smtp-Source: APiQypIgTvGMKEz6sl9HH5RCIXXHFgrPbooYo2mtGh6nZHEr8bqA2Cl1SbFkKe2uV3ThCkN6blwN9w== X-Received: by 2002:adf:e802:: with SMTP id o2mr13655923wrm.110.1588857469130; Thu, 07 May 2020 06:17:49 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id y3sm8079025wrm.64.2020.05.07.06.17.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 May 2020 06:17:48 -0700 (PDT) Message-Id: In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 07 May 2020 13:17:39 +0000 Subject: [PATCH 07/10] sparse-checkout: define in-tree dependencies Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmaill.com, peff@peff.net, me@ttaylorr.com, jrnieder@gmail.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee As mentioned in the definition of the in-tree sparse-checkout definitions, a large repository could have many different roles for the users of that repository. That means that many different in-tree sparse-checkout definitions need to exist. If a directory needs to be added to many of these roles, then many of these files need to be edited. This is too much work to scale properly. Instead, let's make these definitions easier to maintain by following basic principles of build dependencies. Add simple links between these in-tree files using the new "sparse.inherit" config key. This multi- valued config setting specifies more files that are needed to define the sparse-checkout for a role. By separating out the common directories into a base file, the more specialized roles can focus their files on the directories only needed by those roles and leave the common directories to that base file. For example, suppose that .sparse/base has the following data: [sparse] dir = A dir = B/C dir = D/E/F and .sparse/extra has the following data: [sparse] dir = D dir = X inherit = .sparse/base Now, a user can run "git sparse-checkout set --in-tree .sparse/extra" and the resulting directories will define their cone-mode sparse-checkout patterns: A B/C D X Thus, the resulting sparse-checkout definition is the union of the definitions from .sparse/base and .sparse/extra, but the user only has one value of sparse.inTree of ".sparse/extra". It is simple to modify our existing logic to explore the directed graph created by the sparse.inherit values. We simply need to append to the string_list containing the list of in-tree files. Since we never repeat parsing on the same blob oid, this will not lead to infinite loops or exponential blowups in the parsing time. Signed-off-by: Derrick Stolee --- Documentation/git-sparse-checkout.txt | 26 ++++++++++++++++++++++ sparse-checkout.c | 27 ++++++++++++++++++----- sparse-checkout.h | 1 + t/t1091-sparse-checkout-builtin.sh | 31 +++++++++++++++++++++++++++ 4 files changed, 80 insertions(+), 5 deletions(-) diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt index c1713ebb1d2..941658e0011 100644 --- a/Documentation/git-sparse-checkout.txt +++ b/Documentation/git-sparse-checkout.txt @@ -238,6 +238,32 @@ definition according to the files available at the new commit. If any of the specified files do not exist at the new commit, then the sparse-checkout definition will not change. +In a very large repository, there may be need to have multiple of these +in-tree sparse-checkout definitions to fit the roles of multiple types of +users. As each definition grows and the number of user types grow, it can +be difficult to manage updating all of the definitions when a common core +is modified somehow. For this reason, the in-tree pattern sets can inherit +the directories from other in-tree pattern sets. Use the `sparse.inherit` +option to specify other files in the tree. + +For example, suppose the file listed earlier is at `.sparse/core`. Another +file could be stored as `.sparse/extra` with contents + +---------------------------------- +[sparse] + dir = X + dir = Y/Z + inherit = .sparse/core +---------------------------------- + +Then, if you run `git sparse-checkout set --in-tree .sparse/extra`, the +sparse-checkout definition will include `X` and `Y/Z` from `.sparse/extra` +as well as `A`, `B/C`, and `D/E/F` from `.sparse/core`. This is similar +to specifying both `.sparse/core` and `.sparse/extra` in the `set` +subcommand, but has a slight advantage. If the `.sparse/extra` file changes +the set of `inherit` files, then your sparse-checkout definition will +update accordingly as you switch between commits. + SUBMODULES ---------- diff --git a/sparse-checkout.c b/sparse-checkout.c index d01f4d7b525..4edeab49a10 100644 --- a/sparse-checkout.c +++ b/sparse-checkout.c @@ -66,12 +66,29 @@ static int sparse_dir_cb(const char *var, const char *value, void *data) return 0; } +static int sparse_inherit_cb(const char *var, const char *value, void *data) +{ + struct string_list *sl = (struct string_list *)data; + + if (!strcmp(var, SPARSE_CHECKOUT_INHERIT)) + string_list_append(sl, value); + + return 0; +} + static int load_in_tree_from_blob(struct pattern_list *pl, - struct object_id *oid) + struct object_id *oid, + struct string_list *inherit) { - return git_config_from_blob_oid(sparse_dir_cb, - SPARSE_CHECKOUT_DIR, - oid, pl); + if (git_config_from_blob_oid(sparse_dir_cb, + SPARSE_CHECKOUT_DIR, + oid, pl)) + return 1; + if (git_config_from_blob_oid(sparse_inherit_cb, + SPARSE_CHECKOUT_INHERIT, + oid, inherit)) + return 1; + return 0; } int load_in_tree_pattern_list(struct repository *r, @@ -121,7 +138,7 @@ int load_in_tree_pattern_list(struct repository *r, goto cleanup; } - load_in_tree_from_blob(pl, oid); + load_in_tree_from_blob(pl, oid, sl); } cleanup: diff --git a/sparse-checkout.h b/sparse-checkout.h index fb0ba48524a..8b766ea38fb 100644 --- a/sparse-checkout.h +++ b/sparse-checkout.h @@ -5,6 +5,7 @@ #include "repository.h" #define SPARSE_CHECKOUT_DIR "sparse.dir" +#define SPARSE_CHECKOUT_INHERIT "sparse.inherit" #define SPARSE_CHECKOUT_IN_TREE "sparse.intree" struct pattern_list; diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh index fdaafba5377..b2389e5b5e6 100755 --- a/t/t1091-sparse-checkout-builtin.sh +++ b/t/t1091-sparse-checkout-builtin.sh @@ -716,4 +716,35 @@ test_expect_success 'keep definition when in-tree file is missing' ' test_path_is_dir repo/.sparse ' +test_expect_success 'inherit definition from other files' ' + cat >repo/.sparse/inherit <<-EOF && + [sparse] + inherit = .sparse/sparse + inherit = .sparse/deep + inherit = .sparse/deeper1 + EOF + git -C repo add .sparse && + git -C repo commit -m "Add inherited file" && + git -C repo sparse-checkout set --in-tree .sparse/inherit && + check_files repo a deep folder1 && + check_files repo/deep a deeper1 deeper2 && + test_path_is_dir repo/.sparse && + cat >repo/.sparse/sparse <<-EOF && + [sparse] + dir = .sparse + EOF + git -C repo commit -a -m "drop folder1 from sparse" && + check_files repo a deep && + check_files repo/deep a deeper1 deeper2 && + test_path_is_dir repo/.sparse +' + +test_expect_success 'inherit files can have cycles' ' + echo "\tinherit = .sparse/inherit" >>repo/.sparse/sparse && + git -C repo commit -a -m "create inherit cycle" && + check_files repo a deep && + check_files repo/deep a deeper1 deeper2 && + test_path_is_dir repo/.sparse +' + test_done From patchwork Thu May 7 13:17:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11533617 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DEB4681 for ; Thu, 7 May 2020 13:17:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C464D208D6 for ; Thu, 7 May 2020 13:17:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="pAve3YO8" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726913AbgEGNRy (ORCPT ); Thu, 7 May 2020 09:17:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37986 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-FAIL-OK-FAIL) by vger.kernel.org with ESMTP id S1726864AbgEGNRv (ORCPT ); Thu, 7 May 2020 09:17:51 -0400 Received: from mail-wm1-x341.google.com (mail-wm1-x341.google.com [IPv6:2a00:1450:4864:20::341]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 349F1C05BD0C for ; Thu, 7 May 2020 06:17:51 -0700 (PDT) Received: by mail-wm1-x341.google.com with SMTP id e26so6488531wmk.5 for ; Thu, 07 May 2020 06:17:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=QVh2I/ihn8bhGOVhEi5ckFPVrBSzgqLhPRE1/wJdP2k=; b=pAve3YO89/AyFhwI3XabDFL37DyNah82eTk+Up51osCMI8noU7daQg739UjSwhFkqI 7L+5qKWWNojSjYESrk4aEctVRcWQVPGaGSTYMLiaa0+ZAMNVEIXwDcfV+D6vcW+R6+73 dDYPuzDpu+KNfEg96r64OBoWHAeNjW0Zd9ObXlzYL/n1W8UAaGXgDJt8JEHbVoQgv48y 5jGMcckEsLlHiTn9gYvLP5SEQPs01nv0kYNLaeD79yaBpSDTM6rN9rgzEErYAlgNHhkZ bXDo+G20JqTWPO7zCfjIn0L4fYN8leFk/09TB4Jf2mAjAOq6YS+nW/8brE613OY5u8VC kspQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=QVh2I/ihn8bhGOVhEi5ckFPVrBSzgqLhPRE1/wJdP2k=; b=HkvJoW6aNv1YkaNFFcI0/VfG6afeV7n3jdmfmjq66IlpfCVamgI3vRYzkxcPKmpv3d K0+6/6C1ngw40GNi5PikSvaIiq3+kGvmrPiaZhGMKhlHeTKqjtdsD3ATjtWigjEw15vf F7nXNZDDqlgAiGJuyowQ24l3Psw54E7jAOFLxIsUmGLh8O540UFs+rDDdhtw+NXsss+r +rwWpuCLAsDfI4l7S6/S+UkLO5OObHEhWwHhZunFmhF7jts2XA4cK9TdG+//oTUkshvl 6uJwUJhoNEi/Ko9xRGBVxTg0HJACE/4XnT9PXP2xAD7SmhTJO1dqcEg+jDTmc+//f9Dg KGxg== X-Gm-Message-State: AGi0PubYq1r+GRCdvubJNOUnU2BQHWAhc1Ip+Ls37K8R/Lw9A9V4Keny uqxvxcnfG9u60lBxzmnn3245WAmm X-Google-Smtp-Source: APiQypKqE2r2nAoXXhO5pnTf2EnTmpXXDqSriXDqKTJvKYJ+WYdaNRbB3lO4XYHTShxHT7vTujjS8A== X-Received: by 2002:a1c:5502:: with SMTP id j2mr11214041wmb.56.1588857469798; Thu, 07 May 2020 06:17:49 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id c20sm8267057wmd.36.2020.05.07.06.17.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 May 2020 06:17:49 -0700 (PDT) Message-Id: <4ccc5ecbf3d8106ea55b8eb17cd8ad14df3c2680.1588857462.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 07 May 2020 13:17:40 +0000 Subject: [PATCH 08/10] Makefile: skip git-gui if dir is missing Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmaill.com, peff@peff.net, me@ttaylorr.com, jrnieder@gmail.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee As an effort to promote "dogfooding" the sparse-checkout feature within the Git codebase, it is helpful to explore which portions of the codebase are optional. The NO_TCLTK variable in the Makefile allows ignoring the git-gui application at build time. If the local sparse-checkout removes that directory, then the build will fail without manually defining that variable. Instead, check to see if the directory exists at build time to see if we can automatically ignore this directory. With this change, the following list of directories can be supplied to "git sparse-checkout set" and Git will build and test on Linux: Documentation builtin compat/.depend contrib ewah mergetools negotiator perl refs sha1dc sha256 t templates trace2 vcs-svn xdiff The "make install" command requires the "gitweb" and "po" directories. The "po" directory will become optional in the next change. Signed-off-by: Derrick Stolee --- Makefile | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/Makefile b/Makefile index 2e8029b8026..70760d315cb 100644 --- a/Makefile +++ b/Makefile @@ -313,7 +313,8 @@ all:: # # Define NO_PYTHON if you do not want Python scripts or libraries at all. # -# Define NO_TCLTK if you do not want Tcl/Tk GUI. +# Define NO_TCLTK if you do not want Tcl/Tk GUI. This will also be defined +# if the git-gui directory is missing. # # Define SANE_TEXT_GREP to "-a" if you use recent versions of GNU grep # and egrep that are pickier when their input contains non-ASCII data. @@ -1860,6 +1861,12 @@ ifeq ($(TCLTK_PATH),) NO_TCLTK = NoThanks endif +ifndef NO_TCLTK + ifeq ($(ls git-gui),) + NO_TCLTK = NoThanks + endif +endif + ifeq ($(PERL_PATH),) NO_PERL = NoThanks endif From patchwork Thu May 7 13:17:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11533623 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0668781 for ; Thu, 7 May 2020 13:18:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E18C220708 for ; Thu, 7 May 2020 13:17:59 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="JY5mcsdc" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726930AbgEGNR5 (ORCPT ); Thu, 7 May 2020 09:17:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37988 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-FAIL-OK-FAIL) by vger.kernel.org with ESMTP id S1726904AbgEGNRx (ORCPT ); Thu, 7 May 2020 09:17:53 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 07000C05BD43 for ; Thu, 7 May 2020 06:17:52 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id h9so6373721wrt.0 for ; Thu, 07 May 2020 06:17:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=SAG3CfsdYn6lQEMHACFgXhSN7DUoYtsea9kTn51wM78=; b=JY5mcsdcDyKYRXgkS6qQeZ8HDczPeqKTmUqj2bPwp8jMXrAcYi1/hoxslPKL6y3Jbb 1L+KjmEN/x+tqA86Pv4uiPkTF/KOz37HKIeAowJF2wh+OLQUMhr3UQSqoFiGvSDG44Yi HLZ/ZNrZHIa68KVgLeqaXwEhkpyzOS87p/4K+aylQsiXPL6tegClfIhFU1yLVIJEDM83 5gXxHQhDo/C5wYrHhDeGUMSxhnVh6N9JTalHd3aMiA6PCiY/fV3ZpSGyu/sQazUWYth9 9ar7WXlU36op5oLYlC9TizeiH3P6vFEPz/I+1uwxDgqWfp7G8ZXW2L+qwu2ZLgW9K0q5 PAkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=SAG3CfsdYn6lQEMHACFgXhSN7DUoYtsea9kTn51wM78=; b=cClqfmuoEuDvqKEr3DhJWKcggl8CW5r1JBC8TVDcSkOpXllh5CKTuXv37VmeGEnNZF 7w4JL1Pz+Z+9ry5XDXui+j3K6/pKhxcJe0oUhGwxIqmoNZL2a7uF1BFeQlD3HvVCiD0v YEAmGhWzpClfyLTNjOOuyOX5qqgQ/NkAXf9O20QQCbhjw3VcaGUnHHb0XfCGvPOGEa2A 1hcZfPPJGrb3M+oh1MsH1ZRmZVKosmg1oFpfzoP1NFO1thhBYBC9I5XlgJd0B6Z0KQih 200wSu8CCE8ILugclkhRde1JO19ejuKaYd2LQtfEbgLoh3BLVzfxoIztQoQFy8g+UwfW QoaQ== X-Gm-Message-State: AGi0PuY2nkhA7u17sRAjRXjH4BVrITmSRBozdBREAY3I8iH6pNuD2t6C 987/Dtyj0MoI3YJQEXKPa9/ohhLD X-Google-Smtp-Source: APiQypJ/kxRjUnMpsoRiULQzAk3SK6FBr2eZgu0Pgp6pPfo/khxv3kXnl5NVw5l3qEdIh59rpBzklw== X-Received: by 2002:a5d:54c4:: with SMTP id x4mr16694332wrv.73.1588857470620; Thu, 07 May 2020 06:17:50 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id b82sm8447462wmh.1.2020.05.07.06.17.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 May 2020 06:17:50 -0700 (PDT) Message-Id: <5392cb7a1065fae766b6ef3c6f59728263ec6342.1588857462.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 07 May 2020 13:17:41 +0000 Subject: [PATCH 09/10] Makefile: disable GETTEXT when 'po' is missing Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmaill.com, peff@peff.net, me@ttaylorr.com, jrnieder@gmail.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee As an effort to promote "dogfooding" the sparse-checkout feature within the Git codebase, it is helpful to explore which portions of the codebase are optional. The NO_GETTEXT build variable can disable the translation libraries. The test suite then uses the NO_GETTEXT environment variable to disable the GETTEXT prerequisite for some tests. The 'po' directory contains translations for strings in the Git codebase. While this stores extremely important data for Git users, the data is not typically used by Git contributors in their daily work. Thus, it could be removed from the working directory using sparse-checkout. However, doing such a removal causes some tests to fail. Part of the failures are related to the GETTEXT prerequisite being enabled when it probably should not be. The other part is that some tests in t0200-gettext-basic.sh depend on the existence of files in the po directory. In test-lib, disable the GETTEXT prerequisite when the po directory does not exist, then add the GETTEXT prerequisite to these tests that use files in that directory. In Makefile, define NO_GETTEXT when the po directory does not exist. This is necessary for "make install" to work correctly. Signed-off-by: Derrick Stolee --- Makefile | 9 ++++++++- t/lib-gettext.sh | 1 - t/t0200-gettext-basic.sh | 6 +++--- t/test-lib.sh | 7 ++++++- 4 files changed, 17 insertions(+), 6 deletions(-) diff --git a/Makefile b/Makefile index 70760d315cb..38c2d54349a 100644 --- a/Makefile +++ b/Makefile @@ -72,7 +72,8 @@ all:: # # Define NO_GETTEXT if you don't want Git output to be translated. # A translated Git requires GNU libintl or another gettext implementation, -# plus libintl-perl at runtime. +# plus libintl-perl at runtime. This will also be defined if the 'po' +# directory is missing. # # Define USE_GETTEXT_SCHEME and set it to 'fallthrough', if you don't trust # the installed gettext translation of the shell scripts output. @@ -1861,6 +1862,12 @@ ifeq ($(TCLTK_PATH),) NO_TCLTK = NoThanks endif +ifndef NO_GETTEXT + ifeq ($(ls po),) + NO_GETTEXT = NoThanks + endif +endif + ifndef NO_TCLTK ifeq ($(ls git-gui),) NO_TCLTK = NoThanks diff --git a/t/lib-gettext.sh b/t/lib-gettext.sh index 2139b427ca1..beeb45a1387 100644 --- a/t/lib-gettext.sh +++ b/t/lib-gettext.sh @@ -7,7 +7,6 @@ . ./test-lib.sh GIT_TEXTDOMAINDIR="$GIT_BUILD_DIR/po/build/locale" -GIT_PO_PATH="$GIT_BUILD_DIR/po" export GIT_TEXTDOMAINDIR GIT_PO_PATH if test -n "$GIT_TEST_INSTALLED" diff --git a/t/t0200-gettext-basic.sh b/t/t0200-gettext-basic.sh index 8853d8afb92..0f6bc941cbb 100755 --- a/t/t0200-gettext-basic.sh +++ b/t/t0200-gettext-basic.sh @@ -15,17 +15,17 @@ test_expect_success 'sanity: $TEXTDOMAIN is git' ' test $TEXTDOMAIN = "git" ' -test_expect_success 'xgettext sanity: Perl _() strings are not extracted' ' +test_expect_success GETTEXT 'xgettext sanity: Perl _() strings are not extracted' ' ! grep "A Perl string xgettext will not get" "$GIT_PO_PATH"/is.po ' -test_expect_success 'xgettext sanity: Comment extraction with --add-comments' ' +test_expect_success GETTEXT 'xgettext sanity: Comment extraction with --add-comments' ' grep "TRANSLATORS: This is a test" "$TEST_DIRECTORY"/t0200/* | wc -l >expect && grep "TRANSLATORS: This is a test" "$GIT_PO_PATH"/is.po | wc -l >actual && test_cmp expect actual ' -test_expect_success 'xgettext sanity: Comment extraction with --add-comments stops at statements' ' +test_expect_success GETTEXT 'xgettext sanity: Comment extraction with --add-comments stops at statements' ' ! grep "This is a phony" "$GIT_PO_PATH"/is.po && ! grep "the above comment" "$GIT_PO_PATH"/is.po ' diff --git a/t/test-lib.sh b/t/test-lib.sh index 0ea1e5a05ed..ca22d23f0d2 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -1474,7 +1474,12 @@ test -z "$NO_PYTHON" && test_set_prereq PYTHON test -n "$USE_LIBPCRE1$USE_LIBPCRE2" && test_set_prereq PCRE test -n "$USE_LIBPCRE1" && test_set_prereq LIBPCRE1 test -n "$USE_LIBPCRE2" && test_set_prereq LIBPCRE2 -test -z "$NO_GETTEXT" && test_set_prereq GETTEXT + +GIT_PO_PATH="$GIT_BUILD_DIR/po" +if test -d "$GIT_PO_PATH" +then + test -z "$NO_GETTEXT" && test_set_prereq GETTEXT +fi if test -n "$GIT_TEST_GETTEXT_POISON_ORIG" then From patchwork Thu May 7 13:17:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11533621 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4C42917EF for ; Thu, 7 May 2020 13:17:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 33ECC208D6 for ; Thu, 7 May 2020 13:17:57 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="uuuSR9KM" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726924AbgEGNR4 (ORCPT ); Thu, 7 May 2020 09:17:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37990 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-FAIL-OK-FAIL) by vger.kernel.org with ESMTP id S1726776AbgEGNRx (ORCPT ); Thu, 7 May 2020 09:17:53 -0400 Received: from mail-wm1-x344.google.com (mail-wm1-x344.google.com [IPv6:2a00:1450:4864:20::344]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BE7D0C05BD09 for ; Thu, 7 May 2020 06:17:52 -0700 (PDT) Received: by mail-wm1-x344.google.com with SMTP id 188so6481998wmc.2 for ; Thu, 07 May 2020 06:17:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=6oUMy9+0pcbJlA7NWDN/AiA3FX5vH6btDVtujKVxPgE=; b=uuuSR9KMuKZgFhq2nUWWLnBrIX/Cccqv4CMsXPKcvpor4lssSIbM60jUek1G6p228u 08bQTkjwISRyJ0j3WPZgklu58joA6q12aNK/QRXFMacAwQaIZs5GghEgn9p5hjLoESVC hiWFHQ3D+cEWnWDEmecdTkSXI5PIk75xwJgwQ/df9vnxR97eqMshU5xcyGmmehwn3YYJ VXDzRdbfQaE1ugjCcErBKr9mo88YXib9DMWK1kVNfQCkZGXOFyw3h/XY78yxcdIw8fZ+ 4PddZY98297zR09mIdmxzMV0yU+vl6N3s84TdYrsZyz0OSOggtIHNS4juCnXmdHfx/kW ooDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=6oUMy9+0pcbJlA7NWDN/AiA3FX5vH6btDVtujKVxPgE=; b=Vpc2UUbz9W5xyj3U11UAHIp3r2QZb1rSJYHEyb2ycLgIjONxzd1FKWK6LOKfw4iYMv cJwMVZFv3RKunDNe83M6YXddNuUk4BCCAvRx3OaA+/SbGwyBB9lRcqh78rIdGoUBLj1U v6gge6x5AN2HJsC5xLA4HsQRjaFhIZFZ8dNVhAx4SMjgX4kjpfH7adibv69V2upJ4Opf GA8ZSheVa6Fg5MtiyWWl/oosi+PjcV1cd0I0QW6I3TS12ayN4XIRldeaFBaZ2QyxDeCl XfKMP/ugchqBldiUVet5O3MyVRk1Oy7FMV+DJLnZWcivHguJrseCyAtMHj6qDabJCeu+ OxuA== X-Gm-Message-State: AGi0Pua/SUmoEb0bd4M2G38f+pnKeZZZcVigTh3FFbq2MSsePDpuwryF 2B8HwOz6/W6lHYAVPHgtoFppAOyB X-Google-Smtp-Source: APiQypKSDT7da/xmI7eoj8o6gc/SfEQg3ZJYgQpyDl4rqSd3kREV4F+kVMilu9l+Mez9W9bZYVo0nQ== X-Received: by 2002:a7b:cf25:: with SMTP id m5mr11079446wmg.65.1588857471350; Thu, 07 May 2020 06:17:51 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id g74sm7945867wme.44.2020.05.07.06.17.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 07 May 2020 06:17:50 -0700 (PDT) Message-Id: <9078d0872831bd51157b7623070412a3d6f3a1ad.1588857462.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 07 May 2020 13:17:42 +0000 Subject: [PATCH 10/10] .sparse: add in-tree sparse-checkout for Git Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmaill.com, peff@peff.net, me@ttaylorr.com, jrnieder@gmail.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee The in-tree sparse-checkout feature allows architects to adapt their dependentent subsystems into sparse-checkout specificiations. This helps the typical users who do not know the full build system to use the sparse-chekcout in a pain-free way. In particular, if the sparse-checkout dependencies update at the same times as build dependencies, then the users automatically get new sparse-checkout definitions as they switch branches. For the Git codebase, it is not immediately obvious which directories are absolutely required for building Git. From my estimation, the necessary directories for building and testing Git on Linux are listed in the .sparse/base.deps file in this change. A few more directories in the compat/ dir are required for building on Windows. This presents a new possible workflow for Git contributors, especially those that want to test several new features in their workflow. The following allows a user to set up working on Git with partial clone and sparse-checkout: $ git clone --sparse --filter=blob:none https://github.com/git/git $ cd git $ git sparse-checkout set --in-tree .sparse/base.deps $ make test Perhaps there are ways we can further reduce the size of the "bare necessities" by rearranging code and adjusting the Makefile to match. In particular, I noticed that the vcs-svn directory is required for the build. It could be helpful to create a way to build a very small version of "core Git" that doesn't include integrations with other version control systems, and having them automatically disabled if they are missing from the sparse-checkout definition would be a great way to make that more accessible for contributors. Signed-off-by: Derrick Stolee --- .sparse/base.deps | 19 +++++++++++++++++++ .sparse/windows.deps | 3 +++ 2 files changed, 22 insertions(+) create mode 100644 .sparse/base.deps create mode 100644 .sparse/windows.deps diff --git a/.sparse/base.deps b/.sparse/base.deps new file mode 100644 index 00000000000..b0682175dc3 --- /dev/null +++ b/.sparse/base.deps @@ -0,0 +1,19 @@ +[sparse] + dir = Documentation + dir = block-sha1 + dir = builtin + dir = compat/.depend + dir = contrib + dir = ewah + dir = gitweb + dir = mergetools + dir = negotiator + dir = perl + dir = refs + dir = sha1dc + dir = sha256 + dir = t + dir = templates + dir = trace2 + dir = vcs-svn + dir = xdiff diff --git a/.sparse/windows.deps b/.sparse/windows.deps new file mode 100644 index 00000000000..6c9bf1df335 --- /dev/null +++ b/.sparse/windows.deps @@ -0,0 +1,3 @@ +[sparse] + inherit = .sparse/base.deps + dir = compat