From patchwork Wed May 12 17:28:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elijah Newren X-Patchwork-Id: 12254425 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4BDC7C4363C for ; Wed, 12 May 2021 17:55:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2B64961418 for ; Wed, 12 May 2021 17:55:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237660AbhELRnt (ORCPT ); Wed, 12 May 2021 13:43:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45614 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347742AbhELR3g (ORCPT ); Wed, 12 May 2021 13:29:36 -0400 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7048DC06174A for ; Wed, 12 May 2021 10:28:27 -0700 (PDT) Received: by mail-wm1-x333.google.com with SMTP id o26-20020a1c4d1a0000b0290146e1feccdaso341697wmh.0 for ; Wed, 12 May 2021 10:28:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=dm8sgUAbByH21XfkXMiORzWivUxVG7D9c7/7YP9peOs=; b=t0Zqrp0kbQXYqL3MowhGPIRs/TbDgcQT9HOyzN/bHG7gXHyF6qMPBu7BsBUyCGezZz ZqcheTQVbk5VZTkW6hszR1yPi+z0v6vtcCBI0yLv7ZrE5hqGPOHA1isY1MdU7auyJz/f PHkw0SNDSboQpoG5gGqagbVeE73amz/4uwEFRmrSWph0A9HXZYk3wQnyhP1+zgZSYR60 4OdcDfYY4ZUfPrH8ff8ntYE5VonayxRfcI0tG3m58cKim2NfNgprM/W+cB/bF+aSSlmX G34u1aS6UmdwOlfBMkO79jah7xU37exuJumSVCelay2mNFbkn7+P36lOO8Ubiaab7ENN 620g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=dm8sgUAbByH21XfkXMiORzWivUxVG7D9c7/7YP9peOs=; b=hpc8WAI5Thbf9omA9as+hBI5HPHcxaLaxJaSCUdfEIkJLx7omHd6y89ptYgIiVUI+/ kdTG9sU4+Lb85LiC451VrGQiMuIaFKPJl0CAgqhHeGEXl2NpDQT9/gePUh2XkXtTSZvX n1rGhyAMlJLgkuc5je4w0tGFxXC+P8jqCTXTjXmu6Gsu16oKfByaHrIFtmGWpyCCDuXc xXQPWXdJcyIOpvhSezCxQv4/nRNljg4k8jvagGq5HbEmbtO1Lep5pf4n3Ur9CxLlyKTu fBLX69J/jOU9P0N9vhTJb+MEANmjLoc/mjYfqZDbgOsUZqVZGETWbBDYB8S5mVruE3HG +5LQ== X-Gm-Message-State: AOAM5337RPkU4qvemxGGSzLPEMPN+s5ob7hi5fkMmmkhOFFzkSr0DaqV Q+Tjswm9H8eKSqU4HXgPKpKHiFB0I/M= X-Google-Smtp-Source: ABdhPJypTS7Ds6EotjNXASCxjsBcY30rYl+EIOtBewx/47R4QKbYBdeIq3SRTL7bzT3Ssl31+pMU5A== X-Received: by 2002:a1c:bd8a:: with SMTP id n132mr7489904wmf.106.1620840506177; Wed, 12 May 2021 10:28:26 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l22sm6395572wmq.28.2021.05.12.10.28.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 May 2021 10:28:25 -0700 (PDT) Message-Id: <6b1b4820dd20ed0911e049f0edadc2adcc391e9d.1620840502.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 12 May 2021 17:28:14 +0000 Subject: [PATCH v5 1/9] dir: convert trace calls to trace2 equivalents Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , Elijah Newren , Derrick Stolee , Jeff King , Philip Oakley , Jeff Hostetler , Josh Steadmon , Jeff Hostetler , Elijah Newren , Elijah Newren Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Elijah Newren From: Elijah Newren Signed-off-by: Elijah Newren --- dir.c | 50 ++++++-- t/t7063-status-untracked-cache.sh | 205 ++++++++++++++++++------------ t/t7519-status-fsmonitor.sh | 8 +- 3 files changed, 162 insertions(+), 101 deletions(-) diff --git a/dir.c b/dir.c index 3474e67e8f3c..cf19a83d3e2c 100644 --- a/dir.c +++ b/dir.c @@ -2760,15 +2760,46 @@ static struct untracked_cache_dir *validate_untracked_cache(struct dir_struct *d return root; } +static void emit_traversal_statistics(struct dir_struct *dir, + struct repository *repo, + const char *path, + int path_len) +{ + if (!trace2_is_enabled()) + return; + + if (!path_len) { + trace2_data_string("read_directory", repo, "path", ""); + } else { + struct strbuf tmp = STRBUF_INIT; + strbuf_add(&tmp, path, path_len); + trace2_data_string("read_directory", repo, "path", tmp.buf); + strbuf_release(&tmp); + } + + if (!dir->untracked) + return; + trace2_data_intmax("read_directory", repo, + "node-creation", dir->untracked->dir_created); + trace2_data_intmax("read_directory", repo, + "gitignore-invalidation", + dir->untracked->gitignore_invalidated); + trace2_data_intmax("read_directory", repo, + "directory-invalidation", + dir->untracked->dir_invalidated); + trace2_data_intmax("read_directory", repo, + "opendir", dir->untracked->dir_opened); +} + int read_directory(struct dir_struct *dir, struct index_state *istate, const char *path, int len, const struct pathspec *pathspec) { struct untracked_cache_dir *untracked; - trace_performance_enter(); + trace2_region_enter("dir", "read_directory", istate->repo); if (has_symlink_leading_path(path, len)) { - trace_performance_leave("read directory %.*s", len, path); + trace2_region_leave("dir", "read_directory", istate->repo); return dir->nr; } @@ -2784,23 +2815,15 @@ int read_directory(struct dir_struct *dir, struct index_state *istate, QSORT(dir->entries, dir->nr, cmp_dir_entry); QSORT(dir->ignored, dir->ignored_nr, cmp_dir_entry); - trace_performance_leave("read directory %.*s", len, path); + emit_traversal_statistics(dir, istate->repo, path, len); + + trace2_region_leave("dir", "read_directory", istate->repo); if (dir->untracked) { static int force_untracked_cache = -1; - static struct trace_key trace_untracked_stats = TRACE_KEY_INIT(UNTRACKED_STATS); if (force_untracked_cache < 0) force_untracked_cache = git_env_bool("GIT_FORCE_UNTRACKED_CACHE", 0); - trace_printf_key(&trace_untracked_stats, - "node creation: %u\n" - "gitignore invalidation: %u\n" - "directory invalidation: %u\n" - "opendir: %u\n", - dir->untracked->dir_created, - dir->untracked->gitignore_invalidated, - dir->untracked->dir_invalidated, - dir->untracked->dir_opened); if (force_untracked_cache && dir->untracked == istate->untracked && (dir->untracked->dir_opened || @@ -2811,6 +2834,7 @@ int read_directory(struct dir_struct *dir, struct index_state *istate, FREE_AND_NULL(dir->untracked); } } + return dir->nr; } diff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh index accefde72fb1..9710d33b3cd6 100755 --- a/t/t7063-status-untracked-cache.sh +++ b/t/t7063-status-untracked-cache.sh @@ -57,6 +57,19 @@ iuc () { return $ret } +get_relevant_traces () { + # From the GIT_TRACE2_PERF data of the form + # $TIME $FILE:$LINE | d0 | main | data | r1 | ? | ? | read_directo | $RELEVANT_STAT + # extract the $RELEVANT_STAT fields. We don't care about region_enter + # or region_leave, or stats for things outside read_directory. + INPUT_FILE=$1 + OUTPUT_FILE=$2 + grep data.*read_directo $INPUT_FILE | + cut -d "|" -f 9 \ + >"$OUTPUT_FILE" +} + + test_lazy_prereq UNTRACKED_CACHE ' { git update-index --test-untracked-cache; ret=$?; } && test $ret -ne 1 @@ -129,19 +142,21 @@ EOF test_expect_success 'status first time (empty cache)' ' avoid_racy && - : >../trace && - GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \ + : >../trace.output && + GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \ git status --porcelain >../actual && iuc status --porcelain >../status.iuc && test_cmp ../status.expect ../status.iuc && test_cmp ../status.expect ../actual && + get_relevant_traces ../trace.output ../trace.relevant && cat >../trace.expect <../trace && - GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \ + : >../trace.output && + GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \ git status --porcelain >../actual && iuc status --porcelain >../status.iuc && test_cmp ../status.expect ../status.iuc && test_cmp ../status.expect ../actual && + get_relevant_traces ../trace.output ../trace.relevant && cat >../trace.expect <four && - : >../trace && - GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \ + : >../trace.output && + GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \ git status --porcelain >../actual && iuc status --porcelain >../status.iuc && cat >../status.expect <../trace.expect <.gitignore && - : >../trace && - GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \ + : >../trace.output && + GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \ git status --porcelain >../actual && iuc status --porcelain >../status.iuc && cat >../status.expect <../trace.expect <>.git/info/exclude && - : >../trace && - GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \ + : >../trace.output && + GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \ git status --porcelain >../actual && iuc status --porcelain >../status.iuc && cat >../status.expect <../trace.expect <../trace && - GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \ + : >../trace.output && + GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \ git status --porcelain >../actual && iuc status --porcelain >../status.iuc && cat >../status.expect <../trace.expect <../trace && - GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \ + : >../trace.output && + GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \ git status --porcelain >../actual && iuc status --porcelain >../status.iuc && cat >../status.expect <../trace.expect <../trace && - GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \ + : >../trace.output && + GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \ git status --porcelain >../actual && iuc status --porcelain >../status.iuc && cat >../status.expect <../trace.expect <../trace && + : >../trace.output && avoid_racy && - GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \ + GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \ git status --porcelain >../status.actual && iuc status --porcelain >../status.iuc && cat >../status.expect <../trace.expect <../trace && - GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \ + : >../trace.output && + GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \ git status --porcelain >../status.actual && iuc status --porcelain >../status.iuc && cat >../status.expect <../trace.expect <../trace && - GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \ + : >../trace.output && + GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \ git status --porcelain >../status.actual && iuc status --porcelain >../status.iuc && cat >../status.expect <../trace.expect <../trace && - GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \ + : >../trace.output && + GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \ git status --porcelain >../status.actual && iuc status --porcelain >../status.iuc && test_cmp ../status.expect ../status.iuc && test_cmp ../status.expect ../status.actual && + get_relevant_traces ../trace.output ../trace.relevant && cat >../trace.expect <../before ) && @@ -346,12 +346,12 @@ test_expect_success UNTRACKED_CACHE 'ignore .git changes when invalidating UNTR' EOF ( cd dot-git && - GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace-after" \ + GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace-after" \ git status && test-tool dump-untracked-cache >../after ) && - grep "directory invalidation" trace-before >>before && - grep "directory invalidation" trace-after >>after && + grep "directory-invalidation" trace-before | cut -d"|" -f 9 >>before && + grep "directory-invalidation" trace-after | cut -d"|" -f 9 >>after && # UNTR extension unchanged, dir invalidation count unchanged test_cmp before after ' From patchwork Wed May 12 17:28:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elijah Newren X-Patchwork-Id: 12254423 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F1DEC4363F for ; Wed, 12 May 2021 17:55:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CE83B61166 for ; Wed, 12 May 2021 17:55:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237855AbhELRoA (ORCPT ); Wed, 12 May 2021 13:44:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45616 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347746AbhELR3g (ORCPT ); Wed, 12 May 2021 13:29:36 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7A6E6C06175F for ; Wed, 12 May 2021 10:28:28 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id a4so24447109wrr.2 for ; Wed, 12 May 2021 10:28:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=OD7n3L0qIorVlEic5lpEhxsSGS1aewtOJbk93/bYA1I=; b=HNUhZoYc0uVWOz+3ahQPgBSm0R6LFOJbqHnOai/ofaWWXMONdN9r9ry3pD1mXL+g2B CbdYZR9is2x3K6bZCUKm4JbtQVQE32gxrr+53+uAg3kKtBBBUS2XKd+uOxeXYLpVwI3K K5bl7HLBO8ybWjEdOZseGp2bwb3LQblilXmtqUE9Tl43KNXLp5Zn47kq1neFQLiq4zDF Gdhdq1FJmdhJEzstFQaktXLakMqSkc04cm9u0SEpQcrK8TpevrpFtAd57HdRbjbu5+RS wH6QQqufX318yjMutUe6sc0qIF1kkdoJKIeyh6T61NlJtoFC18MEHWZDyD3pMgUjgbqr 6tRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=OD7n3L0qIorVlEic5lpEhxsSGS1aewtOJbk93/bYA1I=; b=A7De5JZekOj4OC5gWhQgFG9HCEwOuLvRDCEpKKImoctAEsKZxzfkUKykSPfomz0Zfv mDlV10wO6NmXXYpnDaVig/S13PC1fAxPGQt1p13e5/j6PFYq9gDg6ylkgcqF0JgGAAVw kshrMCTTsvlbRYAgfuuTirVfs5ysG1wjdQV8ga5ZdfyXNrW15f/8m5xd+kV570l5awIG /ytLkLkxAJd4w8h14IbaoyO5LTusXCRcTyxUmpZMW6991nv+37jhIzLPShMuUDqouVeu bqJkXykeb8Yo/KunmL6n6yq1rlXqAYonSC4XMiXt1G6xantqn63kY5Us2jUmd+ltXIoN b1HQ== X-Gm-Message-State: AOAM5329/Pxz4NRqIj6ZJRNrSn4ULxXgsMMjWC6L0fOcvVIqq7fW4jcZ fW1dpEdbCPudQOr6GSVIHtgA4ntXpEo= X-Google-Smtp-Source: ABdhPJy5wmSczJe+/QFFxIm5xW9PD7VZj54XUFjXP2X0zhvUNmR4SYPnAzfi0VPovtw9dLUqEAKxHA== X-Received: by 2002:a5d:58d0:: with SMTP id o16mr10309602wrf.420.1620840507282; Wed, 12 May 2021 10:28:27 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id s18sm287953wro.95.2021.05.12.10.28.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 May 2021 10:28:26 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 12 May 2021 17:28:15 +0000 Subject: [PATCH v5 2/9] dir: report number of visited directories and paths with trace2 Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , Elijah Newren , Derrick Stolee , Jeff King , Philip Oakley , Jeff Hostetler , Josh Steadmon , Jeff Hostetler , Elijah Newren , Elijah Newren Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Elijah Newren From: Elijah Newren Provide more statistics in trace2 output that include the number of directories and total paths visited by the directory traversal logic. Subsequent patches will take advantage of this to ensure we do not unnecessarily traverse into ignored directories. Signed-off-by: Elijah Newren --- dir.c | 9 +++++++++ dir.h | 4 ++++ t/t7063-status-untracked-cache.sh | 3 ++- 3 files changed, 15 insertions(+), 1 deletion(-) diff --git a/dir.c b/dir.c index cf19a83d3e2c..f6dec5fd4a78 100644 --- a/dir.c +++ b/dir.c @@ -2440,6 +2440,7 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir, if (open_cached_dir(&cdir, dir, untracked, istate, &path, check_only)) goto out; + dir->visited_directories++; if (untracked) untracked->check_only = !!check_only; @@ -2448,6 +2449,7 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir, /* check how the file or directory should be treated */ state = treat_path(dir, untracked, &cdir, istate, &path, baselen, pathspec); + dir->visited_paths++; if (state > dir_state) dir_state = state; @@ -2777,6 +2779,11 @@ static void emit_traversal_statistics(struct dir_struct *dir, strbuf_release(&tmp); } + trace2_data_intmax("read_directory", repo, + "directories-visited", dir->visited_directories); + trace2_data_intmax("read_directory", repo, + "paths-visited", dir->visited_paths); + if (!dir->untracked) return; trace2_data_intmax("read_directory", repo, @@ -2797,6 +2804,8 @@ int read_directory(struct dir_struct *dir, struct index_state *istate, struct untracked_cache_dir *untracked; trace2_region_enter("dir", "read_directory", istate->repo); + dir->visited_paths = 0; + dir->visited_directories = 0; if (has_symlink_leading_path(path, len)) { trace2_region_leave("dir", "read_directory", istate->repo); diff --git a/dir.h b/dir.h index 04d886cfce75..22c67907f689 100644 --- a/dir.h +++ b/dir.h @@ -336,6 +336,10 @@ struct dir_struct { struct oid_stat ss_info_exclude; struct oid_stat ss_excludes_file; unsigned unmanaged_exclude_files; + + /* Stats about the traversal */ + unsigned visited_paths; + unsigned visited_directories; }; /*Count the number of slashes for string s*/ diff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh index 9710d33b3cd6..a0c123b0a77a 100755 --- a/t/t7063-status-untracked-cache.sh +++ b/t/t7063-status-untracked-cache.sh @@ -65,7 +65,8 @@ get_relevant_traces () { INPUT_FILE=$1 OUTPUT_FILE=$2 grep data.*read_directo $INPUT_FILE | - cut -d "|" -f 9 \ + cut -d "|" -f 9 | + grep -v visited \ >"$OUTPUT_FILE" } From patchwork Wed May 12 17:28:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elijah Newren X-Patchwork-Id: 12254437 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4862EC41602 for ; Wed, 12 May 2021 17:55:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1498761413 for ; Wed, 12 May 2021 17:55:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238544AbhELRoJ (ORCPT ); Wed, 12 May 2021 13:44:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45620 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347756AbhELR3i (ORCPT ); Wed, 12 May 2021 13:29:38 -0400 Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4CA8FC061761 for ; Wed, 12 May 2021 10:28:29 -0700 (PDT) Received: by mail-wm1-x336.google.com with SMTP id i128so13315191wma.5 for ; Wed, 12 May 2021 10:28:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=MB7H5JuJEysZqOtYGFNJ+Q3PBPxGokFYW3/0nzpVPpQ=; b=jKTL7GtI+atUsVXLx6Jgc2T0T9QSymapKWqyhdGWsGxZDulxX71ozFwXshk5Ridf2s oqFP4aoW3ZUvjsMsrFQ1dhpqbPe2IZMb5kYUu9Vkb9+6TKjtDoYL6pDa+8XDvOhOf7Gl p+B9OEaWrJhY29Qn9Cb1FxfCKDiMsueF4+otwZOM4kV5DRtuHvIKrINkFL3/fBpuVT/s 4RO4rHRXNaboYBDgRYLR5SybhODtL7jJdDc69roO2llYGd8JlD7xQB/cdXm2S/ciq++h VSbitD1YzaowhG/1ygov/SM0QOLmYKCbMkK33LKwM+Ds7FAL3gzVdkDAEOXZ7uUUsPlE uMQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=MB7H5JuJEysZqOtYGFNJ+Q3PBPxGokFYW3/0nzpVPpQ=; b=GVrOe/YnMK3WbCTN7Y6Ni3i9701/B1+Boxnf7ZRYpjGmq4BCqhLwXORHIRvrX2WvKh 3NsNlWjE1xQRcyhKctNgLzHvFkyyhWnhE5MMprO4JUOUUjDaGakQXKFqL7LQwnufuiLC r06X5fMxapXJD0JTM/MNfSBGzkkQHEVnZ7TEtXiXLFO59H17BOpZFdeCHtow9WBAT5Mq MPtFFqqCH9XNOfa0T/uMNDh/czdBooF5ruiEKax8AbJHSSf7UfGVXPOF5PeWj2H+gF7T 3cnxZvV+rgeRXeHxXe+vm+7nOCTs/pHfeyHpHOjKPZXheaLZhVwgCtITuUHhulbP3mTv Pqcg== X-Gm-Message-State: AOAM530oi0qZkXpN7qrXUogjCiJFIFRoWWRQxEE8YMWuf7j367Rpe9FJ +7EcC450XBF1NtH5/kHTWo+t87l7tb0= X-Google-Smtp-Source: ABdhPJwHq7n8cfaQYxlMeXkVZcuiRh8mURYkacn6mRY/f6UZHXdEcTiGYCf2gXSFLXxgNzZusBEEAw== X-Received: by 2002:a1c:8087:: with SMTP id b129mr1504430wmd.145.1620840508088; Wed, 12 May 2021 10:28:28 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id p10sm291344wmq.14.2021.05.12.10.28.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 May 2021 10:28:27 -0700 (PDT) Message-Id: <279ef30ffbc2145b0bd8edc377366808bd96ff50.1620840502.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 12 May 2021 17:28:16 +0000 Subject: [PATCH v5 3/9] ls-files: error out on -i unless -o or -c are specified Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , Elijah Newren , Derrick Stolee , Jeff King , Philip Oakley , Jeff Hostetler , Josh Steadmon , Jeff Hostetler , Elijah Newren , Elijah Newren Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Elijah Newren From: Elijah Newren ls-files --ignored can be used together with either --others or --cached. After being perplexed for a bit and digging in to the code, I assumed that ls-files -i was just broken and not printing anything and I had a nice patch ready to submit when I finally realized that -i can be used with --cached to find tracked ignores. While that was a mistake on my part, and a careful reading of the documentation could have made this more clear, I suspect this is an error others are likely to make as well. In fact, of two uses in our testsuite, I believe one of the two did make this error. In t1306.13, there are NO tracked files, and all the excludes built up and used in that test and in previous tests thus have to be about untracked files. However, since they were looking for an empty result, the mistake went unnoticed as their erroneous command also just happened to give an empty answer. -i will most the time be used with -o, which would suggest we could just make -i imply -o in the absence of either a -o or -c, but that would be a backward incompatible break. Instead, let's just flag -i without either a -o or -c as an error, and update the two relevant testcases to specify their intent. Signed-off-by: Elijah Newren --- builtin/ls-files.c | 3 +++ t/t1306-xdg-files.sh | 2 +- t/t3003-ls-files-exclude.sh | 4 ++-- 3 files changed, 6 insertions(+), 3 deletions(-) diff --git a/builtin/ls-files.c b/builtin/ls-files.c index 60a2913a01e9..e8e25006c647 100644 --- a/builtin/ls-files.c +++ b/builtin/ls-files.c @@ -748,6 +748,9 @@ int cmd_ls_files(int argc, const char **argv, const char *cmd_prefix) if (pathspec.nr && error_unmatch) ps_matched = xcalloc(pathspec.nr, 1); + if ((dir.flags & DIR_SHOW_IGNORED) && !show_others && !show_cached) + die("ls-files -i must be used with either -o or -c"); + if ((dir.flags & DIR_SHOW_IGNORED) && !exc_given) die("ls-files --ignored needs some exclude pattern"); diff --git a/t/t1306-xdg-files.sh b/t/t1306-xdg-files.sh index dd87b43be1a6..40d3c42618c0 100755 --- a/t/t1306-xdg-files.sh +++ b/t/t1306-xdg-files.sh @@ -116,7 +116,7 @@ test_expect_success 'Exclusion in a non-XDG global ignore file' ' test_expect_success 'Checking XDG ignore file when HOME is unset' ' (sane_unset HOME && git config --unset core.excludesfile && - git ls-files --exclude-standard --ignored >actual) && + git ls-files --exclude-standard --ignored --others >actual) && test_must_be_empty actual ' diff --git a/t/t3003-ls-files-exclude.sh b/t/t3003-ls-files-exclude.sh index d5ec333131f9..c41c4f046abf 100755 --- a/t/t3003-ls-files-exclude.sh +++ b/t/t3003-ls-files-exclude.sh @@ -29,11 +29,11 @@ test_expect_success 'add file to gitignore' ' ' check_all_output -test_expect_success 'ls-files -i lists only tracked-but-ignored files' ' +test_expect_success 'ls-files -i -c lists only tracked-but-ignored files' ' echo content >other-file && git add other-file && echo file >expect && - git ls-files -i --exclude-standard >output && + git ls-files -i -c --exclude-standard >output && test_cmp expect output ' From patchwork Wed May 12 17:28:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elijah Newren X-Patchwork-Id: 12254429 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E22DC41603 for ; Wed, 12 May 2021 17:55:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2F3A261106 for ; Wed, 12 May 2021 17:55:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239189AbhELRoQ (ORCPT ); Wed, 12 May 2021 13:44:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44732 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347753AbhELR3i (ORCPT ); Wed, 12 May 2021 13:29:38 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0535EC061763 for ; Wed, 12 May 2021 10:28:30 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id z17so5364483wrq.7 for ; Wed, 12 May 2021 10:28:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=4KsZA2I7CfIgjH08T88TsFhRmAHld7iU15QGXlFPnMM=; b=i3ame8EZYBfAyroAGudHDkfBG72X0fhylp63LuxOwRBcyK0OIuCUSV7RO75aaWOZ3X 7EGo9OIymOovJYCnlE0HaJXHdVMv9J2G76s2CQuoiMoW47Djryh4eLlGG4oPx/wI7Udi CtWm1vu8PzlXyykbOIIFIW4FcE0gNji9YuBd+6Ruum8phHIpcdS9DXv3uSXZ9JyNqRY6 uoTJOS/HLE+3o48fiIG4bRZgGLREWQuwwG76QukO1w2SLrCnJy9bGkVBk2+oFG2wSFot 64jOb5jgXXJXMdA2eULYNGOtIn+sk98MlN2e6V5J20FLCflgbCuVmfw11nGt7ltRbz1O 05UQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=4KsZA2I7CfIgjH08T88TsFhRmAHld7iU15QGXlFPnMM=; b=lbA4hjL3F4irIntXTnYbDyyyCPWLwvWLbOm2K5Vhrtv+s/qwGVpL+oAfYB7I5jhCAg zko4fjC6SCX9MqY0DfQfJvvKx/Uo6hL0hz3J4YQl1h0EWmacoZDRTU+a+9L+saEPQPE4 ZlXfsIyj3u75PM0/PxPM4nBsX0Vucv66bVXwNInN+XcpGTvBExSsfdr1e3xF59XAeNyB T8yaRNmbgqGMmFbVrVa9WHvzTY8X/7imFh4/gPShf5TKgiPn2UIEWYAP6OTR9KJ5Lq4r xW+nNNtph7mFZf/la7P5kYMI/yKLbnoRtjb3XfHk3ny68u+pNYVF1VeUxwU0A3x94w1A VOvg== X-Gm-Message-State: AOAM532hhUEJOuQuoovolfd6mmdcTRFL71bUOtbBDKsZBMkB6iBIsESL Ig/tldiVCOEujhKNalvLYFnaYIQkhaM= X-Google-Smtp-Source: ABdhPJxi/jZejIzaDkWxWX8u51ddkH6CX6jhPwSaMkXdHAA5sagmKLWrcca8/06at962/7Q+mp3zLg== X-Received: by 2002:adf:f3cd:: with SMTP id g13mr39792137wrp.94.1620840508858; Wed, 12 May 2021 10:28:28 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id b10sm398602wrr.27.2021.05.12.10.28.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 May 2021 10:28:28 -0700 (PDT) Message-Id: <5a8807a1992cd012a8271fd157a757c07254b27a.1620840502.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 12 May 2021 17:28:17 +0000 Subject: [PATCH v5 4/9] t7300: add testcase showing unnecessary traversal into ignored directory Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , Elijah Newren , Derrick Stolee , Jeff King , Philip Oakley , Jeff Hostetler , Josh Steadmon , Jeff Hostetler , Elijah Newren , Elijah Newren Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Elijah Newren From: Elijah Newren The PNPM package manager is apparently creating deeply nested (but ignored) directory structures; traversing them is costly performance-wise, unnecessary, and in some cases is even throwing warnings/errors because the paths are too long to handle on various platforms. Add a testcase that checks for such unnecessary directory traversal. Signed-off-by: Elijah Newren --- t/t7300-clean.sh | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/t/t7300-clean.sh b/t/t7300-clean.sh index a74816ca8b46..07e8ba2d4b85 100755 --- a/t/t7300-clean.sh +++ b/t/t7300-clean.sh @@ -746,4 +746,27 @@ test_expect_success 'clean untracked paths by pathspec' ' test_must_be_empty actual ' +test_expect_failure 'avoid traversing into ignored directories' ' + test_when_finished rm -f output error trace.* && + test_create_repo avoid-traversing-deep-hierarchy && + ( + cd avoid-traversing-deep-hierarchy && + + mkdir -p untracked/subdir/with/a && + >untracked/subdir/with/a/random-file.txt && + + GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \ + git clean -ffdxn -e untracked + ) && + + # Make sure we only visited into the top-level directory, and did + # not traverse into the "untracked" subdirectory since it was excluded + grep data.*read_directo.*directories-visited trace.output | + cut -d "|" -f 9 >trace.relevant && + cat >trace.expect <<-EOF && + ..directories-visited:1 + EOF + test_cmp trace.expect trace.relevant +' + test_done From patchwork Wed May 12 17:28:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elijah Newren X-Patchwork-Id: 12254439 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 65320C43140 for ; Wed, 12 May 2021 17:55:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 47EB361166 for ; Wed, 12 May 2021 17:55:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239396AbhELRo3 (ORCPT ); Wed, 12 May 2021 13:44:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45090 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347757AbhELR3j (ORCPT ); Wed, 12 May 2021 13:29:39 -0400 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B9D4EC0613ED for ; Wed, 12 May 2021 10:28:30 -0700 (PDT) Received: by mail-wr1-x436.google.com with SMTP id v12so24413271wrq.6 for ; Wed, 12 May 2021 10:28:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=/W8yKM0Iad+GJsfgHty7FstnK4TOTYBb4J7fdM1ZrAQ=; b=GI/cp9VTG1mzOgvT9lqsTRYZsORQvpSAu7I7uveWpFMvkJr3uCAAPvvZlgDzs0MjJj hiXXEC+hNm7+qPrzvq97Upan3pBEsCGzcG6P3e58gk7VpvRxrO3JmFb/+UiOFsTFjXKC 5LVzTe/kasTYPv+CvN35uTcShyQwvduAwhIDX1Y5pNo2MedVtcXBANd8eDMfDhb/YgDy yYp5T8WGHrNRrSEHO0o8ikKMt9ImYtzpT33Q2vsTMHCArm48Lx577AX9aRaxhheRV2aI +vfCAhQYuyQoBHFC7hdijNyWeNjuOEtKnD0Ch0coxl/X5ouGkIoGojTHy6uatWY6yi1P zWxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=/W8yKM0Iad+GJsfgHty7FstnK4TOTYBb4J7fdM1ZrAQ=; b=HzLJ7kBHTxOc524RLQfcEenSa/7uzhO8QkYQSblJ9i3I1a5bU1I+5kkLboBasgSY74 ga2liyX5lEf066PhK0IFk49K5ppA2JIGjnF59S7HBuPtjHBiSCequTSYGY3LUhxOknnn vY5rNRyfeWVrKtj2yxMEq//r357iqBOaaNi2Bhg/2SU8BH8cicpytO0vqqKgb0CjTiB3 QOyxHWfEQBWf6Carj6xxeesbJBTqvOCMjGgSfDTycEGw+0SRgLGUC+jouC85CwIs9cxC Z6HEk0QlidE2SNpDvTicxMtWQHkImSOAk5bxO9K6veUtU7g5A8YFZcZAlTZ4UWa7P2gK jmIA== X-Gm-Message-State: AOAM5302mJ1HhV9dLtVp5Q+tMHpBd+qoX69Gli5CS+YQvjveckTpqa34 kfTJzC+EN6+azhtJsk3MVATioSv5EfY= X-Google-Smtp-Source: ABdhPJz+/7cxPLvwfosuOUCJcJusrpHIoQ/zOnjwQBR+Rce3vCtvKVfG1byq8Y+K7Y2K9B8ZtjX7jg== X-Received: by 2002:a5d:50c1:: with SMTP id f1mr8593377wrt.168.1620840509588; Wed, 12 May 2021 10:28:29 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n6sm315674wro.23.2021.05.12.10.28.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 May 2021 10:28:29 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 12 May 2021 17:28:18 +0000 Subject: [PATCH v5 5/9] t3001, t7300: add testcase showcasing missed directory traversal Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , Elijah Newren , Derrick Stolee , Jeff King , Philip Oakley , Jeff Hostetler , Josh Steadmon , Jeff Hostetler , Elijah Newren , Elijah Newren Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Elijah Newren From: Elijah Newren In the last commit, we added a testcase showing that the directory traversal machinery sometimes traverses into directories unnecessarily. Here we show that there are cases where it does the opposite: it does not traverse into directories, despite those directories having important files that need to be flagged. Add a testcase showing that `git ls-files -o -i --directory` can omit some of the files it should be listing, and another showing that `git clean -fX` can fail to clean out some of the expected files. Signed-off-by: Elijah Newren --- t/t3001-ls-files-others-exclude.sh | 5 +++++ t/t7300-clean.sh | 19 +++++++++++++++++++ 2 files changed, 24 insertions(+) diff --git a/t/t3001-ls-files-others-exclude.sh b/t/t3001-ls-files-others-exclude.sh index 1ec7cb57c7a8..ac05d1a17931 100755 --- a/t/t3001-ls-files-others-exclude.sh +++ b/t/t3001-ls-files-others-exclude.sh @@ -292,6 +292,11 @@ EOF test_cmp expect actual ' +test_expect_failure 'ls-files with "**" patterns and --directory' ' + # Expectation same as previous test + git ls-files --directory -o -i --exclude "**/a.1" >actual && + test_cmp expect actual +' test_expect_success 'ls-files with "**" patterns and no slashes' ' git ls-files -o -i --exclude "one**a.1" >actual && diff --git a/t/t7300-clean.sh b/t/t7300-clean.sh index 07e8ba2d4b85..34c08c325407 100755 --- a/t/t7300-clean.sh +++ b/t/t7300-clean.sh @@ -769,4 +769,23 @@ test_expect_failure 'avoid traversing into ignored directories' ' test_cmp trace.expect trace.relevant ' +test_expect_failure 'traverse into directories that may have ignored entries' ' + test_when_finished rm -f output && + test_create_repo need-to-traverse-into-hierarchy && + ( + cd need-to-traverse-into-hierarchy && + mkdir -p modules/foobar/src/generated && + > modules/foobar/src/generated/code.c && + > modules/foobar/Makefile && + echo "/modules/**/src/generated/" >.gitignore && + + git clean -fX modules/foobar >../output && + + grep Removing ../output && + + test_path_is_missing modules/foobar/src/generated/code.c && + test_path_is_file modules/foobar/Makefile + ) +' + test_done From patchwork Wed May 12 17:28:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elijah Newren X-Patchwork-Id: 12254431 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9CDC8C2B9F2 for ; Wed, 12 May 2021 17:55:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7CEC961166 for ; Wed, 12 May 2021 17:55:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239753AbhELRoj (ORCPT ); Wed, 12 May 2021 13:44:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45634 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347762AbhELR3l (ORCPT ); Wed, 12 May 2021 13:29:41 -0400 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8CBAEC06138A for ; Wed, 12 May 2021 10:28:31 -0700 (PDT) Received: by mail-wm1-x330.google.com with SMTP id 82-20020a1c01550000b0290142562ff7c9so3440279wmb.3 for ; Wed, 12 May 2021 10:28:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=ZCsCzRk7QBxJKEuTIHYvP4SktVhXqEjOuJNubhJB0WY=; b=pGWZTfhqTHMjTNUUwDl6tlmcYid+OBwq7vIvaunig67OoI3lQ0WHol06LFbj+CvvwT nZjoqTh1HP5xPFOms+Zv8ThZlLRup83BHRA96dxCmZDkZvptrPb5zN1J2pE/bLzV4XvI lVRhNXobzUPePuEt+/joCS+bA01KUqxB4E1Ik8JjVGJvHnhlJs9gHvRb40n/uboruQMJ 85opE4nstLCvtdhbiYGzxr5NKI5Fw4nr4bq/bI4qwR0eRGdRlm0sACJ5tzr0x4UJODSx 3Q6H3UGNLVOdLsuhWDgb0CyJdTSK/xYzHl6hOKDhbZoI7l/EhMkIbR7dDs5+Iu9QV/W3 0u6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=ZCsCzRk7QBxJKEuTIHYvP4SktVhXqEjOuJNubhJB0WY=; b=YupF1KNx+N0aomY6ekHlOBchGJbiz1aj1ejYE91SOfMZg3qFNmqdJVWrBqB++tka6x CW1ZHyRzpXnF/XNofRFSxLFZmusqkD4OehFzFVgEGGAyeFozRw1Y97wAu36nDD1MlXOW mS30l/OL8omq+dHNBDn6mFN2/RCrj9euOKt2oTuoKyBDjjtq1BuNY1K3g9n4kgbAMHuC +wTIVE9j3r17B29ouuLVsCB9q9gK9CQ0G3Ug6I9aYSEcRARUKZ96sm2/5g63QoxyoBht 0yk5jtLzJnS2N/wmwnM1WXXQ3UPR+Y3/xmnDEoGKdNbH/D4pN2lpj0UpThlBSlnvv+ZU Be+g== X-Gm-Message-State: AOAM531zeUsqU5/eZ2z5f/DPGMfcpaUpDiCeFuOYLObQaJAMfSsuLfEh eko6xYvPC59+dpLqg99+rfhsFNTS+u4= X-Google-Smtp-Source: ABdhPJyVq554MBBH1gO/y0fyRSMYmBuFIofDN2an+I0fwSrOb6/ZIxEkA3sMhiJ8LBFwwtFJBqKnsg== X-Received: by 2002:a7b:c346:: with SMTP id l6mr3009614wmj.109.1620840510262; Wed, 12 May 2021 10:28:30 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id h8sm288714wrw.85.2021.05.12.10.28.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 May 2021 10:28:29 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 12 May 2021 17:28:19 +0000 Subject: [PATCH v5 6/9] dir: avoid unnecessary traversal into ignored directory Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , Elijah Newren , Derrick Stolee , Jeff King , Philip Oakley , Jeff Hostetler , Josh Steadmon , Jeff Hostetler , Elijah Newren , Elijah Newren Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Elijah Newren From: Elijah Newren The show_other_directories case in treat_directory() tried to handle both excludes and untracked files with the same logic, and mishandled both the excludes and the untracked files in the process, in different ways. Split that logic apart, and then focus on the logic for the excludes; a subsequent commit will address the logic for untracked files. For show_other_directories, an excluded directory means that every path underneath that directory will also be excluded. Given that the calling code requested to just show directories when everything under a directory had the same state (that's what the "DIR_SHOW_OTHER_DIRECTORIES" flag means), we generally do not need to traverse into such directories and can just immediately mark them as ignored (i.e. as path_excluded). The only reason we cannot just immediately return path_excluded is the DIR_HIDE_EMPTY_DIRECTORIES flag and the possibility that the ignored directory is an empty directory. The code previously treated DIR_SHOW_IGNORED_TOO in most cases as an exception as well, which was wrong. It can sometimes reduce the number of cases where we need to recurse (namely if DIR_SHOW_IGNORED_TOO_MODE_MATCHING is also set), but should not be able to increase the number of cases where we need to recurse. Fix the logic accordingly. Some sidenotes about possible confusion with dir.c: * "ignored" often refers to an untracked ignore", i.e. a file which is not tracked which matches one of the ignore/exclusion rules. But you can also have a "tracked ignore", a tracked file that happens to match one of the ignore/exclusion rules and which dir.c has to worry about since "git ls-files -c -i" is supposed to list them. * The dir code often uses "ignored" and "excluded" interchangeably, which you need to keep in mind while reading the code. * "exclude" is used multiple ways in the code: * As noted above, "exclude" is often a synonym for "ignored". * The logic for parsing .gitignore files was re-used in .git/info/sparse-checkout, except there it is used to mark paths that the user wants to *keep*. This was mostly addressed by commit 65edd96aec ("treewide: rename 'exclude' methods to 'pattern'", 2019-09-03), but every once in a while you'll find a comment about "exclude" referring to these patterns that might in fact be in use by the sparse-checkout machinery for inclusion rules. * The word "EXCLUDE" is also used for pathspec negation, as in (pathspec->items[3].magic & PATHSPEC_EXCLUDE) Thus if a user had a .gitignore file containing *~ *.log !settings.log And then ran git add -- 'settings.*' ':^settings.log' Then :^settings.log is a pathspec negation making settings.log not be requested to be added even though all other settings.* files are being added. Also, !settings.log in the gitignore file is a negative exclude pattern meaning that settings.log is normally a file we want to track even though all other *.log files are ignored. Sometimes it feels like dir.c needs its own glossary with its many definitions, including the multiply-defined terms. Reported-by: Jason Gore Signed-off-by: Elijah Newren --- dir.c | 44 +++++++++++++++++++++++++++++--------------- t/t7300-clean.sh | 2 +- 2 files changed, 30 insertions(+), 16 deletions(-) diff --git a/dir.c b/dir.c index f6dec5fd4a78..db2ae516a3aa 100644 --- a/dir.c +++ b/dir.c @@ -1844,6 +1844,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir, } /* This is the "show_other_directories" case */ + assert(dir->flags & DIR_SHOW_OTHER_DIRECTORIES); /* * If we have a pathspec which could match something _below_ this @@ -1854,27 +1855,40 @@ static enum path_treatment treat_directory(struct dir_struct *dir, if (matches_how == MATCHED_RECURSIVELY_LEADING_PATHSPEC) return path_recurse; + /* Special cases for where this directory is excluded/ignored */ + if (excluded) { + /* + * In the show_other_directories case, if we're not + * hiding empty directories, there is no need to + * recurse into an ignored directory. + */ + if (!(dir->flags & DIR_HIDE_EMPTY_DIRECTORIES)) + return path_excluded; + + /* + * Even if we are hiding empty directories, we can still avoid + * recursing into ignored directories for DIR_SHOW_IGNORED_TOO + * if DIR_SHOW_IGNORED_TOO_MODE_MATCHING is also set. + */ + if ((dir->flags & DIR_SHOW_IGNORED_TOO) && + (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING)) + return path_excluded; + } + /* - * Other than the path_recurse case immediately above, we only need - * to recurse into untracked/ignored directories if either of the - * following bits is set: + * Other than the path_recurse case above, we only need to + * recurse into untracked directories if either of the following + * bits is set: * - DIR_SHOW_IGNORED_TOO (because then we need to determine if * there are ignored entries below) * - DIR_HIDE_EMPTY_DIRECTORIES (because we have to determine if * the directory is empty) */ - if (!(dir->flags & (DIR_SHOW_IGNORED_TOO | DIR_HIDE_EMPTY_DIRECTORIES))) - return excluded ? path_excluded : path_untracked; - - /* - * ...and even if DIR_SHOW_IGNORED_TOO is set, we can still avoid - * recursing into ignored directories if the path is excluded and - * DIR_SHOW_IGNORED_TOO_MODE_MATCHING is also set. - */ - if (excluded && - (dir->flags & DIR_SHOW_IGNORED_TOO) && - (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING)) - return path_excluded; + if (!excluded && + !(dir->flags & (DIR_SHOW_IGNORED_TOO | + DIR_HIDE_EMPTY_DIRECTORIES))) { + return path_untracked; + } /* * Even if we don't want to know all the paths under an untracked or diff --git a/t/t7300-clean.sh b/t/t7300-clean.sh index 34c08c325407..21e48b3ba591 100755 --- a/t/t7300-clean.sh +++ b/t/t7300-clean.sh @@ -746,7 +746,7 @@ test_expect_success 'clean untracked paths by pathspec' ' test_must_be_empty actual ' -test_expect_failure 'avoid traversing into ignored directories' ' +test_expect_success 'avoid traversing into ignored directories' ' test_when_finished rm -f output error trace.* && test_create_repo avoid-traversing-deep-hierarchy && ( From patchwork Wed May 12 17:28:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elijah Newren X-Patchwork-Id: 12254435 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BEAD4C2B9F5 for ; Wed, 12 May 2021 17:55:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9B6C861413 for ; Wed, 12 May 2021 17:55:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240109AbhELRos (ORCPT ); Wed, 12 May 2021 13:44:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45636 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347765AbhELR3l (ORCPT ); Wed, 12 May 2021 13:29:41 -0400 Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 49B2BC061574 for ; Wed, 12 May 2021 10:28:32 -0700 (PDT) Received: by mail-wm1-x336.google.com with SMTP id 4-20020a05600c26c4b0290146e1feccd8so3441740wmv.1 for ; Wed, 12 May 2021 10:28:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=8SiUoiHeOJgGU6jKM10WleHVXxI8fdxp+WTvsfb1z7o=; b=rN1xtZ4Pee+86W/Xr9boqAAZDbx0h5o/CeVxB3fR9AWnoMGA1ZqfdR876DqwP80FY8 d3HjaWB9uK9KUMAj6sT1zKswHzp6b5d3eva7gtQ+gRClhquuq8+9ircfZR9S3nyaR2Bq GPnT4cVCle8tWdWeHWhB6mHnZqVM7WDwxaUWyioj4lAFm8znnZN9HAAwJyj+2k2N7f/s F/JYgwBBD6fB7bg7eK9XWaiGW+te/Fn3ed6lIeBWwFNwuh67zIU7qrcwGznw//X6OKw8 ofgdlY+Crfk8VD7R4sYYk2+CQrfUU65DNMnXcMEAXgAiAvHAN0Jk06dveQCpjG5MUJZm XgJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=8SiUoiHeOJgGU6jKM10WleHVXxI8fdxp+WTvsfb1z7o=; b=Ht+6+YjrWj5+ZN2158Sq/NeOwCkuSOPbDP2ftFKHyEr8KMZZ+0ax1OrIAfEPX2N7Lh 628d3/S6nfF6b/QX2jKt6FZDVqWx/Ft+JHLNsFK9HrqPv1UURpwgqEYzXynjKpSdq2di SbyxE9bkmfQgTaXKgG8KHYFO00NpoJCaLsrV30qrUgTVgGt00lgQab12AwYNI0L5qxkW AMqP7PIQewTHj1Qc6Aroa33gJBTpwJABfgw3mwQV3bYvv9+OGFAJWp0MceFxVieNRbDd Sk44bU4jnrkukW+aLQ/DyFX2DfzbAsXqZg5VREN2pCi8xd7qD45fqk94Zgc3JX6choos 6B8g== X-Gm-Message-State: AOAM531WgkyL9FoxXKmnXJ1wSVQLB+dA5Zp5cZwm+OC4Bts01vME7r7E q5s1LuRo3ERoLisZnMJAP26KsjNkFlE= X-Google-Smtp-Source: ABdhPJx1cYh/l1OQYLQVyv3BS+NtslsOO5DBVdLvk/vkJIjBvB5YdJXvNk0KiMAIwcLGk718N4ihtQ== X-Received: by 2002:a1c:1f95:: with SMTP id f143mr12764448wmf.77.1620840511052; Wed, 12 May 2021 10:28:31 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n5sm332545wrx.31.2021.05.12.10.28.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 May 2021 10:28:30 -0700 (PDT) Message-Id: <6fa1e85edf2fbfbf902d4f726c9aaee6aa9df308.1620840502.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 12 May 2021 17:28:20 +0000 Subject: [PATCH v5 7/9] dir: traverse into untracked directories if they may have ignored subfiles Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , Elijah Newren , Derrick Stolee , Jeff King , Philip Oakley , Jeff Hostetler , Josh Steadmon , Jeff Hostetler , Elijah Newren , Elijah Newren Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Elijah Newren From: Elijah Newren A directory that is untracked does not imply that all files under it should be categorized as untracked; in particular, if the caller is interested in ignored files, many files or directories underneath the untracked directory may be ignored. We previously partially handled this right with DIR_SHOW_IGNORED_TOO, but missed DIR_SHOW_IGNORED. It was not obvious, though, because the logic for untracked and excluded files had been fused together making it harder to reason about. The previous commit split that logic out, making it easier to notice that DIR_SHOW_IGNORED was missing. Add it. Signed-off-by: Elijah Newren --- dir.c | 10 ++++++---- t/t3001-ls-files-others-exclude.sh | 2 +- t/t7300-clean.sh | 2 +- 3 files changed, 8 insertions(+), 6 deletions(-) diff --git a/dir.c b/dir.c index db2ae516a3aa..c0233bbba36c 100644 --- a/dir.c +++ b/dir.c @@ -1877,15 +1877,17 @@ static enum path_treatment treat_directory(struct dir_struct *dir, /* * Other than the path_recurse case above, we only need to - * recurse into untracked directories if either of the following + * recurse into untracked directories if any of the following * bits is set: - * - DIR_SHOW_IGNORED_TOO (because then we need to determine if - * there are ignored entries below) + * - DIR_SHOW_IGNORED (because then we need to determine if + * there are ignored entries below) + * - DIR_SHOW_IGNORED_TOO (same as above) * - DIR_HIDE_EMPTY_DIRECTORIES (because we have to determine if * the directory is empty) */ if (!excluded && - !(dir->flags & (DIR_SHOW_IGNORED_TOO | + !(dir->flags & (DIR_SHOW_IGNORED | + DIR_SHOW_IGNORED_TOO | DIR_HIDE_EMPTY_DIRECTORIES))) { return path_untracked; } diff --git a/t/t3001-ls-files-others-exclude.sh b/t/t3001-ls-files-others-exclude.sh index ac05d1a17931..516c95ea0e82 100755 --- a/t/t3001-ls-files-others-exclude.sh +++ b/t/t3001-ls-files-others-exclude.sh @@ -292,7 +292,7 @@ EOF test_cmp expect actual ' -test_expect_failure 'ls-files with "**" patterns and --directory' ' +test_expect_success 'ls-files with "**" patterns and --directory' ' # Expectation same as previous test git ls-files --directory -o -i --exclude "**/a.1" >actual && test_cmp expect actual diff --git a/t/t7300-clean.sh b/t/t7300-clean.sh index 21e48b3ba591..0399701e6276 100755 --- a/t/t7300-clean.sh +++ b/t/t7300-clean.sh @@ -769,7 +769,7 @@ test_expect_success 'avoid traversing into ignored directories' ' test_cmp trace.expect trace.relevant ' -test_expect_failure 'traverse into directories that may have ignored entries' ' +test_expect_success 'traverse into directories that may have ignored entries' ' test_when_finished rm -f output && test_create_repo need-to-traverse-into-hierarchy && ( From patchwork Wed May 12 17:28:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12254433 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC7BEC2B9F6 for ; Wed, 12 May 2021 17:55:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BC64561166 for ; Wed, 12 May 2021 17:55:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240405AbhELRox (ORCPT ); Wed, 12 May 2021 13:44:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45614 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347766AbhELR3m (ORCPT ); Wed, 12 May 2021 13:29:42 -0400 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1AB39C06138B for ; Wed, 12 May 2021 10:28:33 -0700 (PDT) Received: by mail-wm1-x332.google.com with SMTP id b11-20020a7bc24b0000b0290148da0694ffso134642wmj.2 for ; Wed, 12 May 2021 10:28:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=B2ZiYssB16i7F8qxNfOEy3ZNesqACrrwJ1aBmmG9MeA=; b=Yd6qwZu9H/hIB7qTk0xek3T0BKSYN9ZAFSJ7hYkNZKsb+yXKbJph8yQ0rT6Sgi33w1 IitkdJPsqJtamclc7SU0ANsr/kmNkvxiy7hwOWNBVumoL7QJPwE3mn1qp6HzAoZRsddL PnWms5REyPsHZDyeRdKavKeE6pzSGIXJk5ZjAstpx3RfDG3igrpptQYZWNv+HIlOQtrx Y2DfJxnsKT35XO4aVvAl7uKyj3T2ya5+dVs5bPROHvAZcQeoH4mb9DCjU8Fsxk6bwbmt GPSj9ZLnMCmbAxY+peixw+wVHsoPFpj822+lRSP/tgy5NADGlLdkDcTTCQO8uP04rWtX cXtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=B2ZiYssB16i7F8qxNfOEy3ZNesqACrrwJ1aBmmG9MeA=; b=NT/HBR4/+sRVhCiIbaLbIawvuvxFcnLX96kuQrXL4mTgywa2ibAQZROn0RVtspo5Gs frJgrYKI/6QfDfwUnZz4ySaTEbivQy5+3mlAp5Emmky+/Cw5AOLCHvPMUSWuTF22qO/X jWVvxeArPfSRiG5th8WHtVgy7nR+9318sh4d+u2C/PpbUNahq5EVpIE1omg2FpeGAgUH 1U3f5M2pe7VsdGFs99hjZEi97lUtDwCbh17flCkoqZv/Rz0BjRQJaeZkCQ+dw4DR2agY xO/m0PtwGycIAn7/LasZDnDFgNN/aF4sby7W2E8C2294bi0N1PN0zOCWe+GNNk4meakJ C2Aw== X-Gm-Message-State: AOAM533zRqyeGJfoLyfHgNf8/1tiQomEz+989MN2iB2kIaAwDAeSGmkZ uJM8LKu97IXhts3fNfnQeYMfErcYySU= X-Google-Smtp-Source: ABdhPJw4HKBSU0yPvMI4MKFrwFgMJK0TlnvYDKVNERvSXT75X37wiLqdHZOerrXyyyzIbbsACKQE6g== X-Received: by 2002:a7b:cc8e:: with SMTP id p14mr5186026wma.74.1620840511863; Wed, 12 May 2021 10:28:31 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m10sm354275wrr.2.2021.05.12.10.28.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 May 2021 10:28:31 -0700 (PDT) Message-Id: <179f992edc9254803252ae10e5d692f3755a40f3.1620840502.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 12 May 2021 17:28:21 +0000 Subject: [PATCH v5 8/9] dir: update stale description of treat_directory() Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , Elijah Newren , Derrick Stolee , Jeff King , Philip Oakley , Jeff Hostetler , Josh Steadmon , Jeff Hostetler , Elijah Newren , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The documentation comment for treat_directory() was originally written in 095952 (Teach directory traversal about subprojects, 2007-04-11) which was before the 'struct dir_struct' split its bitfield of named options into a 'flags' enum in 7c4c97c0 (Turn the flags in struct dir_struct into a single variable, 2009-02-16). When those flags changed, the comment became stale, since members like 'show_other_directories' transitioned into flags like DIR_SHOW_OTHER_DIRECTORIES. Update the comments for treat_directory() to use these flag names rather than the old member names. Signed-off-by: Derrick Stolee Reviewed-by: Elijah Newren --- dir.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/dir.c b/dir.c index c0233bbba36c..4794c822b47f 100644 --- a/dir.c +++ b/dir.c @@ -1749,13 +1749,13 @@ static enum exist_status directory_exists_in_index(struct index_state *istate, * Case 3: if we didn't have it in the index previously, we * have a few sub-cases: * - * (a) if "show_other_directories" is true, we show it as - * just a directory, unless "hide_empty_directories" is + * (a) if DIR_SHOW_OTHER_DIRECTORIES flag is set, we show it as + * just a directory, unless DIR_HIDE_EMPTY_DIRECTORIES is * also true, in which case we need to check if it contains any * untracked and / or ignored files. - * (b) if it looks like a git directory, and we don't have - * 'no_gitlinks' set we treat it as a gitlink, and show it - * as a directory. + * (b) if it looks like a git directory and we don't have the + * DIR_NO_GITLINKS flag, then we treat it as a gitlink, and + * show it as a directory. * (c) otherwise, we recurse into it. */ static enum path_treatment treat_directory(struct dir_struct *dir, @@ -1843,7 +1843,6 @@ static enum path_treatment treat_directory(struct dir_struct *dir, return path_recurse; } - /* This is the "show_other_directories" case */ assert(dir->flags & DIR_SHOW_OTHER_DIRECTORIES); /* @@ -1858,7 +1857,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir, /* Special cases for where this directory is excluded/ignored */ if (excluded) { /* - * In the show_other_directories case, if we're not + * If DIR_SHOW_OTHER_DIRECTORIES is set and we're not * hiding empty directories, there is no need to * recurse into an ignored directory. */ From patchwork Wed May 12 17:28:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elijah Newren X-Patchwork-Id: 12254427 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19050C4161D for ; Wed, 12 May 2021 17:55:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EEB6361166 for ; Wed, 12 May 2021 17:55:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240686AbhELRo6 (ORCPT ); Wed, 12 May 2021 13:44:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45644 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1347769AbhELR3n (ORCPT ); Wed, 12 May 2021 13:29:43 -0400 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 428FAC06174A for ; Wed, 12 May 2021 10:28:34 -0700 (PDT) Received: by mail-wr1-x430.google.com with SMTP id d11so24426664wrw.8 for ; Wed, 12 May 2021 10:28:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=T2Wc+qGui5R3S+fz9nrVYTg7kI8koX4nP8maaIStcus=; b=qXRdOIkxuZei4HXtsW+VlxeG01w8Oy1KO8OoUEUXwhFfDLaxTUBj/JEx7JY+bFVjS2 1Scw9Jm3NC8Xyjrlh0soCDfwTb2yiCiii+/Ep7f56HSJrC5iCtC9u4DL6kgBf8SB1kne shduQiUJb5V/CUeWb6q3b609XdCKfF7goVW0mbsCE/5B5e5AYJ74tieRN2k4JlTfWJv5 WIWJ4Pi42TOHmvpSGZ4FocX65XKqdDSgiJQZXrUz+m6p6ezw3AMbFmArugKvcDqxtOjC dmvaFs1pVNjsJdn2eQv4uS9k+HeCCiJ7tnFFtWuDPUbpUi2Sv2U6/1pCRl4BpPSMIZrw IG7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=T2Wc+qGui5R3S+fz9nrVYTg7kI8koX4nP8maaIStcus=; b=FIx5dgr01H/0ArmNws1asKr+8EzQTOOi36SpFm6MwW85vJ0f2RgxcL3Krqi7dt10KG IOMW7l4ooSjHrp98f5XAwGKN1NbeFuexJ9KJbFvlfdjQMVkRq3T9QlnRCC7GwqUoMdtJ iCZwegMTpIDi6us71m+NtMZszSMmIzJ4wajmw/DtL7ixs1tgep+e4HvCE6jKlZy6m6hr Rh530P/7n4f3oDeSI3U113IauyofvFFQ65lUYXEqQVlRBVHYaddMOUWgQt6ozggtCXHo eyfWrZ1bypeWoKCiYEGgYXPR9aN4mChc7hTiRrpSAXW+dAmNpY/3T4SYeEDVSjE13nYS JYTQ== X-Gm-Message-State: AOAM533vl8NOC+t1wcbcQvQlxKnj5/WhwpfsxGAds8jGfgwquQn/z9+K rVWCoZY4Ba8Wgjqa9sUZ0BSEtVXxeD4= X-Google-Smtp-Source: ABdhPJzmiJ0ddH8FE5SKUQkAZWsxVbVz3nPW+tnlJfOiMUw6bAOqkKPx8tVRKi+5v7rEByToMetknw== X-Received: by 2002:adf:e9c2:: with SMTP id l2mr45541705wrn.323.1620840513051; Wed, 12 May 2021 10:28:33 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id k11sm297225wrm.62.2021.05.12.10.28.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 May 2021 10:28:32 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 12 May 2021 17:28:22 +0000 Subject: [PATCH v5 9/9] dir: introduce readdir_skip_dot_and_dotdot() helper Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , Elijah Newren , Derrick Stolee , Jeff King , Philip Oakley , Jeff Hostetler , Josh Steadmon , Jeff Hostetler , Elijah Newren , Elijah Newren Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Elijah Newren From: Elijah Newren Many places in the code were doing while ((d = readdir(dir)) != NULL) { if (is_dot_or_dotdot(d->d_name)) continue; ...process d... } Introduce a readdir_skip_dot_and_dotdot() helper to make that a one-liner: while ((d = readdir_skip_dot_and_dotdot(dir)) != NULL) { ...process d... } This helper particularly simplifies checks for empty directories. Also use this helper in read_cached_dir() so that our statistics are consistent across platforms. (In other words, read_cached_dir() should have been using is_dot_or_dotdot() and skipping such entries, but did not and left it to treat_path() to detect and mark such entries as path_none.) Signed-off-by: Elijah Newren --- builtin/clean.c | 4 +--- builtin/worktree.c | 4 +--- diff-no-index.c | 5 ++--- dir.c | 26 +++++++++++++++++--------- dir.h | 2 ++ entry.c | 5 +---- notes-merge.c | 5 +---- object-file.c | 4 +--- packfile.c | 5 +---- rerere.c | 4 +--- worktree.c | 12 +++--------- 11 files changed, 31 insertions(+), 45 deletions(-) diff --git a/builtin/clean.c b/builtin/clean.c index 995053b79173..a1a57476153b 100644 --- a/builtin/clean.c +++ b/builtin/clean.c @@ -189,10 +189,8 @@ static int remove_dirs(struct strbuf *path, const char *prefix, int force_flag, strbuf_complete(path, '/'); len = path->len; - while ((e = readdir(dir)) != NULL) { + while ((e = readdir_skip_dot_and_dotdot(dir)) != NULL) { struct stat st; - if (is_dot_or_dotdot(e->d_name)) - continue; strbuf_setlen(path, len); strbuf_addstr(path, e->d_name); diff --git a/builtin/worktree.c b/builtin/worktree.c index 877145349381..ae28249e0f0b 100644 --- a/builtin/worktree.c +++ b/builtin/worktree.c @@ -118,10 +118,8 @@ static void prune_worktrees(void) struct dirent *d; if (!dir) return; - while ((d = readdir(dir)) != NULL) { + while ((d = readdir_skip_dot_and_dotdot(dir)) != NULL) { char *path; - if (is_dot_or_dotdot(d->d_name)) - continue; strbuf_reset(&reason); if (should_prune_worktree(d->d_name, &reason, &path, expire)) prune_worktree(d->d_name, reason.buf); diff --git a/diff-no-index.c b/diff-no-index.c index 7814eabfe028..e5cc87837143 100644 --- a/diff-no-index.c +++ b/diff-no-index.c @@ -26,9 +26,8 @@ static int read_directory_contents(const char *path, struct string_list *list) if (!(dir = opendir(path))) return error("Could not open directory %s", path); - while ((e = readdir(dir))) - if (!is_dot_or_dotdot(e->d_name)) - string_list_insert(list, e->d_name); + while ((e = readdir_skip_dot_and_dotdot(dir))) + string_list_insert(list, e->d_name); closedir(dir); return 0; diff --git a/dir.c b/dir.c index 4794c822b47f..66c8518947dd 100644 --- a/dir.c +++ b/dir.c @@ -59,6 +59,18 @@ void dir_init(struct dir_struct *dir) memset(dir, 0, sizeof(*dir)); } +struct dirent * +readdir_skip_dot_and_dotdot(DIR *dirp) +{ + struct dirent *e; + + while ((e = readdir(dirp)) != NULL) { + if (!is_dot_or_dotdot(e->d_name)) + break; + } + return e; +} + int count_slashes(const char *s) { int cnt = 0; @@ -2341,7 +2353,7 @@ static int read_cached_dir(struct cached_dir *cdir) struct dirent *de; if (cdir->fdir) { - de = readdir(cdir->fdir); + de = readdir_skip_dot_and_dotdot(cdir->fdir); if (!de) { cdir->d_name = NULL; cdir->d_type = DT_UNKNOWN; @@ -2940,11 +2952,9 @@ int is_empty_dir(const char *path) if (!dir) return 0; - while ((e = readdir(dir)) != NULL) - if (!is_dot_or_dotdot(e->d_name)) { - ret = 0; - break; - } + e = readdir_skip_dot_and_dotdot(dir); + if (e) + ret = 0; closedir(dir); return ret; @@ -2984,10 +2994,8 @@ static int remove_dir_recurse(struct strbuf *path, int flag, int *kept_up) strbuf_complete(path, '/'); len = path->len; - while ((e = readdir(dir)) != NULL) { + while ((e = readdir_skip_dot_and_dotdot(dir)) != NULL) { struct stat st; - if (is_dot_or_dotdot(e->d_name)) - continue; strbuf_setlen(path, len); strbuf_addstr(path, e->d_name); diff --git a/dir.h b/dir.h index 22c67907f689..a704e466afd5 100644 --- a/dir.h +++ b/dir.h @@ -342,6 +342,8 @@ struct dir_struct { unsigned visited_directories; }; +struct dirent *readdir_skip_dot_and_dotdot(DIR *dirp); + /*Count the number of slashes for string s*/ int count_slashes(const char *s); diff --git a/entry.c b/entry.c index 2dc94ba5cc2a..6da589696770 100644 --- a/entry.c +++ b/entry.c @@ -57,12 +57,9 @@ static void remove_subtree(struct strbuf *path) if (!dir) die_errno("cannot opendir '%s'", path->buf); - while ((de = readdir(dir)) != NULL) { + while ((de = readdir_skip_dot_and_dotdot(dir)) != NULL) { struct stat st; - if (is_dot_or_dotdot(de->d_name)) - continue; - strbuf_addch(path, '/'); strbuf_addstr(path, de->d_name); if (lstat(path->buf, &st)) diff --git a/notes-merge.c b/notes-merge.c index d2771fa3d43c..e9d6f86d3428 100644 --- a/notes-merge.c +++ b/notes-merge.c @@ -695,13 +695,10 @@ int notes_merge_commit(struct notes_merge_options *o, strbuf_addch(&path, '/'); baselen = path.len; - while ((e = readdir(dir)) != NULL) { + while ((e = readdir_skip_dot_and_dotdot(dir)) != NULL) { struct stat st; struct object_id obj_oid, blob_oid; - if (is_dot_or_dotdot(e->d_name)) - continue; - if (get_oid_hex(e->d_name, &obj_oid)) { if (o->verbosity >= 3) printf("Skipping non-SHA1 entry '%s%s'\n", diff --git a/object-file.c b/object-file.c index 624af408cdcd..77bdcfd21bc8 100644 --- a/object-file.c +++ b/object-file.c @@ -2304,10 +2304,8 @@ int for_each_file_in_obj_subdir(unsigned int subdir_nr, strbuf_addch(path, '/'); baselen = path->len; - while ((de = readdir(dir))) { + while ((de = readdir_skip_dot_and_dotdot(dir))) { size_t namelen; - if (is_dot_or_dotdot(de->d_name)) - continue; namelen = strlen(de->d_name); strbuf_setlen(path, baselen); diff --git a/packfile.c b/packfile.c index 8668345d9309..7c8f1b7202ca 100644 --- a/packfile.c +++ b/packfile.c @@ -813,10 +813,7 @@ void for_each_file_in_pack_dir(const char *objdir, } strbuf_addch(&path, '/'); dirnamelen = path.len; - while ((de = readdir(dir)) != NULL) { - if (is_dot_or_dotdot(de->d_name)) - continue; - + while ((de = readdir_skip_dot_and_dotdot(dir)) != NULL) { strbuf_setlen(&path, dirnamelen); strbuf_addstr(&path, de->d_name); diff --git a/rerere.c b/rerere.c index dee60dc6df63..d83d58df4fbc 100644 --- a/rerere.c +++ b/rerere.c @@ -1190,13 +1190,11 @@ void rerere_gc(struct repository *r, struct string_list *rr) if (!dir) die_errno(_("unable to open rr-cache directory")); /* Collect stale conflict IDs ... */ - while ((e = readdir(dir))) { + while ((e = readdir_skip_dot_and_dotdot(dir))) { struct rerere_dir *rr_dir; struct rerere_id id; int now_empty; - if (is_dot_or_dotdot(e->d_name)) - continue; if (!is_rr_cache_dirname(e->d_name)) continue; /* or should we remove e->d_name? */ diff --git a/worktree.c b/worktree.c index f35ac40a84a5..237517baee67 100644 --- a/worktree.c +++ b/worktree.c @@ -128,10 +128,8 @@ struct worktree **get_worktrees(void) dir = opendir(path.buf); strbuf_release(&path); if (dir) { - while ((d = readdir(dir)) != NULL) { + while ((d = readdir_skip_dot_and_dotdot(dir)) != NULL) { struct worktree *linked = NULL; - if (is_dot_or_dotdot(d->d_name)) - continue; if ((linked = get_linked_worktree(d->d_name))) { ALLOC_GROW(list, counter + 1, alloc); @@ -486,13 +484,9 @@ int submodule_uses_worktrees(const char *path) if (!dir) return 0; - while ((d = readdir(dir)) != NULL) { - if (is_dot_or_dotdot(d->d_name)) - continue; - + d = readdir_skip_dot_and_dotdot(dir); + if (d != NULL) ret = 1; - break; - } closedir(dir); return ret; }