From patchwork Sat May 8 00:08:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elijah Newren X-Patchwork-Id: 12245383 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 87736C433B4 for ; Sat, 8 May 2021 00:08:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4065C6100C for ; Sat, 8 May 2021 00:08:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230245AbhEHAJ1 (ORCPT ); Fri, 7 May 2021 20:09:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43132 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230185AbhEHAJZ (ORCPT ); Fri, 7 May 2021 20:09:25 -0400 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B8A07C061574 for ; Fri, 7 May 2021 17:08:23 -0700 (PDT) Received: by mail-wm1-x32e.google.com with SMTP id t11-20020a05600c198bb02901476e13296aso5754190wmq.0 for ; Fri, 07 May 2021 17:08:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=v5loYWvSZoqGvlhAuSo5IwrvfMT/3yTkKMM6xGuLRpM=; b=k+b1udKWAvAYQclVzngwJh8TxOLXYxNEZFUbAkzGZY3Y67E1bRNNOS8Eh/b7bsBTRr ah5DNO28Hs+DI5Pr48ylWTkaLRtyPer2EHVJb0XUuJybCpFG3U5VWuhI1otwkC43CuQS AdY0ToXKS0fJJPM6oDcQZekD3Hj/TJE2rHFl8GXXhIoSsrp7MbraBom00/ZOvqme73wH 9jLwwPPQW4fopOGymN1+TOZQZN4asZcLmREfq4P63r4wKWgK29JAjiKY+2DTexK7eF+m N1gmG6fqmmi6MHpUAx4HTOq01Ag4CUQmOJMFI8KFqRJW/xTdT0WwgWj5dx82tIrM3rQx WqTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=v5loYWvSZoqGvlhAuSo5IwrvfMT/3yTkKMM6xGuLRpM=; b=JaYHVWDgjRFhcIZV0gxyEOE94AlXiFQ4HO9Hpch7WB5AnL5P1poGSNvzxxtmvAf3LL UvFQHs13xhKpA/F/5iW0Z8+q1mSC5+Sf0LutpvmiMSBt1fAwDyFCXxV92nezf8JNVruX 2tbxzYkZdMytkZMKRmjErx7+97JToTWklNVKIVo5z0B2E/4YC5L8cnGP1dWBUnpXXVdR XUiezsTgSGAn/tbHc5zmO4IYFouSQuidcfQVkKC/270z/W9DzZNW458Jxf888ns3XMQZ tCGtBW6P3mkDEhtDij6R378Y7/iIkCWwt0pM49mv7f79dY6tNU8kQmXVd5nmB5rXgiik Sk3Q== X-Gm-Message-State: AOAM532Qiblj8MJxJ5sJ1hdrwy3wH+DvvW2xzKKoHCDBunN5NiBgdKqv +K+S93MepbvERwNk8TzOt9W0yOEWj3c= X-Google-Smtp-Source: ABdhPJwBFiVWqxgBk6MSaz/+5n5jplIw4i8qnICoIj/WL6h8VYsd8PPErDJCmahPNKeuQ3yzVk6l2A== X-Received: by 2002:a1c:804a:: with SMTP id b71mr12827849wmd.82.1620432502569; Fri, 07 May 2021 17:08:22 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id t7sm10012997wrw.60.2021.05.07.17.08.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 May 2021 17:08:22 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Sat, 08 May 2021 00:08:13 +0000 Subject: [PATCH v2 1/8] t7300: add testcase showing unnecessary traversal into ignored directory Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , Elijah Newren , Derrick Stolee , Jeff King , Elijah Newren , Elijah Newren Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Elijah Newren From: Elijah Newren PNPM is apparently creating deeply nested (but ignored) directory structures; traversing them is costly performance-wise, unnecessary, and in some cases is even throwing warnings/errors because the paths are too long to handle on various platforms. Add a testcase that demonstrates this problem. Initial-test-by: Jason Gore Helped-by: brian m. carlson Signed-off-by: Elijah Newren --- t/t7300-clean.sh | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/t/t7300-clean.sh b/t/t7300-clean.sh index a74816ca8b46..5f1dc397c11e 100755 --- a/t/t7300-clean.sh +++ b/t/t7300-clean.sh @@ -746,4 +746,44 @@ test_expect_success 'clean untracked paths by pathspec' ' test_must_be_empty actual ' +test_expect_failure 'avoid traversing into ignored directories' ' + test_when_finished rm -f output error && + test_create_repo avoid-traversing-deep-hierarchy && + ( + cd avoid-traversing-deep-hierarchy && + + >directory-random-file.txt && + # Put this file under directory400/directory399/.../directory1/ + depth=400 && + for x in $(test_seq 1 $depth); do + mkdir "tmpdirectory$x" && + mv directory* "tmpdirectory$x" && + mv "tmpdirectory$x" "directory$x" + done && + + git clean -ffdxn -e directory$depth >../output 2>../error && + + test_must_be_empty ../output && + # We especially do not want things like + # "warning: could not open directory " + # appearing in the error output. It is true that directories + # that are too long cannot be opened, but we should not be + # recursing into those directories anyway since the very first + # level is ignored. + test_must_be_empty ../error && + + # alpine-linux-musl fails to "rm -rf" a directory with such + # a deeply nested hierarchy. Help it out by deleting the + # leading directories ourselves. Super slow, but, what else + # can we do? Without this, we will hit a + # error: Tests passed but test cleanup failed; aborting + # so do this ugly manual cleanup... + while test ! -f directory-random-file.txt; do + name=$(ls -d directory*) && + mv $name/* . && + rmdir $name + done + ) +' + test_done From patchwork Sat May 8 00:08:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elijah Newren X-Patchwork-Id: 12245387 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 23A4AC43460 for ; Sat, 8 May 2021 00:08:29 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D7D7A6145D for ; Sat, 8 May 2021 00:08:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230252AbhEHAJ2 (ORCPT ); Fri, 7 May 2021 20:09:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43134 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230213AbhEHAJ0 (ORCPT ); Fri, 7 May 2021 20:09:26 -0400 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 59D3AC061761 for ; Fri, 7 May 2021 17:08:24 -0700 (PDT) Received: by mail-wm1-x32b.google.com with SMTP id b11-20020a7bc24b0000b0290148da0694ffso8020983wmj.2 for ; Fri, 07 May 2021 17:08:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=/8i2rA0Y8K4hQ70lLLGfR6UfKhpltRjIYWd7MBEBp4s=; b=L7gHS39+PKt6gtqfuReVNx5w76hv986Ui+QnrLAuxCJGQBPb70jaYr2bshKnSGjX82 4hXQv/qbl2wbrfCvzVOKRJc2lN3QQirONWYI8d5YrIEDMMsRwLDVBib/HV1iT6VGi+Lq rb0GL2jTgzVXkbof+VcVadH8OQiMTMYL4wDpaTDH7t/mayFDzo7cocBhRKI47QmQiIle CHzPZ2Lgd1v8FJSg9jGreUxMiy8vbz5wVTdB4m7PkjhEa1e54GnH+oq2nuj0UFZ7yNg1 eVaoRvGtqpa4hKLXcrWj2/7KDI4RneMprWYmUIlaUh5PU6pMWhLGzdjRv2qPXleBVzji vQ/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=/8i2rA0Y8K4hQ70lLLGfR6UfKhpltRjIYWd7MBEBp4s=; b=B7751tGeUljFkgjwkd4o8UrbiIJJYJxbaobFnStqgN49EnYkO3vqFZvLS63wUof395 7icpjBQ9ZLfDnAi2yaSCu+DMiMN30HCbZ9EuMm8KT5bB5PUll1Vsq1agyT/q7RqY7RD6 IYiu9S3N869FOsUYwIa0HUK3xrOPVpidIjEZnn+pU57wAVS1xIEjTmLM6MqvZw4CVcub /DMC6McN7IZliPd89OocX6bzcFZaSpH1BLPHInXpNhxOaegt6ueOa+GDA++Z/aMcEHEh HKHS2NYX7G3Tnnp6OHYzrre7YQT1VlJydJdhU9+SqrzW2CP3UsgfgZmXR7ARtwMrpVao yJCQ== X-Gm-Message-State: AOAM530y9aGndInYnIURwDdb0K7ztxVCTOmZaaNOQcRB/hNWXQvXKn+f ijCHkdQTnQd4TuaJAKik5KqcOU90fgE= X-Google-Smtp-Source: ABdhPJwzWYI4QK2pJByx03UzgT/5jbsNl5HM0tfjte2Ld83a+/ZSzQVUZOIE7YULK7z78PZt1AZQzw== X-Received: by 2002:a05:600c:4f93:: with SMTP id n19mr9219656wmq.100.1620432503133; Fri, 07 May 2021 17:08:23 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id v20sm8666679wmj.15.2021.05.07.17.08.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 May 2021 17:08:22 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Sat, 08 May 2021 00:08:14 +0000 Subject: [PATCH v2 2/8] t3001, t7300: add testcase showcasing missed directory traversal Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , Elijah Newren , Derrick Stolee , Jeff King , Elijah Newren , Elijah Newren Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Elijah Newren From: Elijah Newren In the last commit, we added a testcase showing that the directory traversal machinery sometimes traverses into directories unnecessarily. Here we show that there are cases where it does the opposite: it does not traverse into directories, despite those directories having important files that need to be flagged. Add a testcase showing that `git ls-files -o -i --directory` can omit some of the files it should be listing, and another showing that `git clean -fX` can fail to clean out some of the expected files. Signed-off-by: Elijah Newren --- t/t3001-ls-files-others-exclude.sh | 5 +++++ t/t7300-clean.sh | 19 +++++++++++++++++++ 2 files changed, 24 insertions(+) diff --git a/t/t3001-ls-files-others-exclude.sh b/t/t3001-ls-files-others-exclude.sh index 1ec7cb57c7a8..ac05d1a17931 100755 --- a/t/t3001-ls-files-others-exclude.sh +++ b/t/t3001-ls-files-others-exclude.sh @@ -292,6 +292,11 @@ EOF test_cmp expect actual ' +test_expect_failure 'ls-files with "**" patterns and --directory' ' + # Expectation same as previous test + git ls-files --directory -o -i --exclude "**/a.1" >actual && + test_cmp expect actual +' test_expect_success 'ls-files with "**" patterns and no slashes' ' git ls-files -o -i --exclude "one**a.1" >actual && diff --git a/t/t7300-clean.sh b/t/t7300-clean.sh index 5f1dc397c11e..337f9af1d74b 100755 --- a/t/t7300-clean.sh +++ b/t/t7300-clean.sh @@ -786,4 +786,23 @@ test_expect_failure 'avoid traversing into ignored directories' ' ) ' +test_expect_failure 'traverse into directories that may have ignored entries' ' + test_when_finished rm -f output && + test_create_repo need-to-traverse-into-hierarchy && + ( + cd need-to-traverse-into-hierarchy && + mkdir -p modules/foobar/src/generated && + > modules/foobar/src/generated/code.c && + > modules/foobar/Makefile && + echo "/modules/**/src/generated/" >.gitignore && + + git clean -fX modules/foobar >../output && + + grep Removing ../output && + + test_path_is_missing modules/foobar/src/generated/code.c && + test_path_is_file modules/foobar/Makefile + ) +' + test_done From patchwork Sat May 8 00:08:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elijah Newren X-Patchwork-Id: 12245385 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F1298C433ED for ; Sat, 8 May 2021 00:08:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A67E160E09 for ; Sat, 8 May 2021 00:08:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230254AbhEHAJ1 (ORCPT ); Fri, 7 May 2021 20:09:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43136 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230200AbhEHAJZ (ORCPT ); Fri, 7 May 2021 20:09:25 -0400 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DDF7EC061763 for ; Fri, 7 May 2021 17:08:24 -0700 (PDT) Received: by mail-wr1-x42c.google.com with SMTP id t18so10839008wry.1 for ; Fri, 07 May 2021 17:08:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=7bCsIv3hR/TUhuX0mUmDqE/q3hvqQQCp2a8FtFwo2KE=; b=sgVpLTQy+OhdzWEDRpPKnnHPb9xll5eY+fJDDNj76GfY3k3ckmZu3al332mNNMaPbd kaUp0HRXwGflstge1cv8mnHhngzbJe9fGNKpXOBVPwMIBrvpJh/BI5N3tfgdCEC516d9 WHjCc1Ij8gAwgVSBP0uk/ZWpDSZK5rB7prndiwkXJOKfdtjG2AR6K1qUKRpN6GC5h1Qu QTEpNC0oMxxUUnyUsMSwaunK6Rj7zwFLgVC7oUSUPG7vhhxk9qkRvIthLZLAkOP+CpnH V0qiAti7uMS6UzPt314r2wClzR044pKT3CUWRE5k8ntrLPclE24GAPzK5Ahg782R7QXG hQZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=7bCsIv3hR/TUhuX0mUmDqE/q3hvqQQCp2a8FtFwo2KE=; b=PXglkmocVq9tvdeR6tgoATzx6pRINl1jVEqbPtV69FPtj9/A/Pm6vV6Q2XTpV4MMqf MylYJeBhLdfcVTmk0V1fqhlYIJTuPz6Ykgc9H/vXqnjeH4loz2QJK3wEUoS+j+tLVDf5 UvnSFafKhHEymAT46xQZi5B5jDKkICaLTGHt2jxCBd83TujT+3kIhN7erPjyLHGNCtia E0yAC2hUt0ZT5iqeTITHPw92DGPH9A2oZtFKUZE3QSeWecvSRFicNjSEgIZBTuvdY8pz BdPibTnkQgaaAjWT3kOESK029uhHUk6HWSJS0WdOpR1TbIQniWqZPLq7K0y4DtQdltDo d2yQ== X-Gm-Message-State: AOAM532vPMczdwzhdR9u89s0CrBEasNH09IQkf1Uq2EBWdjuZi/Vr3ZZ sLgNzMnfLRhgfBm5q3NRSEQ/sty9yZA= X-Google-Smtp-Source: ABdhPJxBWPKdpktU2dTyrUpzutJtGaiE7DwqjFPBQDQs0pRB3zjM3g6VvsYExxAU1feh+Y7ECuH42g== X-Received: by 2002:a5d:6682:: with SMTP id l2mr15656752wru.15.1620432503666; Fri, 07 May 2021 17:08:23 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l21sm14471078wme.10.2021.05.07.17.08.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 May 2021 17:08:23 -0700 (PDT) Message-Id: <3c3f6111da13c885b341053a477d857bd897708f.1620432500.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 08 May 2021 00:08:15 +0000 Subject: [PATCH v2 3/8] dir: avoid unnecessary traversal into ignored directory Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , Elijah Newren , Derrick Stolee , Jeff King , Elijah Newren , Elijah Newren Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Elijah Newren From: Elijah Newren The show_other_directories case in treat_directory() tried to handle both excludes and untracked files with the same logic, and mishandled both the excludes and the untracked files in the process, in different ways. Split that logic apart, and then focus on the logic for the excludes; a subsequent commit will address the logic for untracked files. For show_other_directories, an excluded directory means that every path underneath that directory will also be excluded. Given that the calling code requested to just show directories when everything under a directory had the same state (that's what the "DIR_SHOW_OTHER_DIRECTORIES" flag means), we generally do not need to traverse into such directories and can just immediately mark them as ignored (i.e. as path_excluded). The only reason we cannot just immediately return path_excluded is the DIR_HIDE_EMPTY_DIRECTORIES flag and the possibility that the ignored directory is an empty directory. The code previously treated DIR_SHOW_IGNORED_TOO in most cases as an exception as well, which was wrong. It can sometimes reduce the number of cases where we need to recurse (namely if DIR_SHOW_IGNORED_TOO_MODE_MATCHING is also set), but should not be able to increase the number of cases where we need to recurse. Fix the logic accordingly. Some sidenotes about possible confusion with dir.c: * "ignored" often refers to an untracked ignore", i.e. a file which is not tracked which matches one of the ignore/exclusion rules. But you can also have a "tracked ignore", a tracked file that happens to match one of the ignore/exclusion rules and which dir.c has to worry about since "git ls-files -c -i" is supposed to list them. * The dir code often uses "ignored" and "excluded" interchangeably, which you need to keep in mind while reading the code. Sadly, though, it can get very confusing since ignore rules can have exclusions, as in the last of the following .gitignore rules: .gitignore *~ *.log !settings.log In the last entry above, (pathspec->items[3].magic & PATHSPEC_EXCLUDE) will be true due the the '!' negating the rule. Someone might refer to this as "excluded". That means the file 'settings.log' will not match, and thus not be ignored. So we won't return path_excluded for it. So it's an exclude rule that prevents the file from being an exclude. The non-excluded rules are the ones that result in files being excludes. Great fun, eh? Sometimes it feels like dir.c needs its own glossary with its many definitions, including the multiply-defined terms. Reported-by: Jason Gore Signed-off-by: Elijah Newren --- dir.c | 44 +++++++++++++++++++++++++++++--------------- t/t7300-clean.sh | 2 +- 2 files changed, 30 insertions(+), 16 deletions(-) diff --git a/dir.c b/dir.c index 3474e67e8f3c..4b183749843e 100644 --- a/dir.c +++ b/dir.c @@ -1844,6 +1844,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir, } /* This is the "show_other_directories" case */ + assert(dir->flags & DIR_SHOW_OTHER_DIRECTORIES); /* * If we have a pathspec which could match something _below_ this @@ -1854,27 +1855,40 @@ static enum path_treatment treat_directory(struct dir_struct *dir, if (matches_how == MATCHED_RECURSIVELY_LEADING_PATHSPEC) return path_recurse; + /* Special cases for where this directory is excluded/ignored */ + if (excluded) { + /* + * In the show_other_directories case, if we're not + * hiding empty directories, there is no need to + * recurse into an ignored directory. + */ + if (!(dir->flags & DIR_HIDE_EMPTY_DIRECTORIES)) + return path_excluded; + + /* + * Even if we are hiding empty directories, we can still avoid + * recursing into ignored directories for DIR_SHOW_IGNORED_TOO + * if DIR_SHOW_IGNORED_TOO_MODE_MATCHING is also set. + */ + if ((dir->flags & DIR_SHOW_IGNORED_TOO) && + (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING)) + return path_excluded; + } + /* - * Other than the path_recurse case immediately above, we only need - * to recurse into untracked/ignored directories if either of the - * following bits is set: + * Other than the path_recurse case above, we only need to + * recurse into untracked directories if either of the following + * bits is set: * - DIR_SHOW_IGNORED_TOO (because then we need to determine if * there are ignored entries below) * - DIR_HIDE_EMPTY_DIRECTORIES (because we have to determine if * the directory is empty) */ - if (!(dir->flags & (DIR_SHOW_IGNORED_TOO | DIR_HIDE_EMPTY_DIRECTORIES))) - return excluded ? path_excluded : path_untracked; - - /* - * ...and even if DIR_SHOW_IGNORED_TOO is set, we can still avoid - * recursing into ignored directories if the path is excluded and - * DIR_SHOW_IGNORED_TOO_MODE_MATCHING is also set. - */ - if (excluded && - (dir->flags & DIR_SHOW_IGNORED_TOO) && - (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING)) - return path_excluded; + if (!excluded && + !(dir->flags & (DIR_SHOW_IGNORED_TOO | + DIR_HIDE_EMPTY_DIRECTORIES))) { + return path_untracked; + } /* * Even if we don't want to know all the paths under an untracked or diff --git a/t/t7300-clean.sh b/t/t7300-clean.sh index 337f9af1d74b..00e5fa35dae3 100755 --- a/t/t7300-clean.sh +++ b/t/t7300-clean.sh @@ -746,7 +746,7 @@ test_expect_success 'clean untracked paths by pathspec' ' test_must_be_empty actual ' -test_expect_failure 'avoid traversing into ignored directories' ' +test_expect_success 'avoid traversing into ignored directories' ' test_when_finished rm -f output error && test_create_repo avoid-traversing-deep-hierarchy && ( From patchwork Sat May 8 00:08:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elijah Newren X-Patchwork-Id: 12245389 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43D48C433B4 for ; Sat, 8 May 2021 00:08:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 109DE61469 for ; Sat, 8 May 2021 00:08:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230291AbhEHAJ3 (ORCPT ); Fri, 7 May 2021 20:09:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43138 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230218AbhEHAJ0 (ORCPT ); Fri, 7 May 2021 20:09:26 -0400 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 63561C0613ED for ; Fri, 7 May 2021 17:08:25 -0700 (PDT) Received: by mail-wm1-x335.google.com with SMTP id u5-20020a7bc0450000b02901480e40338bso7274116wmc.1 for ; Fri, 07 May 2021 17:08:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=RCD1WrFJxMrytQ2QmONFIxMjkN4xZtTfbmf/jmcKbn0=; b=pY80lTBim6lH2aQXOf5Ud33JukxBO2tjGiUQlDf9E4gBBYS0oNzhd2Bc1naxIRm0L9 1+b4BI1cS25CqbJA/r4joAz0SM4cfPHPLdEcsQkbjdN0TGzZGGhozEoT2VHqcsdENrtt K6iTFIa/8Khn7l+R34nu74swIqVnBbO37I/2Kai+WE5a9es/FFOj1Qz2MA/E7FWKsJFq ubnY6vRrd2hY4/8nSI7YaymiaZTV07At5ypGMZcdeJuE0fUOE+JJwg6JqGG6NvA0DhfS 9iHISwg2Cf+Qjd6167TUKZgETYucm3fQY576hnCP+kPo/HvrEvRWze+hm5BvX5N9gcTD WOPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=RCD1WrFJxMrytQ2QmONFIxMjkN4xZtTfbmf/jmcKbn0=; b=bAPJYJXSxnFgVmMoij3KDtXEfQCn49XPlNprsmK19SLXDOPWwvqVld9hoZKREMPV/a k1OWfzckpVs2ttOwgDL3+tubj/nWP1tTEcjFk96OOUvWmYneaB7y2a33sCsBWkJ1HAnq rl4dChHP6Jq3ODp6F0xyOoT/FHtzO9EFRBgu2NZGy+cfk7zGsZ6GoEHDacXOuBHMhBNg PHKdnKFGdBeOtc68U1IPt8YWt1J2RycG6Wwu9qRL6BG+Kp5slX5pz5MZbEUwU3UTtcOd KPIPc7008u9XBhWJkjvLr5ZYvB+ertdqwdEHaiP3FMiriD188llG69K5v82VeGXucZep /YNw== X-Gm-Message-State: AOAM533v5JPQPBiyTcEt/XlFIjdp1uXP1TyN9EfnUdR6+nE5lQI66ga6 1IDDocEGrmI9B1NCnfxruddy8m0hEuY= X-Google-Smtp-Source: ABdhPJwrmRWz+08OJrJAGQlF8AeBUDUYm4K2aeyLJ0UEAdI0nPQPrshL8F5Et3gptuX2Red62o9sYA== X-Received: by 2002:a1c:a483:: with SMTP id n125mr8972535wme.159.1620432504155; Fri, 07 May 2021 17:08:24 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l18sm10315831wrx.96.2021.05.07.17.08.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 May 2021 17:08:23 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Sat, 08 May 2021 00:08:16 +0000 Subject: [PATCH v2 4/8] dir: traverse into untracked directories if they may have ignored subfiles Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , Elijah Newren , Derrick Stolee , Jeff King , Elijah Newren , Elijah Newren Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Elijah Newren From: Elijah Newren A directory that is untracked does not imply that all files under it should be categorized as untracked; in particular, if the caller is interested in ignored files, many files or directories underneath the untracked directory may be ignored. We previously partially handled this right with DIR_SHOW_IGNORED_TOO, but missed DIR_SHOW_IGNORED. It was not obvious, though, because the logic for untracked and excluded files had been fused together making it harder to reason about. The previous commit split that logic out, making it easier to notice that DIR_SHOW_IGNORED was missing. Add it. Signed-off-by: Elijah Newren --- dir.c | 10 ++++++---- t/t3001-ls-files-others-exclude.sh | 2 +- t/t7300-clean.sh | 2 +- 3 files changed, 8 insertions(+), 6 deletions(-) diff --git a/dir.c b/dir.c index 4b183749843e..3beb8e17a839 100644 --- a/dir.c +++ b/dir.c @@ -1877,15 +1877,17 @@ static enum path_treatment treat_directory(struct dir_struct *dir, /* * Other than the path_recurse case above, we only need to - * recurse into untracked directories if either of the following + * recurse into untracked directories if any of the following * bits is set: - * - DIR_SHOW_IGNORED_TOO (because then we need to determine if - * there are ignored entries below) + * - DIR_SHOW_IGNORED (because then we need to determine if + * there are ignored entries below) + * - DIR_SHOW_IGNORED_TOO (same as above) * - DIR_HIDE_EMPTY_DIRECTORIES (because we have to determine if * the directory is empty) */ if (!excluded && - !(dir->flags & (DIR_SHOW_IGNORED_TOO | + !(dir->flags & (DIR_SHOW_IGNORED | + DIR_SHOW_IGNORED_TOO | DIR_HIDE_EMPTY_DIRECTORIES))) { return path_untracked; } diff --git a/t/t3001-ls-files-others-exclude.sh b/t/t3001-ls-files-others-exclude.sh index ac05d1a17931..516c95ea0e82 100755 --- a/t/t3001-ls-files-others-exclude.sh +++ b/t/t3001-ls-files-others-exclude.sh @@ -292,7 +292,7 @@ EOF test_cmp expect actual ' -test_expect_failure 'ls-files with "**" patterns and --directory' ' +test_expect_success 'ls-files with "**" patterns and --directory' ' # Expectation same as previous test git ls-files --directory -o -i --exclude "**/a.1" >actual && test_cmp expect actual diff --git a/t/t7300-clean.sh b/t/t7300-clean.sh index 00e5fa35dae3..c2a3b7b6a52b 100755 --- a/t/t7300-clean.sh +++ b/t/t7300-clean.sh @@ -786,7 +786,7 @@ test_expect_success 'avoid traversing into ignored directories' ' ) ' -test_expect_failure 'traverse into directories that may have ignored entries' ' +test_expect_success 'traverse into directories that may have ignored entries' ' test_when_finished rm -f output && test_create_repo need-to-traverse-into-hierarchy && ( From patchwork Sat May 8 00:08:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elijah Newren X-Patchwork-Id: 12245391 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D5E1C433ED for ; Sat, 8 May 2021 00:08:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BA7CD6100C for ; Sat, 8 May 2021 00:08:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230266AbhEHAJa (ORCPT ); Fri, 7 May 2021 20:09:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43140 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230224AbhEHAJ0 (ORCPT ); Fri, 7 May 2021 20:09:26 -0400 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D501DC06138A for ; Fri, 7 May 2021 17:08:25 -0700 (PDT) Received: by mail-wm1-x330.google.com with SMTP id 82-20020a1c01550000b0290142562ff7c9so5737730wmb.3 for ; Fri, 07 May 2021 17:08:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=0e4RrCJuKuAV6s9KmC98iAx1qWZSkyg2U2VQQvqOO0E=; b=XRTM6MSSbaFWMPO1AA+FMSdIYfzmGm60Mq/8E2tSeGUoMoN2Bwt15HOY9lJzil1Ri9 23fe53NLsMX3FgOZYWWJfNndifq1kk5fkdiWokpr/oU3P6/5R3HvkZmrlHmjyCOa/dT4 tBR205sabla83bOFTFMvysqCvu46NxAW+exKjV8wjazpdRcl42vfDK3BlcMPBQbOTXwL tMRTOnLjD9s74/6jK1IateDBlFMi0Xq4hxwa6xuUgnr+8F0jXSFFC7xw12GcrqEnpyBm gS2tlF4o7w3/8hjx9Ai0FSYAyeGbRKZLg1AgUqGBd2/fvKNroYZehPWpW/2aOnjoXiDy 2Ptg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=0e4RrCJuKuAV6s9KmC98iAx1qWZSkyg2U2VQQvqOO0E=; b=j1KbGovksLq6c9/KKaG+7dpyj58jNa9y/0aGbPdceYMganuj8n2xK8yMm5G/QrSRc1 wRt55yjRAIOdE2tW3KrLz1JfKt8UZLye1ZKn7tIoXwzGrYWFbSsYUTcJXGr3yJXB0Llx +CD7ThbhuXr71ZilImq7wean1s0z6WxbyCsNzaf26CKrPCU1y3b3GlIBRHDReP1euh7M vdOC89hQXteb/OVrVfQ+YG0rqAXrEbfsAwB+OQTncmc6a6YTzW/Q3U2jUuujfu0SKnFX JPH/Fj2a/HhMNXGeT2zua1y1CIEr9pOHG4gElQpP9wjZUy1/T2p4F2xn6qJqshv4r1zB GWVA== X-Gm-Message-State: AOAM530Y5jO+5XITKpg63ln4fLotZIVtorMsGxRqF7UDI9vTRfuS2lk4 ggaKYogzCkC0rplZFxxm1i9HHLJSLmU= X-Google-Smtp-Source: ABdhPJyV2qr0EyR75Kn/bLEYTbpVPONXCkv/xV0qj+5NLhqrJtWlNYcAcd76BUtG7Veuw1MRfIBNHw== X-Received: by 2002:a7b:ce19:: with SMTP id m25mr12687101wmc.137.1620432504650; Fri, 07 May 2021 17:08:24 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id s21sm8060854wmc.10.2021.05.07.17.08.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 May 2021 17:08:24 -0700 (PDT) Message-Id: <3d8dd00ccd10d6b7ce9f928f31f8a04e885730fc.1620432501.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 08 May 2021 00:08:17 +0000 Subject: [PATCH v2 5/8] [RFC] ls-files: error out on -i unless -o or -c are specified Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , Elijah Newren , Derrick Stolee , Jeff King , Elijah Newren , Elijah Newren Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Elijah Newren From: Elijah Newren ls-files --ignored can be used together with either --others or --cached. After being perplexed for a bit and digging in to the code, I assumed that ls-files -i was just broken and not printing anything and had a nice patch ready to submit when I finally realized that -i can be used with --cached to find tracked ignores. While that was a mistake on my part, and a careful reading of the documentation could have made this more clear, I suspect this is an error others are likely to make as well. In fact, of two uses in our testsuite, I believe one of the two did make this error. In t1306.13, there are NO tracked files, and all the excludes built up and used in that test and in previous tests thus have to be about untracked files. However, since they were looking for an empty result, the mistake went unnoticed as their erroneous command also just happened to give an empty answer. -i will most the time be used with -o, which would suggest we could just make -i imply -o in the absence of either a -o or -c, but that would be a backward incompatible break. Instead, let's just flag -i without either a -o or -c as an error, and update the two relevant testcases to specify their intent. Signed-off-by: Elijah Newren --- builtin/ls-files.c | 3 +++ t/t1306-xdg-files.sh | 2 +- t/t3003-ls-files-exclude.sh | 4 ++-- 3 files changed, 6 insertions(+), 3 deletions(-) diff --git a/builtin/ls-files.c b/builtin/ls-files.c index 60a2913a01e9..9f74b1ab2e69 100644 --- a/builtin/ls-files.c +++ b/builtin/ls-files.c @@ -748,6 +748,9 @@ int cmd_ls_files(int argc, const char **argv, const char *cmd_prefix) if (pathspec.nr && error_unmatch) ps_matched = xcalloc(pathspec.nr, 1); + if ((dir.flags & DIR_SHOW_IGNORED) && !show_others && !show_cached) + die("ls-files --ignored is usually used with --others, but --cached is the default. Please specify which you want."); + if ((dir.flags & DIR_SHOW_IGNORED) && !exc_given) die("ls-files --ignored needs some exclude pattern"); diff --git a/t/t1306-xdg-files.sh b/t/t1306-xdg-files.sh index dd87b43be1a6..40d3c42618c0 100755 --- a/t/t1306-xdg-files.sh +++ b/t/t1306-xdg-files.sh @@ -116,7 +116,7 @@ test_expect_success 'Exclusion in a non-XDG global ignore file' ' test_expect_success 'Checking XDG ignore file when HOME is unset' ' (sane_unset HOME && git config --unset core.excludesfile && - git ls-files --exclude-standard --ignored >actual) && + git ls-files --exclude-standard --ignored --others >actual) && test_must_be_empty actual ' diff --git a/t/t3003-ls-files-exclude.sh b/t/t3003-ls-files-exclude.sh index d5ec333131f9..c41c4f046abf 100755 --- a/t/t3003-ls-files-exclude.sh +++ b/t/t3003-ls-files-exclude.sh @@ -29,11 +29,11 @@ test_expect_success 'add file to gitignore' ' ' check_all_output -test_expect_success 'ls-files -i lists only tracked-but-ignored files' ' +test_expect_success 'ls-files -i -c lists only tracked-but-ignored files' ' echo content >other-file && git add other-file && echo file >expect && - git ls-files -i --exclude-standard >output && + git ls-files -i -c --exclude-standard >output && test_cmp expect output ' From patchwork Sat May 8 00:08:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12245393 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CD522C43461 for ; Sat, 8 May 2021 00:08:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8EC376145D for ; Sat, 8 May 2021 00:08:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230280AbhEHAJb (ORCPT ); Fri, 7 May 2021 20:09:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43150 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230236AbhEHAJ1 (ORCPT ); Fri, 7 May 2021 20:09:27 -0400 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 71D88C061574 for ; Fri, 7 May 2021 17:08:26 -0700 (PDT) Received: by mail-wr1-x42f.google.com with SMTP id x5so10819119wrv.13 for ; Fri, 07 May 2021 17:08:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=1ThQwWIUnfUmXZXm3iHsPscyieySk2PI+0bWDitXriE=; b=nbtFsgyrqwSuMNOrrX2DQMWiAYP94DmyCygTkfYnMrCMbOAVQNPSpZ5X/igUoZVrgU 31dvNDcy4vm5sxfJ9ZRqE31KKJHEm7vKUK770dr4zQKPNlsvJK3xyenaIKlkbSu0vwKV SORXDZ6uftRGFdQfxOLtf/NyuEvfX0nNUX3byu0KMpWPhv6nX2U+HPkcfISqzkVKvlSS H7LCv2R7LBvyyiQyvEi3M9Ihyfyp4wzvnMHYIWZJgMUm7uEX9O9xeLD5LyqeNAtAu8C/ 9HAAho21xUA6sls9Zo54hQtnB7wEPtVeokt8jMhmGP0wyObUSnunisKXcYiSd6NX3IsD qIVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=1ThQwWIUnfUmXZXm3iHsPscyieySk2PI+0bWDitXriE=; b=PEvoiL1spXxNrTB0x4M9kf2zQ+yO+vrvbI6hwnpvyw9Y0cSKrpReRMcihNSlXky5qu zl9aiPBBTDgH5p3pb5WHOukLLoVR2G4IMez8oQfz6c+h5Nf6pIxHhXdjkz8pouBysNST DgA6LdXolGXmk2/YvKI1RgntApeoGO3PbBJsGfn2ZyhjJjqLlNU/3yvGvgwrl/eEn1oE 3Cp2nYaW1M1sgYOq38GSaLxy7HII9yAG8aQkolFfn4yzW9FNwsaIG/iup2GSWypGRpyg 7uRABHItelLAYRHYUWlAUmBrRPsGsEIEieWIkxSlcOjn73a6U+FOL/Fm1jxgRkjyvauw FD3A== X-Gm-Message-State: AOAM530nK5vDelDICgIJxCybXeI4nFvVyA7c+Tz0kAq+qRuHHEHofsuI 5qVUWn3h2tgVGoNgE34+k/w2V0G5ErU= X-Google-Smtp-Source: ABdhPJyuRoKmtTXvyZqCiCJ9kpffA9XXplPoMJrq+id3CiJ6vtRIChY64n9hC9A1wmThlsBhpBQIUA== X-Received: by 2002:a5d:648e:: with SMTP id o14mr15247692wri.27.1620432505233; Fri, 07 May 2021 17:08:25 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l18sm10315884wrx.96.2021.05.07.17.08.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 May 2021 17:08:24 -0700 (PDT) Message-Id: <1d825dfdc70b8c658c4b6317310706bb6386f468.1620432501.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 08 May 2021 00:08:18 +0000 Subject: [PATCH v2 6/8] dir: update stale description of treat_directory() Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , Elijah Newren , Derrick Stolee , Jeff King , Elijah Newren , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The documentation comment for treat_directory() was originally written in 095952 (Teach directory traversal about subprojects, 2007-04-11) which was before the 'struct dir_struct' split its bitfield of named options into a 'flags' enum in 7c4c97c0 (Turn the flags in struct dir_struct into a single variable, 2009-02-16). When those flags changed, the comment became stale, since members like 'show_other_directories' transitioned into flags like DIR_SHOW_OTHER_DIRECTORIES. Update the comments for treat_directory() to use these flag names rather than the old member names. Signed-off-by: Derrick Stolee Reviewed-by: Elijah Newren --- dir.c | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/dir.c b/dir.c index 3beb8e17a839..0a0138bc1aa6 100644 --- a/dir.c +++ b/dir.c @@ -1749,13 +1749,13 @@ static enum exist_status directory_exists_in_index(struct index_state *istate, * Case 3: if we didn't have it in the index previously, we * have a few sub-cases: * - * (a) if "show_other_directories" is true, we show it as - * just a directory, unless "hide_empty_directories" is + * (a) if DIR_SHOW_OTHER_DIRECTORIES flag is set, we show it as + * just a directory, unless DIR_HIDE_EMPTY_DIRECTORIES is * also true, in which case we need to check if it contains any * untracked and / or ignored files. - * (b) if it looks like a git directory, and we don't have - * 'no_gitlinks' set we treat it as a gitlink, and show it - * as a directory. + * (b) if it looks like a git directory and we don't have the + * DIR_NO_GITLINKS flag, then we treat it as a gitlink, and + * show it as a directory. * (c) otherwise, we recurse into it. */ static enum path_treatment treat_directory(struct dir_struct *dir, @@ -1843,7 +1843,6 @@ static enum path_treatment treat_directory(struct dir_struct *dir, return path_recurse; } - /* This is the "show_other_directories" case */ assert(dir->flags & DIR_SHOW_OTHER_DIRECTORIES); /* @@ -1858,7 +1857,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir, /* Special cases for where this directory is excluded/ignored */ if (excluded) { /* - * In the show_other_directories case, if we're not + * If DIR_SHOW_OTHER_DIRECTORIES is set and we're not * hiding empty directories, there is no need to * recurse into an ignored directory. */ From patchwork Sat May 8 00:08:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elijah Newren X-Patchwork-Id: 12245395 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A58EBC433B4 for ; Sat, 8 May 2021 00:08:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 729736145E for ; Sat, 8 May 2021 00:08:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230303AbhEHAJb (ORCPT ); Fri, 7 May 2021 20:09:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43154 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230267AbhEHAJ2 (ORCPT ); Fri, 7 May 2021 20:09:28 -0400 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 09D26C061574 for ; Fri, 7 May 2021 17:08:27 -0700 (PDT) Received: by mail-wm1-x32e.google.com with SMTP id p14-20020a05600c358eb029015c01f207d7so15996wmq.5 for ; Fri, 07 May 2021 17:08:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=rYNxJzdZdKYwECH2HKImBF+CpT8pMjlB0AmKWi9ihcM=; b=ghruqQaFTs9NysnyJjk3TBY5OwCtgCyAGgqkQAq3mMBLxa/mjvGVrpIg6EJzGqqPRR M5wP5oiYXAwlZ8yRhbARdcydzLft5rGaiqNrYgT/PupXtjinwVXEfBZLrHyBcObniQEA sq63OPGs5ULZJRmi6TWgiGqUBs9XhlDjg2Vwx2bou4cLutpeNALllmVISCd6R6Z6RbOB 1zch9S6ENYpVUjpZEGJhgkvEgy5UKGFXYq2lZ2IjFrtK0tCgLqNBi7lY9iwOyf3JXsEu DoTksAgN1em0749aLfpEJs1zPlbQJysS9dRdQvarcTt8poS+C3oxIQM+wWDBD6nhpXHQ zZNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=rYNxJzdZdKYwECH2HKImBF+CpT8pMjlB0AmKWi9ihcM=; b=Lf5aHICrm9VqjtfkLH+O5ADQGMrCauVXSH5kUlRhdiKjTpramT2MTnmwxQbLOwvnQZ FQ1b9ZrU+oTsiNMm6du9nsHZ3/NCbh1Pg782On5HA/VYpAE22MUW1XezCoUlRSVFr8Z3 8//Kpcb8qC7RMo3VTbi1KFPFq5SncWgcAxLgQWvkmAYKiVbUMv/xldL1K/RSyltl5t57 CEEN/a3ezYrRx2eITwvUdj3QqQ+0DKEVECZtwpgdMOYRWcEJVFPEeFjyZFzU37oYPoBv F3Y/vB/G1JnnSyjwnpKg7j7JGwE4JPYjKxAdWMh1BxGe8Sn7Rx76pb70n9F+Q59F1UrT 6fcg== X-Gm-Message-State: AOAM533rYFSZ4ZEXwGQUXsRDZUEX/eT/7TWUiUwwbv30wCjHJymwY+Xq XkzRekthNzdXkIC8jqwxIbW1aWsoY4k= X-Google-Smtp-Source: ABdhPJxo56ap8+kU8NINGcGrGzWDkW9KsZB1dTyUUdbifQMH3Y6c08oVWEHZtlUWynqdy2ixYDN0dQ== X-Received: by 2002:a05:600c:4f8b:: with SMTP id n11mr12870689wmq.180.1620432505817; Fri, 07 May 2021 17:08:25 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n5sm10453341wrx.31.2021.05.07.17.08.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 May 2021 17:08:25 -0700 (PDT) Message-Id: <3a2394506a536707788744a8b082077a017f8c8c.1620432501.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Sat, 08 May 2021 00:08:19 +0000 Subject: [PATCH v2 7/8] [RFC] dir: convert trace calls to trace2 equivalents Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , Elijah Newren , Derrick Stolee , Jeff King , Elijah Newren , Elijah Newren Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Elijah Newren From: Elijah Newren Signed-off-by: Elijah Newren --- dir.c | 34 ++++-- t/t7063-status-untracked-cache.sh | 193 +++++++++++++++++------------- t/t7519-status-fsmonitor.sh | 8 +- 3 files changed, 135 insertions(+), 100 deletions(-) diff --git a/dir.c b/dir.c index 0a0138bc1aa6..23c71ab7e9a1 100644 --- a/dir.c +++ b/dir.c @@ -2775,12 +2775,29 @@ static struct untracked_cache_dir *validate_untracked_cache(struct dir_struct *d return root; } +static void trace2_read_directory_statistics(struct dir_struct *dir, + struct repository *repo) +{ + if (!dir->untracked) + return; + trace2_data_intmax("read_directory", repo, + "node-creation", dir->untracked->dir_created); + trace2_data_intmax("read_directory", repo, + "gitignore-invalidation", + dir->untracked->gitignore_invalidated); + trace2_data_intmax("read_directory", repo, + "directory-invalidation", + dir->untracked->dir_invalidated); + trace2_data_intmax("read_directory", repo, + "opendir", dir->untracked->dir_opened); +} + int read_directory(struct dir_struct *dir, struct index_state *istate, const char *path, int len, const struct pathspec *pathspec) { struct untracked_cache_dir *untracked; - trace_performance_enter(); + trace2_region_enter("dir", "read_directory", istate->repo); if (has_symlink_leading_path(path, len)) { trace_performance_leave("read directory %.*s", len, path); @@ -2799,23 +2816,13 @@ int read_directory(struct dir_struct *dir, struct index_state *istate, QSORT(dir->entries, dir->nr, cmp_dir_entry); QSORT(dir->ignored, dir->ignored_nr, cmp_dir_entry); - trace_performance_leave("read directory %.*s", len, path); + trace2_region_leave("dir", "read_directory", istate->repo); if (dir->untracked) { static int force_untracked_cache = -1; - static struct trace_key trace_untracked_stats = TRACE_KEY_INIT(UNTRACKED_STATS); if (force_untracked_cache < 0) force_untracked_cache = git_env_bool("GIT_FORCE_UNTRACKED_CACHE", 0); - trace_printf_key(&trace_untracked_stats, - "node creation: %u\n" - "gitignore invalidation: %u\n" - "directory invalidation: %u\n" - "opendir: %u\n", - dir->untracked->dir_created, - dir->untracked->gitignore_invalidated, - dir->untracked->dir_invalidated, - dir->untracked->dir_opened); if (force_untracked_cache && dir->untracked == istate->untracked && (dir->untracked->dir_opened || @@ -2826,6 +2833,9 @@ int read_directory(struct dir_struct *dir, struct index_state *istate, FREE_AND_NULL(dir->untracked); } } + + if (trace2_is_enabled()) + trace2_read_directory_statistics(dir, istate->repo); return dir->nr; } diff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh index accefde72fb1..6bce65b439e3 100755 --- a/t/t7063-status-untracked-cache.sh +++ b/t/t7063-status-untracked-cache.sh @@ -57,6 +57,19 @@ iuc () { return $ret } +get_relevant_traces() { + # From the GIT_TRACE2_PERF data of the form + # $TIME $FILE:$LINE | d0 | main | data | r1 | ? | ? | read_directo | $RELEVANT_STAT + # extract the $RELEVANT_STAT fields. We don't care about region_enter + # or region_leave, or stats for things outside read_directory. + INPUT_FILE=$1 + OUTPUT_FILE=$2 + grep data.*read_directo $INPUT_FILE \ + | cut -d "|" -f 9 \ + >$OUTPUT_FILE +} + + test_lazy_prereq UNTRACKED_CACHE ' { git update-index --test-untracked-cache; ret=$?; } && test $ret -ne 1 @@ -129,19 +142,20 @@ EOF test_expect_success 'status first time (empty cache)' ' avoid_racy && - : >../trace && - GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \ + : >../trace.output && + GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \ git status --porcelain >../actual && iuc status --porcelain >../status.iuc && test_cmp ../status.expect ../status.iuc && test_cmp ../status.expect ../actual && + get_relevant_traces ../trace.output ../trace.relevant && cat >../trace.expect <../trace && - GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \ + : >../trace.output && + GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \ git status --porcelain >../actual && iuc status --porcelain >../status.iuc && test_cmp ../status.expect ../status.iuc && test_cmp ../status.expect ../actual && + get_relevant_traces ../trace.output ../trace.relevant && cat >../trace.expect <four && - : >../trace && - GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \ + : >../trace.output && + GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \ git status --porcelain >../actual && iuc status --porcelain >../status.iuc && cat >../status.expect <../trace.expect <.gitignore && - : >../trace && - GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \ + : >../trace.output && + GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \ git status --porcelain >../actual && iuc status --porcelain >../status.iuc && cat >../status.expect <../trace.expect <>.git/info/exclude && - : >../trace && - GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \ + : >../trace.output && + GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \ git status --porcelain >../actual && iuc status --porcelain >../status.iuc && cat >../status.expect <../trace.expect <../trace && - GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \ + : >../trace.output && + GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \ git status --porcelain >../actual && iuc status --porcelain >../status.iuc && cat >../status.expect <../trace.expect <../trace && - GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \ + : >../trace.output && + GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \ git status --porcelain >../actual && iuc status --porcelain >../status.iuc && cat >../status.expect <../trace.expect <../trace && - GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \ + : >../trace.output && + GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \ git status --porcelain >../actual && iuc status --porcelain >../status.iuc && cat >../status.expect <../trace.expect <../trace && + : >../trace.output && avoid_racy && - GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \ + GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \ git status --porcelain >../status.actual && iuc status --porcelain >../status.iuc && cat >../status.expect <../trace.expect <../trace && - GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \ + : >../trace.output && + GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \ git status --porcelain >../status.actual && iuc status --porcelain >../status.iuc && cat >../status.expect <../trace.expect <../trace && - GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \ + : >../trace.output && + GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \ git status --porcelain >../status.actual && iuc status --porcelain >../status.iuc && cat >../status.expect <../trace.expect <../trace && - GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace" \ + : >../trace.output && + GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \ git status --porcelain >../status.actual && iuc status --porcelain >../status.iuc && test_cmp ../status.expect ../status.iuc && test_cmp ../status.expect ../status.actual && + get_relevant_traces ../trace.output ../trace.relevant && cat >../trace.expect <../before ) && @@ -346,12 +346,12 @@ test_expect_success UNTRACKED_CACHE 'ignore .git changes when invalidating UNTR' EOF ( cd dot-git && - GIT_TRACE_UNTRACKED_STATS="$TRASH_DIRECTORY/trace-after" \ + GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace-after" \ git status && test-tool dump-untracked-cache >../after ) && - grep "directory invalidation" trace-before >>before && - grep "directory invalidation" trace-after >>after && + grep "directory-invalidation" trace-before | cut -d"|" -f 9 >>before && + grep "directory-invalidation" trace-after | cut -d"|" -f 9 >>after && # UNTR extension unchanged, dir invalidation count unchanged test_cmp before after ' From patchwork Sat May 8 00:08:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elijah Newren X-Patchwork-Id: 12245397 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD261C433ED for ; Sat, 8 May 2021 00:08:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 852676145D for ; Sat, 8 May 2021 00:08:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230294AbhEHAJe (ORCPT ); Fri, 7 May 2021 20:09:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43156 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230289AbhEHAJ3 (ORCPT ); Fri, 7 May 2021 20:09:29 -0400 Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1F377C061763 for ; Fri, 7 May 2021 17:08:28 -0700 (PDT) Received: by mail-wm1-x331.google.com with SMTP id t11-20020a05600c198bb02901476e13296aso5754251wmq.0 for ; Fri, 07 May 2021 17:08:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=jESdGkKuLXbijtDvf5e2HewiIfotNeXrl5Hx64zKT1g=; b=pEmI5zCdrSAWjlyKMdDRuG+brRomTnBdoUfbYt6tu44s0qylSEJ6HfBRE26H/Npx+4 N0QNR9GQ8UMeNxFM41QdyDbH7UHNtmpOGtunCfO+ii513MWHc7yaarhi63HqXwGFulsF 6dPjdzH3lDleKx7y+LZFy2foh4OI0CXiGBO8noujQXa20simdu9gclSXssATenoVS3zU haOH0rXLnzY4MF9ZHUt3xnndt9BzTKU/TCMYGauD+ctCdHXT3/LNPkuBecacduosRK6Q iBfqNKya7BbIQ5y3+WRgP6lpmBrW+CVm+40x4pvHuHKQDaFWk+EJ3AhccxVGfI+FOhiD bzOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=jESdGkKuLXbijtDvf5e2HewiIfotNeXrl5Hx64zKT1g=; b=EgQjVS+MwxBOC0Y0rY8ON4IReIpEM2ZryOY7O5GiUtLJaeaIsi9WlHLIVMmTbwn8Wb pBk5JSdvl7I9SxCeUXx11keZ5R9JPYZW0x0HbNC8jEKwvSiwVmu8G7Irh+cUKLXPv+2X ISpTwh6oOu7aEb9WaCVVrly6wUkKCy8ARDyqk8Auc2984nsxnSYAAPsI1vJPb8q7LvS1 t4CqdgFGcOns2z6qhEqN6wL8pWh0vogD+Z/plWq9SgXvPjFHENdYRyqLxk5W67Xrg2uQ ih3VCkRDvwSv7WwH+JgMAe+lqI+FOA3jPJ/v74Ds5a1q7D6QxYB36LqoIEFKjgGsjf7E PbYA== X-Gm-Message-State: AOAM531QJ46h89meYyh2SFsaoJBXfkS8i8hJh03Ir4b2L93leSuhH1ci NPDWNE+IoBubYbAcdFtcnPb8bv0C1HQ= X-Google-Smtp-Source: ABdhPJzkuMbf1alFg4+T2izfkOf5reantovmwWdiM1bgNPRifVJGWvzVKoEO2Ct23PpdqbMGWzmTxA== X-Received: by 2002:a7b:c74d:: with SMTP id w13mr12448234wmk.25.1620432506927; Fri, 07 May 2021 17:08:26 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m6sm10655639wrp.29.2021.05.07.17.08.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 May 2021 17:08:26 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Sat, 08 May 2021 00:08:20 +0000 Subject: [PATCH v2 8/8] [RFC] dir: reported number of visited directories and paths with trace2 Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , Elijah Newren , Derrick Stolee , Jeff King , Elijah Newren , Elijah Newren Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Elijah Newren From: Elijah Newren Previously, tests that wanted to verify that we don't traverse into a deep directory hierarchy that is ignored had no easy way to verify and enforce that behavior. Record information about the number of directories and paths we inspect while traversing the directory hierarchy in read_directory(), and when trace2 is enabled, print these statistics. Make use of these statistics in t7300 to simplify (and vastly improve the performance of) the "avoid traversing into ignored directories" test. Signed-off-by: Elijah Newren --- dir.c | 8 ++++++ dir.h | 4 +++ t/t7063-status-untracked-cache.sh | 1 + t/t7300-clean.sh | 46 ++++++++++--------------------- 4 files changed, 27 insertions(+), 32 deletions(-) diff --git a/dir.c b/dir.c index 23c71ab7e9a1..896a9a62b2c7 100644 --- a/dir.c +++ b/dir.c @@ -2455,6 +2455,7 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir, if (open_cached_dir(&cdir, dir, untracked, istate, &path, check_only)) goto out; + dir->visited_directories++; if (untracked) untracked->check_only = !!check_only; @@ -2463,6 +2464,7 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir, /* check how the file or directory should be treated */ state = treat_path(dir, untracked, &cdir, istate, &path, baselen, pathspec); + dir->visited_paths++; if (state > dir_state) dir_state = state; @@ -2778,6 +2780,10 @@ static struct untracked_cache_dir *validate_untracked_cache(struct dir_struct *d static void trace2_read_directory_statistics(struct dir_struct *dir, struct repository *repo) { + trace2_data_intmax("read_directory", repo, + "directories-visited", dir->visited_directories); + trace2_data_intmax("read_directory", repo, + "paths-visited", dir->visited_paths); if (!dir->untracked) return; trace2_data_intmax("read_directory", repo, @@ -2798,6 +2804,8 @@ int read_directory(struct dir_struct *dir, struct index_state *istate, struct untracked_cache_dir *untracked; trace2_region_enter("dir", "read_directory", istate->repo); + dir->visited_paths = 0; + dir->visited_directories = 0; if (has_symlink_leading_path(path, len)) { trace_performance_leave("read directory %.*s", len, path); diff --git a/dir.h b/dir.h index 04d886cfce75..22c67907f689 100644 --- a/dir.h +++ b/dir.h @@ -336,6 +336,10 @@ struct dir_struct { struct oid_stat ss_info_exclude; struct oid_stat ss_excludes_file; unsigned unmanaged_exclude_files; + + /* Stats about the traversal */ + unsigned visited_paths; + unsigned visited_directories; }; /*Count the number of slashes for string s*/ diff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh index 6bce65b439e3..1517c316892f 100755 --- a/t/t7063-status-untracked-cache.sh +++ b/t/t7063-status-untracked-cache.sh @@ -65,6 +65,7 @@ get_relevant_traces() { INPUT_FILE=$1 OUTPUT_FILE=$2 grep data.*read_directo $INPUT_FILE \ + | grep -v visited \ | cut -d "|" -f 9 \ >$OUTPUT_FILE } diff --git a/t/t7300-clean.sh b/t/t7300-clean.sh index c2a3b7b6a52b..2c10a7b64f11 100755 --- a/t/t7300-clean.sh +++ b/t/t7300-clean.sh @@ -747,42 +747,24 @@ test_expect_success 'clean untracked paths by pathspec' ' ' test_expect_success 'avoid traversing into ignored directories' ' - test_when_finished rm -f output error && + test_when_finished rm -f output error trace.* && test_create_repo avoid-traversing-deep-hierarchy && ( cd avoid-traversing-deep-hierarchy && - >directory-random-file.txt && - # Put this file under directory400/directory399/.../directory1/ - depth=400 && - for x in $(test_seq 1 $depth); do - mkdir "tmpdirectory$x" && - mv directory* "tmpdirectory$x" && - mv "tmpdirectory$x" "directory$x" - done && - - git clean -ffdxn -e directory$depth >../output 2>../error && - - test_must_be_empty ../output && - # We especially do not want things like - # "warning: could not open directory " - # appearing in the error output. It is true that directories - # that are too long cannot be opened, but we should not be - # recursing into those directories anyway since the very first - # level is ignored. - test_must_be_empty ../error && - - # alpine-linux-musl fails to "rm -rf" a directory with such - # a deeply nested hierarchy. Help it out by deleting the - # leading directories ourselves. Super slow, but, what else - # can we do? Without this, we will hit a - # error: Tests passed but test cleanup failed; aborting - # so do this ugly manual cleanup... - while test ! -f directory-random-file.txt; do - name=$(ls -d directory*) && - mv $name/* . && - rmdir $name - done + mkdir -p untracked/subdir/with/a && + >untracked/subdir/with/a/random-file.txt && + + GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace.output" \ + git clean -ffdxn -e untracked && + + grep data.*read_directo.*visited ../trace.output \ + | cut -d "|" -f 9 >../trace.relevant && + cat >../trace.expect <<-EOF && + directories-visited:1 + paths-visited:4 + EOF + test_cmp ../trace.expect ../trace.relevant ) '