From patchwork Thu Nov 3 23:05:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anh Le X-Patchwork-Id: 13031108 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EAD49C433FE for ; Thu, 3 Nov 2022 23:05:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231379AbiKCXFZ (ORCPT ); Thu, 3 Nov 2022 19:05:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40870 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231328AbiKCXFG (ORCPT ); Thu, 3 Nov 2022 19:05:06 -0400 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C6D86DF13 for ; Thu, 3 Nov 2022 16:05:05 -0700 (PDT) Received: by mail-wr1-x436.google.com with SMTP id k8so4885857wrh.1 for ; Thu, 03 Nov 2022 16:05:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=D974zv1IGlEZeh+8MXjYMZ76BSH79SkJr+0VkSUuDz8=; b=N8o37F/DGkS6Bm8kJxH53VyPxIxBuMSUeu66MVDz5vVEDNJ9EsHVn3NiurxUX9zr+4 uXlqG8SiR83Co7DQo6211fGXgayOfBO5a/1TZGHYfi/Xym18wlkmye3v8BWf7CITpKjj 9f/5BIM5YbpP/MhGKAk70roK+zTGx+lls8pulCukNKSiQc1BdK9VsBetj6bkiQiplbTy VydT3MWQ85Ez25prCdELuR+QPwYp412N84mOiROEPpYa2hhiMw+fQaRAQOXqTSo9SD8u KZoAOX3Ilx06g/PTlwQM1vLybC8E1pj5sYHB9zT7X9/0m7fypuu5fCE84WRNV2ozcf9d zdSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=D974zv1IGlEZeh+8MXjYMZ76BSH79SkJr+0VkSUuDz8=; b=RkJ2r1hDc889FJL3TMHvINJIpb44fvj/0VyikkdreXlVA7z9RbI7nmeczgdZkF0vN9 +4wJWq7zbkVjKkAHG6dM38LLcX9+zlomwPo7ZAI+c1WQMGk9p9KJKLtMMe1dLaKOtOls LwTEJh8VYWW3UwHHmLqsitLb1TvNyuuxm1SVLbCJ6uggdF5KHLwtZmfYLOPnf/sbVyRR RmNS//c+NdLy5F7Stwa7mKwXZFvd609VCeHUDVSyoLQchFM1nQx2aVUteM8wGilk3Rfb /XwPEYpogIP3fyCabGaIUeMW1IQV2GmAQiEu96t5K3PkZh1i4jVQQ5yZQE48HVsuIfSd Z3Pw== X-Gm-Message-State: ACrzQf2huOceki7cmSua2PUu3TudYKJHVYYKRxoyuv51jVJ4WL4y1yRd HrIuN4Dfrc87YcChq7cx+TCobpivHFc= X-Google-Smtp-Source: AMsMyM4YO6mKUi3iGeCygjvzGhSEOKSfbtoHTwalvzBUHx57EUXiea0BkeUHlMoY4KJtfLvbnVOv5Q== X-Received: by 2002:adf:de8d:0:b0:236:6087:e07e with SMTP id w13-20020adfde8d000000b002366087e07emr20489652wrl.533.1667516704085; Thu, 03 Nov 2022 16:05:04 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id h8-20020a05600c350800b003c6f426467fsm1129073wmq.40.2022.11.03.16.05.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 03 Nov 2022 16:05:03 -0700 (PDT) Message-Id: <33e9b2afd91f4376ef9a64fd267fa61f1a8de07f.1667516701.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 03 Nov 2022 23:05:00 +0000 Subject: [PATCH v4 1/2] index: add trace2 region for clear skip worktree Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Timothy Jones , Jeff Hostetler , Jeff Hostetler , Derrick Stolee , Taylor Blau , Anh Le , Anh Le Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Anh Le From: Anh Le When using sparse checkout, clear_skip_worktree_from_present_files() must enumerate index entries to find ones with the SKIP_WORKTREE bit to determine whether those index entries exist on disk (in which case their SKIP_WORKTREE bit should be removed). In a large repository, this may take considerable time depending on the size of the index. Add a trace2 region to surface this information, keeping a count of how many paths have been checked. Separately, keep counts after a full index is materialized. Signed-off-by: Anh Le --- sparse-index.c | 28 ++++++++++++++++++++++------ 1 file changed, 22 insertions(+), 6 deletions(-) diff --git a/sparse-index.c b/sparse-index.c index e4a54ce1943..8713a15611d 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -493,24 +493,40 @@ void clear_skip_worktree_from_present_files(struct index_state *istate) int dir_found = 1; int i; + int path_count[2] = {0, 0}; + int restarted = 0; if (!core_apply_sparse_checkout || sparse_expect_files_outside_of_patterns) return; + trace2_region_enter("index", "clear_skip_worktree_from_present_files", + istate->repo); restart: for (i = 0; i < istate->cache_nr; i++) { struct cache_entry *ce = istate->cache[i]; - if (ce_skip_worktree(ce) && - path_found(ce->name, &last_dirname, &dir_len, &dir_found)) { - if (S_ISSPARSEDIR(ce->ce_mode)) { - ensure_full_index(istate); - goto restart; + if (ce_skip_worktree(ce)) { + path_count[restarted]++; + if (path_found(ce->name, &last_dirname, &dir_len, &dir_found)) { + if (S_ISSPARSEDIR(ce->ce_mode)) { + ensure_full_index(istate); + restarted = 1; + goto restart; + } + ce->ce_flags &= ~CE_SKIP_WORKTREE; } - ce->ce_flags &= ~CE_SKIP_WORKTREE; } } + + if (path_count[0]) + trace2_data_intmax("index", istate->repo, + "sparse_path_count", path_count[0]); + if (restarted) + trace2_data_intmax("index", istate->repo, + "sparse_path_count_full", path_count[1]); + trace2_region_leave("index", "clear_skip_worktree_from_present_files", + istate->repo); } /* From patchwork Thu Nov 3 23:05:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anh Le X-Patchwork-Id: 13031107 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC4B0C433FE for ; Thu, 3 Nov 2022 23:05:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231497AbiKCXFV (ORCPT ); Thu, 3 Nov 2022 19:05:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41022 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231354AbiKCXFN (ORCPT ); Thu, 3 Nov 2022 19:05:13 -0400 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AE1801F2FA for ; Thu, 3 Nov 2022 16:05:06 -0700 (PDT) Received: by mail-wr1-x436.google.com with SMTP id cl5so4825553wrb.9 for ; Thu, 03 Nov 2022 16:05:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=yqS3P2hMx5OGT369nE6Yevhwy/bc0bvFdSzfLAB2Joc=; b=GzcRdtBTV0SuXRojq3E5OLngZv3F/2VXru39UhpfCm7MCCkZVkOZl4buglOrnLiBd8 pE9uiLTs7mHrk7DGWAoZkXvwAThJycXzIPjfCuzHzM6P7i4f0yyeeLUpXffBB1UJDTpz R1N2rBwgt8wC7MEY8vHIkI7NlgwKf/N/7d1Zv7doaExD8V6HOnLMvoqvM2I7kaLG6U7Y zEgqSw+lJ0r4AU5lljr7P0l2B2gy63NDFFfKLreoC5f02ObzTpPjAQu85uD3rFss4MYE tzh+7aGEFuhTqv7aE6/RmUW4Dnub91hlJeZP4cISDcJX78LiGJb4+jVGGdzPbUr9ebBx Gukg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yqS3P2hMx5OGT369nE6Yevhwy/bc0bvFdSzfLAB2Joc=; b=G7JQU1VXlC0ntrrsxjJ8FEZjEFUhEpEZk6d7mSS/XhRGsBeJBVItcpugUbJ8AiRydW xHXH82b8vQbHpyYWFW+wBrU1+O0juQ0DScCwmDodCNPBa8smPhAa/ZP14jRaA7T3q9wJ y15JwwEdwV6+3EIrQJCbEfyS2OSupzm36oByniBQdI9JvshZ44D55dvvKEkpu1a/IOfw mWkbTs6rVtN1UfqiG1cpKdvR4YoUXcyt6gyN/eUc6dLxAFoWPWzJI2MhBZMhiKIwmNFM SsTdLfkbqJ6XIxNkbTbQi0ms0rxEizej6M20W1CPi/PW+Eu8duJ92jK6YKseN237fBc0 goBA== X-Gm-Message-State: ACrzQf0TnmpAzx3JCMfdrTbB1BRIlxcs3e1tQNkWF6K8X12G1CE2bNhD wRFAdC/JCaeTxWRzpiGzjttoce6n0GY= X-Google-Smtp-Source: AMsMyM6NX5EYk7PwQwDo55u9gL1NULSUikfoYcA+X7ZNjDNCsd8qWZeF24ddb7ZHguIZQNPX10AwSA== X-Received: by 2002:a05:6000:719:b0:236:73ff:9ca3 with SMTP id bs25-20020a056000071900b0023673ff9ca3mr20260778wrb.603.1667516705112; Thu, 03 Nov 2022 16:05:05 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id b21-20020a05600c06d500b003b95ed78275sm1070302wmn.20.2022.11.03.16.05.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 03 Nov 2022 16:05:04 -0700 (PDT) Message-Id: <91ad797330731e48907767d13213c8d8f899d996.1667516701.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 03 Nov 2022 23:05:01 +0000 Subject: [PATCH v4 2/2] index: raise a bug if the index is materialised more than once Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Timothy Jones , Jeff Hostetler , Jeff Hostetler , Derrick Stolee , Taylor Blau , Anh Le , Anh Le Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Anh Le From: Anh Le If clear_skip_worktree_from_present_files() encounter a sparse directory, it fully materialise the index which should expand any sparse directories and start going through each entries again. If this happens more than once, raise it with a BUG. Signed-off-by: Anh Le --- sparse-index.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparse-index.c b/sparse-index.c index 8713a15611d..8c269dab803 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -510,6 +510,8 @@ restart: path_count[restarted]++; if (path_found(ce->name, &last_dirname, &dir_len, &dir_found)) { if (S_ISSPARSEDIR(ce->ce_mode)) { + if (restarted) + BUG("ensure-full-index did not fully flatten?"); ensure_full_index(istate); restarted = 1; goto restart;