From patchwork Wed Oct 27 18:20:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victoria Dye X-Patchwork-Id: 12587705 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7C47BC43219 for ; Wed, 27 Oct 2021 18:20:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 62DF4610C8 for ; Wed, 27 Oct 2021 18:20:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243451AbhJ0SWl (ORCPT ); Wed, 27 Oct 2021 14:22:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41136 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236965AbhJ0SWk (ORCPT ); Wed, 27 Oct 2021 14:22:40 -0400 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DB9E3C061745 for ; Wed, 27 Oct 2021 11:20:14 -0700 (PDT) Received: by mail-wr1-x42b.google.com with SMTP id v17so5622974wrv.9 for ; Wed, 27 Oct 2021 11:20:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=fBwFOyZpb1FNiGNmlBqIz8szMCFUAyTVo8d9LrANmrg=; b=hCWIZ1tTSV7RdHiMlNiSPB4poAyJY18+db6JRI0ofs9pPdMRRtgmDF1sxtAk3n3K/o tc8yUsTAPwEWe624Rm6rcAQLWuHAwkkTqusDSn8SjiSZlA3tj+E9ctQKyENmFfDFPhoc nGtoUTAkjLcJdqS7R+dT8aLAhMo0QAWumF4vM6sCYV6C5fhM6KuPn1snymTxyry/5ZJJ hcbgeAm2JIownuFO6mUpk9G8T/zJ8vaTnteUx14/5UGtwEghIAbZEewAYYdhf97hjbiT j5IhnOguvyKW0E1k/nytfWU79zzyXaHq7yGmQ7L+RZpySvt+hDUlpJMlxLIZQtRPr6ot e5NQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=fBwFOyZpb1FNiGNmlBqIz8szMCFUAyTVo8d9LrANmrg=; b=MBtNz+Tuc98iK8Z9eWnaGAfGnaVKxhYpQMAGqgSe90iJxIV8zY/zfr7jjocUxDozpv ETJepX5GzgsFtaEYvMcWrjNq4TXdDALoZkDgnBfX6bezx/hEnMmpLMYQJJ2peGnC6+91 JKwYzVFKnHBtd4bSTguXuNmZmcvxbVDrMDK68UFjyonUO7nnqmGSW9J3WbyYPCGJQXqu dpRYep1Q/MorCET/UlWaO8C2iMeyeF9HAawGTHexkiLhV2fSjY5ZhD47rvRF5qqK2kK6 JqUbUrlwPDHh+vMiwqT0fhYqfEVtee3O2n/z9B0LEPx0f7i0lmpDkm3CdWGn7z/ZEol1 Fkdg== X-Gm-Message-State: AOAM533A/8U5W04H0Ujy5NqZTFDp8zLxBbIYSzLugbP8VOu1Dm1rN8Pe SfJ9m5CD86WhEUwX36rfKH1MsA4kiKU= X-Google-Smtp-Source: ABdhPJwjRCxLtfkGxqqQsmwzMVX5tSN802CeLS5xtxg9RpB+GqYfgKtO3bsfR/cFy7bcPQGQZYtZZA== X-Received: by 2002:a5d:6e8d:: with SMTP id k13mr42294404wrz.295.1635358813521; Wed, 27 Oct 2021 11:20:13 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id t19sm611026wry.89.2021.10.27.11.20.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Oct 2021 11:20:13 -0700 (PDT) Message-Id: <6974ce7e7f568444518f9ff405747653516b2613.1635358812.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 27 Oct 2021 18:20:09 +0000 Subject: [PATCH v3 1/3] test-read-cache.c: prepare_repo_settings after config init Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: stolee@gmail.com, gitster@pobox.com, Victoria Dye , Victoria Dye Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Victoria Dye From: Victoria Dye Move `prepare_repo_settings` after the git directory has been set up in `test-read-cache.c`. The git directory settings must be initialized to properly assign repo settings using the worktree-level git config. Signed-off-by: Victoria Dye --- t/helper/test-read-cache.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/t/helper/test-read-cache.c b/t/helper/test-read-cache.c index b52c174acc7..0d9f08931a1 100644 --- a/t/helper/test-read-cache.c +++ b/t/helper/test-read-cache.c @@ -39,8 +39,6 @@ int cmd__read_cache(int argc, const char **argv) int table = 0, expand = 0; initialize_the_repository(); - prepare_repo_settings(r); - r->settings.command_requires_full_index = 0; for (++argv, --argc; *argv && starts_with(*argv, "--"); ++argv, --argc) { if (skip_prefix(*argv, "--print-and-refresh=", &name)) @@ -56,6 +54,9 @@ int cmd__read_cache(int argc, const char **argv) setup_git_directory(); git_config(git_default_config, NULL); + prepare_repo_settings(r); + r->settings.command_requires_full_index = 0; + for (i = 0; i < cnt; i++) { repo_read_index(r); From patchwork Wed Oct 27 18:20:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victoria Dye X-Patchwork-Id: 12587709 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 02B0AC433F5 for ; Wed, 27 Oct 2021 18:20:20 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E1A08610A4 for ; Wed, 27 Oct 2021 18:20:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243468AbhJ0SWo (ORCPT ); Wed, 27 Oct 2021 14:22:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41138 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240277AbhJ0SWl (ORCPT ); Wed, 27 Oct 2021 14:22:41 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A93AAC061570 for ; Wed, 27 Oct 2021 11:20:15 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id u18so5665758wrg.5 for ; Wed, 27 Oct 2021 11:20:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=VR2Z2fLSfarik6H2oV1f213mqXM+f4ls0tb3YM3OJfA=; b=mlAGq1uT5BN555gD3Swe9Hk+OtlOu4cozG76WCYnmkFPwqMB+4+v5R/s02MHN1HDYu XELPkxPHNM0JrDnIPV/1tw+c7hQZp6kDnJRyajsDermRT1wCZyCCUE+uro2gmW2rCVLw Lbn79FERH4ZMjnWvY1EEjYu0awfcrfhkZOKi4Z7QCFYTDBdJzms8bVd31f2v4pdgy7Tm ODX2rJoYekY6USaYH2CudBHYcSWcJeZnM2NhCiujTSFMpAY8YhoeLSe6n7CNzKqe986R n6/g8ephObYcaqLop5529087jrFJJ53sPXKbmjRGLVd3CmCmhoydIa25RHv/ew9AlfxS qk/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=VR2Z2fLSfarik6H2oV1f213mqXM+f4ls0tb3YM3OJfA=; b=cQFHI1Izf2p1CkzYPHO7fGHLRhw9cPR304CL300ikVJ41Ov2IVSFDh1vzDjPdPvBA4 dJvoJLIG6OIKfZNTS3fgE/1Xf0ipmzFnq8WPVLm+/VofgK336v/jGiguAZuesYuvRaV9 xImq7b6XAZCuvlwY2eKgl7rG1/z0MpQ2PYcooFx3QiHsHeiexXxARk4+lAPoenBiBQaj gVJ8D5aF/02xV5XKDtbejk2/9ty3MXeN4bZseAyN/flHODLCS9DoY3Whihbf5sC2IIw4 g7kkrW554FrMJ86cShnqmi+U7+r6zu7UdoDKCe4RnQ+7B6R2buaKpWohVyVsOx81Zm6s p66Q== X-Gm-Message-State: AOAM530vphjnKeZIV0rW++0clV07qOH5O8MLBNwi95tUhmM+k0LBz6n3 zLaAzzH5zT6YreQ49tV28xkqxO8dP2Q= X-Google-Smtp-Source: ABdhPJwTjoA15MPBoSTGPdkKnpVjwjUkq1Nqpk6g4sylRB2ZbAW1zQKFd7kN3c72Hg8+BWqPv7rGvw== X-Received: by 2002:adf:d0cf:: with SMTP id z15mr17252136wrh.219.1635358814177; Wed, 27 Oct 2021 11:20:14 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m3sm1793316wms.25.2021.10.27.11.20.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Oct 2021 11:20:13 -0700 (PDT) Message-Id: <9d6511db0728e9880a96f3d9e3a025a9ddc5bc9e.1635358812.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 27 Oct 2021 18:20:10 +0000 Subject: [PATCH v3 2/3] sparse-index: add ensure_correct_sparsity function Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: stolee@gmail.com, gitster@pobox.com, Victoria Dye , Victoria Dye Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Victoria Dye From: Victoria Dye The `ensure_correct_sparsity` function is intended to provide a means of aligning the in-core index with the sparsity required by the repository settings and other properties of the index. The function first checks whether a sparse index is allowed (per repository & sparse checkout pattern settings). If the sparse index may be used, the index is converted to sparse; otherwise, it is explicitly expanded with `ensure_full_index`. Helped-by: Junio C Hamano Signed-off-by: Victoria Dye --- sparse-index.c | 33 +++++++++++++++++++++++++++++---- sparse-index.h | 1 + 2 files changed, 30 insertions(+), 4 deletions(-) diff --git a/sparse-index.c b/sparse-index.c index 7b7ff79e044..bc3ee358c63 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -122,17 +122,17 @@ static int index_has_unmerged_entries(struct index_state *istate) return 0; } -int convert_to_sparse(struct index_state *istate, int flags) +static int is_sparse_index_allowed(struct index_state *istate, int flags) { - int test_env; - if (istate->sparse_index || !istate->cache_nr || - !core_apply_sparse_checkout || !core_sparse_checkout_cone) + if (!core_apply_sparse_checkout || !core_sparse_checkout_cone) return 0; if (!istate->repo) istate->repo = the_repository; if (!(flags & SPARSE_INDEX_MEMORY_ONLY)) { + int test_env; + /* * The sparse index is not (yet) integrated with a split index. */ @@ -168,6 +168,19 @@ int convert_to_sparse(struct index_state *istate, int flags) if (!istate->sparse_checkout_patterns->use_cone_patterns) return 0; + return 1; +} + +int convert_to_sparse(struct index_state *istate, int flags) +{ + /* + * If the index is already sparse, empty, or otherwise + * cannot be converted to sparse, do not convert. + */ + if (istate->sparse_index || !istate->cache_nr || + !is_sparse_index_allowed(istate, flags)) + return 0; + /* * NEEDSWORK: If we have unmerged entries, then stay full. * Unmerged entries prevent the cache-tree extension from working. @@ -313,6 +326,18 @@ void ensure_full_index(struct index_state *istate) trace2_region_leave("index", "ensure_full_index", istate->repo); } +void ensure_correct_sparsity(struct index_state *istate) +{ + /* + * If the index can be sparse, make it sparse. Otherwise, + * ensure the index is full. + */ + if (is_sparse_index_allowed(istate, 0)) + convert_to_sparse(istate, 0); + else + ensure_full_index(istate); +} + /* * This static global helps avoid infinite recursion between * expand_to_path() and index_file_exists(). diff --git a/sparse-index.h b/sparse-index.h index 9f3d7bc7faf..656bd835b25 100644 --- a/sparse-index.h +++ b/sparse-index.h @@ -4,6 +4,7 @@ struct index_state; #define SPARSE_INDEX_MEMORY_ONLY (1 << 0) int convert_to_sparse(struct index_state *istate, int flags); +void ensure_correct_sparsity(struct index_state *istate); /* * Some places in the codebase expect to search for a specific path. From patchwork Wed Oct 27 18:20:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Victoria Dye X-Patchwork-Id: 12587711 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 443B4C433EF for ; Wed, 27 Oct 2021 18:20:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 26E27610A4 for ; Wed, 27 Oct 2021 18:20:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243486AbhJ0SWp (ORCPT ); Wed, 27 Oct 2021 14:22:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41144 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243465AbhJ0SWm (ORCPT ); Wed, 27 Oct 2021 14:22:42 -0400 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2836FC061745 for ; Wed, 27 Oct 2021 11:20:16 -0700 (PDT) Received: by mail-wm1-x32b.google.com with SMTP id b82-20020a1c8055000000b0032ccc728d63so3091172wmd.1 for ; Wed, 27 Oct 2021 11:20:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=qkq4mQztxITq/GOSkhCIwELaNOZlIiv3dFjIv3bSuQU=; b=hm4wVJqpMGr8bbBAatl5LFOlqR5HKxgjSyLVL69fIrxAszaK+pSRUiFzzNudblVYVs 9GtFQkvtOOlMpC/5wwoXZb+ydBTT1QE1Ny1ri+1C8m+GOfh5CR5vQZaypaRkCLcpXf/D jJoNmO0MD7lkxHJL/Kc12KjyTPaG9zxTo1nipI0QpzSbSmezNPJY3wUGxDeF2Zy9tZ81 zkzDCG9CR2E19tu0cayqfmcSeWKUAg+0VbwumfPccmxDOUbbCYuobB1NOLq2fcPvA37N XB/+D0EDuJayo40bj2c3dARLjuCgzYQC1Voijj5VlaTffFFO2efnLEVaTo1jx8DpxSUz lcOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=qkq4mQztxITq/GOSkhCIwELaNOZlIiv3dFjIv3bSuQU=; b=n5KMFBd5UstVy9vL31FpKFWTy4MD4mC+IfbSZ+hMs7C8lPcc96BnlXKYJfn3MukBBv 003Q7sc4C9c4tvEwptIcQSbaTstv3V+R7iHt0k7OYV1g26SIcNN0b4pXbvak8Dt9Un0S IQ6vge4163ar2x3p7H1op1bi1nncNHabcs5dfHVxy0cVRVAkTohXuaDE5B76Y0ZUyVZ6 cD/qQV7O7os019dLr5HXfZW53Yv2MUA6jU9WnlUlqKH33lyj1vPsRV/oZ6ZbSTes98lF Ea/kefEePGDZ+Ue5QHjJI28L7eNHg4hyPYQTF14gixB2CN1GyZFRrFzIJGsPnGDALNgH zhRg== X-Gm-Message-State: AOAM5310A0MIKY3Nd/yO9sRlQCGWrkO73gTeK43pLQ/rBGsfaVLCOWgh MV/Sa1QDsu7JJwExHo/iFFk7fY3bcOM= X-Google-Smtp-Source: ABdhPJzD1Vl8oV7Kq5MNfWCUnQea/Orf12vKzDkr06cQ1W9AyzT2NuUjrLvjjuCYQvgunbNcKtIgGA== X-Received: by 2002:a05:600c:350b:: with SMTP id h11mr7140149wmq.123.1635358814770; Wed, 27 Oct 2021 11:20:14 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o2sm620459wrg.1.2021.10.27.11.20.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Oct 2021 11:20:14 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 27 Oct 2021 18:20:11 +0000 Subject: [PATCH v3 3/3] sparse-index: update do_read_index to ensure correct sparsity Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: stolee@gmail.com, gitster@pobox.com, Victoria Dye , Victoria Dye Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Victoria Dye From: Victoria Dye If `command_requires_full_index` is false, ensure correct in-core index sparsity on read by calling `ensure_correct_sparsity`. This change is meant to update the how the index is read in a command after sparse index-related repository settings are modified. Previously, for example, if `index.sparse` were changed from `true` to `false`, the in-core index on the next command would be sparse. The index would only be expanded to full when it was next written to disk. By adding a call to `ensure_correct_sparsity`, the in-core index now matches the sparsity dictated by the relevant repository settings as soon as it is read into memory, rather than when it is later written to disk. Helped-by: Junio C Hamano Co-authored-by: Derrick Stolee Signed-off-by: Victoria Dye --- read-cache.c | 8 ++++++ t/t1092-sparse-checkout-compatibility.sh | 31 ++++++++++++++++++++++++ 2 files changed, 39 insertions(+) diff --git a/read-cache.c b/read-cache.c index a78b88a41bf..b3772ba70a1 100644 --- a/read-cache.c +++ b/read-cache.c @@ -2337,9 +2337,17 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist) if (!istate->repo) istate->repo = the_repository; + + /* + * If the command explicitly requires a full index, force it + * to be full. Otherwise, correct the sparsity based on repository + * settings and other properties of the index (if necessary). + */ prepare_repo_settings(istate->repo); if (istate->repo->settings.command_requires_full_index) ensure_full_index(istate); + else + ensure_correct_sparsity(istate); return istate->cache_nr; diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index ca91c6a67f8..59accde1fa3 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -694,6 +694,37 @@ test_expect_success 'sparse-index is expanded and converted back' ' test_region index ensure_full_index trace2.txt ' +test_expect_success 'index.sparse disabled inline uses full index' ' + init_repos && + + # When index.sparse is disabled inline with `git status`, the + # index is expanded at the beginning of the execution then never + # converted back to sparse. It is then written to disk as a full index. + rm -f trace2.txt && + GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ + git -C sparse-index -c index.sparse=false status && + ! test_region index convert_to_sparse trace2.txt && + test_region index ensure_full_index trace2.txt && + + # Since index.sparse is set to true at a repo level, the index + # is converted from full to sparse when read, then never expanded + # over the course of `git status`. It is written to disk as a sparse + # index. + rm -f trace2.txt && + GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ + git -C sparse-index status && + test_region index convert_to_sparse trace2.txt && + ! test_region index ensure_full_index trace2.txt && + + # Now that the index has been written to disk as sparse, it is not + # converted to sparse (or expanded to full) when read by `git status`. + rm -f trace2.txt && + GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ + git -C sparse-index status && + ! test_region index convert_to_sparse trace2.txt && + ! test_region index ensure_full_index trace2.txt +' + ensure_not_expanded () { rm -f trace2.txt && echo >>sparse-index/untracked.txt &&