From patchwork Mon Apr 10 22:53:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13206810 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2A17C76196 for ; Mon, 10 Apr 2023 22:53:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229780AbjDJWxc (ORCPT ); Mon, 10 Apr 2023 18:53:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42898 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229591AbjDJWx2 (ORCPT ); Mon, 10 Apr 2023 18:53:28 -0400 Received: from mail-yw1-x1131.google.com (mail-yw1-x1131.google.com [IPv6:2607:f8b0:4864:20::1131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 19B2E1BF7 for ; Mon, 10 Apr 2023 15:53:28 -0700 (PDT) Received: by mail-yw1-x1131.google.com with SMTP id 00721157ae682-54bfa5e698eso279789707b3.13 for ; Mon, 10 Apr 2023 15:53:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20210112.gappssmtp.com; s=20210112; t=1681167207; x=1683759207; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=pYYglhFGC6dAPiBth3kdOOoXPhN+QeLt6UuSgkb6z+A=; b=rgNCw+gLdTpHnf+uPfqGwxpZUr6N2V9RYWXW6i41kbRQ/dWNs8uBlPPttSJsfj4XXn uXXywTRlpf3eN9UNxuJtrS7BbmXLQrQ5NXOOGzrz8Um7cgwJcGTViB5TsZqyShweU/Xl X5aQJJ6PggLoRX6bfgFTc/VLWcLzmSX0da9hSFS0Cc9do/UMNKYQwRod8slEchtXdHsA pw7hwFAYQWjv0eQ+jFAm1H2s2I76id5V8KQtUnXk5I6oG7XVe2anKpFbEC5i+/nUWX4r 8VXi2k0I2014NmTrLQAQkzRO0VyrTvnt2oIKLKWBD38Di0eamFgazOHTVmvs/+ueqEtu /mmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1681167207; x=1683759207; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=pYYglhFGC6dAPiBth3kdOOoXPhN+QeLt6UuSgkb6z+A=; b=vQE5IuEo+C++LbOReOzd/GeY13x4lytdudXKwR2Z9idmM/EURgvoND9u+gTBQcBi83 0XYDFIAPRy0uuUo9SpUd/Kcf5dny8ts5e04Gzv4Cap3zTTYg5kXrxWonmLXyP5iVTpPC /kcrXnrP0NZyZNaC15dJwFmK/fg1WC9IiJyzXKEeIMON/eeM8D3LNq3hDOd6c4HClheo 04LYdWPfBa6aOrNSj6ljGTbm9oCw+Ena305Fj0hAY17Bf1yxi0bnEDRmnrrTySu1/KQl Pp/hJH2MnD4zcNoWYe5F/fhQPw5CQeYqkDWcEXpJ0DU5/q0IRT6CUJj05BtPpMq+L3RE XYZw== X-Gm-Message-State: AAQBX9e+Wc2aQsZuxS6LBizd4jbHcL0YuXdYmtQ0d24V/GHeFdYnBaXf UnT8BM5O9zSpkv/XufcMHWvSvF1KYbEcmYSvUCZkDg== X-Google-Smtp-Source: AKy350YCGkEg3d75yJQU/luKLjMYItFFL/YlDwEVCn65HQHV7vBXgax8/Q8E2ehxnjB/wP34Crz5Qw== X-Received: by 2002:a81:84d5:0:b0:541:6b00:f68d with SMTP id u204-20020a8184d5000000b005416b00f68dmr10046356ywf.16.1681167207142; Mon, 10 Apr 2023 15:53:27 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id z18-20020a81ac52000000b00545a0818474sm3125767ywj.4.2023.04.10.15.53.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Apr 2023 15:53:26 -0700 (PDT) Date: Mon, 10 Apr 2023 18:53:25 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Derrick Stolee , Jeff King , Junio C Hamano Subject: [PATCH 1/7] pack-write.c: plug a leak in stage_tmp_packfiles() Message-ID: <18be29c3988295cd58521f8cc4a729897df074c8.1681166596.git.me@ttaylorr.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org The function `stage_tmp_packfiles()` generates a filename to use for staging the contents of what will become the pack's ".rev" file. The name is generated in `write_rev_file_order()` (via its caller `write_rev_file()`) in a string buffer, and the result is returned back to `stage_tmp_packfiles()` which uses it to rename the temporary file into place via `rename_tmp_packfiles()`. That name is not visible outside of `stage_tmp_packfiles()`, so it can (and should) be `free()`'d at the end of that function. We can't free it in `rename_tmp_packfile()` since not all of its `source` arguments are unreachable after calling it. Instead, simply free() `rev_tmp_name` at the end of `stage_tmp_packfiles()`. (Note that the same leak exists for `mtimes_tmp_name`, but we do not address it in this commit). Signed-off-by: Taylor Blau --- pack-write.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/pack-write.c b/pack-write.c index f171405495..f27c1f7f28 100644 --- a/pack-write.c +++ b/pack-write.c @@ -568,6 +568,8 @@ void stage_tmp_packfiles(struct strbuf *name_buffer, rename_tmp_packfile(name_buffer, rev_tmp_name, "rev"); if (mtimes_tmp_name) rename_tmp_packfile(name_buffer, mtimes_tmp_name, "mtimes"); + + free((char *)rev_tmp_name); } void write_promisor_file(const char *promisor_name, struct ref **sought, int nr_sought) From patchwork Mon Apr 10 22:53:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13206811 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C85F9C77B6F for ; Mon, 10 Apr 2023 22:53:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229845AbjDJWxe (ORCPT ); Mon, 10 Apr 2023 18:53:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43002 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229880AbjDJWxc (ORCPT ); Mon, 10 Apr 2023 18:53:32 -0400 Received: from mail-yw1-x112c.google.com (mail-yw1-x112c.google.com [IPv6:2607:f8b0:4864:20::112c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0BA701BFD for ; Mon, 10 Apr 2023 15:53:31 -0700 (PDT) Received: by mail-yw1-x112c.google.com with SMTP id 00721157ae682-54ee9ad5eb3so121540377b3.1 for ; Mon, 10 Apr 2023 15:53:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20210112.gappssmtp.com; s=20210112; t=1681167210; x=1683759210; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=z1fFUZEnhk8mc5WSf+cuRlV1BkCsAwGZhwQWdDsFwOs=; b=i9TJZDDOg74Myuw64w8iV9zxuDPO7L4tJYdLnT9PSGwj2EMpEMgzXMJKfIGmM8d3EY ruND0E+A6H2ffLRf4RQpOjFyslP/lgnkbPyd9qthBx4C6qXwyjFawu3m0qUv39tC14Qp k3uTSQQkT1ketjhnKgWqjiNngIbAb6T1N8/Yz/UBdcyh4qnCETlug43kpp9dauPbnIGp IH3watLmCs3IyZItJvGGXTSHquqeTNnxfAhSl1vEVrmCaHL8xvP7kPnIKtH7d0nUzLF0 AnI4GS/xoRR02SzDYFcnl80nGkxCds5JWOjvlOYVApRxypzY03RM0vQL8+521gV2XXym pt1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1681167210; x=1683759210; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=z1fFUZEnhk8mc5WSf+cuRlV1BkCsAwGZhwQWdDsFwOs=; b=zmmRjy9lXmyGHdpKqGN8aWHnO1YOSwUixxt+pRemloEW5F0824r9zkhFUXc3Llw8o/ cAPB6cSx7taxbDge4OYTjiLkCLp8Yq9ZC7KOcNf8GY/ywDvBIFUb3AbRXQAdmd3Bm/is td2eoJZOmaBtdY7djvJcElFzl6VZcJcN5AfWEDYs4ZoHtttK0sVzrgnTJAuCTbaUBky6 H9OHk/Il0SbYmOSELtBdqkkcIdQWZTG2QQOGfIJs5mC4KYvPcL19aGqKl4b7U5ZmMf0W DNhVBiQFlwTfeXl5qG4FNsiHhnodr5G3O9ZyjlXqHgyWS+1LJFUMQu/XZSWjdEdHuqdX irHw== X-Gm-Message-State: AAQBX9cfPD6N9Xa2ug8C6F6IhpJV5WLUi25T5D7Wrj4vvwN7pP+aqYDh e/5k8zNjsTeC/7lkTMxaMSGXOmJMHaeBNS9RGqB62A== X-Google-Smtp-Source: AKy350ayKKtnZDo40XCJ5x4kvL5m0jmmIA+krKvTtEx6FxqkD9OTNlmErpsPsnAkeZkARBN8PCv2Wg== X-Received: by 2002:a0d:ca86:0:b0:546:aa9:8e42 with SMTP id m128-20020a0dca86000000b005460aa98e42mr6740230ywd.7.1681167210163; Mon, 10 Apr 2023 15:53:30 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id v189-20020a8148c6000000b0054d8263d642sm1824719ywa.113.2023.04.10.15.53.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Apr 2023 15:53:29 -0700 (PDT) Date: Mon, 10 Apr 2023 18:53:28 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Derrick Stolee , Jeff King , Junio C Hamano Subject: [PATCH 2/7] t5325: mark as leak-free Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org This test is leak-free as of the previous commit, so let's mark it as such to ensure we don't regress and introduce a leak in the future. Signed-off-by: Taylor Blau --- t/t5325-reverse-index.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/t/t5325-reverse-index.sh b/t/t5325-reverse-index.sh index d042d26f2b..48c14d8576 100755 --- a/t/t5325-reverse-index.sh +++ b/t/t5325-reverse-index.sh @@ -1,6 +1,8 @@ #!/bin/sh test_description='on-disk reverse index' + +TEST_PASSES_SANITIZE_LEAK=true . ./test-lib.sh # The below tests want control over the 'pack.writeReverseIndex' setting From patchwork Mon Apr 10 22:53:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13206812 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 211BBC76196 for ; Mon, 10 Apr 2023 22:53:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229903AbjDJWxk (ORCPT ); Mon, 10 Apr 2023 18:53:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43074 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229851AbjDJWxf (ORCPT ); Mon, 10 Apr 2023 18:53:35 -0400 Received: from mail-yw1-x112e.google.com (mail-yw1-x112e.google.com [IPv6:2607:f8b0:4864:20::112e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1E525210D for ; Mon, 10 Apr 2023 15:53:34 -0700 (PDT) Received: by mail-yw1-x112e.google.com with SMTP id 00721157ae682-54c0c86a436so245956337b3.6 for ; Mon, 10 Apr 2023 15:53:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20210112.gappssmtp.com; s=20210112; t=1681167213; x=1683759213; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=6Qguh8l2CvGX8dlC/4PABDvsvoriHxwl7xwHwVE/Erk=; b=Y00B+MgxYMBouIaypcc/4KWtEYr+xYGXjRfwrwELaNhUOhhqY5K/9F8JwlILXQzDOX n0gz34FLeR2+B1WNcM9UZuxZjvqObzUqJ6PaG1mGvIfd0pZnvyKw9U0OpETj+ECBouha Pfd/ZiikYNKyC1a7b50a/PMxtWl/jqht2omWKXAnu0cd7gtSOYELxZ2krU6Izs/rprH7 utIDUnY89Xyj+cq5O6FMFBuktFg6rZkQu9gSoPbfKa1jfPTVytoBM8Z7pgcpZXeBqcfr Knnebh2OyY3XYHgvMyVomU6arhtPuk+3ZbxJT17hypduVWzEWJp4P5eH9zZktnzanliC ejlQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1681167213; x=1683759213; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=6Qguh8l2CvGX8dlC/4PABDvsvoriHxwl7xwHwVE/Erk=; b=rTfS6WBbkiIgAmzC+WADNiGS0K3C53ZmFc5iMuMoah1dlL5X+Cju2DFHV1fOUmv8PU PVpVDGtkmfTTjmu+6xVipkDxMhXhLCYrOuaKAe6VLopJHDEPQPQ7EbPW2p4hrwGKq+Qm 7AtqjUb9OkWCCbSjfaA0OEGJZZj0nhK0nL4a6AG3pbC0hxug+LkYiHEOD2SYDht8xJtG lkbS/Tu5cJkLRuLDl3BSR1KWukz1KzMIQGGdvDil7XBorSy9nVxyB+U91i8BjcgUjdPi oJAM2cizaycP5MKLNKbtHzb2tymc6G+S7hrYBTrWqX3g3j5YPbvWSE7O0oEyNGm/DQec aLLw== X-Gm-Message-State: AAQBX9chALYGD5sQlo9hQ6BNZXlTU7Ho+5of0yyq+aRQv0szYnDsVpbn w6hoeLhk+T17mfZvFZZVrrWelxoAnS6u63yRUiWOxA== X-Google-Smtp-Source: AKy350bA8SFBLSr/AN5whzOM1jqnra4lsFimYg2Gb+zAjGZqQ3N0MhAj8H/P5dxMUrn4sf3BiXWYHg== X-Received: by 2002:a0d:ddcc:0:b0:54f:baa:2c with SMTP id g195-20020a0dddcc000000b0054f0baa002cmr810000ywe.0.1681167213098; Mon, 10 Apr 2023 15:53:33 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id y207-20020a814bd8000000b00545a0818496sm3105739ywa.38.2023.04.10.15.53.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Apr 2023 15:53:32 -0700 (PDT) Date: Mon, 10 Apr 2023 18:53:31 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Derrick Stolee , Jeff King , Junio C Hamano Subject: [PATCH 3/7] pack-revindex: make `load_pack_revindex` take a repository Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org In a future commit, we will introduce a `pack.readReverseIndex` configuration, which forces Git to generate the reverse index from scratch instead of loading it from disk. In order to avoid reading this configuration value more than once, we'll use the `repo_settings` struct to lazily load this value. In order to access the `struct repo_settings`, add a repository argument to `load_pack_revindex`, and update all callers to pass the correct instance (in all cases, `the_repository`). Signed-off-by: Taylor Blauy Signed-off-by: Derrick Stolee --- pack-bitmap.c | 5 +++-- pack-revindex.c | 4 ++-- pack-revindex.h | 3 ++- packfile.c | 2 +- 4 files changed, 8 insertions(+), 6 deletions(-) diff --git a/pack-bitmap.c b/pack-bitmap.c index b2e7d06d60..8e3bb00931 100644 --- a/pack-bitmap.c +++ b/pack-bitmap.c @@ -477,13 +477,14 @@ static int load_reverse_index(struct bitmap_index *bitmap_git) * since we will need to make use of them in pack-objects. */ for (i = 0; i < bitmap_git->midx->num_packs; i++) { - ret = load_pack_revindex(bitmap_git->midx->packs[i]); + ret = load_pack_revindex(the_repository, + bitmap_git->midx->packs[i]); if (ret) return ret; } return 0; } - return load_pack_revindex(bitmap_git->pack); + return load_pack_revindex(the_repository, bitmap_git->pack); } static int load_bitmap(struct bitmap_index *bitmap_git) diff --git a/pack-revindex.c b/pack-revindex.c index 03c7e81f9d..e3d69cc0f7 100644 --- a/pack-revindex.c +++ b/pack-revindex.c @@ -283,7 +283,7 @@ static int load_pack_revindex_from_disk(struct packed_git *p) return ret; } -int load_pack_revindex(struct packed_git *p) +int load_pack_revindex(struct repository *r, struct packed_git *p) { if (p->revindex || p->revindex_data) return 0; @@ -356,7 +356,7 @@ int offset_to_pack_pos(struct packed_git *p, off_t ofs, uint32_t *pos) { unsigned lo, hi; - if (load_pack_revindex(p) < 0) + if (load_pack_revindex(the_repository, p) < 0) return -1; lo = 0; diff --git a/pack-revindex.h b/pack-revindex.h index 4974e75eb4..3d1969ce8b 100644 --- a/pack-revindex.h +++ b/pack-revindex.h @@ -39,6 +39,7 @@ struct packed_git; struct multi_pack_index; +struct repository; /* * load_pack_revindex populates the revindex's internal data-structures for the @@ -47,7 +48,7 @@ struct multi_pack_index; * If a '.rev' file is present it is mmap'd, and pointers are assigned into it * (instead of using the in-memory variant). */ -int load_pack_revindex(struct packed_git *p); +int load_pack_revindex(struct repository *r, struct packed_git *p); /* * load_midx_revindex loads the '.rev' file corresponding to the given diff --git a/packfile.c b/packfile.c index b120405ccc..717fd685c9 100644 --- a/packfile.c +++ b/packfile.c @@ -2151,7 +2151,7 @@ int for_each_object_in_pack(struct packed_git *p, int r = 0; if (flags & FOR_EACH_OBJECT_PACK_ORDER) { - if (load_pack_revindex(p)) + if (load_pack_revindex(the_repository, p)) return -1; } From patchwork Mon Apr 10 22:53:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13206813 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 372E4C77B61 for ; Mon, 10 Apr 2023 22:53:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229914AbjDJWxq (ORCPT ); Mon, 10 Apr 2023 18:53:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43292 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229899AbjDJWxj (ORCPT ); Mon, 10 Apr 2023 18:53:39 -0400 Received: from mail-yb1-xb2f.google.com (mail-yb1-xb2f.google.com [IPv6:2607:f8b0:4864:20::b2f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2C7AE1FF9 for ; Mon, 10 Apr 2023 15:53:37 -0700 (PDT) Received: by mail-yb1-xb2f.google.com with SMTP id ch3so7303950ybb.4 for ; Mon, 10 Apr 2023 15:53:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20210112.gappssmtp.com; s=20210112; t=1681167216; x=1683759216; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=B4FTR6K5zz6BX79OARH6llxCs8zdC5FF/Zl7jOYTtVQ=; b=EYIRmaDPlRuhwTiHZeY1XAcf4QBA2/Susvp/bN+LS5JDqrwUt+z092guKDBDKRllwJ RI9Af3fKtxm6r0KAD7GSx+6O7AnqYr4qmTKgqHgjvM9OvtIHGi3Rmrs6Uf+4CSpLS7ht ZheBjdiFRpyf1u938Ote25N5uitp3XeOirBDFwP1CU6cR2NfyBkS7iJtQsq8SBZLw2Ot eqfqy2zieXSskFDLhloPt6PRGYBtl0yiQ8DqA0mnT5aHN82ev7VPzZGkbtEdU/Lf0wU4 9CJHDkYAtmccyES4rHeCHk+/vayDGEO/rJlv8fh+RJ47kAlUv8uwLLF2SWfRNV6QZBht +lCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1681167216; x=1683759216; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=B4FTR6K5zz6BX79OARH6llxCs8zdC5FF/Zl7jOYTtVQ=; b=l/+Qo1q1ROdisqc47X4Th3viLWFKMqVNmnW/uOOAR5yMvlGYel0OUE6VrfZviNlMz/ SgDtgQzuXE+4OFYwl5lPIWLRZNQqvuwPNoxObM02VdaVpvq6bCdfPga/8414x+2blwjt qhl4Qu7ExdeR0EX6bPmY97CmNIi40j6W5FeD5iWcKks83wQISQ5ioYQZld6moovg7RxB uQVwAJWFL3y59/AJVXJWzlj2uuWbhPQeNmWVfE21Zurp5vY/Txs620Twp1B7ofgoaHYr C3O5CcNWU6Nmg2WVblgV4vshHcjmVfziYxSbOfW0jfGw4IZszDKfyqGdbOy5yZy5bnjO kgNA== X-Gm-Message-State: AAQBX9cbxwFbrf9VQQkFQGDfAB7qACXzb8g4TiWyITn/Otwoa+afOqax c6K5vq3eNxYz2/G6TM7kk8MGeewK4dntQaAFYOWmXA== X-Google-Smtp-Source: AKy350byeQKpc1nqH8UqrhWsYwAhhpTrp77OPStNK5kPP31T3F60zEr8fipaqifyd5ltcmnAyBzhiQ== X-Received: by 2002:a25:348d:0:b0:b8e:411c:b85e with SMTP id b135-20020a25348d000000b00b8e411cb85emr8420291yba.23.1681167216044; Mon, 10 Apr 2023 15:53:36 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id l5-20020a25bcc5000000b00b7767ca7484sm2132971ybm.33.2023.04.10.15.53.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Apr 2023 15:53:35 -0700 (PDT) Date: Mon, 10 Apr 2023 18:53:34 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Derrick Stolee , Jeff King , Junio C Hamano Subject: [PATCH 4/7] pack-revindex: introduce GIT_TEST_REV_INDEX_DIE_ON_DISK Message-ID: <0f368e2347e10325d0c5a34939191994aa8899b9.1681166596.git.me@ttaylorr.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org In ec8e7760ac (pack-revindex: ensure that on-disk reverse indexes are given precedence, 2021-01-25), we introduced GIT_TEST_REV_INDEX_DIE_IN_MEMORY to abort the process when Git generated a reverse index from scratch. ec8e7760ac was about ensuring that Git prefers a .rev file when available over generating the same information in memory from scratch. In a subsequent patch, we'll introduce `pack.readReverseIndex`, which may be used to disable reading ".rev" files when available. In order to ensure that those files are indeed being ignored, introduce an analogous option to abort the process when Git reads a ".rev" file from disk. Signed-off-by: Taylor Blau --- pack-revindex.c | 3 +++ pack-revindex.h | 1 + 2 files changed, 4 insertions(+) diff --git a/pack-revindex.c b/pack-revindex.c index e3d69cc0f7..44e1b3fed9 100644 --- a/pack-revindex.c +++ b/pack-revindex.c @@ -205,6 +205,9 @@ static int load_revindex_from_disk(char *revindex_name, size_t revindex_size; struct revindex_header *hdr; + if (git_env_bool(GIT_TEST_REV_INDEX_DIE_ON_DISK, 0)) + die("dying as requested by '%s'", GIT_TEST_REV_INDEX_DIE_ON_DISK); + fd = git_open(revindex_name); if (fd < 0) { diff --git a/pack-revindex.h b/pack-revindex.h index 3d1969ce8b..ef8afee88b 100644 --- a/pack-revindex.h +++ b/pack-revindex.h @@ -36,6 +36,7 @@ #define GIT_TEST_WRITE_REV_INDEX "GIT_TEST_WRITE_REV_INDEX" #define GIT_TEST_REV_INDEX_DIE_IN_MEMORY "GIT_TEST_REV_INDEX_DIE_IN_MEMORY" +#define GIT_TEST_REV_INDEX_DIE_ON_DISK "GIT_TEST_REV_INDEX_DIE_ON_DISK" struct packed_git; struct multi_pack_index; From patchwork Mon Apr 10 22:53:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13206814 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37437C76196 for ; Mon, 10 Apr 2023 22:53:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229794AbjDJWxs (ORCPT ); Mon, 10 Apr 2023 18:53:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43004 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229827AbjDJWxl (ORCPT ); Mon, 10 Apr 2023 18:53:41 -0400 Received: from mail-yw1-x112b.google.com (mail-yw1-x112b.google.com [IPv6:2607:f8b0:4864:20::112b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2B48C210B for ; Mon, 10 Apr 2023 15:53:40 -0700 (PDT) Received: by mail-yw1-x112b.google.com with SMTP id 00721157ae682-54ee108142eso128411327b3.2 for ; Mon, 10 Apr 2023 15:53:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20210112.gappssmtp.com; s=20210112; t=1681167219; x=1683759219; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=qnGxzQd7zwZW5czww1ysMWxcUE7CVzfMnPgoYP+PK88=; b=D0NB0ynPF8UYH8sGD98cZXev7OMbVWgb5VRiZOJu5NdOYY/NRmrPulkt6G5XfGqfzd NFdy5Lhx8UMpixq47Um3uDzzc5f8yl/gC+rEEO7mCEdYtpCzdcu0VG5L/QIte+aQRIPk 12fIdiWAW3LIGlrgki7gAVA4VVexQNYX6vCrFMuFlIYIBhbaMoSgjT3D0FvukqVkXaI5 QOeoMdld3tMPs/bk5FeBW1OpmCggR+Im9uDx8drw5bhTr6tr+3KMXAKN4tzvv9AVWdZa Qu4U2lQXa9aJ68TpNrZoU4KF6iNqLz+1O6ZdZyQjv3HjFRwmeSdJk/H4ya49bH1e2who Y1mA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1681167219; x=1683759219; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=qnGxzQd7zwZW5czww1ysMWxcUE7CVzfMnPgoYP+PK88=; b=gfB5F0bcla+6sEZGOHbk3mR6UalFGniPx7Gl9emp3Ay7TCQlWEWwZhQjtGIP6yvBCJ 8gClIkvy31Ku76Pxe6mDeexE4yciDKIZb7oUDIldGicsx/9KVpGplFh+/vzewSGGt66g XOMRgPElZ3mLdE51pYAfcutUfN2g9UW4k212p7eR8Cdjj+B67smdm9RM9k7jaUm4ZNHN B2dou3yJ3owskMUGdADIFQwFDGXSkSXtiL2nuvmSERdtW0HLyYw/9FpNRgkH85iGWrx1 kebfRIo/oEMY4sHV2NGStUrfArrboqL2HDTzb/R0pLaC2vLwHX0sC4+9QWniJn2yq9Sq UA/w== X-Gm-Message-State: AAQBX9dnNOdwcyErzHQFKhcfbDGRqfMFTIG3XH4NLxIZXqr3vlmHT0U/ a5uBXayDdaEemWN+m6kkCNvqpzaQR8T0WNcU3Uy2iQ== X-Google-Smtp-Source: AKy350YayoSaskf6RbDXHH0n+LWPs2G4144jiYoPkqbVq1/EFelDKfhwPaTrptU3ilVnGMTg99ypkw== X-Received: by 2002:a0d:c347:0:b0:54c:1e9e:9033 with SMTP id f68-20020a0dc347000000b0054c1e9e9033mr9129101ywd.22.1681167219032; Mon, 10 Apr 2023 15:53:39 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id bh30-20020a05690c039e00b0054c0ddb3611sm3103610ywb.87.2023.04.10.15.53.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Apr 2023 15:53:38 -0700 (PDT) Date: Mon, 10 Apr 2023 18:53:37 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Derrick Stolee , Jeff King , Junio C Hamano Subject: [PATCH 5/7] pack-revindex: introduce `pack.readReverseIndex` Message-ID: <6f692d470cb691402020e2e423e2ec660aba26ce.1681166596.git.me@ttaylorr.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Since 1615c567b8 (Documentation/config/pack.txt: advertise 'pack.writeReverseIndex', 2021-01-25), we have had the `pack.writeReverseIndex` configuration option, which tells Git whether or not it is allowed to write a ".rev" file when indexing a pack. Introduce a complementary configuration knob, `pack.readReverseIndex` to control whether or not Git will read any ".rev" file(s) that may be available on disk. This option is useful for debugging, as well as disabling the effect of ".rev" files in certain instances. This is useful because of the trade-off[^1] between the time it takes to generate a reverse index (slow from scratch, fast when reading an existing ".rev" file), and the time it takes to access a record (the opposite). For example, even though it is faster to use the on-disk reverse index when computing the on-disk size of a packed object, it is slower to enumerate the same value for all objects. Here are a couple of examples from linux.git. When computing the above for a single object, using the on-disk reverse index is significantly faster: $ git rev-parse HEAD >in $ hyperfine -L v false,true 'git.compile -c pack.readReverseIndex={v} cat-file --batch-check="%(objectsize:disk)" --- Documentation/config/pack.txt | 6 ++++++ pack-revindex.c | 5 ++++- repo-settings.c | 1 + repository.h | 1 + t/t5325-reverse-index.sh | 11 +++++++++++ 5 files changed, 23 insertions(+), 1 deletion(-) diff --git a/Documentation/config/pack.txt b/Documentation/config/pack.txt index 53093d9996..7db7fed466 100644 --- a/Documentation/config/pack.txt +++ b/Documentation/config/pack.txt @@ -171,6 +171,12 @@ pack.writeBitmapLookupTable:: beneficial in repositories that have relatively large bitmap indexes. Defaults to false. +pack.readReverseIndex:: + When true, git will read any .rev file(s) that may be available + (see: linkgit:gitformat-pack[5]). When false, the reverse index + will be generated from scratch and stored in memory. Defaults to + true. + pack.writeReverseIndex:: When true, git will write a corresponding .rev file (see: linkgit:gitformat-pack[5]) diff --git a/pack-revindex.c b/pack-revindex.c index 44e1b3fed9..29f5358b25 100644 --- a/pack-revindex.c +++ b/pack-revindex.c @@ -291,7 +291,10 @@ int load_pack_revindex(struct repository *r, struct packed_git *p) if (p->revindex || p->revindex_data) return 0; - if (!load_pack_revindex_from_disk(p)) + prepare_repo_settings(r); + + if (r->settings.pack_read_reverse_index && + !load_pack_revindex_from_disk(p)) return 0; else if (!create_pack_revindex_in_memory(p)) return 0; diff --git a/repo-settings.c b/repo-settings.c index 0a6c0b381f..bdd7640ab0 100644 --- a/repo-settings.c +++ b/repo-settings.c @@ -63,6 +63,7 @@ void prepare_repo_settings(struct repository *r) repo_cfg_bool(r, "core.multipackindex", &r->settings.core_multi_pack_index, 1); repo_cfg_bool(r, "index.sparse", &r->settings.sparse_index, 0); repo_cfg_bool(r, "index.skiphash", &r->settings.index_skip_hash, r->settings.index_skip_hash); + repo_cfg_bool(r, "pack.readreverseindex", &r->settings.pack_read_reverse_index, 1); /* * The GIT_TEST_MULTI_PACK_INDEX variable is special in that diff --git a/repository.h b/repository.h index 15a8afc5fb..ed73e799b6 100644 --- a/repository.h +++ b/repository.h @@ -37,6 +37,7 @@ struct repo_settings { int fetch_write_commit_graph; int command_requires_full_index; int sparse_index; + int pack_read_reverse_index; struct fsmonitor_settings *fsmonitor; /* lazily loaded */ diff --git a/t/t5325-reverse-index.sh b/t/t5325-reverse-index.sh index 48c14d8576..66171c1d67 100755 --- a/t/t5325-reverse-index.sh +++ b/t/t5325-reverse-index.sh @@ -96,6 +96,17 @@ test_expect_success 'reverse index is not generated when available on disk' ' --batch-check="%(objectsize:disk)" tip && + GIT_TEST_REV_INDEX_DIE_ON_DISK=1 git cat-file \ + --batch-check="%(objectsize:disk)" X-Patchwork-Id: 13206815 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 122D6C77B61 for ; Mon, 10 Apr 2023 22:53:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229868AbjDJWx4 (ORCPT ); Mon, 10 Apr 2023 18:53:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43292 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229901AbjDJWxp (ORCPT ); Mon, 10 Apr 2023 18:53:45 -0400 Received: from mail-yw1-x1135.google.com (mail-yw1-x1135.google.com [IPv6:2607:f8b0:4864:20::1135]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2ADA2212E for ; Mon, 10 Apr 2023 15:53:43 -0700 (PDT) Received: by mail-yw1-x1135.google.com with SMTP id 00721157ae682-54c12009c30so233389577b3.9 for ; Mon, 10 Apr 2023 15:53:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20210112.gappssmtp.com; s=20210112; t=1681167222; x=1683759222; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=WPfbEcaOwGFeNtY9Wa7/E9QTZEQLc5FFABXfxZzDLWA=; b=6icYCugtnp+tMUayyASz65lzeqQ04JQ5L/6AZo/NRjL8BxEhDTDpvzpKOWN3tGFbPN viCR0wuqtfd8noQqPD5Pr4TFULCrz3nkhPecjujOkxVPLVbif/KB0HLlWG6TPKnIZ7ZV 1OCGQGZXporUB1d1d+YJNvQYfG7R6cVlAk28dmMi/mLOynag7QvOiUWkl7CagOtfg850 O6oPzoQZOENf9G+44Qi4ngAYFmYYO9W1Sy4NVEslxurmwCt5CN/9DBolmFTjAX14zCe+ dUtU2xqWIc1pM+xJnTabudyQL7ajdtMofAGzZh0I4YIzksfouSpVCu6Zu5GLSausvFrn 9Lbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1681167222; x=1683759222; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=WPfbEcaOwGFeNtY9Wa7/E9QTZEQLc5FFABXfxZzDLWA=; b=3c9SUPXV5hMp398X9wWWJYka6SeGt2SA9hSyzPa//nGB1CmX35uj1sAy1qGzEKlEBZ efgIB8ES1M5rTeM/9u+YNCZA+EfbQOmw+QF8KsauiLSZYe+8ycabGp9ixbfnjUjSGc9o emBWBCvA0iih1nlilBcdnJsO++uj2Mrg8VO6vYCCeBCY619xjdP3owy9VoLdInG5UsRc ++V1kVJ5e2e9C8WBVvONrLPZD6N+cdjFlzgHEDe3MjN7TU/+NGrGSnGSIEqqnJy261BD DzXS8DRazuneKkmb9vy6qcWHPT0aFH/spjHwPy67Bmkon3nMvt+dlBF0eOcjc71ObOhp SQLg== X-Gm-Message-State: AAQBX9c3KwqO9UNuuHAiP5/F1WA+KK2L9oUXyQ7a1TzSvhZyJbttjC6g LpriiG1nJ1AvdvLqOpo3UTuTybyaUkV4bsRKg9Vchg== X-Google-Smtp-Source: AKy350a4XvlgW3BbIGYo/2z4FYe7jntpp8YYsm/KyG0J0fgz6MsQjt3iTbuYgUgDCnPXYH4U3UeQKA== X-Received: by 2002:a81:728a:0:b0:533:9c8d:81d1 with SMTP id n132-20020a81728a000000b005339c8d81d1mr10290780ywc.3.1681167222012; Mon, 10 Apr 2023 15:53:42 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 82-20020a810a55000000b00545a0818486sm3106418ywk.22.2023.04.10.15.53.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Apr 2023 15:53:41 -0700 (PDT) Date: Mon, 10 Apr 2023 18:53:40 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Derrick Stolee , Jeff King , Junio C Hamano Subject: [PATCH 6/7] config: enable `pack.writeReverseIndex` by default Message-ID: <56a0fc0098e0b0551e01414c8e67c17fb1ba3054.1681166596.git.me@ttaylorr.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Back in e37d0b8730 (builtin/index-pack.c: write reverse indexes, 2021-01-25), Git learned how to read and write a pack's reverse index from a file instead of in-memory. A pack's reverse index is a mapping from pack position (that is, the order that objects appear together in a ".pack") to their position in lexical order (that is, the order that objects are listed in an ".idx" file). Reverse indexes are consulted often during pack-objects, as well as during auxiliary operations that require mapping between pack offsets, pack order, and index index. They are useful in GitHub's infrastructure, where we have seen a dramatic increase in performance when writing ".rev" files[1]. In particular: - an ~80% reduction in the time it takes to serve fetches on a popular repository, Homebrew/homebrew-core. - a ~60% reduction in the peak memory usage to serve fetches on that same repository. - a collective savings of ~35% in CPU time across all pack-objects invocations serving fetches across all repositories in a single datacenter. Reverse indexes are also beneficial to end-users as well as forges. For example, the time it takes to generate a pack containing the objects for the 10 most recent commits in linux.git (representing a typical push) is significantly faster when on-disk reverse indexes are available: $ { git rev-parse HEAD && printf '^' && git rev-parse HEAD~10 } >in $ hyperfine -L v false,true 'git.compile -c pack.readReverseIndex={v} pack-objects --delta-base-offset --revs --stdout /dev/null' Benchmark 1: git.compile -c pack.readReverseIndex=false pack-objects --delta-base-offset --revs --stdout /dev/null Time (mean ± σ): 543.0 ms ± 20.3 ms [User: 616.2 ms, System: 58.8 ms] Range (min … max): 521.0 ms … 577.9 ms 10 runs Benchmark 2: git.compile -c pack.readReverseIndex=true pack-objects --delta-base-offset --revs --stdout /dev/null Time (mean ± σ): 245.0 ms ± 11.4 ms [User: 335.6 ms, System: 31.3 ms] Range (min … max): 226.0 ms … 259.6 ms 13 runs Summary 'git.compile -c pack.readReverseIndex=true pack-objects --delta-base-offset --revs --stdout /dev/null' ran 2.22 ± 0.13 times faster than 'git.compile -c pack.readReverseIndex=false pack-objects --delta-base-offset --revs --stdout /dev/null' The same is true of writing a pack containing the objects for the 30 most-recent commits: $ { git rev-parse HEAD && printf '^' && git rev-parse HEAD~30 } >in $ hyperfine -L v false,true 'git.compile -c pack.readReverseIndex={v} pack-objects --delta-base-offset --revs --stdout /dev/null' Benchmark 1: git.compile -c pack.readReverseIndex=false pack-objects --delta-base-offset --revs --stdout /dev/null Time (mean ± σ): 866.5 ms ± 16.2 ms [User: 1414.5 ms, System: 97.0 ms] Range (min … max): 839.3 ms … 886.9 ms 10 runs Benchmark 2: git.compile -c pack.readReverseIndex=true pack-objects --delta-base-offset --revs --stdout /dev/null Time (mean ± σ): 581.6 ms ± 10.2 ms [User: 1181.7 ms, System: 62.6 ms] Range (min … max): 567.5 ms … 599.3 ms 10 runs Summary 'git.compile -c pack.readReverseIndex=true pack-objects --delta-base-offset --revs --stdout /dev/null' ran 1.49 ± 0.04 times faster than 'git.compile -c pack.readReverseIndex=false pack-objects --delta-base-offset --revs --stdout /dev/null' ...and savings on trivial operations like computing the on-disk size of a single (packed) object are even more dramatic: $ git rev-parse HEAD >in $ hyperfine -L v false,true 'git.compile -c pack.readReverseIndex={v} cat-file --batch-check="%(objectsize:disk)" --- Documentation/config/pack.txt | 2 +- builtin/index-pack.c | 1 + builtin/pack-objects.c | 1 + t/perf/p5312-pack-bitmaps-revs.sh | 3 +-- t/t5325-reverse-index.sh | 1 + 5 files changed, 5 insertions(+), 3 deletions(-) diff --git a/Documentation/config/pack.txt b/Documentation/config/pack.txt index 7db7fed466..d4c7c9d4e4 100644 --- a/Documentation/config/pack.txt +++ b/Documentation/config/pack.txt @@ -182,4 +182,4 @@ pack.writeReverseIndex:: linkgit:gitformat-pack[5]) for each new packfile that it writes in all places except for linkgit:git-fast-import[1] and in the bulk checkin mechanism. - Defaults to false. + Defaults to true. diff --git a/builtin/index-pack.c b/builtin/index-pack.c index b17e79cd40..323c063f9d 100644 --- a/builtin/index-pack.c +++ b/builtin/index-pack.c @@ -1753,6 +1753,7 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix) fsck_options.walk = mark_link; reset_pack_idx_option(&opts); + opts.flags |= WRITE_REV; git_config(git_index_pack_config, &opts); if (prefix && chdir(prefix)) die(_("Cannot come back to cwd")); diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 77d88f85b0..dbaa04482f 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -4293,6 +4293,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix) } reset_pack_idx_option(&pack_idx_opts); + pack_idx_opts.flags |= WRITE_REV; git_config(git_pack_config, NULL); if (git_env_bool(GIT_TEST_WRITE_REV_INDEX, 0)) pack_idx_opts.flags |= WRITE_REV; diff --git a/t/perf/p5312-pack-bitmaps-revs.sh b/t/perf/p5312-pack-bitmaps-revs.sh index 0684b690af..ceec60656b 100755 --- a/t/perf/p5312-pack-bitmaps-revs.sh +++ b/t/perf/p5312-pack-bitmaps-revs.sh @@ -12,8 +12,7 @@ test_lookup_pack_bitmap () { test_perf_large_repo test_expect_success 'setup bitmap config' ' - git config pack.writebitmaps true && - git config pack.writeReverseIndex true + git config pack.writebitmaps true ' # we need to create the tag up front such that it is covered by the repack and diff --git a/t/t5325-reverse-index.sh b/t/t5325-reverse-index.sh index 66171c1d67..149dcf5193 100755 --- a/t/t5325-reverse-index.sh +++ b/t/t5325-reverse-index.sh @@ -14,6 +14,7 @@ packdir=.git/objects/pack test_expect_success 'setup' ' test_commit base && + test_config pack.writeReverseIndex false && pack=$(git pack-objects --all $packdir/pack) && rev=$packdir/pack-$pack.rev && From patchwork Mon Apr 10 22:53:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13206816 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C19DC77B6F for ; Mon, 10 Apr 2023 22:54:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229946AbjDJWx7 (ORCPT ); Mon, 10 Apr 2023 18:53:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42994 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229507AbjDJWxr (ORCPT ); Mon, 10 Apr 2023 18:53:47 -0400 Received: from mail-yb1-xb31.google.com (mail-yb1-xb31.google.com [IPv6:2607:f8b0:4864:20::b31]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 08613211B for ; Mon, 10 Apr 2023 15:53:46 -0700 (PDT) Received: by mail-yb1-xb31.google.com with SMTP id z9so6211241ybs.9 for ; Mon, 10 Apr 2023 15:53:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20210112.gappssmtp.com; s=20210112; t=1681167225; x=1683759225; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=fziDqg2Z1UHsm3H8cJvq5x//xNnwMbr1AO0i2gwl/KM=; b=urg/siUdW0I7aIf4JPsYkRwueUWqDaZVoo038D+eM5Y0/Cmz/JmwjT5G/59HbtEkXE bA8ZijxiXmsxBcouwa2cH6kYM4r66nSIAkefP/irzwOVdmPzj+cQEQ9kzCizAdbeq0Jc +9EzwtbTN0WH+HFMIXzctQsS1iFGZruaP4LtwLfu9mj2MNAhYJy+VXmleIR1ydTwXfGP Fq2O9IBqxeOhkMIQVRMrvReUoQXSHuGMCvK+CU/Rp41JEYKrpyOZofI7gf/ljhmu99tt ZH5D6CTrlfC1kdYQhXtrPUY12zucnt+w1uoahkimKx4Gt+rEo+3zv6UlxBEpjiJUhgOA A8+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1681167225; x=1683759225; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=fziDqg2Z1UHsm3H8cJvq5x//xNnwMbr1AO0i2gwl/KM=; b=1B27jRMzANeH/Q+obJguRtg3Uks7L6ruOcmnFM3PKjuHwaRU8aV4+1C+p4p/lNSvgd Wka7s3Na6GWK8Go+KSEQJTVzBgnsrhCMAr3/kSbeXAoEbCUPhpgZ4pQM9LkQovGZ/+kY /CZIko+NBDQwSGuWi6v5v/MFEugR7yNQVbzZ65aZLD0mCvM8cH9vX4041Lo+vgRUWxA/ YtQJq7LI7juGJgU7JEY5PaNdVgBzJtoPacsKuZeOtRnBHVyscgtv0irbnbYRGPTPm7NE A/eBgLPWplD2OonuaAt0cUlmulMS4JGZ+vPDQDhLyCmL03uueK3X4oryEDmc7kKECBeH Fedw== X-Gm-Message-State: AAQBX9dMUnX0D4QG1FAwoScLIC3XR2Gpi6eN12SYmIceiRMo3hPn0vYy 53ywhOwAaME6ujbLWHQyY8IpJVHiIrnvdbitS5iOjA== X-Google-Smtp-Source: AKy350aOqGa+XksLVRi2JBbah/eeyTFQt1EOg6fB3np4zVJR01FYsTUiF4UgCwUYhQj4rdzp5jPHvw== X-Received: by 2002:a25:a265:0:b0:b8c:49c:85cf with SMTP id b92-20020a25a265000000b00b8c049c85cfmr9928419ybi.3.1681167225036; Mon, 10 Apr 2023 15:53:45 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id v72-20020a252f4b000000b00b7767ca747csm3194951ybv.25.2023.04.10.15.53.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Apr 2023 15:53:44 -0700 (PDT) Date: Mon, 10 Apr 2023 18:53:43 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Derrick Stolee , Jeff King , Junio C Hamano Subject: [PATCH 7/7] t: invert `GIT_TEST_WRITE_REV_INDEX` Message-ID: <9c80379958824ac8fb7834f4f98000d11c3dc4e0.1681166596.git.me@ttaylorr.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Back in e8c58f894b (t: support GIT_TEST_WRITE_REV_INDEX, 2021-01-25), we added a test knob to conditionally enable writing a ".rev" file when indexing a pack. At the time, this was used to ensure that the test suite worked even when ".rev" files were written, which served as a stress-test for the on-disk reverse index implementation. Now that reading from on-disk ".rev" files is enabled by default, the test knob `GIT_TEST_WRITE_REV_INDEX` no longer has any meaning. We could get rid of the option entirely, but there would be no convenient way to test Git when ".rev" files *aren't* in place. Instead of getting rid of the option, invert its meaning to instead disable writing ".rev" files, thereby running the test suite in a mode where the reverse index is generated from scratch. This ensures that we are still running and exercising Git's behavior when forced to generate reverse indexes from scratch. Signed-off-by: Taylor Blau --- builtin/index-pack.c | 4 ++-- builtin/pack-objects.c | 4 ++-- ci/run-build-and-tests.sh | 1 - pack-revindex.h | 2 +- t/README | 2 +- t/t5325-reverse-index.sh | 2 +- 6 files changed, 7 insertions(+), 8 deletions(-) diff --git a/builtin/index-pack.c b/builtin/index-pack.c index 323c063f9d..9e36c985cf 100644 --- a/builtin/index-pack.c +++ b/builtin/index-pack.c @@ -1758,8 +1758,8 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix) if (prefix && chdir(prefix)) die(_("Cannot come back to cwd")); - if (git_env_bool(GIT_TEST_WRITE_REV_INDEX, 0)) - rev_index = 1; + if (git_env_bool(GIT_TEST_NO_WRITE_REV_INDEX, 0)) + rev_index = 0; else rev_index = !!(opts.flags & (WRITE_REV_VERIFY | WRITE_REV)); diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index dbaa04482f..1797871ce9 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -4295,8 +4295,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix) reset_pack_idx_option(&pack_idx_opts); pack_idx_opts.flags |= WRITE_REV; git_config(git_pack_config, NULL); - if (git_env_bool(GIT_TEST_WRITE_REV_INDEX, 0)) - pack_idx_opts.flags |= WRITE_REV; + if (git_env_bool(GIT_TEST_NO_WRITE_REV_INDEX, 0)) + pack_idx_opts.flags &= ~WRITE_REV; progress = isatty(2); argc = parse_options(argc, argv, prefix, pack_objects_options, diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh index b098e10f52..e9fbfb6345 100755 --- a/ci/run-build-and-tests.sh +++ b/ci/run-build-and-tests.sh @@ -27,7 +27,6 @@ linux-TEST-vars) export GIT_TEST_MULTI_PACK_INDEX=1 export GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=1 export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master - export GIT_TEST_WRITE_REV_INDEX=1 export GIT_TEST_CHECKOUT_WORKERS=2 ;; linux-clang) diff --git a/pack-revindex.h b/pack-revindex.h index ef8afee88b..46e834064e 100644 --- a/pack-revindex.h +++ b/pack-revindex.h @@ -34,7 +34,7 @@ #define RIDX_SIGNATURE 0x52494458 /* "RIDX" */ #define RIDX_VERSION 1 -#define GIT_TEST_WRITE_REV_INDEX "GIT_TEST_WRITE_REV_INDEX" +#define GIT_TEST_NO_WRITE_REV_INDEX "GIT_TEST_NO_WRITE_REV_INDEX" #define GIT_TEST_REV_INDEX_DIE_IN_MEMORY "GIT_TEST_REV_INDEX_DIE_IN_MEMORY" #define GIT_TEST_REV_INDEX_DIE_ON_DISK "GIT_TEST_REV_INDEX_DIE_ON_DISK" diff --git a/t/README b/t/README index 29576c3748..bdfac4cceb 100644 --- a/t/README +++ b/t/README @@ -475,7 +475,7 @@ GIT_TEST_DEFAULT_HASH= specifies which hash algorithm to use in the test scripts. Recognized values for are "sha1" and "sha256". -GIT_TEST_WRITE_REV_INDEX=, when true enables the +GIT_TEST_NO_WRITE_REV_INDEX=, when true disables the 'pack.writeReverseIndex' setting. GIT_TEST_SPARSE_INDEX=, when true enables index writes to use the diff --git a/t/t5325-reverse-index.sh b/t/t5325-reverse-index.sh index 149dcf5193..0548fce1aa 100755 --- a/t/t5325-reverse-index.sh +++ b/t/t5325-reverse-index.sh @@ -7,7 +7,7 @@ TEST_PASSES_SANITIZE_LEAK=true # The below tests want control over the 'pack.writeReverseIndex' setting # themselves to assert various combinations of it with other options. -sane_unset GIT_TEST_WRITE_REV_INDEX +sane_unset GIT_TEST_NO_WRITE_REV_INDEX packdir=.git/objects/pack