From patchwork Thu May 2 06:51:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13651323 Received: from fout4-smtp.messagingengine.com (fout4-smtp.messagingengine.com [103.168.172.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DA86221350 for ; Thu, 2 May 2024 06:51:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.147 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714632699; cv=none; b=K357xuIKDMRJgeRFaL1WaPsvdP4UGVy+H3JqQz44UnlviymsMZpFXo5By6KvaGLN7JhGlQXkwiL5mnMgRFNHOQjODQCe7B20JpkWkS5I17yh+1faRzFoayp1YiQDN0SqBOFB76egSsLyl2D+S3TpED8dAwti1OdwIB1RvS0Y0kc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714632699; c=relaxed/simple; bh=+/VLM4ZRH9dlTzCe/ZC5CCxiD2BM4WNu4PoccHTfgdA=; h=Date:From:To:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=F/qc9dHFBoBP9r1igqgQ+1YCC8qlNgBASRbe/8BABICBPe11Txe6kdP0baLvzg3UBJ4D7ZERHcHtkF1Gc70zL37dFXS0f5Dzj0hkQe1OrAt5uOVp0hgwyzaANyQ51E3lvjI5hytLwWc74NZHCmql3J0o20yTcKWYeSH8BSBxZew= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=jOZ1/USG; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=LXlrPuXo; arc=none smtp.client-ip=103.168.172.147 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="jOZ1/USG"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="LXlrPuXo" Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailfout.nyi.internal (Postfix) with ESMTP id E73B31380395 for ; Thu, 2 May 2024 02:51:35 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Thu, 02 May 2024 02:51:35 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1714632695; x=1714719095; bh=8wfiyAtqFI UlSo3lWfIz27RONxU2B+2CkTZzKEZ7C+U=; b=jOZ1/USGnCVg8DmSCLD9WfK7ln L+iwpHHj/yZaMLqA5qXoVRPvLVA6WY5HD9t4cBXIk8qkizU70ONLE4ayVlcvWEsP 89JzFQpf2Dbz24vqitQ20Dm94sEj0s8y7pCwCxFyExAAG2QVO0A3S8881y5dvFIa U6rgYr7p15StFH1IrBiUx94QF9ZKlfJgsKcKl9I6TEkp2XThcm8dICIAXC1JJ/5G ADLFp7Lf9auxPdCQjI101fFtNvknEI3obEwcCt5Iv7+gu85IPd0DJo3AhGRXspm1 bjEAZU+wNjj0x0hFvmhmiHuL92MDiocCG/bYPAD9iON+TSS0z6s9XYd4gl5w== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1714632695; x=1714719095; bh=8wfiyAtqFIUlSo3lWfIz27RONxU2 B+2CkTZzKEZ7C+U=; b=LXlrPuXo4E9D5a2736MjJyHKCXSfK//J0XQfGZa6ZdqO DRC8Aoo4+kgbTgIp1f+Y46smTS135dqtgr5iGREL7zeRGgrN57vyMJO22tvh3bAi poHqf14T5Qp9ZnKFlfXZKk43ALXekZSZ8s3rEz+tTDNssVJT41rLqee9Wuu4LL+N WTMhj7f2TQ3Oixne/dy1xfHxI12G8fNWtitQQ7I1kvPyXiE9zsbLXredG2MTV0ts ihdJmzsigxgw0zekpkkbKhP3j3zRI2HRGO0d8s0r0lbmEkkJlVG/XFwy83Zn3u1x p96f4pQ3btXKZ8FG47fR4HkipIGFH6ggjFpJmr9QAQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvddujedguddutdcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeehgefhtdefueffheekgfffudelffejtd fhvdejkedthfehvdelgfetgfdvtedthfenucevlhhushhtvghrufhiiigvpedtnecurfgr rhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Thu, 2 May 2024 02:51:35 -0400 (EDT) Received: by localhost (OpenSMTPD) with ESMTPSA id bf17e69f (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Thu, 2 May 2024 06:51:09 +0000 (UTC) Date: Thu, 2 May 2024 08:51:32 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Subject: [PATCH 01/11] reftable: consistently refer to `reftable_write_options` as `opts` Message-ID: <47cee6e25ebaac5af9fd0c9eaad7fc50b2547cad.1714630191.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Throughout the reftable library the `reftable_write_options` are sometimes referred to as `cfg` and sometimes as `opts`. Unify these to consistently use `opts` to avoid confusion. While at it, touch up the coding style a bit by removing unneeded braces around one-line statements and newlines between variable declarations. Signed-off-by: Patrick Steinhardt --- reftable/dump.c | 4 +- reftable/reftable-stack.h | 2 +- reftable/stack.c | 43 +++++++------- reftable/stack.h | 2 +- reftable/stack_test.c | 114 +++++++++++++++++--------------------- 5 files changed, 75 insertions(+), 90 deletions(-) diff --git a/reftable/dump.c b/reftable/dump.c index 26e0393c7d..9c770a10cc 100644 --- a/reftable/dump.c +++ b/reftable/dump.c @@ -27,9 +27,9 @@ license that can be found in the LICENSE file or at static int compact_stack(const char *stackdir) { struct reftable_stack *stack = NULL; - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options opts = { 0 }; - int err = reftable_new_stack(&stack, stackdir, cfg); + int err = reftable_new_stack(&stack, stackdir, opts); if (err < 0) goto done; diff --git a/reftable/reftable-stack.h b/reftable/reftable-stack.h index 1b602dda58..9c8e4eef49 100644 --- a/reftable/reftable-stack.h +++ b/reftable/reftable-stack.h @@ -29,7 +29,7 @@ struct reftable_stack; * stored in 'dir'. Typically, this should be .git/reftables. */ int reftable_new_stack(struct reftable_stack **dest, const char *dir, - struct reftable_write_options config); + struct reftable_write_options opts); /* returns the update_index at which a next table should be written. */ uint64_t reftable_stack_next_update_index(struct reftable_stack *st); diff --git a/reftable/stack.c b/reftable/stack.c index 80266bcbab..3979657793 100644 --- a/reftable/stack.c +++ b/reftable/stack.c @@ -56,15 +56,14 @@ static int reftable_fd_flush(void *arg) } int reftable_new_stack(struct reftable_stack **dest, const char *dir, - struct reftable_write_options config) + struct reftable_write_options opts) { struct reftable_stack *p = reftable_calloc(1, sizeof(*p)); struct strbuf list_file_name = STRBUF_INIT; int err = 0; - if (config.hash_id == 0) { - config.hash_id = GIT_SHA1_FORMAT_ID; - } + if (opts.hash_id == 0) + opts.hash_id = GIT_SHA1_FORMAT_ID; *dest = NULL; @@ -75,7 +74,7 @@ int reftable_new_stack(struct reftable_stack **dest, const char *dir, p->list_file = strbuf_detach(&list_file_name, NULL); p->list_fd = -1; p->reftable_dir = xstrdup(dir); - p->config = config; + p->opts = opts; err = reftable_stack_reload_maybe_reuse(p, 1); if (err < 0) { @@ -257,7 +256,7 @@ static int reftable_stack_reload_once(struct reftable_stack *st, char **names, /* success! */ err = reftable_new_merged_table(&new_merged, new_tables, - new_readers_len, st->config.hash_id); + new_readers_len, st->opts.hash_id); if (err < 0) goto done; @@ -580,8 +579,8 @@ static int reftable_stack_init_addition(struct reftable_addition *add, } goto done; } - if (st->config.default_permissions) { - if (chmod(add->lock_file->filename.buf, st->config.default_permissions) < 0) { + if (st->opts.default_permissions) { + if (chmod(add->lock_file->filename.buf, st->opts.default_permissions) < 0) { err = REFTABLE_IO_ERROR; goto done; } @@ -680,7 +679,7 @@ int reftable_addition_commit(struct reftable_addition *add) if (err) goto done; - if (!add->stack->config.disable_auto_compact) { + if (!add->stack->opts.disable_auto_compact) { /* * Auto-compact the stack to keep the number of tables in * control. It is possible that a concurrent writer is already @@ -758,9 +757,9 @@ int reftable_addition_add(struct reftable_addition *add, err = REFTABLE_IO_ERROR; goto done; } - if (add->stack->config.default_permissions) { + if (add->stack->opts.default_permissions) { if (chmod(get_tempfile_path(tab_file), - add->stack->config.default_permissions)) { + add->stack->opts.default_permissions)) { err = REFTABLE_IO_ERROR; goto done; } @@ -768,7 +767,7 @@ int reftable_addition_add(struct reftable_addition *add, tab_fd = get_tempfile_fd(tab_file); wr = reftable_new_writer(reftable_fd_write, reftable_fd_flush, &tab_fd, - &add->stack->config); + &add->stack->opts); err = write_table(wr, arg); if (err < 0) goto done; @@ -855,14 +854,14 @@ static int stack_compact_locked(struct reftable_stack *st, } tab_fd = get_tempfile_fd(tab_file); - if (st->config.default_permissions && - chmod(get_tempfile_path(tab_file), st->config.default_permissions) < 0) { + if (st->opts.default_permissions && + chmod(get_tempfile_path(tab_file), st->opts.default_permissions) < 0) { err = REFTABLE_IO_ERROR; goto done; } wr = reftable_new_writer(reftable_fd_write, reftable_fd_flush, - &tab_fd, &st->config); + &tab_fd, &st->opts); err = stack_write_compact(st, wr, first, last, config); if (err < 0) goto done; @@ -910,7 +909,7 @@ static int stack_write_compact(struct reftable_stack *st, st->readers[last]->max_update_index); err = reftable_new_merged_table(&mt, subtabs, subtabs_len, - st->config.hash_id); + st->opts.hash_id); if (err < 0) { reftable_free(subtabs); goto done; @@ -1100,9 +1099,9 @@ static int stack_compact_range(struct reftable_stack *st, goto done; } - if (st->config.default_permissions) { + if (st->opts.default_permissions) { if (chmod(get_lock_file_path(&tables_list_lock), - st->config.default_permissions) < 0) { + st->opts.default_permissions) < 0) { err = REFTABLE_IO_ERROR; goto done; } @@ -1292,7 +1291,7 @@ static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st) { uint64_t *sizes = reftable_calloc(st->merged->stack_len, sizeof(*sizes)); - int version = (st->config.hash_id == GIT_SHA1_FORMAT_ID) ? 1 : 2; + int version = (st->opts.hash_id == GIT_SHA1_FORMAT_ID) ? 1 : 2; int overhead = header_size(version) - 1; int i = 0; for (i = 0; i < st->merged->stack_len; i++) { @@ -1368,7 +1367,7 @@ static int stack_check_addition(struct reftable_stack *st, int len = 0; int i = 0; - if (st->config.skip_name_check) + if (st->opts.skip_name_check) return 0; err = reftable_block_source_from_file(&src, new_tab_name); @@ -1500,11 +1499,11 @@ int reftable_stack_clean(struct reftable_stack *st) int reftable_stack_print_directory(const char *stackdir, uint32_t hash_id) { struct reftable_stack *stack = NULL; - struct reftable_write_options cfg = { .hash_id = hash_id }; + struct reftable_write_options opts = { .hash_id = hash_id }; struct reftable_merged_table *merged = NULL; struct reftable_table table = { NULL }; - int err = reftable_new_stack(&stack, stackdir, cfg); + int err = reftable_new_stack(&stack, stackdir, opts); if (err < 0) goto done; diff --git a/reftable/stack.h b/reftable/stack.h index d43efa4760..97d7ebc043 100644 --- a/reftable/stack.h +++ b/reftable/stack.h @@ -20,7 +20,7 @@ struct reftable_stack { char *reftable_dir; - struct reftable_write_options config; + struct reftable_write_options opts; struct reftable_reader **readers; size_t readers_len; diff --git a/reftable/stack_test.c b/reftable/stack_test.c index 1df3ffce52..3316d55f19 100644 --- a/reftable/stack_test.c +++ b/reftable/stack_test.c @@ -150,7 +150,7 @@ static void test_reftable_stack_add_one(void) char *dir = get_tmp_dir(__LINE__); struct strbuf scratch = STRBUF_INIT; int mask = umask(002); - struct reftable_write_options cfg = { + struct reftable_write_options opts = { .default_permissions = 0660, }; struct reftable_stack *st = NULL; @@ -163,7 +163,7 @@ static void test_reftable_stack_add_one(void) }; struct reftable_ref_record dest = { NULL }; struct stat stat_result = { 0 }; - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); err = reftable_stack_add(st, &write_test_ref, &ref); @@ -186,7 +186,7 @@ static void test_reftable_stack_add_one(void) strbuf_addstr(&scratch, "/tables.list"); err = stat(scratch.buf, &stat_result); EXPECT(!err); - EXPECT((stat_result.st_mode & 0777) == cfg.default_permissions); + EXPECT((stat_result.st_mode & 0777) == opts.default_permissions); strbuf_reset(&scratch); strbuf_addstr(&scratch, dir); @@ -195,7 +195,7 @@ static void test_reftable_stack_add_one(void) strbuf_addstr(&scratch, st->readers[0]->name); err = stat(scratch.buf, &stat_result); EXPECT(!err); - EXPECT((stat_result.st_mode & 0777) == cfg.default_permissions); + EXPECT((stat_result.st_mode & 0777) == opts.default_permissions); #else (void) stat_result; #endif @@ -209,7 +209,7 @@ static void test_reftable_stack_add_one(void) static void test_reftable_stack_uptodate(void) { - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options opts = { 0 }; struct reftable_stack *st1 = NULL; struct reftable_stack *st2 = NULL; char *dir = get_tmp_dir(__LINE__); @@ -232,10 +232,10 @@ static void test_reftable_stack_uptodate(void) /* simulate multi-process access to the same stack by creating two stacks for the same directory. */ - err = reftable_new_stack(&st1, dir, cfg); + err = reftable_new_stack(&st1, dir, opts); EXPECT_ERR(err); - err = reftable_new_stack(&st2, dir, cfg); + err = reftable_new_stack(&st2, dir, opts); EXPECT_ERR(err); err = reftable_stack_add(st1, &write_test_ref, &ref1); @@ -257,8 +257,7 @@ static void test_reftable_stack_uptodate(void) static void test_reftable_stack_transaction_api(void) { char *dir = get_tmp_dir(__LINE__); - - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options opts = { 0 }; struct reftable_stack *st = NULL; int err; struct reftable_addition *add = NULL; @@ -271,8 +270,7 @@ static void test_reftable_stack_transaction_api(void) }; struct reftable_ref_record dest = { NULL }; - - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); reftable_addition_destroy(add); @@ -301,12 +299,12 @@ static void test_reftable_stack_transaction_api(void) static void test_reftable_stack_transaction_api_performs_auto_compaction(void) { char *dir = get_tmp_dir(__LINE__); - struct reftable_write_options cfg = {0}; + struct reftable_write_options opts = {0}; struct reftable_addition *add = NULL; struct reftable_stack *st = NULL; int i, n = 20, err; - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); for (i = 0; i <= n; i++) { @@ -325,7 +323,7 @@ static void test_reftable_stack_transaction_api_performs_auto_compaction(void) * we can ensure that we indeed honor this setting and have * better control over when exactly auto compaction runs. */ - st->config.disable_auto_compact = i != n; + st->opts.disable_auto_compact = i != n; err = reftable_stack_new_addition(&add, st); EXPECT_ERR(err); @@ -361,13 +359,13 @@ static void test_reftable_stack_auto_compaction_fails_gracefully(void) .value_type = REFTABLE_REF_VAL1, .value.val1 = {0x01}, }; - struct reftable_write_options cfg = {0}; + struct reftable_write_options opts = {0}; struct reftable_stack *st; struct strbuf table_path = STRBUF_INIT; char *dir = get_tmp_dir(__LINE__); int err; - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); err = reftable_stack_add(st, write_test_ref, &ref); @@ -442,8 +440,7 @@ static int write_error(struct reftable_writer *wr, void *arg) static void test_reftable_stack_update_index_check(void) { char *dir = get_tmp_dir(__LINE__); - - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options opts = { 0 }; struct reftable_stack *st = NULL; int err; struct reftable_ref_record ref1 = { @@ -459,7 +456,7 @@ static void test_reftable_stack_update_index_check(void) .value.symref = "master", }; - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); err = reftable_stack_add(st, &write_test_ref, &ref1); @@ -474,12 +471,11 @@ static void test_reftable_stack_update_index_check(void) static void test_reftable_stack_lock_failure(void) { char *dir = get_tmp_dir(__LINE__); - - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options opts = { 0 }; struct reftable_stack *st = NULL; int err, i; - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); for (i = -1; i != REFTABLE_EMPTY_TABLE_ERROR; i--) { err = reftable_stack_add(st, &write_error, &i); @@ -494,7 +490,7 @@ static void test_reftable_stack_add(void) { int i = 0; int err = 0; - struct reftable_write_options cfg = { + struct reftable_write_options opts = { .exact_log_message = 1, .default_permissions = 0660, .disable_auto_compact = 1, @@ -507,7 +503,7 @@ static void test_reftable_stack_add(void) struct stat stat_result; int N = ARRAY_SIZE(refs); - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); for (i = 0; i < N; i++) { @@ -566,7 +562,7 @@ static void test_reftable_stack_add(void) strbuf_addstr(&path, "/tables.list"); err = stat(path.buf, &stat_result); EXPECT(!err); - EXPECT((stat_result.st_mode & 0777) == cfg.default_permissions); + EXPECT((stat_result.st_mode & 0777) == opts.default_permissions); strbuf_reset(&path); strbuf_addstr(&path, dir); @@ -575,7 +571,7 @@ static void test_reftable_stack_add(void) strbuf_addstr(&path, st->readers[0]->name); err = stat(path.buf, &stat_result); EXPECT(!err); - EXPECT((stat_result.st_mode & 0777) == cfg.default_permissions); + EXPECT((stat_result.st_mode & 0777) == opts.default_permissions); #else (void) stat_result; #endif @@ -593,7 +589,7 @@ static void test_reftable_stack_add(void) static void test_reftable_stack_log_normalize(void) { int err = 0; - struct reftable_write_options cfg = { + struct reftable_write_options opts = { 0, }; struct reftable_stack *st = NULL; @@ -617,7 +613,7 @@ static void test_reftable_stack_log_normalize(void) .update_index = 1, }; - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); input.value.update.message = "one\ntwo"; @@ -650,8 +646,7 @@ static void test_reftable_stack_tombstone(void) { int i = 0; char *dir = get_tmp_dir(__LINE__); - - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options opts = { 0 }; struct reftable_stack *st = NULL; int err; struct reftable_ref_record refs[2] = { { NULL } }; @@ -660,8 +655,7 @@ static void test_reftable_stack_tombstone(void) struct reftable_ref_record dest = { NULL }; struct reftable_log_record log_dest = { NULL }; - - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); /* even entries add the refs, odd entries delete them. */ @@ -729,8 +723,7 @@ static void test_reftable_stack_tombstone(void) static void test_reftable_stack_hash_id(void) { char *dir = get_tmp_dir(__LINE__); - - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options opts = { 0 }; struct reftable_stack *st = NULL; int err; @@ -740,24 +733,24 @@ static void test_reftable_stack_hash_id(void) .value.symref = "target", .update_index = 1, }; - struct reftable_write_options cfg32 = { .hash_id = GIT_SHA256_FORMAT_ID }; + struct reftable_write_options opts32 = { .hash_id = GIT_SHA256_FORMAT_ID }; struct reftable_stack *st32 = NULL; - struct reftable_write_options cfg_default = { 0 }; + struct reftable_write_options opts_default = { 0 }; struct reftable_stack *st_default = NULL; struct reftable_ref_record dest = { NULL }; - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); err = reftable_stack_add(st, &write_test_ref, &ref); EXPECT_ERR(err); /* can't read it with the wrong hash ID. */ - err = reftable_new_stack(&st32, dir, cfg32); + err = reftable_new_stack(&st32, dir, opts32); EXPECT(err == REFTABLE_FORMAT_ERROR); - /* check that we can read it back with default config too. */ - err = reftable_new_stack(&st_default, dir, cfg_default); + /* check that we can read it back with default opts too. */ + err = reftable_new_stack(&st_default, dir, opts_default); EXPECT_ERR(err); err = reftable_stack_read_ref(st_default, "master", &dest); @@ -790,8 +783,7 @@ static void test_suggest_compaction_segment_nothing(void) static void test_reflog_expire(void) { char *dir = get_tmp_dir(__LINE__); - - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options opts = { 0 }; struct reftable_stack *st = NULL; struct reftable_log_record logs[20] = { { NULL } }; int N = ARRAY_SIZE(logs) - 1; @@ -802,8 +794,7 @@ static void test_reflog_expire(void) }; struct reftable_log_record log = { NULL }; - - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); for (i = 1; i <= N; i++) { @@ -866,21 +857,19 @@ static int write_nothing(struct reftable_writer *wr, void *arg) static void test_empty_add(void) { - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options opts = { 0 }; struct reftable_stack *st = NULL; int err; char *dir = get_tmp_dir(__LINE__); - struct reftable_stack *st2 = NULL; - - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); err = reftable_stack_add(st, &write_nothing, NULL); EXPECT_ERR(err); - err = reftable_new_stack(&st2, dir, cfg); + err = reftable_new_stack(&st2, dir, opts); EXPECT_ERR(err); clear_dir(dir); reftable_stack_destroy(st); @@ -899,16 +888,15 @@ static int fastlog2(uint64_t sz) static void test_reftable_stack_auto_compaction(void) { - struct reftable_write_options cfg = { + struct reftable_write_options opts = { .disable_auto_compact = 1, }; struct reftable_stack *st = NULL; char *dir = get_tmp_dir(__LINE__); - int err, i; int N = 100; - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); for (i = 0; i < N; i++) { @@ -938,13 +926,13 @@ static void test_reftable_stack_auto_compaction(void) static void test_reftable_stack_add_performs_auto_compaction(void) { - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options opts = { 0 }; struct reftable_stack *st = NULL; struct strbuf refname = STRBUF_INIT; char *dir = get_tmp_dir(__LINE__); int err, i, n = 20; - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); for (i = 0; i <= n; i++) { @@ -959,7 +947,7 @@ static void test_reftable_stack_add_performs_auto_compaction(void) * we can ensure that we indeed honor this setting and have * better control over when exactly auto compaction runs. */ - st->config.disable_auto_compact = i != n; + st->opts.disable_auto_compact = i != n; strbuf_reset(&refname); strbuf_addf(&refname, "branch-%04d", i); @@ -986,14 +974,13 @@ static void test_reftable_stack_add_performs_auto_compaction(void) static void test_reftable_stack_compaction_concurrent(void) { - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options opts = { 0 }; struct reftable_stack *st1 = NULL, *st2 = NULL; char *dir = get_tmp_dir(__LINE__); - int err, i; int N = 3; - err = reftable_new_stack(&st1, dir, cfg); + err = reftable_new_stack(&st1, dir, opts); EXPECT_ERR(err); for (i = 0; i < N; i++) { @@ -1010,7 +997,7 @@ static void test_reftable_stack_compaction_concurrent(void) EXPECT_ERR(err); } - err = reftable_new_stack(&st2, dir, cfg); + err = reftable_new_stack(&st2, dir, opts); EXPECT_ERR(err); err = reftable_stack_compact_all(st1, NULL); @@ -1036,14 +1023,13 @@ static void unclean_stack_close(struct reftable_stack *st) static void test_reftable_stack_compaction_concurrent_clean(void) { - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options opts = { 0 }; struct reftable_stack *st1 = NULL, *st2 = NULL, *st3 = NULL; char *dir = get_tmp_dir(__LINE__); - int err, i; int N = 3; - err = reftable_new_stack(&st1, dir, cfg); + err = reftable_new_stack(&st1, dir, opts); EXPECT_ERR(err); for (i = 0; i < N; i++) { @@ -1060,7 +1046,7 @@ static void test_reftable_stack_compaction_concurrent_clean(void) EXPECT_ERR(err); } - err = reftable_new_stack(&st2, dir, cfg); + err = reftable_new_stack(&st2, dir, opts); EXPECT_ERR(err); err = reftable_stack_compact_all(st1, NULL); @@ -1069,7 +1055,7 @@ static void test_reftable_stack_compaction_concurrent_clean(void) unclean_stack_close(st1); unclean_stack_close(st2); - err = reftable_new_stack(&st3, dir, cfg); + err = reftable_new_stack(&st3, dir, opts); EXPECT_ERR(err); err = reftable_stack_clean(st3); From patchwork Thu May 2 06:51:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13651324 Received: from fhigh8-smtp.messagingengine.com (fhigh8-smtp.messagingengine.com [103.168.172.159]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DE7741CAA5 for ; Thu, 2 May 2024 06:51:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.159 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714632703; cv=none; b=p8CjjckDRXsv7Exbvm+dvalVh1m0/QF9zBTtBzFAGdJVVVwIX2T15NSpbBuNUN0JmC9XUNlgFslPXH16iKMZNZz3ArCEraqx14CvdU69SknuaFm3eZ3duhKGS7EjweB6CoRj6//eiGxgY7ZXkGpVO+3/IeOTawlgQ8g2x5EMx6c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714632703; c=relaxed/simple; bh=ynRGB7lJG8sNhHFo5sRXSb6Apy316AbP2dmhNzAiKXY=; h=Date:From:To:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=KzDEe6maDU29qHzsNug3Ul4sGvnsB1Cp3h9V2YiCP7ftO283dzuNBnjY+O89ZaXIBZQqMC6L4Lve4oeYAEvxCuEH6ytNNmKvogsSJjp000x3Tk0vT4VzTaZFz9k6GVBDRB8tKdaCtg5qTXBSOc+gsd10h9LtDakeo/AatHv2mW4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=ocq5LfwY; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=VkiaQNhE; arc=none smtp.client-ip=103.168.172.159 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="ocq5LfwY"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="VkiaQNhE" Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailfhigh.nyi.internal (Postfix) with ESMTP id D141311400CA for ; Thu, 2 May 2024 02:51:40 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Thu, 02 May 2024 02:51:40 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1714632700; x=1714719100; bh=OIC2Jw9GSe kd3bCME2UpPW8YcZmAwwuASVSTATkHWvQ=; b=ocq5LfwYQwfE/JK6xjdYNjX7xK 01mx/EMpp3giQoofjd5vflgLcX+uVYMMbkgkMIpVXN2e29bNQzWZl/wnoSyHdDk9 bbUWfGIlMOTX1+bx+rLT7bBzmG/O6BLYyR+XSu7kv76A584JIbTb7Apg8asLy46w F67Bk1Po+opcUXrt/1GEhQ28QElNnX5P9Sqgt0R14Dx47C/hi5ulXU83Ipv8vQla UsPNZrza2jRCmlEdej1/eAETEDdXbxrTmaAuDYnSiHPEunMbCdTAi8D8JuP7UaTJ Su0+CRJ+qJON4w7xsaZpXVemd8Esr3qrxPajBJUyDaVaPgEhCwBu3Xd2fFaw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1714632700; x=1714719100; bh=OIC2Jw9GSekd3bCME2UpPW8YcZmA wwuASVSTATkHWvQ=; b=VkiaQNhE6AAdppSpXq3vnG5TMilGM0lRkHQbAO0FJTyZ pSIMgUw/wIFxL0QlynmcGazoSM9Vd1ZWpvXr21mJH5z1X1gKnbPWA22w2I7o6AGv 4GU8LsNs+BZS8OUPVeIsEppun11xFdMf6+zXZ0R0bfOAdeuBgCQlsBRVVKE+6XBK h8UDFxxWfkIcuwP+PQUuwrbLMGqk/F+KnDwEijTzfK3trQBmd0uG+6mpWniysbLb IIjl5a0EPnpCk/8j100P1r145QFBnRWPMBK0S32kLwRjKcgiFZPmoQC8MbjjhSfr v/P6EoS4NuaygXtDu1mtINzl6jQmDZ3IWzq941iWrQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvddujedgudduudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeehgefhtdefueffheekgfffudelffejtd fhvdejkedthfehvdelgfetgfdvtedthfenucevlhhushhtvghrufhiiigvpedtnecurfgr rhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Thu, 2 May 2024 02:51:40 -0400 (EDT) Received: by localhost (OpenSMTPD) with ESMTPSA id e628a170 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Thu, 2 May 2024 06:51:14 +0000 (UTC) Date: Thu, 2 May 2024 08:51:37 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Subject: [PATCH 02/11] reftable: consistently pass write opts as value Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: We sometimes pass the refatble write options as value and sometimes as a pointer. This is quite confusing and makes the reader wonder whether the options get modified sometimes. In fact, `reftable_new_writer()` does cause the caller-provided options to get updated when some values aren't set up. This is quite unexpected, but didn't cause any harm until now. Refactor the code to consistently pass the options as a value so that they are local to the subsystem they are being passed into so that we can avoid weirdness like this. Signed-off-by: Patrick Steinhardt --- reftable/merged_test.c | 6 +++--- reftable/readwrite_test.c | 26 +++++++++++++------------- reftable/refname_test.c | 2 +- reftable/reftable-writer.h | 2 +- reftable/stack.c | 4 ++-- reftable/writer.c | 12 +++++++----- 6 files changed, 27 insertions(+), 25 deletions(-) diff --git a/reftable/merged_test.c b/reftable/merged_test.c index 530fc82d1c..4ac81de8d4 100644 --- a/reftable/merged_test.c +++ b/reftable/merged_test.c @@ -42,7 +42,7 @@ static void write_test_table(struct strbuf *buf, } } - w = reftable_new_writer(&strbuf_add_void, &noop_flush, buf, &opts); + w = reftable_new_writer(&strbuf_add_void, &noop_flush, buf, opts); reftable_writer_set_limits(w, min, max); for (i = 0; i < n; i++) { @@ -70,7 +70,7 @@ static void write_test_log_table(struct strbuf *buf, .exact_log_message = 1, }; struct reftable_writer *w = NULL; - w = reftable_new_writer(&strbuf_add_void, &noop_flush, buf, &opts); + w = reftable_new_writer(&strbuf_add_void, &noop_flush, buf, opts); reftable_writer_set_limits(w, update_index, update_index); for (i = 0; i < n; i++) { @@ -403,7 +403,7 @@ static void test_default_write_opts(void) struct reftable_write_options opts = { 0 }; struct strbuf buf = STRBUF_INIT; struct reftable_writer *w = - reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts); + reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts); struct reftable_ref_record rec = { .refname = "master", diff --git a/reftable/readwrite_test.c b/reftable/readwrite_test.c index a6dbd214c5..27631a041b 100644 --- a/reftable/readwrite_test.c +++ b/reftable/readwrite_test.c @@ -51,7 +51,7 @@ static void write_table(char ***names, struct strbuf *buf, int N, .hash_id = hash_id, }; struct reftable_writer *w = - reftable_new_writer(&strbuf_add_void, &noop_flush, buf, &opts); + reftable_new_writer(&strbuf_add_void, &noop_flush, buf, opts); struct reftable_ref_record ref = { NULL }; int i = 0, n; struct reftable_log_record log = { NULL }; @@ -129,7 +129,7 @@ static void test_log_buffer_size(void) .message = "commit: 9\n", } } }; struct reftable_writer *w = - reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts); + reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts); /* This tests buffer extension for log compression. Must use a random hash, to ensure that the compressed part is larger than the original. @@ -172,7 +172,7 @@ static void test_log_overflow(void) }, }; struct reftable_writer *w = - reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts); + reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts); memset(msg, 'x', sizeof(msg) - 1); reftable_writer_set_limits(w, update_index, update_index); @@ -199,7 +199,7 @@ static void test_log_write_read(void) struct reftable_block_source source = { NULL }; struct strbuf buf = STRBUF_INIT; struct reftable_writer *w = - reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts); + reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts); const struct reftable_stats *stats = NULL; reftable_writer_set_limits(w, 0, N); for (i = 0; i < N; i++) { @@ -288,7 +288,7 @@ static void test_log_zlib_corruption(void) struct reftable_block_source source = { 0 }; struct strbuf buf = STRBUF_INIT; struct reftable_writer *w = - reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts); + reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts); const struct reftable_stats *stats = NULL; char message[100] = { 0 }; int err, i, n; @@ -526,7 +526,7 @@ static void test_table_refs_for(int indexed) struct strbuf buf = STRBUF_INIT; struct reftable_writer *w = - reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts); + reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts); struct reftable_iterator it = { NULL }; int j; @@ -619,7 +619,7 @@ static void test_write_empty_table(void) struct reftable_write_options opts = { 0 }; struct strbuf buf = STRBUF_INIT; struct reftable_writer *w = - reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts); + reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts); struct reftable_block_source source = { NULL }; struct reftable_reader *rd = NULL; struct reftable_ref_record rec = { NULL }; @@ -657,7 +657,7 @@ static void test_write_object_id_min_length(void) }; struct strbuf buf = STRBUF_INIT; struct reftable_writer *w = - reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts); + reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts); struct reftable_ref_record ref = { .update_index = 1, .value_type = REFTABLE_REF_VAL1, @@ -692,7 +692,7 @@ static void test_write_object_id_length(void) }; struct strbuf buf = STRBUF_INIT; struct reftable_writer *w = - reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts); + reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts); struct reftable_ref_record ref = { .update_index = 1, .value_type = REFTABLE_REF_VAL1, @@ -726,7 +726,7 @@ static void test_write_empty_key(void) struct reftable_write_options opts = { 0 }; struct strbuf buf = STRBUF_INIT; struct reftable_writer *w = - reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts); + reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts); struct reftable_ref_record ref = { .refname = "", .update_index = 1, @@ -749,7 +749,7 @@ static void test_write_key_order(void) struct reftable_write_options opts = { 0 }; struct strbuf buf = STRBUF_INIT; struct reftable_writer *w = - reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts); + reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts); struct reftable_ref_record refs[2] = { { .refname = "b", @@ -792,7 +792,7 @@ static void test_write_multiple_indices(void) struct reftable_reader *reader; int err, i; - writer = reftable_new_writer(&strbuf_add_void, &noop_flush, &writer_buf, &opts); + writer = reftable_new_writer(&strbuf_add_void, &noop_flush, &writer_buf, opts); reftable_writer_set_limits(writer, 1, 1); for (i = 0; i < 100; i++) { struct reftable_ref_record ref = { @@ -869,7 +869,7 @@ static void test_write_multi_level_index(void) struct reftable_reader *reader; int err; - writer = reftable_new_writer(&strbuf_add_void, &noop_flush, &writer_buf, &opts); + writer = reftable_new_writer(&strbuf_add_void, &noop_flush, &writer_buf, opts); reftable_writer_set_limits(writer, 1, 1); for (size_t i = 0; i < 200; i++) { struct reftable_ref_record ref = { diff --git a/reftable/refname_test.c b/reftable/refname_test.c index b9cc62554e..3468253be7 100644 --- a/reftable/refname_test.c +++ b/reftable/refname_test.c @@ -30,7 +30,7 @@ static void test_conflict(void) struct reftable_write_options opts = { 0 }; struct strbuf buf = STRBUF_INIT; struct reftable_writer *w = - reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, &opts); + reftable_new_writer(&strbuf_add_void, &noop_flush, &buf, opts); struct reftable_ref_record rec = { .refname = "a/b", .value_type = REFTABLE_REF_SYMREF, diff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h index 155bf0bbe2..44cb986465 100644 --- a/reftable/reftable-writer.h +++ b/reftable/reftable-writer.h @@ -92,7 +92,7 @@ struct reftable_stats { struct reftable_writer * reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t), int (*flush_func)(void *), - void *writer_arg, struct reftable_write_options *opts); + void *writer_arg, struct reftable_write_options opts); /* Set the range of update indices for the records we will add. When writing a table into a stack, the min should be at least diff --git a/reftable/stack.c b/reftable/stack.c index 3979657793..7b4fff7c9e 100644 --- a/reftable/stack.c +++ b/reftable/stack.c @@ -767,7 +767,7 @@ int reftable_addition_add(struct reftable_addition *add, tab_fd = get_tempfile_fd(tab_file); wr = reftable_new_writer(reftable_fd_write, reftable_fd_flush, &tab_fd, - &add->stack->opts); + add->stack->opts); err = write_table(wr, arg); if (err < 0) goto done; @@ -861,7 +861,7 @@ static int stack_compact_locked(struct reftable_stack *st, } wr = reftable_new_writer(reftable_fd_write, reftable_fd_flush, - &tab_fd, &st->opts); + &tab_fd, st->opts); err = stack_write_compact(st, wr, first, last, config); if (err < 0) goto done; diff --git a/reftable/writer.c b/reftable/writer.c index 1d9ff0fbfa..ad2f2e6c65 100644 --- a/reftable/writer.c +++ b/reftable/writer.c @@ -122,20 +122,22 @@ static struct strbuf reftable_empty_strbuf = STRBUF_INIT; struct reftable_writer * reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t), int (*flush_func)(void *), - void *writer_arg, struct reftable_write_options *opts) + void *writer_arg, struct reftable_write_options opts) { struct reftable_writer *wp = reftable_calloc(1, sizeof(*wp)); strbuf_init(&wp->block_writer_data.last_key, 0); - options_set_defaults(opts); - if (opts->block_size >= (1 << 24)) { + + options_set_defaults(&opts); + if (opts.block_size >= (1 << 24)) { /* TODO - error return? */ abort(); } + wp->last_key = reftable_empty_strbuf; - REFTABLE_CALLOC_ARRAY(wp->block, opts->block_size); + REFTABLE_CALLOC_ARRAY(wp->block, opts.block_size); wp->write = writer_func; wp->write_arg = writer_arg; - wp->opts = *opts; + wp->opts = opts; wp->flush = flush_func; writer_reinit_block_writer(wp, BLOCK_TYPE_REF); From patchwork Thu May 2 06:51:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13651325 Received: from fout4-smtp.messagingengine.com (fout4-smtp.messagingengine.com [103.168.172.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AD3AF225D6 for ; Thu, 2 May 2024 06:51:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.147 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714632708; cv=none; b=ur5Mioo+YTLRhVwNoHHQafaGc6uBjSz8J/HXPSg8WVGzZvbVdqA+jFwhXLl3X66mvk4iMGujnt8SZrz7MKP4QEGl9qry69e9DH6MVJrp5lq4uuMliIia7pqeaeslexDAm7lVXWqcYzfn1ZlmkqPogXERp08c8lXJtu4YxXxxZqQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714632708; c=relaxed/simple; bh=u263LjXluUEDJfg1Jt1LBJ/dfzS9bYocg0nPZv/yWYE=; h=Date:From:To:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=W4dp1Oqodc4W5wPtJAd6NLBqMt7Gk1myBH44bjQwlhSRdgvEcLSZReUIuyNclnQ4M8bC8HUfryD0711rs74WuJHpS2SIK0Whzd/epVfMGdBL4mRAUt/7gVTE9Vu3KbCgA55ChifVydMp5BygSl3BgKWYEn25Gv9ozh+5qMQOKsA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=hK9wHrdS; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=Eg01xCly; arc=none smtp.client-ip=103.168.172.147 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="hK9wHrdS"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="Eg01xCly" Received: from compute7.internal (compute7.nyi.internal [10.202.2.48]) by mailfout.nyi.internal (Postfix) with ESMTP id C7EE21380394 for ; Thu, 2 May 2024 02:51:45 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute7.internal (MEProxy); Thu, 02 May 2024 02:51:45 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1714632705; x=1714719105; bh=wsVg0F0Vpb pWrQ5OGAYm1ym135bAYULGznb7wQXP+Is=; b=hK9wHrdSp6CE2zvKs9aINnZV2e gAur2oT7T3M2u6Ts+5EEIojvwTxypwnge9wcjr0h25KE4rEAoGh1yRiM/4mzBmFn 0Lb7KhpSo5Xm4wefkrTrt7MD8XXnWOhtqdO4e1tRILvNDhbdRryyy46s3f9jT5yL PzmrsLv1rV5NQGOmletkcHTBfRFS2CRM83GSmNUQVirzZpCPy+Wn7HWgP67l6p/E Ed/5nJr3s51fqnS8dRq/msAzCkOL4INBf5f8xrHyrbt9atEuPGR38z6sxrnLplh1 GpzpaU3CF33pGnQT6vLzNGa5SGCy7csP0sGOdu0jjjjrc3VEFBW9Ke/yu7lg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1714632705; x=1714719105; bh=wsVg0F0VpbpWrQ5OGAYm1ym135bA YULGznb7wQXP+Is=; b=Eg01xCly13hpfuD+fHx2CJt8Ru4pBwUUIcuRdRbvYb9j w7F6y5zgSQd7RMpMKe5Lce3FK6VAX+tQ2id+apYnGsjfHhFhMo2bpEGMAf+G7TCW iqiPFbmoS0xhx3j/0FSJdYqe7GwwuYwbXgdMYTqW0QeAdt4i3Khi1BcV6oZt/oXR lverN5f7ISZGWm69D8G4DGat/Z61roTcvAtqujl0RJV79LRzxCtafyL5bJewhUBx nBilr6shjsO2Za2PdMiCv51T7tlWAEBQ3BPVT+eiXo4QxiCg27NEgXyW2Lljd7M0 K3nXxJq4svUdMO2tqQHPpKe0CbYUFnUhMKUB/SuIQw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvddujedgudduudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeehgefhtdefueffheekgfffudelffejtd fhvdejkedthfehvdelgfetgfdvtedthfenucevlhhushhtvghrufhiiigvpedtnecurfgr rhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Thu, 2 May 2024 02:51:44 -0400 (EDT) Received: by localhost (OpenSMTPD) with ESMTPSA id 2821365e (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Thu, 2 May 2024 06:51:19 +0000 (UTC) Date: Thu, 2 May 2024 08:51:42 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Subject: [PATCH 03/11] reftable/writer: drop static variable used to initialize strbuf Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: We have a static variable in the reftable writer code that is merely used to initialize the `last_key` of the writer. Convert the code to instead use `strbuf_init()` and drop the variable. Signed-off-by: Patrick Steinhardt --- reftable/writer.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/reftable/writer.c b/reftable/writer.c index ad2f2e6c65..7df6e53699 100644 --- a/reftable/writer.c +++ b/reftable/writer.c @@ -117,15 +117,12 @@ static void writer_reinit_block_writer(struct reftable_writer *w, uint8_t typ) w->block_writer->restart_interval = w->opts.restart_interval; } -static struct strbuf reftable_empty_strbuf = STRBUF_INIT; - struct reftable_writer * reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t), int (*flush_func)(void *), void *writer_arg, struct reftable_write_options opts) { struct reftable_writer *wp = reftable_calloc(1, sizeof(*wp)); - strbuf_init(&wp->block_writer_data.last_key, 0); options_set_defaults(&opts); if (opts.block_size >= (1 << 24)) { @@ -133,7 +130,8 @@ reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t), abort(); } - wp->last_key = reftable_empty_strbuf; + strbuf_init(&wp->block_writer_data.last_key, 0); + strbuf_init(&wp->last_key, 0); REFTABLE_CALLOC_ARRAY(wp->block, opts.block_size); wp->write = writer_func; wp->write_arg = writer_arg; From patchwork Thu May 2 06:51:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13651326 Received: from fout4-smtp.messagingengine.com (fout4-smtp.messagingengine.com [103.168.172.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 932B828DC3 for ; Thu, 2 May 2024 06:51:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.147 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714632713; cv=none; b=nBqR1Jd/kGbL9pPaq1D/bTQAEZ48WOjSsXUCrN9E/FuxgLps2zF9Lf2206PIfp9TvJztuHzke6RghMAUgRJsrfPN3oNMxPX4ATQax3umTGwncWzMQWfaWdhFlpThOJS4v0C7NhvmZJkzQGYgfGAsvTnsrAtQQ10RqnJgzB+IUko= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714632713; c=relaxed/simple; bh=Zsi3P/Ka3OntghPbCcI74VWsQsYecX9nG5snxyHfOJM=; h=Date:From:To:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=DWZ95hnGfU93vZ8ZD9rSgsSdH13ipvjmQ6caWofTnQ7Ct0mhwzqCkMV8OrBmwcJOrqLoxCdA6HhQDSO4kY/1tFw87Dk4b7iFIjEORaOlrQejPt8bwcgJMmbQAqM0JUfXi/qbaTk7yJ0W2jh4UjyP3DdEqeyv7dpObOUo2IrjA10= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=I7dzE6lu; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=UOd5XHw2; arc=none smtp.client-ip=103.168.172.147 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="I7dzE6lu"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="UOd5XHw2" Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailfout.nyi.internal (Postfix) with ESMTP id B1DAB1380394 for ; Thu, 2 May 2024 02:51:50 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Thu, 02 May 2024 02:51:50 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1714632710; x=1714719110; bh=UGjiQj9FCH ZDK9/bLNveM9hY036D08Xx3Kn/ywiZv7I=; b=I7dzE6lu5LXxur0Vb0OsmlJPI+ iysofW7IobIQhwgRIsMorK2WYmQguu5TNGHeqSWs9Qj/excelV2VEIfP2Mg4rRtr GpWapI6R0Wu/MShE3IFk3MIYpoPU8TVP89iUpHTqm933b3cX/w51ZW7fEzlJRMEN mnIt8FiSWgc9Vs/DW7dfAdstGgPzf/OtXAGvs9Ha1kSo1kRdQWjplAb/r5KEUm4c Szi+uCSRP6EbEstU/3uMxJrC5Y2ZLXKomJy4mGdYX+TwwkKBm2mNoYM+bCYvxn80 UIwhGWqeV887D2tjskW+V1ZhSFpPVIZ7Xs4fze5pJ58CwyUz3pPlcErqv8Cw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1714632710; x=1714719110; bh=UGjiQj9FCHZDK9/bLNveM9hY036D 08Xx3Kn/ywiZv7I=; b=UOd5XHw2g+MUWPTtYVBk+YBbQwY5u+3FjjvAWBur2h/q c1kJwFtrwHQpOjB+7F/KiHfuOR4YdraeFTUvQVqMzWWjPs10rJvfkYNXgXO8Wz7P MTtMlZe6UWYssEPnMv7DXBlmTtK/n3GHcdBtOriPj0Jqf+FaBHTtid4rI+Mm7EPc /2LDJMQfrHvIAoWu9Yo9rhWeDihDWn4DKztsC64H51mWglDLMmp3l6nFY3fr/Y8C oDpQhBHcl3tVqNyCMxKxUnbfMvVCvDcgIsLqBlfmLYfYxYmzNjSN6b+HWLwI6KtB KSVjtE4mq4+rNFBjaDCdXtXtZL4Eo3Ye4zecwqF9TA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvddujedgudduudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeehgefhtdefueffheekgfffudelffejtd fhvdejkedthfehvdelgfetgfdvtedthfenucevlhhushhtvghrufhiiigvpedunecurfgr rhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Thu, 2 May 2024 02:51:49 -0400 (EDT) Received: by localhost (OpenSMTPD) with ESMTPSA id bddd43f1 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Thu, 2 May 2024 06:51:24 +0000 (UTC) Date: Thu, 2 May 2024 08:51:47 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Subject: [PATCH 04/11] reftable/writer: improve error when passed an invalid block size Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The reftable format only supports block sizes up to 16MB. When the writer is being passed a value bigger than that it simply calls abort(3P), which isn't all that helpful due to the lack of a proper error message. Improve this by calling `BUG()` instead. Signed-off-by: Patrick Steinhardt --- reftable/writer.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/reftable/writer.c b/reftable/writer.c index 7df6e53699..374b7d15ed 100644 --- a/reftable/writer.c +++ b/reftable/writer.c @@ -125,10 +125,8 @@ reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t), struct reftable_writer *wp = reftable_calloc(1, sizeof(*wp)); options_set_defaults(&opts); - if (opts.block_size >= (1 << 24)) { - /* TODO - error return? */ - abort(); - } + if (opts.block_size >= (1 << 24)) + BUG("configured block size exceeds 16MB"); strbuf_init(&wp->block_writer_data.last_key, 0); strbuf_init(&wp->last_key, 0); From patchwork Thu May 2 06:51:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13651327 Received: from fhigh8-smtp.messagingengine.com (fhigh8-smtp.messagingengine.com [103.168.172.159]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8505021A04 for ; Thu, 2 May 2024 06:51:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.159 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714632718; cv=none; b=rk1NJMvIY57iynpsMHj2X31+896YWmCccBYt2C1RvX811xY0/5K69SP40OqSNbqAyi8JJxkFQNNM9vH6nxS9SZouH1LTgyRk0tNH0SZeAX4kSD5ndEz/ye0rqhDxiYDLGBD8EzN/d9X9XwrriylG/YARJSbwusGkv1J25RtI2Cs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714632718; c=relaxed/simple; bh=8ZZLRJkV0OWEktJ8YDMyMo3vhx7JEDyQHXzIGHE27kw=; h=Date:From:To:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=j7xasEGgtlnIpkFmnMbxfN5gUejFQeeBAKQZX9UCd3hNfi6gcqADRnGlNkYAFTr8OWvsKHZVUYp8m2RO57UpUE+dYDhis75MRIesUrb/pdOoCH+ToYUo2vJUZY9jDy1lHELyHEv+IkurfhQEeagMT+DcAabvjmDY3VC2NYylOhQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=NNJ8cqwr; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=P26dXn/3; arc=none smtp.client-ip=103.168.172.159 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="NNJ8cqwr"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="P26dXn/3" Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailfhigh.nyi.internal (Postfix) with ESMTP id 9D9E411400F9 for ; Thu, 2 May 2024 02:51:55 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Thu, 02 May 2024 02:51:55 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1714632715; x=1714719115; bh=BGAazfA8S/ I1+2DAsPXiosk/lreELDQhmG+U5dUBPTk=; b=NNJ8cqwr/sIPWXsKNgXS2SHjbL 3OgkHJq/ZRpQX8IIFB5vVu96pO79L7py2nljsRBo2TCotHE/X7x8GfIu6myAp3QP F/l5htgrgAuzzeFrrGiKh6dQJx/fOF1+24YK2kBqcurFPSnvWuWdoDPihhWKCJbF OccOFxUV3yuW51ZHMx6EMk/CYu1UPgMEAoHxEfn9bH/HA8J9CsH6dgiUdtmq4OK3 /mh9dQbYodjcyBoR44bHeVwPepPfbBmdZQy8xIhCjXyLwVhX48bcBO2VqhSYcU5Y +taBjR6p16EDduxoqz0oES/wA3IiktfsZ4Tgjv2SEH/YpnNF+2fi6FjzDOww== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1714632715; x=1714719115; bh=BGAazfA8S/I1+2DAsPXiosk/lreE LDQhmG+U5dUBPTk=; b=P26dXn/3nfvlegbgMdGyXtr+joEkxoD1ra0YILSePSEw XKAnM8buqxiFhWCSwyHVTheNvkz9DqSzHoXHeAgSSWJyBpuBO6w9OU3tw1oRw+Tv wz8Y1V4bUKwFIsup0ql++s5sS2q8AluSrV+eZF+V4brIixh08ekkn0BRJZ7UAKhZ Iu9VsTxh8hvPuOveBNGoeiWfl9hCpk52/izKnI1fAJnrWUTcawUIPmWmsh70Dt24 e2khe6etbTurFCD5murvWTDuwmAdZtIWXiRvr+ftH8n5f8IHlizk/o3AJ39S1Q3Z eBYJTiibghR5TqJIoobvhguEGsa3j2upfGgjfuxZoQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvddujedgudduudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeehgefhtdefueffheekgfffudelffejtd fhvdejkedthfehvdelgfetgfdvtedthfenucevlhhushhtvghrufhiiigvpedunecurfgr rhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Thu, 2 May 2024 02:51:54 -0400 (EDT) Received: by localhost (OpenSMTPD) with ESMTPSA id a3513db5 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Thu, 2 May 2024 06:51:28 +0000 (UTC) Date: Thu, 2 May 2024 08:51:52 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Subject: [PATCH 05/11] reftable/dump: support dumping a table's block structure Message-ID: <4d4407d4a468a81f7775595c1ea4c611c1ca2482.1714630191.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: We're about to introduce new configs that will allow users to have more control over how exactly reftables are written. To verify that these configs are effective we will need to take a peak into the actual blocks written by the reftable backend. Introduce a new mode to the dumping logic that prints out the block structure. This logic can be invoked via `test-tool dump-reftables -b`. Signed-off-by: Patrick Steinhardt --- reftable/dump.c | 8 ++- reftable/reader.c | 63 ++++++++++++++++++ reftable/reftable-reader.h | 2 + t/t0613-reftable-write-options.sh | 102 ++++++++++++++++++++++++++++++ 4 files changed, 174 insertions(+), 1 deletion(-) create mode 100755 t/t0613-reftable-write-options.sh diff --git a/reftable/dump.c b/reftable/dump.c index 9c770a10cc..24476cc2a9 100644 --- a/reftable/dump.c +++ b/reftable/dump.c @@ -48,6 +48,7 @@ static void print_help(void) printf("usage: dump [-cst] arg\n\n" "options: \n" " -c compact\n" + " -b dump blocks\n" " -t dump table\n" " -s dump stack\n" " -6 sha256 hash format\n" @@ -58,6 +59,7 @@ static void print_help(void) int reftable_dump_main(int argc, char *const *argv) { int err = 0; + int opt_dump_blocks = 0; int opt_dump_table = 0; int opt_dump_stack = 0; int opt_compact = 0; @@ -67,6 +69,8 @@ int reftable_dump_main(int argc, char *const *argv) for (; argc > 1; argv++, argc--) if (*argv[1] != '-') break; + else if (!strcmp("-b", argv[1])) + opt_dump_blocks = 1; else if (!strcmp("-t", argv[1])) opt_dump_table = 1; else if (!strcmp("-6", argv[1])) @@ -88,7 +92,9 @@ int reftable_dump_main(int argc, char *const *argv) arg = argv[1]; - if (opt_dump_table) { + if (opt_dump_blocks) { + err = reftable_reader_print_blocks(arg); + } else if (opt_dump_table) { err = reftable_reader_print_file(arg); } else if (opt_dump_stack) { err = reftable_stack_print_directory(arg, opt_hash_id); diff --git a/reftable/reader.c b/reftable/reader.c index 481dff10d4..f23c8523db 100644 --- a/reftable/reader.c +++ b/reftable/reader.c @@ -856,3 +856,66 @@ int reftable_reader_print_file(const char *tablename) reftable_reader_free(r); return err; } + +int reftable_reader_print_blocks(const char *tablename) +{ + struct { + const char *name; + int type; + } sections[] = { + { + .name = "ref", + .type = BLOCK_TYPE_REF, + }, + { + .name = "obj", + .type = BLOCK_TYPE_OBJ, + }, + { + .name = "log", + .type = BLOCK_TYPE_LOG, + }, + }; + struct reftable_block_source src = { 0 }; + struct table_iter ti = TABLE_ITER_INIT; + struct reftable_reader *r = NULL; + size_t i; + int err; + + err = reftable_block_source_from_file(&src, tablename); + if (err < 0) + goto done; + + err = reftable_new_reader(&r, &src, tablename); + if (err < 0) + goto done; + + printf("header:\n"); + printf(" block_size: %d\n", r->block_size); + + for (i = 0; i < ARRAY_SIZE(sections); i++) { + err = reader_start(r, &ti, sections[i].type, 0); + if (err < 0) + goto done; + if (err > 0) + continue; + + printf("%s:\n", sections[i].name); + + while (1) { + printf(" - length: %u\n", ti.br.block_len); + printf(" restarts: %u\n", ti.br.restart_count); + + err = table_iter_next_block(&ti); + if (err < 0) + goto done; + if (err > 0) + break; + } + } + +done: + reftable_reader_free(r); + table_iter_close(&ti); + return err; +} diff --git a/reftable/reftable-reader.h b/reftable/reftable-reader.h index 4a4bc2fdf8..4a04857773 100644 --- a/reftable/reftable-reader.h +++ b/reftable/reftable-reader.h @@ -97,5 +97,7 @@ void reftable_table_from_reader(struct reftable_table *tab, /* print table onto stdout for debugging. */ int reftable_reader_print_file(const char *tablename); +/* print blocks onto stdout for debugging. */ +int reftable_reader_print_blocks(const char *tablename); #endif diff --git a/t/t0613-reftable-write-options.sh b/t/t0613-reftable-write-options.sh new file mode 100755 index 0000000000..462980c37c --- /dev/null +++ b/t/t0613-reftable-write-options.sh @@ -0,0 +1,102 @@ +#!/bin/sh + +test_description='reftable write options' + +GIT_TEST_DEFAULT_REF_FORMAT=reftable +export GIT_TEST_DEFAULT_REF_FORMAT +# Disable auto-compaction for all tests as we explicitly control repacking of +# refs. +GIT_TEST_REFTABLE_AUTOCOMPACTION=false +export GIT_TEST_REFTABLE_AUTOCOMPACTION +# Block sizes depend on the hash function, so we force SHA1 here. +GIT_TEST_DEFAULT_HASH=sha1 +export GIT_TEST_DEFAULT_HASH +# Block sizes also depend on the actual refs we write, so we force "master" to +# be the default initial branch name. +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME + +. ./test-lib.sh + +test_expect_success 'default write options' ' + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + test_commit initial && + git pack-refs && + cat >expect <<-EOF && + header: + block_size: 4096 + ref: + - length: 129 + restarts: 2 + log: + - length: 262 + restarts: 2 + EOF + test-tool dump-reftable -b .git/reftable/*.ref >actual && + test_cmp expect actual + ) +' + +test_expect_success 'disabled reflog writes no log blocks' ' + test_config_global core.logAllRefUpdates false && + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + test_commit initial && + git pack-refs && + cat >expect <<-EOF && + header: + block_size: 4096 + ref: + - length: 129 + restarts: 2 + EOF + test-tool dump-reftable -b .git/reftable/*.ref >actual && + test_cmp expect actual + ) +' + +test_expect_success 'many refs results in multiple blocks' ' + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + test_commit initial && + for i in $(test_seq 200) + do + printf "update refs/heads/branch-%d HEAD\n" "$i" || + return 1 + done >input && + git update-ref --stdin expect <<-EOF && + header: + block_size: 4096 + ref: + - length: 4049 + restarts: 11 + - length: 1136 + restarts: 3 + log: + - length: 4041 + restarts: 4 + - length: 4015 + restarts: 3 + - length: 4014 + restarts: 3 + - length: 4012 + restarts: 3 + - length: 3289 + restarts: 3 + EOF + test-tool dump-reftable -b .git/reftable/*.ref >actual && + test_cmp expect actual + ) +' + +test_done From patchwork Thu May 2 06:51:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13651328 Received: from fout4-smtp.messagingengine.com (fout4-smtp.messagingengine.com [103.168.172.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6D25A225D6 for ; Thu, 2 May 2024 06:52:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.147 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714632722; cv=none; b=hQCqIIDArSSCfLvEenEdCZU9orzlK9MFJ004FiuMRRJwp0uPu5XgicKvpy3dTPmuDCTYTr00A2drQuFxkglHekYy37DjTbV4FYVdbjehpg6UpvwKD8S5qNbBOj/SSJEPoGYzaiAcDzQ4MCTHBxdiWhK3Zd9P7uyUHxkxJo021sQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714632722; c=relaxed/simple; bh=XLoPOw/8E5VFsaZPOZv/blGplVCS3/mNkcnhwGbWY9o=; h=Date:From:To:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=SnBpOQo8yxdn10rXwo+PIT3PP+C2u1VJDiZd3yiLwzSBNY5UAm2HJY77ySAuPA96A65EGi0RSD42nd8RKbim/yopYIQxID6NDpf+9tKUjiBBnov/1wDkkAVnS15bIgaD7RZLD2qcHpGBoCxqhY6AEnHoOFo/kfbfbo/sqjg1ZHQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=ARfQSHQv; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=UN2uD4rf; arc=none smtp.client-ip=103.168.172.147 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="ARfQSHQv"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="UN2uD4rf" Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailfout.nyi.internal (Postfix) with ESMTP id 8E4261380394 for ; Thu, 2 May 2024 02:51:59 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Thu, 02 May 2024 02:51:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1714632719; x=1714719119; bh=QjDF2hIjlH PeCZW8wPvM7UFTJ8TyyVUGAGGAghgCHqc=; b=ARfQSHQvl5sykd9F+wSUfBIHBu B/TNQd/3TdOLdnBhm0LT/EF13sP4vWsdNmxWbwGQbQPRZP1d+mdmOjzOFw0GGSj6 LwKaWtWxc8QyY5069eQBnkjZ8qN3GchUU4u9j2OAdEvIf6dOHRbpmo+EMAT3N1zv 06hIaEePiCDbCqywY7vk1k9FctwT+6n4N5Q7vvBrQNcVx63zFGPzS8TQAhxepmRV nIVa7YtpjK6jjJCPszu9A1VcnrhXFFgTSGfvsLF41q5nWyBctRBGsPLi8D4siY/x foUjug5b8/+LRlQcMofNYJ0eEsfOhBbq+8NZHj/VmeXfB2Xhr/P2PJAGxvKQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1714632719; x=1714719119; bh=QjDF2hIjlHPeCZW8wPvM7UFTJ8Ty yVUGAGGAghgCHqc=; b=UN2uD4rfWhb244SltUKEhs7pjGfQRxygeGaTXm3sme3A mmAURdthIv1u027JumdmV3wtoxxiyQy94vkTN+iKVv6XbcvJGkqfpkgrTmQetmvk SbOU8oHsGKtenqU655k/dk5UoLswc4t/w3kKKX9zTghxDX4WPHIkLCPnJugSgH6/ hyGkmYUb+OuXGPioPyfxhMLf6+5NAW/sWILTZ5rFTt1iMKPUPWa/eukuLv2mEkBt 4M8+b+w6UQlscBuPhYILQ8RX7XigOw3uH8MAnJ+bkIChYY2H9dIiUsPU1beu3N8U D5u5UgpWoq/iKOOsaERpJgEvBnyG7H4Dc9bB5ffdAQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvddujedgudduudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeehgefhtdefueffheekgfffudelffejtd fhvdejkedthfehvdelgfetgfdvtedthfenucevlhhushhtvghrufhiiigvpedtnecurfgr rhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Thu, 2 May 2024 02:51:58 -0400 (EDT) Received: by localhost (OpenSMTPD) with ESMTPSA id fc33b6c7 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Thu, 2 May 2024 06:51:33 +0000 (UTC) Date: Thu, 2 May 2024 08:51:56 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Subject: [PATCH 06/11] refs/reftable: allow configuring block size Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Add a new option `reftable.blockSize` that allows the user to control the block size used by the reftable library. Signed-off-by: Patrick Steinhardt --- Documentation/config.txt | 2 + Documentation/config/reftable.txt | 14 ++++++ refs/reftable-backend.c | 32 +++++++++++++- t/t0613-reftable-write-options.sh | 72 +++++++++++++++++++++++++++++++ 4 files changed, 119 insertions(+), 1 deletion(-) create mode 100644 Documentation/config/reftable.txt diff --git a/Documentation/config.txt b/Documentation/config.txt index 70b448b132..fa1469e5e7 100644 --- a/Documentation/config.txt +++ b/Documentation/config.txt @@ -497,6 +497,8 @@ include::config/rebase.txt[] include::config/receive.txt[] +include::config/reftable.txt[] + include::config/remote.txt[] include::config/remotes.txt[] diff --git a/Documentation/config/reftable.txt b/Documentation/config/reftable.txt new file mode 100644 index 0000000000..fa7c4be014 --- /dev/null +++ b/Documentation/config/reftable.txt @@ -0,0 +1,14 @@ +reftable.blockSize:: + The size in bytes used by the reftable backend when writing blocks. + The block size is determined by the writer, and does not have to be a + power of 2. The block size must be larger than the longest reference + name or log entry used in the repository, as references cannot span + blocks. ++ +Powers of two that are friendly to the virtual memory system or +filesystem (such as 4kB or 8kB) are recommended. Larger sizes (64kB) can +yield better compression, with a possible increased cost incurred by +readers during access. ++ +The largest block size is `16777215` bytes (15.99 MiB). The default value is +`4096` bytes (4kB). A value of `0` will use the default value. diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c index 1cda48c504..c2c47a3bc1 100644 --- a/refs/reftable-backend.c +++ b/refs/reftable-backend.c @@ -1,6 +1,7 @@ #include "../git-compat-util.h" #include "../abspath.h" #include "../chdir-notify.h" +#include "../config.h" #include "../environment.h" #include "../gettext.h" #include "../hash.h" @@ -230,6 +231,23 @@ static int read_ref_without_reload(struct reftable_stack *stack, return ret; } +static int reftable_be_config(const char *var, const char *value, + const struct config_context *ctx, + void *_opts) +{ + struct reftable_write_options *opts = _opts; + + if (!strcmp(var, "reftable.blocksize")) { + unsigned long block_size = git_config_ulong(var, value, ctx->kvi); + if (block_size > 16777215) + die("reftable block size cannot exceed 16MB"); + opts->block_size = block_size; + return 0; + } + + return 0; +} + static struct ref_store *reftable_be_init(struct repository *repo, const char *gitdir, unsigned int store_flags) @@ -245,12 +263,24 @@ static struct ref_store *reftable_be_init(struct repository *repo, base_ref_store_init(&refs->base, repo, gitdir, &refs_be_reftable); strmap_init(&refs->worktree_stacks); refs->store_flags = store_flags; - refs->write_options.block_size = 4096; + refs->write_options.hash_id = repo->hash_algo->format_id; refs->write_options.default_permissions = calc_shared_perm(0666 & ~mask); refs->write_options.disable_auto_compact = !git_env_bool("GIT_TEST_REFTABLE_AUTOCOMPACTION", 1); + git_config(reftable_be_config, &refs->write_options); + + /* + * It is somewhat unfortunate that we have to mirror the default block + * size of the reftable library here. But given that the write options + * wouldn't be updated by the library here, and given that we require + * the proper block size to trim reflog message so that they fit, we + * must set up a proper value here. + */ + if (!refs->write_options.block_size) + refs->write_options.block_size = 4096; + /* * Set up the main reftable stack that is hosted in GIT_COMMON_DIR. * This stack contains both the shared and the main worktree refs. diff --git a/t/t0613-reftable-write-options.sh b/t/t0613-reftable-write-options.sh index 462980c37c..8bdbc6ec70 100755 --- a/t/t0613-reftable-write-options.sh +++ b/t/t0613-reftable-write-options.sh @@ -99,4 +99,76 @@ test_expect_success 'many refs results in multiple blocks' ' ) ' +test_expect_success 'tiny block size leads to error' ' + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + test_commit initial && + cat >expect <<-EOF && + error: unable to compact stack: entry too large + EOF + test_must_fail git -c reftable.blockSize=50 pack-refs 2>err && + test_cmp expect err + ) +' + +test_expect_success 'small block size leads to multiple ref blocks' ' + test_config_global core.logAllRefUpdates false && + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + test_commit A && + test_commit B && + git -c reftable.blockSize=100 pack-refs && + + cat >expect <<-EOF && + header: + block_size: 100 + ref: + - length: 53 + restarts: 1 + - length: 74 + restarts: 1 + - length: 38 + restarts: 1 + EOF + test-tool dump-reftable -b .git/reftable/*.ref >actual && + test_cmp expect actual + ) +' + +test_expect_success 'small block size fails with large reflog message' ' + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + test_commit A && + perl -e "print \"a\" x 500" >logmsg && + cat >expect <<-EOF && + fatal: update_ref failed for ref ${SQ}refs/heads/logme${SQ}: reftable: transaction failure: entry too large + EOF + test_must_fail git -c reftable.blockSize=100 \ + update-ref -m "$(cat logmsg)" refs/heads/logme HEAD 2>err && + test_cmp expect err + ) +' + +test_expect_success 'block size exceeding maximum supported size' ' + test_config_global core.logAllRefUpdates false && + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + test_commit A && + test_commit B && + cat >expect <<-EOF && + fatal: reftable block size cannot exceed 16MB + EOF + test_must_fail git -c reftable.blockSize=16777216 pack-refs 2>err && + test_cmp expect err + ) +' + test_done From patchwork Thu May 2 06:52:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13651329 Received: from fout4-smtp.messagingengine.com (fout4-smtp.messagingengine.com [103.168.172.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5F90E225D6 for ; Thu, 2 May 2024 06:52:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.147 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714632726; cv=none; b=I/Fx+yxGC/PVpNwxF9KAaOUkwjRfnALLkkm939pTadJ1Eh3JaYOupriRsAEQMMvfX+NUubxNJofS6EDYIm3BVlgOuZ8Omu+I8RWyXhzTSNy3OIBC7OGKyn2Yu8+nE9t5Hflk3jMopVyrGStvZIuPGtryQD9a9EUcK0KLRtA9ma4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714632726; c=relaxed/simple; bh=VIikiLqrQIA3iJ8tj10pb3LCV1AJJPcbCTVS2Pwu3iI=; h=Date:From:To:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=C36aPDuu74zq5767IVgTz22nTy0eOJmDkWRcmy8/QSygQ95Fz6INqBg+t89c8/tWgkBn/etfDLLmpfR0DVpziv4gsZ5OTGyeKCM4gGNpheMNwSX+KnbFSSqrYFIQ9KKXwmKTqFfMdlZfcVTZCUUKzouq0DLLfqxrifZtg4uqvm0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=vlbMrutt; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=ck9U2sGh; arc=none smtp.client-ip=103.168.172.147 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="vlbMrutt"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="ck9U2sGh" Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailfout.nyi.internal (Postfix) with ESMTP id 82ADB1380388 for ; Thu, 2 May 2024 02:52:04 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Thu, 02 May 2024 02:52:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1714632724; x=1714719124; bh=KuRt2mfo/k HbfDB2W2AMwumLHOhcF7qO5bMwumE4LSc=; b=vlbMruttc3Ay/iA4JyQIt99ydb Dq3JeHf9i1sR9hyy8cGc/+Oxo2ilCdF0tQdc049EzsDmhFPUbB73rQ6cpzT2MteN gvkdgYbsJsvOIa2mmwOYrcpjsKOAwGOYgUf93hQtr/yIrQDc9FCYOImrR6uWPJk4 yobgD/DD3SdzhUStOBY1PO7A+rfm89vW9q9mYOVzt7NiTcOIOoSwVmK5kaPlfG+c 3qWYiZHDxxhnQtJUGJBLbRO4/1nzyChC81iPgyfgZpdFxJkGr+eZambZeqVGC0BI VS49wJ0xOljt7ggnD1XInPsCb0NYOlSWcrJMIFWK5xPlrm7HmfkdpS8+5QQA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1714632724; x=1714719124; bh=KuRt2mfo/kHbfDB2W2AMwumLHOhc F7qO5bMwumE4LSc=; b=ck9U2sGhw0T5ojaEY3zKz1y4DG/W5tY8OgFb9TVLfqcO VTK5d5V32sfpYmNFU6FCJ0xBB6SkAgs+IVVIbyZrWTz1sUpJISGvFhDNIKE8bLpz dfI6iR2Oc0XEvqcnejZIFc981vQAddfKg6Fvl8BnNb/0ptDLERtbaOVJINHqwjDM LaJO53CA9CDpT+Ihg5O8Yiyg3kn38OLBNapVThT9owyeprjeJFrvIvzCbkoVxLrd BayKM11XDUNz4xPMCmHfF4bNnrWveBeczl6XenZhr43Jvz3kkHr4iBgkKyER+6dV I4Ju1cDI81Ab/XIydtg+q9mT0B8vVuHkKsfV0VcKGQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvddujedgudduudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeehgefhtdefueffheekgfffudelffejtd fhvdejkedthfehvdelgfetgfdvtedthfenucevlhhushhtvghrufhiiigvpedtnecurfgr rhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Thu, 2 May 2024 02:52:03 -0400 (EDT) Received: by localhost (OpenSMTPD) with ESMTPSA id 46023e75 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Thu, 2 May 2024 06:51:38 +0000 (UTC) Date: Thu, 2 May 2024 08:52:01 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Subject: [PATCH 07/11] reftable: use `uint16_t` to track restart interval Message-ID: <79d9e07ca92baace4b729b0b9b9227735a84286b.1714630191.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The restart interval can at most be `UINT16_MAX` as specified in the technical documentation of the reftable format. Furthermore, it cannot ever be negative. Regardless of that we use an `int` to track the restart interval. Change the type to use an `uint16_t` instead. Signed-off-by: Patrick Steinhardt --- reftable/block.h | 2 +- reftable/reftable-writer.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/reftable/block.h b/reftable/block.h index ea4384a7e2..cd5577105d 100644 --- a/reftable/block.h +++ b/reftable/block.h @@ -25,7 +25,7 @@ struct block_writer { uint32_t header_off; /* How often to restart keys. */ - int restart_interval; + uint16_t restart_interval; int hash_size; /* Offset of next uint8_t to write. */ diff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h index 44cb986465..4cd8ebe6c7 100644 --- a/reftable/reftable-writer.h +++ b/reftable/reftable-writer.h @@ -28,7 +28,7 @@ struct reftable_write_options { unsigned skip_index_objects : 1; /* how often to write complete keys in each block. */ - int restart_interval; + uint16_t restart_interval; /* 4-byte identifier ("sha1", "s256") of the hash. * Defaults to SHA1 if unset From patchwork Thu May 2 06:52:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13651330 Received: from fhigh8-smtp.messagingengine.com (fhigh8-smtp.messagingengine.com [103.168.172.159]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6CFA524A0E for ; Thu, 2 May 2024 06:52:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.159 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714632732; cv=none; b=FBPUsZktldN1H9xsP501s9YkCByTJWiXX/KKUGY2jvtPOYODs5uerZ5qT1/IMU/xBzqAyAXw3IMdcCB+ZYJ32SvoPxNKKHSIoXBLiOBqxKUSBDXjt7srpqYIr5JWXsRT1Y/kVqB1+A8ml9XlLlTXQTC7X0IjfKG7O0huQjTKdew= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714632732; c=relaxed/simple; bh=DMa7YorWs6qN2HC4D/sOzWM8AZexY4TFGppjp3KhHUM=; h=Date:From:To:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=Tm0LhFVWCQJ60TFg2Wu8jSK7fmcVl2aQUeCdMFBSIZTi9ejP+jsKPUbOuUso+TqEpraED4i6C6SmvMQqoubCYgQsajYwYCVC6FoBwUkG4q6uNL7c13/xw1x5wfSrRD50EaysqGUb5u8D7/nEvW455zkx1RGSL/Jo7oPmsDmdhMs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=roc7V4lK; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=Grd9aj6V; arc=none smtp.client-ip=103.168.172.159 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="roc7V4lK"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="Grd9aj6V" Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailfhigh.nyi.internal (Postfix) with ESMTP id 6AEFD11400CA for ; Thu, 2 May 2024 02:52:09 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Thu, 02 May 2024 02:52:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1714632729; x=1714719129; bh=jOZ/nVo3ci uX/XcMkTwPXvB6JqMdUK802enhZbN2XHs=; b=roc7V4lK8HrY60yqf5rXndOfX2 pTyjjktQYdgl0InsI3JbCWbL38RvgMTG1QzxCgWNjD/+KABTPZeJVjP1+yZI6esa nV3EpMGMt74fZ1EjbA4ELSlGgtTOZJccStqVVr85XQH4Pg9bMsDoyB0Ygao/Antu 018NitrKytvqr6Cc7M3I23gUEuA3nazhHUpm8XxK9jXkudasua19uFYQ7rVIkq6t +eelCF2G43XJ7B8xCrYOaAD7t7rIaLfS+xjV24G5wk00r8/6zfHkdcF9hvBIV3ou 758oKOD1Hx6nIZlWZJ92FDUwTzYTIqxjS3Z7IE1u8SuzvjhWDYf1eUJOwgTg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1714632729; x=1714719129; bh=jOZ/nVo3ciuX/XcMkTwPXvB6JqMd UK802enhZbN2XHs=; b=Grd9aj6Ved1TdyFRwoqJrjEggEpOmBIRnCq2WJtGRHGp z3fMvV1YRfJdNlbEBa1+OflSn6VjfJXADM9hTPPrjO2ZlQKcZ7MswdwOYwZJyIn6 oRO2vyEM2Zz5JcocZWeKZXojwnMa3pP1Qs5CNLkuedqhctkC8oF8ngGhye5VSKBi bl9QcvuHLUz0KCztS5Mog197iRn9vdnBVAQ5g8r6nFmBt3yrjd5drg5G+00JPxHY ASS3FlyZ7TxQ0YEEsDKOKjWjfk0cn6wSlW7V6RgJFrd0u43kCwT/UF3zmh+IEIIa 2vev3KFaqfMBkt3Yv7RQWGbbjisyvQGZmlVZ/HeRCA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvddujedgudduudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeehgefhtdefueffheekgfffudelffejtd fhvdejkedthfehvdelgfetgfdvtedthfenucevlhhushhtvghrufhiiigvpedtnecurfgr rhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Thu, 2 May 2024 02:52:08 -0400 (EDT) Received: by localhost (OpenSMTPD) with ESMTPSA id d92da9b5 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Thu, 2 May 2024 06:51:43 +0000 (UTC) Date: Thu, 2 May 2024 08:52:06 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Subject: [PATCH 08/11] refs/reftable: allow configuring restart interval Message-ID: <653ec4dfa59b20e0ca9ef76bb5f2c1725c2bc078.1714630191.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Add a new option `reftable.restartInterval` that allows the user to control the restart interval when writing reftable records used by the reftable library. Signed-off-by: Patrick Steinhardt --- Documentation/config/reftable.txt | 19 ++++++++++++++ refs/reftable-backend.c | 6 +++++ t/t0613-reftable-write-options.sh | 43 +++++++++++++++++++++++++++++++ 3 files changed, 68 insertions(+) diff --git a/Documentation/config/reftable.txt b/Documentation/config/reftable.txt index fa7c4be014..16b915c75e 100644 --- a/Documentation/config/reftable.txt +++ b/Documentation/config/reftable.txt @@ -12,3 +12,22 @@ readers during access. + The largest block size is `16777215` bytes (15.99 MiB). The default value is `4096` bytes (4kB). A value of `0` will use the default value. + +reftable.restartInterval:: + The interval at which to create restart points. The reftable backend + determines the restart points at file creation. The process is + arbitrary, but every 16 or 64 records is recommended. Every 16 may be + more suitable for smaller block sizes (4k or 8k), every 64 for larger + block sizes (64k). ++ +More frequent restart points reduces prefix compression and increases +space consumed by the restart table, both of which increase file size. ++ +Less frequent restart points makes prefix compression more effective, +decreasing overall file size, with increased penalties for readers +walking through more records after the binary search step. ++ +A maximum of `65535` restart points per block is supported. ++ +The default value is to create restart points every 16 records. A value of `0` +will use the default value. diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c index c2c47a3bc1..a786143de2 100644 --- a/refs/reftable-backend.c +++ b/refs/reftable-backend.c @@ -243,6 +243,12 @@ static int reftable_be_config(const char *var, const char *value, die("reftable block size cannot exceed 16MB"); opts->block_size = block_size; return 0; + } else if (!strcmp(var, "reftable.restartinterval")) { + unsigned long restart_interval = git_config_ulong(var, value, ctx->kvi); + if (restart_interval > UINT16_MAX) + die("reftable block size cannot exceed %u", (unsigned)UINT16_MAX); + opts->restart_interval = restart_interval; + return 0; } return 0; diff --git a/t/t0613-reftable-write-options.sh b/t/t0613-reftable-write-options.sh index 8bdbc6ec70..e0a5b26f58 100755 --- a/t/t0613-reftable-write-options.sh +++ b/t/t0613-reftable-write-options.sh @@ -171,4 +171,47 @@ test_expect_success 'block size exceeding maximum supported size' ' ) ' +test_expect_success 'restart interval at every single record' ' + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + test_commit initial && + for i in $(test_seq 10) + do + printf "update refs/heads/branch-%d HEAD\n" "$i" || + return 1 + done >input && + git update-ref --stdin expect <<-EOF && + header: + block_size: 4096 + ref: + - length: 566 + restarts: 13 + log: + - length: 1393 + restarts: 12 + EOF + test-tool dump-reftable -b .git/reftable/*.ref >actual && + test_cmp expect actual + ) +' + +test_expect_success 'restart interval exceeding maximum supported interval' ' + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + test_commit initial && + cat >expect <<-EOF && + fatal: reftable block size cannot exceed 65535 + EOF + test_must_fail git -c reftable.restartInterval=65536 pack-refs 2>err && + test_cmp expect err + ) +' + test_done From patchwork Thu May 2 06:52:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13651331 Received: from fout4-smtp.messagingengine.com (fout4-smtp.messagingengine.com [103.168.172.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5DE3224B4A for ; Thu, 2 May 2024 06:52:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.147 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714632737; cv=none; b=EkTUYttnw58yTCPX91TJV6O3cG4R59+ObphluF0dPAnwS7I6xwNPbdrCXQtdEPLWv7hHTp2OA+MCs1LlcszbtstT6m52Z6M8bMuUQqGwOWPCk1abd/54HvC7PqRR/p7KhX3m7aO/ttoybNWLEo6KWBUYRB1ICrMeqhmzP5Y/3po= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714632737; c=relaxed/simple; bh=fAu8sGz/ZCUJVNwjWebgHdh3OPwETVUBVl6Vx4yRqLc=; h=Date:From:To:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=VG3/WI0bUGsvM/3T4YZ/07aRQDLdidRFdBrhZsgCffIzKSuWOwCdU1nohtfhTUYRxHtzD2rPY1miYo75q4pEsDTPCRvPhkAVUztZ2BTYyVT6Tcf5mo3i101qlNkNkJYTTfvnBGi9zkcROEMM9pji20zrWNYgHlmUdjMVSJHCgnU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=ruh+AE2b; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=XbHUFS6i; arc=none smtp.client-ip=103.168.172.147 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="ruh+AE2b"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="XbHUFS6i" Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailfout.nyi.internal (Postfix) with ESMTP id 51C721380395 for ; Thu, 2 May 2024 02:52:15 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Thu, 02 May 2024 02:52:15 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1714632735; x=1714719135; bh=ZLgk95HqTM v7VUrDl5/sSfpraZJMJHGJZsnLD/9A75E=; b=ruh+AE2bY9LRGJSkr3GGnJb9+X LKYcjuXgzkum0205CKhizKL00oAE3sqmt+4f7G6LSf0OriP44UyjFz7Sdmb3l2Cd tEYy8bfTkneLiD2uWN0ZAtddUCZDgLDN65TcIMOUo11pvipOkYU626cWybKbxdIz iRMHyinY8bhcaUnZL2hCBIV3JN0LWi1xoUcC576ViKgHZonicQdxB0hA8zhO4umm souIDOuiPDYXdlUlDzJ0IAG+62A7IXYLOrHK6ELxhGqC15PaIg7iT+yb4ab3cdPT IVjykcuKiT/TOaDnwv4jcL2rI6v9BA8gbL956mEB0pbTBHSJl1YWGE0RfBaA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1714632735; x=1714719135; bh=ZLgk95HqTMv7VUrDl5/sSfpraZJM JHGJZsnLD/9A75E=; b=XbHUFS6il8T8GR36MqP5GPVoBXji7uLdYi+PwcsBdf8x LY5Pb9Vnul9vniV0k4np9qlEVFK72lkoIXDtgMf6nHi7yI5MbffRum/YCQII3ZbC PqATe1HoEzwf3U3VM8/WU0dCvI08bmrqsgfDoXuceCs6H1o1LyL4hTIt7irsvu+w 4ZBRETBgdEiPCjhLS5HXToI7GH4tdk7kN8Pm/W36ECrSJRXV9kvFg0hWXNuRj3an nfLgiHDpvv2kgtfpwtYGqqZz1VMMck7h3xUlvyr+gZ9Pjk/kr31+4VsNWCEnx/sC pD2FKzDKktuhyBwO3vsWvNALfMf8Sez86vUmsWOWHw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvddujedgudduudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeelleegheeuueehveehkefhffegteevie dtkedtleekvddugeeukeelveejtefftdenucffohhmrghinheprghllhhofigrnhihshhh rgduihhnfigrnhhtrdhithenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmh grihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Thu, 2 May 2024 02:52:14 -0400 (EDT) Received: by localhost (OpenSMTPD) with ESMTPSA id 4e9c3f53 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Thu, 2 May 2024 06:51:48 +0000 (UTC) Date: Thu, 2 May 2024 08:52:11 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Subject: [PATCH 09/11] refs/reftable: allow disabling writing the object index Message-ID: <6f2c481acce1ba7728a9a559fbf01fff331153a0.1714630191.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Besides the expected "ref" and "log" records, the reftable library also writes "obj" records. These are basically a reverse mapping of object IDs to their respective ref records so that it becomes efficient to figure out which references point to a specific object. The motivation for this data structure is the "uploadpack.allowTipSHA1InWant" config, which allows a client to fetch any object by its hash that has a ref pointing to it. This reverse index is not used by Git at all though, and the expectation is that most hosters nowadays use "uploadpack.allowAnySHA1InWant". It may thus be preferable for many users to disable writing these optional object indices altogether to safe some precious disk space. Add a new config "reftable.indexObjects" that allows the user to disable the object index altogether. Signed-off-by: Patrick Steinhardt --- Documentation/config/reftable.txt | 6 +++ refs/reftable-backend.c | 3 ++ t/t0613-reftable-write-options.sh | 69 +++++++++++++++++++++++++++++++ 3 files changed, 78 insertions(+) diff --git a/Documentation/config/reftable.txt b/Documentation/config/reftable.txt index 16b915c75e..6e4466f3c5 100644 --- a/Documentation/config/reftable.txt +++ b/Documentation/config/reftable.txt @@ -31,3 +31,9 @@ A maximum of `65535` restart points per block is supported. + The default value is to create restart points every 16 records. A value of `0` will use the default value. + +reftable.indexObjects:: + Whether the reftable backend shall write object blocks. Object blocks + are a reverse mapping of object ID to the references pointing to them. ++ +The default value is `true`. diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c index a786143de2..5298fcef6e 100644 --- a/refs/reftable-backend.c +++ b/refs/reftable-backend.c @@ -249,6 +249,9 @@ static int reftable_be_config(const char *var, const char *value, die("reftable block size cannot exceed %u", (unsigned)UINT16_MAX); opts->restart_interval = restart_interval; return 0; + } else if (!strcmp(var, "reftable.indexobjects")) { + opts->skip_index_objects = !git_config_bool(var, value); + return 0; } return 0; diff --git a/t/t0613-reftable-write-options.sh b/t/t0613-reftable-write-options.sh index e0a5b26f58..e2708e11d5 100755 --- a/t/t0613-reftable-write-options.sh +++ b/t/t0613-reftable-write-options.sh @@ -214,4 +214,73 @@ test_expect_success 'restart interval exceeding maximum supported interval' ' ) ' +test_expect_success 'object index gets written by default with ref index' ' + test_config_global core.logAllRefUpdates false && + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + test_commit initial && + for i in $(test_seq 5) + do + printf "update refs/heads/branch-%d HEAD\n" "$i" || + return 1 + done >input && + git update-ref --stdin expect <<-EOF && + header: + block_size: 100 + ref: + - length: 53 + restarts: 1 + - length: 95 + restarts: 1 + - length: 71 + restarts: 1 + - length: 80 + restarts: 1 + obj: + - length: 11 + restarts: 1 + EOF + test-tool dump-reftable -b .git/reftable/*.ref >actual && + test_cmp expect actual + ) +' + +test_expect_success 'object index can be disabled' ' + test_config_global core.logAllRefUpdates false && + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + test_commit initial && + for i in $(test_seq 5) + do + printf "update refs/heads/branch-%d HEAD\n" "$i" || + return 1 + done >input && + git update-ref --stdin expect <<-EOF && + header: + block_size: 100 + ref: + - length: 53 + restarts: 1 + - length: 95 + restarts: 1 + - length: 71 + restarts: 1 + - length: 80 + restarts: 1 + EOF + test-tool dump-reftable -b .git/reftable/*.ref >actual && + test_cmp expect actual + ) +' + test_done From patchwork Thu May 2 06:52:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13651332 Received: from fout4-smtp.messagingengine.com (fout4-smtp.messagingengine.com [103.168.172.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3BF37219E0 for ; Thu, 2 May 2024 06:52:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.147 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714632742; cv=none; b=rhzCcNOmxwQNRtQ2GkXvalwVcK7HjjGh5iHGNkwrWQKljwS0tXqB4ZF7CdYQ0FricXEvMbipOQPXDvV0cRFJUGcxE9V4wMT0DMXFxhrR9WxEKSblGiopP792T1kvCpmgd+dbnF7SvOwM7QR1lHR2ERLSwTDSkJZW4XTS8vWvvlo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714632742; c=relaxed/simple; bh=VDQoA3a+5Za7OGZ4jRKYfu+LjUY1nTIrDnyI7CELch4=; h=Date:From:To:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=Boyu44TKvW6xWbieDL/UacxGaSBj83JvsAlWxAFtgBGanvSFKKcH/TPlIuAbNg+FDEp6NTbVcdUvupAhoc1CWi3va+y+RnGZUKH49tGioT644hGHpl4AQqs53oRjw6ABc1T5eVtjZmgYLYB76oQfRtoPDBo7RfeTHAurcNWKwqo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=iS9jBE2r; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=Hddm0+2W; arc=none smtp.client-ip=103.168.172.147 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="iS9jBE2r"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="Hddm0+2W" Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailfout.nyi.internal (Postfix) with ESMTP id 4B3681380395 for ; Thu, 2 May 2024 02:52:20 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Thu, 02 May 2024 02:52:20 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1714632740; x=1714719140; bh=8R8GoFO2SD tOs8Uv59r7cJHXfmw8BMKfAsRgeWiuRvA=; b=iS9jBE2rL0mpRUvCmzLfukWsDe bX1OwSOzMU6WuUkqkmYseIKRDAD0nSzXmzWuWO93VGsqUt6ocbsPecJ3HQmWAOjp bU72qIJ6nT96mb1yuVuqoXf2UgsX1++cfsYmBE2IMRwlSeyFBgi4DHbx0QkiallG zfmAaT8CwOmFoRwvb6GL4e64qgjrS9OG9klomhtE5kqNaCplCoKtUe+MQoa+AU++ MfgXqD5ByKPeA8kqu0NCZmQ8Vrh6Mq+kjh5NxCArEPuC7kT7y9gJWX7auTUCRcDL I+xrajLEryNJa4C6RaKXtRARP1WwrFLzxbDeBIYHjbRtDTHyxnfoGLjQtKhg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1714632740; x=1714719140; bh=8R8GoFO2SDtOs8Uv59r7cJHXfmw8 BMKfAsRgeWiuRvA=; b=Hddm0+2W0BXCvmmYc5ckBz/k0hLR8tkg0LxTj0VwCXes xKgTshcH73M4iBQXk0e1q1rJtV6nSyEL28bG7WGDrAr0JDlHChF/qkInvEEEN9zv 7Z91X89X9PLZ4E/Y+FP0PdxVM7aT+WMmdHOU0n+5NAt9LlJADtvJJDKxnSMuRq26 w/bf3jhGzTNaceY5xLg7mrESnJvdXq1e/urvsndJF4u+xU3XoORq2nhVtSZdJ5j3 mFMhVy+EdWmP2p6RpeF2wXDtK0QdlMn7G+TqNWeVM4x0ajhaQGPmpOAiBnu+o4ob QgDhLVimgZ5NqzH9x5Ai4ln0B1ED1lHSgfSdMjFH4Q== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvddujedgudduudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeehgefhtdefueffheekgfffudelffejtd fhvdejkedthfehvdelgfetgfdvtedthfenucevlhhushhtvghrufhiiigvpedunecurfgr rhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Thu, 2 May 2024 02:52:19 -0400 (EDT) Received: by localhost (OpenSMTPD) with ESMTPSA id fa366084 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Thu, 2 May 2024 06:51:54 +0000 (UTC) Date: Thu, 2 May 2024 08:52:17 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Subject: [PATCH 10/11] reftable: make the compaction factor configurable Message-ID: <30e2e33479a6b6b07a4652e84eebd78f71fb363a.1714630191.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: When auto-compacting, the reftable library packs references such that the sizes of the tables form a geometric sequence. The factor for this geometric sequence is hardcoded to 2 right now. We're about to expose this as a config option though, so let's expose the factor via write options. Signed-off-by: Patrick Steinhardt --- reftable/reftable-writer.h | 6 ++++++ reftable/stack.c | 13 +++++++++---- reftable/stack.h | 3 ++- reftable/stack_test.c | 4 ++-- 4 files changed, 19 insertions(+), 7 deletions(-) diff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h index 4cd8ebe6c7..155457b042 100644 --- a/reftable/reftable-writer.h +++ b/reftable/reftable-writer.h @@ -49,6 +49,12 @@ struct reftable_write_options { /* boolean: Prevent auto-compaction of tables. */ unsigned disable_auto_compact : 1; + + /* + * Geometric sequence factor used by auto-compaction to decide which + * tables to compact. Defaults to 2 if unset. + */ + uint8_t auto_compaction_factor; }; /* reftable_block_stats holds statistics for a single block type */ diff --git a/reftable/stack.c b/reftable/stack.c index 7b4fff7c9e..6b0f8e13e7 100644 --- a/reftable/stack.c +++ b/reftable/stack.c @@ -1215,12 +1215,16 @@ static int segment_size(struct segment *s) return s->end - s->start; } -struct segment suggest_compaction_segment(uint64_t *sizes, size_t n) +struct segment suggest_compaction_segment(uint64_t *sizes, size_t n, + uint8_t factor) { struct segment seg = { 0 }; uint64_t bytes; size_t i; + if (!factor) + factor = 2; + /* * If there are no tables or only a single one then we don't have to * compact anything. The sequence is geometric by definition already. @@ -1252,7 +1256,7 @@ struct segment suggest_compaction_segment(uint64_t *sizes, size_t n) * 64, 32, 16, 8, 4, 3, 1 */ for (i = n - 1; i > 0; i--) { - if (sizes[i - 1] < sizes[i] * 2) { + if (sizes[i - 1] < sizes[i] * factor) { seg.end = i + 1; bytes = sizes[i]; break; @@ -1278,7 +1282,7 @@ struct segment suggest_compaction_segment(uint64_t *sizes, size_t n) uint64_t curr = bytes; bytes += sizes[i - 1]; - if (sizes[i - 1] < curr * 2) { + if (sizes[i - 1] < curr * factor) { seg.start = i - 1; seg.bytes = bytes; } @@ -1304,7 +1308,8 @@ int reftable_stack_auto_compact(struct reftable_stack *st) { uint64_t *sizes = stack_table_sizes_for_compaction(st); struct segment seg = - suggest_compaction_segment(sizes, st->merged->stack_len); + suggest_compaction_segment(sizes, st->merged->stack_len, + st->opts.auto_compaction_factor); reftable_free(sizes); if (segment_size(&seg) > 0) return stack_compact_range_stats(st, seg.start, seg.end - 1, diff --git a/reftable/stack.h b/reftable/stack.h index 97d7ebc043..5b45cff4f7 100644 --- a/reftable/stack.h +++ b/reftable/stack.h @@ -35,6 +35,7 @@ struct segment { uint64_t bytes; }; -struct segment suggest_compaction_segment(uint64_t *sizes, size_t n); +struct segment suggest_compaction_segment(uint64_t *sizes, size_t n, + uint8_t factor); #endif diff --git a/reftable/stack_test.c b/reftable/stack_test.c index 3316d55f19..f6c11ef18d 100644 --- a/reftable/stack_test.c +++ b/reftable/stack_test.c @@ -767,7 +767,7 @@ static void test_suggest_compaction_segment(void) { uint64_t sizes[] = { 512, 64, 17, 16, 9, 9, 9, 16, 2, 16 }; struct segment min = - suggest_compaction_segment(sizes, ARRAY_SIZE(sizes)); + suggest_compaction_segment(sizes, ARRAY_SIZE(sizes), 2); EXPECT(min.start == 1); EXPECT(min.end == 10); } @@ -776,7 +776,7 @@ static void test_suggest_compaction_segment_nothing(void) { uint64_t sizes[] = { 64, 32, 16, 8, 4, 2 }; struct segment result = - suggest_compaction_segment(sizes, ARRAY_SIZE(sizes)); + suggest_compaction_segment(sizes, ARRAY_SIZE(sizes), 2); EXPECT(result.start == result.end); } From patchwork Thu May 2 06:52:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13651333 Received: from fhigh8-smtp.messagingengine.com (fhigh8-smtp.messagingengine.com [103.168.172.159]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 014A1224F6 for ; Thu, 2 May 2024 06:52:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.159 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714632748; cv=none; b=Ky1DwCo48aQQEXLJNo+Uk+fZh/QuCkZaEafthxgXzlUNaB+5cpRdYXVvk1kIaVjuTpCmTdMt/3r84M/tUMtzWdm9RcNF9Oa2jb3/hEtE4dAZsAF0vlPxZiGPhbhSDbHbdLmWXTRYkedpnIISGHWM7wABq4s0p2nIoY2EyGB+tdw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714632748; c=relaxed/simple; bh=0ll4ABHhRHsV7uKDT0fXH8g157isfXUbjNGCuPqQJnc=; h=Date:From:To:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=ttwNr9KCvSQyQTp65y/uYi2tv9BDFuTuyfV0E65NhQs36RVH3llLe9n5Xpejz7kY86f0re3QGhuxjMWr4MDQDN4NQAycjme6QnSWEG9jh/uvc8VX5zgVrepIKEU9y+egV0/TBxPFbsmrGE2hsvk/50cifltbWDYZUJeil7K2U1o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=Kw+QqnNK; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=YfrxstI9; arc=none smtp.client-ip=103.168.172.159 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="Kw+QqnNK"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="YfrxstI9" Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailfhigh.nyi.internal (Postfix) with ESMTP id 2EFAC11400CA for ; Thu, 2 May 2024 02:52:26 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Thu, 02 May 2024 02:52:26 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1714632746; x=1714719146; bh=bAlinNZkch pStXNjco5GLy2cq4H6QDT9yTU/9ay7/mI=; b=Kw+QqnNKvf9UIKcCdFJIwd0AmY C9ibTSykw/PUh+2mc5nKJa/SW1hk3dCdnPZ6MHWm8GBYWkAC8LM8F+EURWsRJ3QY m2qncVRqoZCVeDuwIuGiHcqjoFCLZEmiVf1E8lBFaThi+wtPV/d6jgp/nVsRApVm TouVgoRkdT6OBmUjbpVav+GBaZMIRCi8xsoh1oepYHWQMjIzCSspzEuZfkW90Nqg HJNoaasCmJV6aZzclRxwHOivrKAYuLl+FkQUbqGKtvXquAOkKonXo5FIFRMd8KB3 mygGMu2Jwq5NOXUo/5ooyWqqUzP+FRj8gxYc7Rs8dpC5rsUw2Jo1hhC1QvVg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1714632746; x=1714719146; bh=bAlinNZkchpStXNjco5GLy2cq4H6 QDT9yTU/9ay7/mI=; b=YfrxstI94HLOBa0bx/mKJ1b11KVhFnxdUtWsY3MjHH6H UfNbBSJUJHWf5G8ajQPrmkS9JrB5K5wPnTVylPFw4f3smyJs5a9PEeRmHzE3Nuoa NfaIKF17+7ySrjZay3yAh8THX+pzOvTIxI8qkRVoZ9BcAq60stl1llUsvQGeVNNH yG18kiDw0gZvq3w0iON806kHKRnEsf6IKyRqvvbsPysron4SIXDco4W6AZqsiZYa 5tU58dsrGQuwY3brTfqGal+Zg2ZWhDDDGkObYR02dUv9Sb+5tckHaKzzyLtIqLpd m3dJCwbUEIx4vyRbIlrHmygEEv8U/eIx5eiNA1O5uw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvddujedgudduudcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjsehgtd erredttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshes phhkshdrihhmqeenucggtffrrghtthgvrhhnpeehgefhtdefueffheekgfffudelffejtd fhvdejkedthfehvdelgfetgfdvtedthfenucevlhhushhtvghrufhiiigvpedunecurfgr rhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Thu, 2 May 2024 02:52:25 -0400 (EDT) Received: by localhost (OpenSMTPD) with ESMTPSA id 0a6ea4cc (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Thu, 2 May 2024 06:51:59 +0000 (UTC) Date: Thu, 2 May 2024 08:52:23 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Subject: [PATCH 11/11] refs/reftable: allow configuring geometric factor Message-ID: <861f2e72d987d17b00d68cdaf400d743e2f8c118.1714630191.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Allow configuring the geometric factor used by the auto-compaction algorithm whenever a new table is appended to the stack of tables. Signed-off-by: Patrick Steinhardt --- Documentation/config/reftable.txt | 10 ++++++++++ refs/reftable-backend.c | 5 +++++ 2 files changed, 15 insertions(+) diff --git a/Documentation/config/reftable.txt b/Documentation/config/reftable.txt index 6e4466f3c5..1c381dda04 100644 --- a/Documentation/config/reftable.txt +++ b/Documentation/config/reftable.txt @@ -37,3 +37,13 @@ reftable.indexObjects:: are a reverse mapping of object ID to the references pointing to them. + The default value is `true`. + +reftable.geometricFactor:: + Whenever the reftable backend appends a new table to the table it + performs auto compaction to ensure that there is only a handful of + tables. The backend does this by ensuring that tables form a geometric + sequence regarding the respective sizes of each table. ++ +By default, the geometric sequence uses a factor of 2, meaning that for any +table, the next-biggest table must at least be twice as big. A maximum factor +of 256 is supported. diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c index 5298fcef6e..657d227c12 100644 --- a/refs/reftable-backend.c +++ b/refs/reftable-backend.c @@ -252,6 +252,11 @@ static int reftable_be_config(const char *var, const char *value, } else if (!strcmp(var, "reftable.indexobjects")) { opts->skip_index_objects = !git_config_bool(var, value); return 0; + } else if (!strcmp(var, "reftable.geometricfactor")) { + unsigned long factor = git_config_ulong(var, value, ctx->kvi); + if (factor > UINT8_MAX) + die("reftable geometric factor cannot exceed %u", (unsigned)UINT8_MAX); + opts->auto_compaction_factor = factor; } return 0;