From patchwork Mon May 13 08:17:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13663169 Received: from wfout2-smtp.messagingengine.com (wfout2-smtp.messagingengine.com [64.147.123.145]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AEC23146D6B for ; Mon, 13 May 2024 08:17:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=64.147.123.145 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715588281; cv=none; b=ha6Domux5obH3fOFkcOxQ/P2M460/j7DbpwHz4lyZt9cgHNz66qelBka08ApGE+wost7W8uSkK0InhxaZX930MJ8Q79BbLSb+PqnWzZnFFAWfugO6CWQtK2Eg4zki12Luo9JGl1PYTfAta2B/lkSBlWvaTMbiKEsS24eHE4mN2M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715588281; c=relaxed/simple; bh=vV+qWDOx/XpFE9wVIX94qQBHDwMyUGjSrprgEFpgueE=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=d9Mp2B67jc2HfkgBgCwuHEdmPqqRoo6b2N1XM+Ux4kzCc08JZ9WTPAwqEsSRgelafkgdggMDsUlp6uUEo9rI3D4Z+rRLwOLkhgPSD+1T+LZsI9ZhqPgmakXy/UbaJFqiom0mhKIjZqw68vdIQYA3t0Fh6faWSBtoo00m0Bzyw1Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=AyMe2rA8; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=etRKtyTn; arc=none smtp.client-ip=64.147.123.145 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="AyMe2rA8"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="etRKtyTn" Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailfout.west.internal (Postfix) with ESMTP id A00E51C00111; Mon, 13 May 2024 04:17:58 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Mon, 13 May 2024 04:17:58 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1715588278; x=1715674678; bh=pfmSqCkthq 8J6nonuCQmQQOrFuU4ZevyJT0SZsXIFzg=; b=AyMe2rA8ZtLO/IVwaJOKtRZqnT rm+5/5Kqn9D1mFPL1f5Rn1PFX0VtniiT9TkTlWIkYjb0qduws+lLuGSiiEXfKDxC D+fdsB3QERzQUEfNKSA1mBIDiSmRKOuNIQKJxpSZIyo4pT7qjlm3ibxpt8QQreJu NNKFNJDv/4Onk/nLAN138+rUvzAwJ7syKUsXv61izaBqeRwgniatWz0nPikDwsyD KShFm7k4sdO6gKK0ZIWHigDBTEQZC8lSy0YJUuY1JIajKqUvtUan1Ce0B0J4V9on EtsxFHIT4QMISkC9Ja7d7g6irhHNnM14RlBJAxFl2cWlLc9SlnpBtvcIkmsQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1715588278; x=1715674678; bh=pfmSqCkthq8J6nonuCQmQQOrFuU4 ZevyJT0SZsXIFzg=; b=etRKtyTnM5X9gCgjtJe9BQ8I++lXdNDd/ZCErv8eBR8R VcxSFdfRPvpWqlDY3vqi5Dsty70ywwQmuH006YibLi9iIDlc2GFjOs/WH4zirhjQ xz/o/065vKgT826JUdVUSaWg4dnkqfr91M31jp1heYN+IojpHDvD7qUMxbdea8Sj WfI8Kn9x1EXY3Orwk9s+Ph6F/TQV+sKtX0n9DHg7oIsSWYy6+Qg+X34/30IDMwrC CThTNdReuCgtFlMAlbY660TLqRExyltUCCt4hHcSc4jdFVsMoodYy63s70Se4cj4 yOnoNaLmKTuRI/a6n1fFr5gmhGI3Bszz04oGToixNg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdeggedgtdduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgepudenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 13 May 2024 04:17:57 -0400 (EDT) Received: by localhost (OpenSMTPD) with ESMTPSA id 6fe717c9 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 13 May 2024 08:17:38 +0000 (UTC) Date: Mon, 13 May 2024 10:17:54 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Karthik Nayak , Justin Tobler , Junio C Hamano Subject: [PATCH v3 01/11] reftable: consistently refer to `reftable_write_options` as `opts` Message-ID: <71f4e31cf7c03b7679fc7d5c018063b938a6ee35.1715587849.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Throughout the reftable library the `reftable_write_options` are sometimes referred to as `cfg` and sometimes as `opts`. Unify these to consistently use `opts` to avoid confusion. While at it, touch up the coding style a bit by removing unneeded braces around one-line statements and newlines between variable declarations. Signed-off-by: Patrick Steinhardt --- reftable/dump.c | 4 +- reftable/reftable-stack.h | 2 +- reftable/stack.c | 41 +++++++------- reftable/stack.h | 2 +- reftable/stack_test.c | 114 +++++++++++++++++--------------------- 5 files changed, 74 insertions(+), 89 deletions(-) diff --git a/reftable/dump.c b/reftable/dump.c index 26e0393c7d..9c770a10cc 100644 --- a/reftable/dump.c +++ b/reftable/dump.c @@ -27,9 +27,9 @@ license that can be found in the LICENSE file or at static int compact_stack(const char *stackdir) { struct reftable_stack *stack = NULL; - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options opts = { 0 }; - int err = reftable_new_stack(&stack, stackdir, cfg); + int err = reftable_new_stack(&stack, stackdir, opts); if (err < 0) goto done; diff --git a/reftable/reftable-stack.h b/reftable/reftable-stack.h index 1b602dda58..9c8e4eef49 100644 --- a/reftable/reftable-stack.h +++ b/reftable/reftable-stack.h @@ -29,7 +29,7 @@ struct reftable_stack; * stored in 'dir'. Typically, this should be .git/reftables. */ int reftable_new_stack(struct reftable_stack **dest, const char *dir, - struct reftable_write_options config); + struct reftable_write_options opts); /* returns the update_index at which a next table should be written. */ uint64_t reftable_stack_next_update_index(struct reftable_stack *st); diff --git a/reftable/stack.c b/reftable/stack.c index a59ebe038d..54e7473f3a 100644 --- a/reftable/stack.c +++ b/reftable/stack.c @@ -54,15 +54,14 @@ static int reftable_fd_flush(void *arg) } int reftable_new_stack(struct reftable_stack **dest, const char *dir, - struct reftable_write_options config) + struct reftable_write_options opts) { struct reftable_stack *p = reftable_calloc(1, sizeof(*p)); struct strbuf list_file_name = STRBUF_INIT; int err = 0; - if (config.hash_id == 0) { - config.hash_id = GIT_SHA1_FORMAT_ID; - } + if (opts.hash_id == 0) + opts.hash_id = GIT_SHA1_FORMAT_ID; *dest = NULL; @@ -73,7 +72,7 @@ int reftable_new_stack(struct reftable_stack **dest, const char *dir, p->list_file = strbuf_detach(&list_file_name, NULL); p->list_fd = -1; p->reftable_dir = xstrdup(dir); - p->config = config; + p->opts = opts; err = reftable_stack_reload_maybe_reuse(p, 1); if (err < 0) { @@ -255,7 +254,7 @@ static int reftable_stack_reload_once(struct reftable_stack *st, char **names, /* success! */ err = reftable_new_merged_table(&new_merged, new_tables, - new_readers_len, st->config.hash_id); + new_readers_len, st->opts.hash_id); if (err < 0) goto done; @@ -578,8 +577,8 @@ static int reftable_stack_init_addition(struct reftable_addition *add, } goto done; } - if (st->config.default_permissions) { - if (chmod(add->lock_file->filename.buf, st->config.default_permissions) < 0) { + if (st->opts.default_permissions) { + if (chmod(add->lock_file->filename.buf, st->opts.default_permissions) < 0) { err = REFTABLE_IO_ERROR; goto done; } @@ -678,7 +677,7 @@ int reftable_addition_commit(struct reftable_addition *add) if (err) goto done; - if (!add->stack->config.disable_auto_compact) { + if (!add->stack->opts.disable_auto_compact) { /* * Auto-compact the stack to keep the number of tables in * control. It is possible that a concurrent writer is already @@ -756,9 +755,9 @@ int reftable_addition_add(struct reftable_addition *add, err = REFTABLE_IO_ERROR; goto done; } - if (add->stack->config.default_permissions) { + if (add->stack->opts.default_permissions) { if (chmod(get_tempfile_path(tab_file), - add->stack->config.default_permissions)) { + add->stack->opts.default_permissions)) { err = REFTABLE_IO_ERROR; goto done; } @@ -766,7 +765,7 @@ int reftable_addition_add(struct reftable_addition *add, tab_fd = get_tempfile_fd(tab_file); wr = reftable_new_writer(reftable_fd_write, reftable_fd_flush, &tab_fd, - &add->stack->config); + &add->stack->opts); err = write_table(wr, arg); if (err < 0) goto done; @@ -849,14 +848,14 @@ static int stack_compact_locked(struct reftable_stack *st, } tab_fd = get_tempfile_fd(tab_file); - if (st->config.default_permissions && - chmod(get_tempfile_path(tab_file), st->config.default_permissions) < 0) { + if (st->opts.default_permissions && + chmod(get_tempfile_path(tab_file), st->opts.default_permissions) < 0) { err = REFTABLE_IO_ERROR; goto done; } wr = reftable_new_writer(reftable_fd_write, reftable_fd_flush, - &tab_fd, &st->config); + &tab_fd, &st->opts); err = stack_write_compact(st, wr, first, last, config); if (err < 0) goto done; @@ -904,7 +903,7 @@ static int stack_write_compact(struct reftable_stack *st, st->readers[last]->max_update_index); err = reftable_new_merged_table(&mt, subtabs, subtabs_len, - st->config.hash_id); + st->opts.hash_id); if (err < 0) { reftable_free(subtabs); goto done; @@ -1094,9 +1093,9 @@ static int stack_compact_range(struct reftable_stack *st, goto done; } - if (st->config.default_permissions) { + if (st->opts.default_permissions) { if (chmod(get_lock_file_path(&tables_list_lock), - st->config.default_permissions) < 0) { + st->opts.default_permissions) < 0) { err = REFTABLE_IO_ERROR; goto done; } @@ -1286,7 +1285,7 @@ static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st) { uint64_t *sizes = reftable_calloc(st->merged->stack_len, sizeof(*sizes)); - int version = (st->config.hash_id == GIT_SHA1_FORMAT_ID) ? 1 : 2; + int version = (st->opts.hash_id == GIT_SHA1_FORMAT_ID) ? 1 : 2; int overhead = header_size(version) - 1; int i = 0; for (i = 0; i < st->merged->stack_len; i++) { @@ -1435,11 +1434,11 @@ int reftable_stack_clean(struct reftable_stack *st) int reftable_stack_print_directory(const char *stackdir, uint32_t hash_id) { struct reftable_stack *stack = NULL; - struct reftable_write_options cfg = { .hash_id = hash_id }; + struct reftable_write_options opts = { .hash_id = hash_id }; struct reftable_merged_table *merged = NULL; struct reftable_table table = { NULL }; - int err = reftable_new_stack(&stack, stackdir, cfg); + int err = reftable_new_stack(&stack, stackdir, opts); if (err < 0) goto done; diff --git a/reftable/stack.h b/reftable/stack.h index d43efa4760..97d7ebc043 100644 --- a/reftable/stack.h +++ b/reftable/stack.h @@ -20,7 +20,7 @@ struct reftable_stack { char *reftable_dir; - struct reftable_write_options config; + struct reftable_write_options opts; struct reftable_reader **readers; size_t readers_len; diff --git a/reftable/stack_test.c b/reftable/stack_test.c index 7889f818d1..e17ad4dc62 100644 --- a/reftable/stack_test.c +++ b/reftable/stack_test.c @@ -150,7 +150,7 @@ static void test_reftable_stack_add_one(void) char *dir = get_tmp_dir(__LINE__); struct strbuf scratch = STRBUF_INIT; int mask = umask(002); - struct reftable_write_options cfg = { + struct reftable_write_options opts = { .default_permissions = 0660, }; struct reftable_stack *st = NULL; @@ -163,7 +163,7 @@ static void test_reftable_stack_add_one(void) }; struct reftable_ref_record dest = { NULL }; struct stat stat_result = { 0 }; - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); err = reftable_stack_add(st, &write_test_ref, &ref); @@ -186,7 +186,7 @@ static void test_reftable_stack_add_one(void) strbuf_addstr(&scratch, "/tables.list"); err = stat(scratch.buf, &stat_result); EXPECT(!err); - EXPECT((stat_result.st_mode & 0777) == cfg.default_permissions); + EXPECT((stat_result.st_mode & 0777) == opts.default_permissions); strbuf_reset(&scratch); strbuf_addstr(&scratch, dir); @@ -195,7 +195,7 @@ static void test_reftable_stack_add_one(void) strbuf_addstr(&scratch, st->readers[0]->name); err = stat(scratch.buf, &stat_result); EXPECT(!err); - EXPECT((stat_result.st_mode & 0777) == cfg.default_permissions); + EXPECT((stat_result.st_mode & 0777) == opts.default_permissions); #else (void) stat_result; #endif @@ -209,7 +209,7 @@ static void test_reftable_stack_add_one(void) static void test_reftable_stack_uptodate(void) { - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options opts = { 0 }; struct reftable_stack *st1 = NULL; struct reftable_stack *st2 = NULL; char *dir = get_tmp_dir(__LINE__); @@ -232,10 +232,10 @@ static void test_reftable_stack_uptodate(void) /* simulate multi-process access to the same stack by creating two stacks for the same directory. */ - err = reftable_new_stack(&st1, dir, cfg); + err = reftable_new_stack(&st1, dir, opts); EXPECT_ERR(err); - err = reftable_new_stack(&st2, dir, cfg); + err = reftable_new_stack(&st2, dir, opts); EXPECT_ERR(err); err = reftable_stack_add(st1, &write_test_ref, &ref1); @@ -257,8 +257,7 @@ static void test_reftable_stack_uptodate(void) static void test_reftable_stack_transaction_api(void) { char *dir = get_tmp_dir(__LINE__); - - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options opts = { 0 }; struct reftable_stack *st = NULL; int err; struct reftable_addition *add = NULL; @@ -271,8 +270,7 @@ static void test_reftable_stack_transaction_api(void) }; struct reftable_ref_record dest = { NULL }; - - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); reftable_addition_destroy(add); @@ -301,12 +299,12 @@ static void test_reftable_stack_transaction_api(void) static void test_reftable_stack_transaction_api_performs_auto_compaction(void) { char *dir = get_tmp_dir(__LINE__); - struct reftable_write_options cfg = {0}; + struct reftable_write_options opts = {0}; struct reftable_addition *add = NULL; struct reftable_stack *st = NULL; int i, n = 20, err; - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); for (i = 0; i <= n; i++) { @@ -325,7 +323,7 @@ static void test_reftable_stack_transaction_api_performs_auto_compaction(void) * we can ensure that we indeed honor this setting and have * better control over when exactly auto compaction runs. */ - st->config.disable_auto_compact = i != n; + st->opts.disable_auto_compact = i != n; err = reftable_stack_new_addition(&add, st); EXPECT_ERR(err); @@ -361,13 +359,13 @@ static void test_reftable_stack_auto_compaction_fails_gracefully(void) .value_type = REFTABLE_REF_VAL1, .value.val1 = {0x01}, }; - struct reftable_write_options cfg = {0}; + struct reftable_write_options opts = {0}; struct reftable_stack *st; struct strbuf table_path = STRBUF_INIT; char *dir = get_tmp_dir(__LINE__); int err; - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); err = reftable_stack_add(st, write_test_ref, &ref); @@ -404,8 +402,7 @@ static int write_error(struct reftable_writer *wr, void *arg) static void test_reftable_stack_update_index_check(void) { char *dir = get_tmp_dir(__LINE__); - - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options opts = { 0 }; struct reftable_stack *st = NULL; int err; struct reftable_ref_record ref1 = { @@ -421,7 +418,7 @@ static void test_reftable_stack_update_index_check(void) .value.symref = "master", }; - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); err = reftable_stack_add(st, &write_test_ref, &ref1); @@ -436,12 +433,11 @@ static void test_reftable_stack_update_index_check(void) static void test_reftable_stack_lock_failure(void) { char *dir = get_tmp_dir(__LINE__); - - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options opts = { 0 }; struct reftable_stack *st = NULL; int err, i; - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); for (i = -1; i != REFTABLE_EMPTY_TABLE_ERROR; i--) { err = reftable_stack_add(st, &write_error, &i); @@ -456,7 +452,7 @@ static void test_reftable_stack_add(void) { int i = 0; int err = 0; - struct reftable_write_options cfg = { + struct reftable_write_options opts = { .exact_log_message = 1, .default_permissions = 0660, .disable_auto_compact = 1, @@ -469,7 +465,7 @@ static void test_reftable_stack_add(void) struct stat stat_result; int N = ARRAY_SIZE(refs); - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); for (i = 0; i < N; i++) { @@ -528,7 +524,7 @@ static void test_reftable_stack_add(void) strbuf_addstr(&path, "/tables.list"); err = stat(path.buf, &stat_result); EXPECT(!err); - EXPECT((stat_result.st_mode & 0777) == cfg.default_permissions); + EXPECT((stat_result.st_mode & 0777) == opts.default_permissions); strbuf_reset(&path); strbuf_addstr(&path, dir); @@ -537,7 +533,7 @@ static void test_reftable_stack_add(void) strbuf_addstr(&path, st->readers[0]->name); err = stat(path.buf, &stat_result); EXPECT(!err); - EXPECT((stat_result.st_mode & 0777) == cfg.default_permissions); + EXPECT((stat_result.st_mode & 0777) == opts.default_permissions); #else (void) stat_result; #endif @@ -555,7 +551,7 @@ static void test_reftable_stack_add(void) static void test_reftable_stack_log_normalize(void) { int err = 0; - struct reftable_write_options cfg = { + struct reftable_write_options opts = { 0, }; struct reftable_stack *st = NULL; @@ -579,7 +575,7 @@ static void test_reftable_stack_log_normalize(void) .update_index = 1, }; - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); input.value.update.message = "one\ntwo"; @@ -612,8 +608,7 @@ static void test_reftable_stack_tombstone(void) { int i = 0; char *dir = get_tmp_dir(__LINE__); - - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options opts = { 0 }; struct reftable_stack *st = NULL; int err; struct reftable_ref_record refs[2] = { { NULL } }; @@ -622,8 +617,7 @@ static void test_reftable_stack_tombstone(void) struct reftable_ref_record dest = { NULL }; struct reftable_log_record log_dest = { NULL }; - - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); /* even entries add the refs, odd entries delete them. */ @@ -691,8 +685,7 @@ static void test_reftable_stack_tombstone(void) static void test_reftable_stack_hash_id(void) { char *dir = get_tmp_dir(__LINE__); - - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options opts = { 0 }; struct reftable_stack *st = NULL; int err; @@ -702,24 +695,24 @@ static void test_reftable_stack_hash_id(void) .value.symref = "target", .update_index = 1, }; - struct reftable_write_options cfg32 = { .hash_id = GIT_SHA256_FORMAT_ID }; + struct reftable_write_options opts32 = { .hash_id = GIT_SHA256_FORMAT_ID }; struct reftable_stack *st32 = NULL; - struct reftable_write_options cfg_default = { 0 }; + struct reftable_write_options opts_default = { 0 }; struct reftable_stack *st_default = NULL; struct reftable_ref_record dest = { NULL }; - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); err = reftable_stack_add(st, &write_test_ref, &ref); EXPECT_ERR(err); /* can't read it with the wrong hash ID. */ - err = reftable_new_stack(&st32, dir, cfg32); + err = reftable_new_stack(&st32, dir, opts32); EXPECT(err == REFTABLE_FORMAT_ERROR); - /* check that we can read it back with default config too. */ - err = reftable_new_stack(&st_default, dir, cfg_default); + /* check that we can read it back with default opts too. */ + err = reftable_new_stack(&st_default, dir, opts_default); EXPECT_ERR(err); err = reftable_stack_read_ref(st_default, "master", &dest); @@ -752,8 +745,7 @@ static void test_suggest_compaction_segment_nothing(void) static void test_reflog_expire(void) { char *dir = get_tmp_dir(__LINE__); - - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options opts = { 0 }; struct reftable_stack *st = NULL; struct reftable_log_record logs[20] = { { NULL } }; int N = ARRAY_SIZE(logs) - 1; @@ -764,8 +756,7 @@ static void test_reflog_expire(void) }; struct reftable_log_record log = { NULL }; - - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); for (i = 1; i <= N; i++) { @@ -828,21 +819,19 @@ static int write_nothing(struct reftable_writer *wr, void *arg) static void test_empty_add(void) { - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options opts = { 0 }; struct reftable_stack *st = NULL; int err; char *dir = get_tmp_dir(__LINE__); - struct reftable_stack *st2 = NULL; - - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); err = reftable_stack_add(st, &write_nothing, NULL); EXPECT_ERR(err); - err = reftable_new_stack(&st2, dir, cfg); + err = reftable_new_stack(&st2, dir, opts); EXPECT_ERR(err); clear_dir(dir); reftable_stack_destroy(st); @@ -861,16 +850,15 @@ static int fastlog2(uint64_t sz) static void test_reftable_stack_auto_compaction(void) { - struct reftable_write_options cfg = { + struct reftable_write_options opts = { .disable_auto_compact = 1, }; struct reftable_stack *st = NULL; char *dir = get_tmp_dir(__LINE__); - int err, i; int N = 100; - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); for (i = 0; i < N; i++) { @@ -900,13 +888,13 @@ static void test_reftable_stack_auto_compaction(void) static void test_reftable_stack_add_performs_auto_compaction(void) { - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options opts = { 0 }; struct reftable_stack *st = NULL; struct strbuf refname = STRBUF_INIT; char *dir = get_tmp_dir(__LINE__); int err, i, n = 20; - err = reftable_new_stack(&st, dir, cfg); + err = reftable_new_stack(&st, dir, opts); EXPECT_ERR(err); for (i = 0; i <= n; i++) { @@ -921,7 +909,7 @@ static void test_reftable_stack_add_performs_auto_compaction(void) * we can ensure that we indeed honor this setting and have * better control over when exactly auto compaction runs. */ - st->config.disable_auto_compact = i != n; + st->opts.disable_auto_compact = i != n; strbuf_reset(&refname); strbuf_addf(&refname, "branch-%04d", i); @@ -948,14 +936,13 @@ static void test_reftable_stack_add_performs_auto_compaction(void) static void test_reftable_stack_compaction_concurrent(void) { - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options opts = { 0 }; struct reftable_stack *st1 = NULL, *st2 = NULL; char *dir = get_tmp_dir(__LINE__); - int err, i; int N = 3; - err = reftable_new_stack(&st1, dir, cfg); + err = reftable_new_stack(&st1, dir, opts); EXPECT_ERR(err); for (i = 0; i < N; i++) { @@ -972,7 +959,7 @@ static void test_reftable_stack_compaction_concurrent(void) EXPECT_ERR(err); } - err = reftable_new_stack(&st2, dir, cfg); + err = reftable_new_stack(&st2, dir, opts); EXPECT_ERR(err); err = reftable_stack_compact_all(st1, NULL); @@ -998,14 +985,13 @@ static void unclean_stack_close(struct reftable_stack *st) static void test_reftable_stack_compaction_concurrent_clean(void) { - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options opts = { 0 }; struct reftable_stack *st1 = NULL, *st2 = NULL, *st3 = NULL; char *dir = get_tmp_dir(__LINE__); - int err, i; int N = 3; - err = reftable_new_stack(&st1, dir, cfg); + err = reftable_new_stack(&st1, dir, opts); EXPECT_ERR(err); for (i = 0; i < N; i++) { @@ -1022,7 +1008,7 @@ static void test_reftable_stack_compaction_concurrent_clean(void) EXPECT_ERR(err); } - err = reftable_new_stack(&st2, dir, cfg); + err = reftable_new_stack(&st2, dir, opts); EXPECT_ERR(err); err = reftable_stack_compact_all(st1, NULL); @@ -1031,7 +1017,7 @@ static void test_reftable_stack_compaction_concurrent_clean(void) unclean_stack_close(st1); unclean_stack_close(st2); - err = reftable_new_stack(&st3, dir, cfg); + err = reftable_new_stack(&st3, dir, opts); EXPECT_ERR(err); err = reftable_stack_clean(st3); From patchwork Mon May 13 08:17:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13663170 Received: from wfout2-smtp.messagingengine.com (wfout2-smtp.messagingengine.com [64.147.123.145]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CE5D6146D5D for ; Mon, 13 May 2024 08:18:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=64.147.123.145 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715588286; cv=none; b=hHnmZU3LsCS8xLEjDKhMJrYA0oQDmIgh8PEwUgCLDxl4TXxI4Cmw/M/DkBxqxxxOw3rdaBsmdtlbNZz90LHdqvBnWk6DLH+55mk7cO6hBYZVw0ljIyq6YhcfFUBMB1+ki7aphXrctiAoqaL8aZZtu/4+IVyO4O8xF3zFU9W50w8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715588286; c=relaxed/simple; bh=tFfHR21NRcDmKzyptCPdQG8VDMMBi2RaMn8yh2J9Avk=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=naVODy9wUWtKFL3PkhHGIzaTVKsZG5MMdagDM2xZ8RVbruAhV/REYg3rKrz+vYV8OmrbCaWc+hI/cfTKt66h2Zh8TXV5xD5z4doeZECKxQ02eOO9FuHoKG8AYE5rOVaoe0LSBa/czfnxcTCkkpstOjgSssxh4MZWqZEeHOibXAQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=hmGSdqgE; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=UhLJiVMy; arc=none smtp.client-ip=64.147.123.145 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="hmGSdqgE"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="UhLJiVMy" Received: from compute7.internal (compute7.nyi.internal [10.202.2.48]) by mailfout.west.internal (Postfix) with ESMTP id BFBA81C000E5; Mon, 13 May 2024 04:18:03 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute7.internal (MEProxy); Mon, 13 May 2024 04:18:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1715588283; x=1715674683; bh=T8bkwO9wPg UsRV5Jrf5E1hzcW3zYfh4ZIbltPC3tN/4=; b=hmGSdqgERskBjwPAjN3HUcCkU5 n1FlDMvzmhVPStu1curx/oHNJSLWlQCcaturtFqX7L48ml7qngOWs5lKi5aMhelb 9zj6R7enqvhA/1R0Zi8xGWdR9AOheiKjbAsRzi+i8PHRIcXjzfYZD+EM4Sjfet+9 INwDhsqcs4t7R9cKHcJ32uP4G6miFSMhR67bTYEyJuqroIDcRefSjmZxeDhDktFe NYKYoClCrJV1eMU+l2HpI7BdsZ922cGQTvMCEgLqNvxc3qKqCknH0YhsMMiUxUQv 7AF0QHRO8qoxP0kOfjdEToQQbddQ9eukX0vr+lOaqx2a9HeYwgb3ouet2Bfg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1715588283; x=1715674683; bh=T8bkwO9wPgUsRV5Jrf5E1hzcW3zY fh4ZIbltPC3tN/4=; b=UhLJiVMycoMEFuHbi7dfgFDCs8MEpFfF4c+wTcfPNACf iAdmT/2ORH6Be3XzmeLXt4kNYw65+pwAeCZdayvCWhb0QVtGTB/ns2BrYF8pG4J1 PJHvizGcfGxV3IQoKWTUHwOBCUTKavxBFcqAQ+NomRhVq0DOtSu2g0HA/hR5TzkV UNyFhltcEveBppHvE2Dc5f5H9AXo28QjMLo5wB0Oxb1pc/65JWRxpC4Je6s+/G8j Rx14gw0O0POlBVAI01kV2dEkYfJUQvxtkNXwIjKbv8/BOvmZKrY/JzQkHuRacn6i Ji2mignnondnBoat8jf+p1EoGQn7t0CNnLjhIOBrmw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdeggedgtdduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 13 May 2024 04:18:02 -0400 (EDT) Received: by localhost (OpenSMTPD) with ESMTPSA id b4d9c65e (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 13 May 2024 08:17:43 +0000 (UTC) Date: Mon, 13 May 2024 10:17:59 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Karthik Nayak , Justin Tobler , Junio C Hamano Subject: [PATCH v3 02/11] reftable: pass opts as constant pointer Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: We sometimes pass the refatble write options as value and sometimes as a pointer. This is quite confusing and makes the reader wonder whether the options get modified sometimes. In fact, `reftable_new_writer()` does cause the caller-provided options to get updated when some values aren't set up. This is quite unexpected, but didn't cause any harm until now. Adapt the code so that we do not modify the caller-provided values anymore. While at it, refactor the code to code to consistently pass the options as a constant pointer to clarify that the caller-provided opts will not ever get modified. Signed-off-by: Patrick Steinhardt --- refs/reftable-backend.c | 6 ++--- reftable/dump.c | 2 +- reftable/reftable-stack.h | 2 +- reftable/reftable-writer.h | 2 +- reftable/stack.c | 7 ++++-- reftable/stack_test.c | 48 +++++++++++++++++++------------------- reftable/writer.c | 17 +++++++++----- 7 files changed, 46 insertions(+), 38 deletions(-) diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c index 010ef811b6..f8f930380d 100644 --- a/refs/reftable-backend.c +++ b/refs/reftable-backend.c @@ -129,7 +129,7 @@ static struct reftable_stack *stack_for(struct reftable_ref_store *store, store->base.repo->commondir, wtname_buf.buf); store->err = reftable_new_stack(&stack, wt_dir.buf, - store->write_options); + &store->write_options); assert(store->err != REFTABLE_API_ERROR); strmap_put(&store->worktree_stacks, wtname_buf.buf, stack); } @@ -263,7 +263,7 @@ static struct ref_store *reftable_be_init(struct repository *repo, } strbuf_addstr(&path, "/reftable"); refs->err = reftable_new_stack(&refs->main_stack, path.buf, - refs->write_options); + &refs->write_options); if (refs->err) goto done; @@ -280,7 +280,7 @@ static struct ref_store *reftable_be_init(struct repository *repo, strbuf_addf(&path, "%s/reftable", gitdir); refs->err = reftable_new_stack(&refs->worktree_stack, path.buf, - refs->write_options); + &refs->write_options); if (refs->err) goto done; } diff --git a/reftable/dump.c b/reftable/dump.c index 9c770a10cc..586f3eb288 100644 --- a/reftable/dump.c +++ b/reftable/dump.c @@ -29,7 +29,7 @@ static int compact_stack(const char *stackdir) struct reftable_stack *stack = NULL; struct reftable_write_options opts = { 0 }; - int err = reftable_new_stack(&stack, stackdir, opts); + int err = reftable_new_stack(&stack, stackdir, &opts); if (err < 0) goto done; diff --git a/reftable/reftable-stack.h b/reftable/reftable-stack.h index 9c8e4eef49..c15632c401 100644 --- a/reftable/reftable-stack.h +++ b/reftable/reftable-stack.h @@ -29,7 +29,7 @@ struct reftable_stack; * stored in 'dir'. Typically, this should be .git/reftables. */ int reftable_new_stack(struct reftable_stack **dest, const char *dir, - struct reftable_write_options opts); + const struct reftable_write_options *opts); /* returns the update_index at which a next table should be written. */ uint64_t reftable_stack_next_update_index(struct reftable_stack *st); diff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h index b601a69a40..03df3a4963 100644 --- a/reftable/reftable-writer.h +++ b/reftable/reftable-writer.h @@ -88,7 +88,7 @@ struct reftable_stats { struct reftable_writer * reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t), int (*flush_func)(void *), - void *writer_arg, struct reftable_write_options *opts); + void *writer_arg, const struct reftable_write_options *opts); /* Set the range of update indices for the records we will add. When writing a table into a stack, the min should be at least diff --git a/reftable/stack.c b/reftable/stack.c index 54e7473f3a..d2e68be7e8 100644 --- a/reftable/stack.c +++ b/reftable/stack.c @@ -54,12 +54,15 @@ static int reftable_fd_flush(void *arg) } int reftable_new_stack(struct reftable_stack **dest, const char *dir, - struct reftable_write_options opts) + const struct reftable_write_options *_opts) { struct reftable_stack *p = reftable_calloc(1, sizeof(*p)); struct strbuf list_file_name = STRBUF_INIT; + struct reftable_write_options opts = {0}; int err = 0; + if (_opts) + opts = *_opts; if (opts.hash_id == 0) opts.hash_id = GIT_SHA1_FORMAT_ID; @@ -1438,7 +1441,7 @@ int reftable_stack_print_directory(const char *stackdir, uint32_t hash_id) struct reftable_merged_table *merged = NULL; struct reftable_table table = { NULL }; - int err = reftable_new_stack(&stack, stackdir, opts); + int err = reftable_new_stack(&stack, stackdir, &opts); if (err < 0) goto done; diff --git a/reftable/stack_test.c b/reftable/stack_test.c index e17ad4dc62..d15f11d712 100644 --- a/reftable/stack_test.c +++ b/reftable/stack_test.c @@ -163,7 +163,7 @@ static void test_reftable_stack_add_one(void) }; struct reftable_ref_record dest = { NULL }; struct stat stat_result = { 0 }; - err = reftable_new_stack(&st, dir, opts); + err = reftable_new_stack(&st, dir, &opts); EXPECT_ERR(err); err = reftable_stack_add(st, &write_test_ref, &ref); @@ -232,10 +232,10 @@ static void test_reftable_stack_uptodate(void) /* simulate multi-process access to the same stack by creating two stacks for the same directory. */ - err = reftable_new_stack(&st1, dir, opts); + err = reftable_new_stack(&st1, dir, &opts); EXPECT_ERR(err); - err = reftable_new_stack(&st2, dir, opts); + err = reftable_new_stack(&st2, dir, &opts); EXPECT_ERR(err); err = reftable_stack_add(st1, &write_test_ref, &ref1); @@ -270,7 +270,7 @@ static void test_reftable_stack_transaction_api(void) }; struct reftable_ref_record dest = { NULL }; - err = reftable_new_stack(&st, dir, opts); + err = reftable_new_stack(&st, dir, &opts); EXPECT_ERR(err); reftable_addition_destroy(add); @@ -304,7 +304,7 @@ static void test_reftable_stack_transaction_api_performs_auto_compaction(void) struct reftable_stack *st = NULL; int i, n = 20, err; - err = reftable_new_stack(&st, dir, opts); + err = reftable_new_stack(&st, dir, &opts); EXPECT_ERR(err); for (i = 0; i <= n; i++) { @@ -365,7 +365,7 @@ static void test_reftable_stack_auto_compaction_fails_gracefully(void) char *dir = get_tmp_dir(__LINE__); int err; - err = reftable_new_stack(&st, dir, opts); + err = reftable_new_stack(&st, dir, &opts); EXPECT_ERR(err); err = reftable_stack_add(st, write_test_ref, &ref); @@ -418,7 +418,7 @@ static void test_reftable_stack_update_index_check(void) .value.symref = "master", }; - err = reftable_new_stack(&st, dir, opts); + err = reftable_new_stack(&st, dir, &opts); EXPECT_ERR(err); err = reftable_stack_add(st, &write_test_ref, &ref1); @@ -437,7 +437,7 @@ static void test_reftable_stack_lock_failure(void) struct reftable_stack *st = NULL; int err, i; - err = reftable_new_stack(&st, dir, opts); + err = reftable_new_stack(&st, dir, &opts); EXPECT_ERR(err); for (i = -1; i != REFTABLE_EMPTY_TABLE_ERROR; i--) { err = reftable_stack_add(st, &write_error, &i); @@ -465,7 +465,7 @@ static void test_reftable_stack_add(void) struct stat stat_result; int N = ARRAY_SIZE(refs); - err = reftable_new_stack(&st, dir, opts); + err = reftable_new_stack(&st, dir, &opts); EXPECT_ERR(err); for (i = 0; i < N; i++) { @@ -575,7 +575,7 @@ static void test_reftable_stack_log_normalize(void) .update_index = 1, }; - err = reftable_new_stack(&st, dir, opts); + err = reftable_new_stack(&st, dir, &opts); EXPECT_ERR(err); input.value.update.message = "one\ntwo"; @@ -617,7 +617,7 @@ static void test_reftable_stack_tombstone(void) struct reftable_ref_record dest = { NULL }; struct reftable_log_record log_dest = { NULL }; - err = reftable_new_stack(&st, dir, opts); + err = reftable_new_stack(&st, dir, &opts); EXPECT_ERR(err); /* even entries add the refs, odd entries delete them. */ @@ -701,18 +701,18 @@ static void test_reftable_stack_hash_id(void) struct reftable_stack *st_default = NULL; struct reftable_ref_record dest = { NULL }; - err = reftable_new_stack(&st, dir, opts); + err = reftable_new_stack(&st, dir, &opts); EXPECT_ERR(err); err = reftable_stack_add(st, &write_test_ref, &ref); EXPECT_ERR(err); /* can't read it with the wrong hash ID. */ - err = reftable_new_stack(&st32, dir, opts32); + err = reftable_new_stack(&st32, dir, &opts32); EXPECT(err == REFTABLE_FORMAT_ERROR); /* check that we can read it back with default opts too. */ - err = reftable_new_stack(&st_default, dir, opts_default); + err = reftable_new_stack(&st_default, dir, &opts_default); EXPECT_ERR(err); err = reftable_stack_read_ref(st_default, "master", &dest); @@ -756,7 +756,7 @@ static void test_reflog_expire(void) }; struct reftable_log_record log = { NULL }; - err = reftable_new_stack(&st, dir, opts); + err = reftable_new_stack(&st, dir, &opts); EXPECT_ERR(err); for (i = 1; i <= N; i++) { @@ -825,13 +825,13 @@ static void test_empty_add(void) char *dir = get_tmp_dir(__LINE__); struct reftable_stack *st2 = NULL; - err = reftable_new_stack(&st, dir, opts); + err = reftable_new_stack(&st, dir, &opts); EXPECT_ERR(err); err = reftable_stack_add(st, &write_nothing, NULL); EXPECT_ERR(err); - err = reftable_new_stack(&st2, dir, opts); + err = reftable_new_stack(&st2, dir, &opts); EXPECT_ERR(err); clear_dir(dir); reftable_stack_destroy(st); @@ -858,7 +858,7 @@ static void test_reftable_stack_auto_compaction(void) int err, i; int N = 100; - err = reftable_new_stack(&st, dir, opts); + err = reftable_new_stack(&st, dir, &opts); EXPECT_ERR(err); for (i = 0; i < N; i++) { @@ -894,7 +894,7 @@ static void test_reftable_stack_add_performs_auto_compaction(void) char *dir = get_tmp_dir(__LINE__); int err, i, n = 20; - err = reftable_new_stack(&st, dir, opts); + err = reftable_new_stack(&st, dir, &opts); EXPECT_ERR(err); for (i = 0; i <= n; i++) { @@ -942,7 +942,7 @@ static void test_reftable_stack_compaction_concurrent(void) int err, i; int N = 3; - err = reftable_new_stack(&st1, dir, opts); + err = reftable_new_stack(&st1, dir, &opts); EXPECT_ERR(err); for (i = 0; i < N; i++) { @@ -959,7 +959,7 @@ static void test_reftable_stack_compaction_concurrent(void) EXPECT_ERR(err); } - err = reftable_new_stack(&st2, dir, opts); + err = reftable_new_stack(&st2, dir, &opts); EXPECT_ERR(err); err = reftable_stack_compact_all(st1, NULL); @@ -991,7 +991,7 @@ static void test_reftable_stack_compaction_concurrent_clean(void) int err, i; int N = 3; - err = reftable_new_stack(&st1, dir, opts); + err = reftable_new_stack(&st1, dir, &opts); EXPECT_ERR(err); for (i = 0; i < N; i++) { @@ -1008,7 +1008,7 @@ static void test_reftable_stack_compaction_concurrent_clean(void) EXPECT_ERR(err); } - err = reftable_new_stack(&st2, dir, opts); + err = reftable_new_stack(&st2, dir, &opts); EXPECT_ERR(err); err = reftable_stack_compact_all(st1, NULL); @@ -1017,7 +1017,7 @@ static void test_reftable_stack_compaction_concurrent_clean(void) unclean_stack_close(st1); unclean_stack_close(st2); - err = reftable_new_stack(&st3, dir, opts); + err = reftable_new_stack(&st3, dir, &opts); EXPECT_ERR(err); err = reftable_stack_clean(st3); diff --git a/reftable/writer.c b/reftable/writer.c index 10eccaaa07..4cc6e2ebd8 100644 --- a/reftable/writer.c +++ b/reftable/writer.c @@ -122,20 +122,25 @@ static struct strbuf reftable_empty_strbuf = STRBUF_INIT; struct reftable_writer * reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t), int (*flush_func)(void *), - void *writer_arg, struct reftable_write_options *opts) + void *writer_arg, const struct reftable_write_options *_opts) { struct reftable_writer *wp = reftable_calloc(1, sizeof(*wp)); - strbuf_init(&wp->block_writer_data.last_key, 0); - options_set_defaults(opts); - if (opts->block_size >= (1 << 24)) { + struct reftable_write_options opts = {0}; + + if (_opts) + opts = *_opts; + options_set_defaults(&opts); + if (opts.block_size >= (1 << 24)) { /* TODO - error return? */ abort(); } + + strbuf_init(&wp->block_writer_data.last_key, 0); wp->last_key = reftable_empty_strbuf; - REFTABLE_CALLOC_ARRAY(wp->block, opts->block_size); + REFTABLE_CALLOC_ARRAY(wp->block, opts.block_size); wp->write = writer_func; wp->write_arg = writer_arg; - wp->opts = *opts; + wp->opts = opts; wp->flush = flush_func; writer_reinit_block_writer(wp, BLOCK_TYPE_REF); From patchwork Mon May 13 08:18:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13663171 Received: from wfhigh5-smtp.messagingengine.com (wfhigh5-smtp.messagingengine.com [64.147.123.156]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 05F731474CF for ; Mon, 13 May 2024 08:18:08 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=64.147.123.156 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715588290; cv=none; b=DtvBKHJ2tmjBf0wu+JVAN69nl4e2GmmGW3smcEJI0OjyVylMgc0yJjeO//fXpbz2yBS0kl3+r4ug6/3lke3mpWaZgjEvqnThkIgVzFxfPNEktdV87uZf2M/HvDE3/6Pj6/zdAHt1EInbEgR68Ot/OcBM/Y0WVoBhW/wWjHExxWY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715588290; c=relaxed/simple; bh=5V5JSM8I51DRprmeCT0ZMwEouH53ApNc7C/gtHsypgI=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=RO1tUEcQzg6NmTBEaLMw7zuq8jZWWn3md5+AgKSpBir4eO5HlC75mGOI4yMylS8uKCh/NY5CYZGapebA6jJbL2oPxGwCtxiuAFtyAxkVVjuWtZcHPOmtjcnVNMTVqiZ1RKRbt6QzUEAVan3G824NiUQOxVG0nJ5/pBgR4BJQnrs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=nlwatANk; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=MWWE/K93; arc=none smtp.client-ip=64.147.123.156 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="nlwatANk"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="MWWE/K93" Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailfhigh.west.internal (Postfix) with ESMTP id 1D45F1800105; Mon, 13 May 2024 04:18:08 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute6.internal (MEProxy); Mon, 13 May 2024 04:18:08 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1715588287; x=1715674687; bh=+4TcLo5b8F h9SOg0oXSjaFRgcLRtWd7Prtc2R4jJv7A=; b=nlwatANkz9DgH6Pz/iRy6jLcTk k6KCNI0B8AcseWe0AGSlIHubK+XQjkxps2BN0euP/PjhBrMCZVFpyOq70dROLDbp ykNRDOIzzjyjnuVSE0abfIsLtY2iTAMEzTfHOEUwaV6vlAuR9mq9UeOyuatfKwo+ duMHaWfD7u98a+jkdwXCxnI0hbFuRUsEdSY0LEbIqhD8gZIg7G+gTxo31J6Ehw8I 7dFwDUPXVASoNGr9vZdooPaa+wdRCVlf3x/4UVCjKPvR9NmX+WonYfDycyegXEHo zHuoqGujF3UQqYoUj51md+8oKraAGgUixVUDHpI25ci5GQt+IalCbrx2EY2A== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1715588287; x=1715674687; bh=+4TcLo5b8Fh9SOg0oXSjaFRgcLRt Wd7Prtc2R4jJv7A=; b=MWWE/K93MClQ+X0y5IvxD/NNR7f+0F0tzj0WEp236q8b g1GyGKo49KGgdEC8QsD30Af/bPGnoDT0zF7W7/QDOyhOLP4/UVERvTP8I2vXBZXI EnIjb8nDBPNcLyH3L1hxV9C/v3MP/1xy84hFd9z9Hqn0n7Wb+Frt6qiVkUwwxp3S LptsbJrWuTiebH+z5pAk7Ba0AhQa82CfyhitYbR/MlU3UTwK3oScwOHK6Qr960R/ wjiQTohWGF484AjuiQ9T8CzA/O8PpF/feHl2z+xaei7iLmwykQAZJ9GBY253jbDo SBeBahpg+6JsOqGRsk9Ob/d0JOC8dG9j7GTCIznbSw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdeggedgtddvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 13 May 2024 04:18:06 -0400 (EDT) Received: by localhost (OpenSMTPD) with ESMTPSA id 846757ed (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 13 May 2024 08:17:48 +0000 (UTC) Date: Mon, 13 May 2024 10:18:04 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Karthik Nayak , Justin Tobler , Junio C Hamano Subject: [PATCH v3 03/11] reftable/writer: drop static variable used to initialize strbuf Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: We have a static variable in the reftable writer code that is merely used to initialize the `last_key` of the writer. Convert the code to instead use `strbuf_init()` and drop the variable. Signed-off-by: Patrick Steinhardt --- reftable/writer.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/reftable/writer.c b/reftable/writer.c index 4cc6e2ebd8..a043025b01 100644 --- a/reftable/writer.c +++ b/reftable/writer.c @@ -117,8 +117,6 @@ static void writer_reinit_block_writer(struct reftable_writer *w, uint8_t typ) w->block_writer->restart_interval = w->opts.restart_interval; } -static struct strbuf reftable_empty_strbuf = STRBUF_INIT; - struct reftable_writer * reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t), int (*flush_func)(void *), @@ -136,7 +134,7 @@ reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t), } strbuf_init(&wp->block_writer_data.last_key, 0); - wp->last_key = reftable_empty_strbuf; + strbuf_init(&wp->last_key, 0); REFTABLE_CALLOC_ARRAY(wp->block, opts.block_size); wp->write = writer_func; wp->write_arg = writer_arg; From patchwork Mon May 13 08:18:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13663172 Received: from wfout2-smtp.messagingengine.com (wfout2-smtp.messagingengine.com [64.147.123.145]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 31FBC1474B7 for ; Mon, 13 May 2024 08:18:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=64.147.123.145 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715588295; cv=none; b=bj+aRHE+WywAGM1kYRtzzYwrAjmvStNeloNqnBIeaIwltzM/ifHWD7hnXDzWyUIcaAmCsg57LxYAEh4CCFC+l7CoccMIYl4lWozMbkx2hH9PTgxaa6FK2rMdrXQumGLjPhc02+ihWmc70y09KNQwCsZomU9/gFi7FKFotpQN8ZU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715588295; c=relaxed/simple; bh=mRkSdMQnGTRzzOuCy151L5vlutbmWrOHiPporAr/Fk0=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=kgjE03ljLKCdA2yEMK6y9BolnJ0FgRxfoKomFI7jDA00BRj9oAqgUurU1tQWL/r0wQNMIy8pEOQtUZ44oz34WkgZie68tzvDNIJRbAOPiKV5bALFQawlCJ0VAyKZywApMOYqFzlieDTnpz3dXg2vFP0R5MnPROC0Vmk7gkA0bvg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=k2DsGKdp; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=f5b7ch3Q; arc=none smtp.client-ip=64.147.123.145 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="k2DsGKdp"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="f5b7ch3Q" Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailfout.west.internal (Postfix) with ESMTP id 4E4611C000E9; Mon, 13 May 2024 04:18:13 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute6.internal (MEProxy); Mon, 13 May 2024 04:18:13 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1715588292; x=1715674692; bh=L+FnBqk1Gp h5hGc8E31RP3v/aBf2VUnJhRsPUBPROuo=; b=k2DsGKdpkAhFOUBYg0rkjtJP0z NudRSMW+2RRYGYJXTaSQRSDp1vdRycrdnwCBEXv3JjW4SkMnYhKYwDUU4CwJb5EH NwyZwOzlhkagXqvBFOieGIXZYWIcVTxrjfPbnzLV1WmUZQHOnv6b88AkAJ+2JkTX 06bc6DfnSIomLB079HO2I81+Qeh3yG1n6/ggh1a0r4LCGWR5X9zQwDVp3lnH/xEF cwRHo2nQNPwyYH3VCuvO+VD87Tjb8Td+kMcn+td2+wdK+u8T+nXJYh4gKgN5mMIQ D8l7eJMrFbttxgvr2qJXESvnNGejS/nOaioZceJTtsAZibOXi6tnqiPleeqA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1715588292; x=1715674692; bh=L+FnBqk1Gph5hGc8E31RP3v/aBf2 VUnJhRsPUBPROuo=; b=f5b7ch3QYGIIa8qDjA8UZxCeGzSdHrnxoAa+flrexbi7 ngEZBVR8CD5eEZyYnoGKZM3Reb+GEs0XFg2qkYq4WNtGlMA6NUEaaQxZmjO0kIGW Hpt08kwyKpX31RsscGjNPJRRrvRJ1JHr8uDrIf1SDNHvi9l0c+mgRBq+CUouaAMG fts/X8VG4VKNdLHE03/HfOzVj0zRkLhEGUsn2UqOi14AMATKvjteRcmS9CRzRmiD dmZV28pgdt3YNxaPL84s9QjARMpPY1a96yxKRo+Gp29pFfMj6TUb5NeA/PGPsifG 7FtqI3xk/PmdnRlDEowg0GElKMEnVY/eU+CFfQ+tEQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdeggedgtddvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgepudenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 13 May 2024 04:18:11 -0400 (EDT) Received: by localhost (OpenSMTPD) with ESMTPSA id 45894a5d (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 13 May 2024 08:17:52 +0000 (UTC) Date: Mon, 13 May 2024 10:18:09 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Karthik Nayak , Justin Tobler , Junio C Hamano Subject: [PATCH v3 04/11] reftable/writer: improve error when passed an invalid block size Message-ID: <8ec26646f2653aa77076ca6498d7618e7c7dffa9.1715587849.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The reftable format only supports block sizes up to 16MB. When the writer is being passed a value bigger than that it simply calls abort(3P), which isn't all that helpful due to the lack of a proper error message. Improve this by calling `BUG()` instead. Signed-off-by: Patrick Steinhardt --- reftable/writer.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/reftable/writer.c b/reftable/writer.c index a043025b01..45b3e9ce1f 100644 --- a/reftable/writer.c +++ b/reftable/writer.c @@ -128,10 +128,8 @@ reftable_new_writer(ssize_t (*writer_func)(void *, const void *, size_t), if (_opts) opts = *_opts; options_set_defaults(&opts); - if (opts.block_size >= (1 << 24)) { - /* TODO - error return? */ - abort(); - } + if (opts.block_size >= (1 << 24)) + BUG("configured block size exceeds 16MB"); strbuf_init(&wp->block_writer_data.last_key, 0); strbuf_init(&wp->last_key, 0); From patchwork Mon May 13 08:18:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13663173 Received: from wfhigh5-smtp.messagingengine.com (wfhigh5-smtp.messagingengine.com [64.147.123.156]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6CA12146D6B for ; Mon, 13 May 2024 08:18:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=64.147.123.156 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715588300; cv=none; b=LRXJDV2hM6wLHhxDPHmz/LumoxjQ1RdHACE760Sxr9z6iJuvrEHYeJm3E6u3maLI+QWX+8nz4xzu1SXlJQK10EUs3wDNIFVIbBnlr02a+BH+h7yUF8yXd0zGsOFRlt0RzGLijb1LF11iTC1gTwSSNQdbtzNph7LsZBlx3gginBs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715588300; c=relaxed/simple; bh=SwdWlZ4cniXrgpFHFsqQB0nAjq4V8DiuXKf+7IywUwI=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=oACEAc2G3OPaW8GYGIv5/nA3qpMCJI8+ZHfWCvQYfRZJv3aLWau9NVU5ZUh8NRCgjx6yyLX43qDjz8y6RW9DvFtlXzuXgHJQCQpN2G98Om3MOX44ocN+9whT1gtrPkQTnv/XYyUcUHN6wnkWtlliDBp8nklc6vCQztkYHI11Dno= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=oxvkJcLh; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=J490yu15; arc=none smtp.client-ip=64.147.123.156 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="oxvkJcLh"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="J490yu15" Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailfhigh.west.internal (Postfix) with ESMTP id 69D131800101; Mon, 13 May 2024 04:18:17 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute5.internal (MEProxy); Mon, 13 May 2024 04:18:18 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1715588297; x=1715674697; bh=CEa3txDYsJ gLgngLd9sP/X9Zc5ykZIyUhZxMDL3tKkw=; b=oxvkJcLhbl0lLFshPwtjnwysRb 8jQkPRthuZQkIjbtxhPfmPJb2p98Nb35zJCc7NXTEbWzjXWrX3VehlCh5M+NueTv 04vsJCXj+YCZeEfoVad2Yn9xhSiRoB8f18lhdvL3wZdEk38iDTMdshXxlAGwrRIq uYUhbDT9sTwuxTglN0lmbi59mnLx0Y84Q0S/fsgZA3VJFv9ILB77MwzSK/fYecO4 pHFTAH2MntS/H4rrF7sYsPMgWs7A1zLX5DJWh+uNFNwUB3q9eqo6bbp6dW5gvo7d fPw+2Le95PAbgTW3fmXIGxnJ6CCyfQwNrrEJMIqrud1JEpifbRU1iYzAIYSQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1715588297; x=1715674697; bh=CEa3txDYsJgLgngLd9sP/X9Zc5yk ZIyUhZxMDL3tKkw=; b=J490yu15ljuCFxzf+afoMVQtGLHCbg4jIM5zNeZuTKkI RcD7ozx9hRP5ZdqtWyWAn97RoKtbqYfZQO2GcgaiaejFACCYyezxW4Hp/73ALVr/ 4nsrHg6j9hsS7tGPtLkfPePc6ZcsPQvVtVVcQeMtraXzVSSTJm9o3QP2DJSE/k9L RD3sSI9EW2hJUuK4dRwUlQlO1Ir2jXHbIuV+vf0kEAN/TBcyp+cVj62jJ1qNONZm qBIH8rJ+Kaa22EXVjj5ZcDeRJAe0GjhIoYyyLmBktJoqd6lv0+5xAWn+39vu5tmu ErhtnthNdYaKf0d/c08wsNxMIUzw2To21+gImvjC9w== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdeggedgtdduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 13 May 2024 04:18:15 -0400 (EDT) Received: by localhost (OpenSMTPD) with ESMTPSA id 34a544e6 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 13 May 2024 08:17:57 +0000 (UTC) Date: Mon, 13 May 2024 10:18:13 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Karthik Nayak , Justin Tobler , Junio C Hamano Subject: [PATCH v3 05/11] reftable/dump: support dumping a table's block structure Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: We're about to introduce new configs that will allow users to have more control over how exactly reftables are written. To verify that these configs are effective we will need to take a peak into the actual blocks written by the reftable backend. Introduce a new mode to the dumping logic that prints out the block structure. This logic can be invoked via `test-tool dump-reftables -b`. Signed-off-by: Patrick Steinhardt --- reftable/dump.c | 8 ++- reftable/reader.c | 63 ++++++++++++++++++ reftable/reftable-reader.h | 2 + t/t0613-reftable-write-options.sh | 102 ++++++++++++++++++++++++++++++ 4 files changed, 174 insertions(+), 1 deletion(-) create mode 100755 t/t0613-reftable-write-options.sh diff --git a/reftable/dump.c b/reftable/dump.c index 586f3eb288..41abbb8ecf 100644 --- a/reftable/dump.c +++ b/reftable/dump.c @@ -48,6 +48,7 @@ static void print_help(void) printf("usage: dump [-cst] arg\n\n" "options: \n" " -c compact\n" + " -b dump blocks\n" " -t dump table\n" " -s dump stack\n" " -6 sha256 hash format\n" @@ -58,6 +59,7 @@ static void print_help(void) int reftable_dump_main(int argc, char *const *argv) { int err = 0; + int opt_dump_blocks = 0; int opt_dump_table = 0; int opt_dump_stack = 0; int opt_compact = 0; @@ -67,6 +69,8 @@ int reftable_dump_main(int argc, char *const *argv) for (; argc > 1; argv++, argc--) if (*argv[1] != '-') break; + else if (!strcmp("-b", argv[1])) + opt_dump_blocks = 1; else if (!strcmp("-t", argv[1])) opt_dump_table = 1; else if (!strcmp("-6", argv[1])) @@ -88,7 +92,9 @@ int reftable_dump_main(int argc, char *const *argv) arg = argv[1]; - if (opt_dump_table) { + if (opt_dump_blocks) { + err = reftable_reader_print_blocks(arg); + } else if (opt_dump_table) { err = reftable_reader_print_file(arg); } else if (opt_dump_stack) { err = reftable_stack_print_directory(arg, opt_hash_id); diff --git a/reftable/reader.c b/reftable/reader.c index 481dff10d4..f23c8523db 100644 --- a/reftable/reader.c +++ b/reftable/reader.c @@ -856,3 +856,66 @@ int reftable_reader_print_file(const char *tablename) reftable_reader_free(r); return err; } + +int reftable_reader_print_blocks(const char *tablename) +{ + struct { + const char *name; + int type; + } sections[] = { + { + .name = "ref", + .type = BLOCK_TYPE_REF, + }, + { + .name = "obj", + .type = BLOCK_TYPE_OBJ, + }, + { + .name = "log", + .type = BLOCK_TYPE_LOG, + }, + }; + struct reftable_block_source src = { 0 }; + struct table_iter ti = TABLE_ITER_INIT; + struct reftable_reader *r = NULL; + size_t i; + int err; + + err = reftable_block_source_from_file(&src, tablename); + if (err < 0) + goto done; + + err = reftable_new_reader(&r, &src, tablename); + if (err < 0) + goto done; + + printf("header:\n"); + printf(" block_size: %d\n", r->block_size); + + for (i = 0; i < ARRAY_SIZE(sections); i++) { + err = reader_start(r, &ti, sections[i].type, 0); + if (err < 0) + goto done; + if (err > 0) + continue; + + printf("%s:\n", sections[i].name); + + while (1) { + printf(" - length: %u\n", ti.br.block_len); + printf(" restarts: %u\n", ti.br.restart_count); + + err = table_iter_next_block(&ti); + if (err < 0) + goto done; + if (err > 0) + break; + } + } + +done: + reftable_reader_free(r); + table_iter_close(&ti); + return err; +} diff --git a/reftable/reftable-reader.h b/reftable/reftable-reader.h index 4a4bc2fdf8..4a04857773 100644 --- a/reftable/reftable-reader.h +++ b/reftable/reftable-reader.h @@ -97,5 +97,7 @@ void reftable_table_from_reader(struct reftable_table *tab, /* print table onto stdout for debugging. */ int reftable_reader_print_file(const char *tablename); +/* print blocks onto stdout for debugging. */ +int reftable_reader_print_blocks(const char *tablename); #endif diff --git a/t/t0613-reftable-write-options.sh b/t/t0613-reftable-write-options.sh new file mode 100755 index 0000000000..462980c37c --- /dev/null +++ b/t/t0613-reftable-write-options.sh @@ -0,0 +1,102 @@ +#!/bin/sh + +test_description='reftable write options' + +GIT_TEST_DEFAULT_REF_FORMAT=reftable +export GIT_TEST_DEFAULT_REF_FORMAT +# Disable auto-compaction for all tests as we explicitly control repacking of +# refs. +GIT_TEST_REFTABLE_AUTOCOMPACTION=false +export GIT_TEST_REFTABLE_AUTOCOMPACTION +# Block sizes depend on the hash function, so we force SHA1 here. +GIT_TEST_DEFAULT_HASH=sha1 +export GIT_TEST_DEFAULT_HASH +# Block sizes also depend on the actual refs we write, so we force "master" to +# be the default initial branch name. +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME + +. ./test-lib.sh + +test_expect_success 'default write options' ' + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + test_commit initial && + git pack-refs && + cat >expect <<-EOF && + header: + block_size: 4096 + ref: + - length: 129 + restarts: 2 + log: + - length: 262 + restarts: 2 + EOF + test-tool dump-reftable -b .git/reftable/*.ref >actual && + test_cmp expect actual + ) +' + +test_expect_success 'disabled reflog writes no log blocks' ' + test_config_global core.logAllRefUpdates false && + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + test_commit initial && + git pack-refs && + cat >expect <<-EOF && + header: + block_size: 4096 + ref: + - length: 129 + restarts: 2 + EOF + test-tool dump-reftable -b .git/reftable/*.ref >actual && + test_cmp expect actual + ) +' + +test_expect_success 'many refs results in multiple blocks' ' + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + test_commit initial && + for i in $(test_seq 200) + do + printf "update refs/heads/branch-%d HEAD\n" "$i" || + return 1 + done >input && + git update-ref --stdin expect <<-EOF && + header: + block_size: 4096 + ref: + - length: 4049 + restarts: 11 + - length: 1136 + restarts: 3 + log: + - length: 4041 + restarts: 4 + - length: 4015 + restarts: 3 + - length: 4014 + restarts: 3 + - length: 4012 + restarts: 3 + - length: 3289 + restarts: 3 + EOF + test-tool dump-reftable -b .git/reftable/*.ref >actual && + test_cmp expect actual + ) +' + +test_done From patchwork Mon May 13 08:18:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13663174 Received: from wfhigh5-smtp.messagingengine.com (wfhigh5-smtp.messagingengine.com [64.147.123.156]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 86060146D5D for ; Mon, 13 May 2024 08:18:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=64.147.123.156 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715588305; cv=none; b=kMdlOJHWPnwlTimoA6uCi7ZuiG4VdvaQznRisoET71+gpf2sT51bnfGwLFoB/m1/AyD7zy1RyN5EevZJ7H8H1HW0hM2rpprxXS+Wy9bH9MAMArgR39VsrGbPH8YdVujX6dTzWnSK5+4n7KCxTl780YnbqTd5jRNLwQgT9nPoC2s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715588305; c=relaxed/simple; bh=XQR5yLqmZSmLzaHqnT0zaTXQh0Dw2+Qxo8Db5XoWjgw=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=l7v6t6r6BAYM3DVmVvVN02EDVwQIu8F++k0lb/3pbKsJ8E/2iLucLYiRx+YBnd/odtPFDnu6YFdpcrwqSky0t5UTWtWHPIM/MLrJk5bS9GdNB7RHLeCxNo3NDf1d0WAWVgcaAlYzoAhIJZtfFkKLPDJfs63sll0epGCZkZyHStA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=Mj22e9iV; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=joE56jZN; arc=none smtp.client-ip=64.147.123.156 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="Mj22e9iV"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="joE56jZN" Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailfhigh.west.internal (Postfix) with ESMTP id CE9551800105; Mon, 13 May 2024 04:18:22 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute5.internal (MEProxy); Mon, 13 May 2024 04:18:23 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1715588302; x=1715674702; bh=+/Hzv9gnSv 70yU7tvLjErEnGZhAaJpD+jENrr3I0c0k=; b=Mj22e9iVtiLZRj2Jb86DzKWU75 yF/mWgyXkSNUxd/kOOGEYrONoo2yZ9YXF+JmSo2X8h5B3b2ikGVaTmBDZApr2+yk rlMIfiGaB7NTiqhVXlwGqKDBne6vRSW9tmseoGp3LBZNYgQuq8HBpLA7He/qc9e2 DT+kDFNnyqlVDqBeXUhsR6XaEI/TLFVBoArNB5E6Yu/HLblGWUr1vkDESUA4kGcv vtczcxN8DspitaZA5itmnk1ZgFzPB9QZikCUr7SgyJ5jgrgn5Fhd23yvwNsRz7Sh G45SLG61sEzhzVqUwOTu9XKHBBOKFILug5N0Za/lT0SKFMMh4MvnmPFo3gaw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1715588302; x=1715674702; bh=+/Hzv9gnSv70yU7tvLjErEnGZhAa JpD+jENrr3I0c0k=; b=joE56jZNs7vxELkVoFhRadzVskqGV312KZU0I5JsmUQG DfhwPn49Wb5Muof509ifOIIT2fvzD8P1mFkd9wLOLDohXUgO4xGH4f7qoFnAjsP3 gxf1jIpxYAlnGVRTOCT+v010/14KrCF6Y/b+SlqL7ge8SqDiHUEVeXhS8dWG/9fp Di/KqKZmTfAxfQfD4AWKD1N+RhoiVx90FxZowHimtSGWmnHzMBwve8229FXzOqXb yOzd9VoI0V4e0XvAw6PAN93e7ppkMvqBBCCip3WeNdj4cjbPHdjN78LyqMfqzztI Jb4mXIV9j+JjDB+0QSl8ynsGBdqxAaP1HrlWvyoebw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdeggedgtdduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgepudenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 13 May 2024 04:18:21 -0400 (EDT) Received: by localhost (OpenSMTPD) with ESMTPSA id 5defbb1d (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 13 May 2024 08:18:03 +0000 (UTC) Date: Mon, 13 May 2024 10:18:19 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Karthik Nayak , Justin Tobler , Junio C Hamano Subject: [PATCH v3 06/11] refs/reftable: allow configuring block size Message-ID: <70720af4d3476e30eba2a85c2691d664e02ade31.1715587849.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Add a new option `reftable.blockSize` that allows the user to control the block size used by the reftable library. Signed-off-by: Patrick Steinhardt --- Documentation/config.txt | 2 + Documentation/config/reftable.txt | 14 ++++++ refs/reftable-backend.c | 31 ++++++++++++- t/t0613-reftable-write-options.sh | 72 +++++++++++++++++++++++++++++++ 4 files changed, 118 insertions(+), 1 deletion(-) create mode 100644 Documentation/config/reftable.txt diff --git a/Documentation/config.txt b/Documentation/config.txt index 6f649c997c..cbf0b99c44 100644 --- a/Documentation/config.txt +++ b/Documentation/config.txt @@ -498,6 +498,8 @@ include::config/rebase.txt[] include::config/receive.txt[] +include::config/reftable.txt[] + include::config/remote.txt[] include::config/remotes.txt[] diff --git a/Documentation/config/reftable.txt b/Documentation/config/reftable.txt new file mode 100644 index 0000000000..fa7c4be014 --- /dev/null +++ b/Documentation/config/reftable.txt @@ -0,0 +1,14 @@ +reftable.blockSize:: + The size in bytes used by the reftable backend when writing blocks. + The block size is determined by the writer, and does not have to be a + power of 2. The block size must be larger than the longest reference + name or log entry used in the repository, as references cannot span + blocks. ++ +Powers of two that are friendly to the virtual memory system or +filesystem (such as 4kB or 8kB) are recommended. Larger sizes (64kB) can +yield better compression, with a possible increased cost incurred by +readers during access. ++ +The largest block size is `16777215` bytes (15.99 MiB). The default value is +`4096` bytes (4kB). A value of `0` will use the default value. diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c index f8f930380d..8d0ae9e285 100644 --- a/refs/reftable-backend.c +++ b/refs/reftable-backend.c @@ -1,6 +1,7 @@ #include "../git-compat-util.h" #include "../abspath.h" #include "../chdir-notify.h" +#include "../config.h" #include "../environment.h" #include "../gettext.h" #include "../hash.h" @@ -228,6 +229,22 @@ static int read_ref_without_reload(struct reftable_stack *stack, return ret; } +static int reftable_be_config(const char *var, const char *value, + const struct config_context *ctx, + void *_opts) +{ + struct reftable_write_options *opts = _opts; + + if (!strcmp(var, "reftable.blocksize")) { + unsigned long block_size = git_config_ulong(var, value, ctx->kvi); + if (block_size > 16777215) + die("reftable block size cannot exceed 16MB"); + opts->block_size = block_size; + } + + return 0; +} + static struct ref_store *reftable_be_init(struct repository *repo, const char *gitdir, unsigned int store_flags) @@ -243,12 +260,24 @@ static struct ref_store *reftable_be_init(struct repository *repo, base_ref_store_init(&refs->base, repo, gitdir, &refs_be_reftable); strmap_init(&refs->worktree_stacks); refs->store_flags = store_flags; - refs->write_options.block_size = 4096; + refs->write_options.hash_id = repo->hash_algo->format_id; refs->write_options.default_permissions = calc_shared_perm(0666 & ~mask); refs->write_options.disable_auto_compact = !git_env_bool("GIT_TEST_REFTABLE_AUTOCOMPACTION", 1); + git_config(reftable_be_config, &refs->write_options); + + /* + * It is somewhat unfortunate that we have to mirror the default block + * size of the reftable library here. But given that the write options + * wouldn't be updated by the library here, and given that we require + * the proper block size to trim reflog message so that they fit, we + * must set up a proper value here. + */ + if (!refs->write_options.block_size) + refs->write_options.block_size = 4096; + /* * Set up the main reftable stack that is hosted in GIT_COMMON_DIR. * This stack contains both the shared and the main worktree refs. diff --git a/t/t0613-reftable-write-options.sh b/t/t0613-reftable-write-options.sh index 462980c37c..8bdbc6ec70 100755 --- a/t/t0613-reftable-write-options.sh +++ b/t/t0613-reftable-write-options.sh @@ -99,4 +99,76 @@ test_expect_success 'many refs results in multiple blocks' ' ) ' +test_expect_success 'tiny block size leads to error' ' + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + test_commit initial && + cat >expect <<-EOF && + error: unable to compact stack: entry too large + EOF + test_must_fail git -c reftable.blockSize=50 pack-refs 2>err && + test_cmp expect err + ) +' + +test_expect_success 'small block size leads to multiple ref blocks' ' + test_config_global core.logAllRefUpdates false && + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + test_commit A && + test_commit B && + git -c reftable.blockSize=100 pack-refs && + + cat >expect <<-EOF && + header: + block_size: 100 + ref: + - length: 53 + restarts: 1 + - length: 74 + restarts: 1 + - length: 38 + restarts: 1 + EOF + test-tool dump-reftable -b .git/reftable/*.ref >actual && + test_cmp expect actual + ) +' + +test_expect_success 'small block size fails with large reflog message' ' + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + test_commit A && + perl -e "print \"a\" x 500" >logmsg && + cat >expect <<-EOF && + fatal: update_ref failed for ref ${SQ}refs/heads/logme${SQ}: reftable: transaction failure: entry too large + EOF + test_must_fail git -c reftable.blockSize=100 \ + update-ref -m "$(cat logmsg)" refs/heads/logme HEAD 2>err && + test_cmp expect err + ) +' + +test_expect_success 'block size exceeding maximum supported size' ' + test_config_global core.logAllRefUpdates false && + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + test_commit A && + test_commit B && + cat >expect <<-EOF && + fatal: reftable block size cannot exceed 16MB + EOF + test_must_fail git -c reftable.blockSize=16777216 pack-refs 2>err && + test_cmp expect err + ) +' + test_done From patchwork Mon May 13 08:18:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13663175 Received: from wfhigh5-smtp.messagingengine.com (wfhigh5-smtp.messagingengine.com [64.147.123.156]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BD2C0146D5D for ; Mon, 13 May 2024 08:18:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=64.147.123.156 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715588310; cv=none; b=WiEWVVQzBExJw1e3VQ2fEuh2xWpKRqXppejOpF3qGa+l2NDPTHa/le13cDwfjutqaMhEPe+rYMihLPAEGDZLUR28pundYdC1IlEeDwCfAA7GBjHutO1T0LLqrjQUZleOSNbvQzUfbhD1gU1pLGXRJ+/BjG/bVxZhGKrrKDxhW5E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715588310; c=relaxed/simple; bh=0nwwyIrkFFvFbP0oeO2sHsl45gaByKAmmBCvKlDsgFI=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=eOp9zkYrwUljdFjRjPVaHlqjq+2lv8t4uEHd8NWH0SvE0ayFe358DrpiXV7Wm/MTIEtarVuGuYgrZoLxoQmY86l+TVoWa7UQmOHcyXIxPB8DhEhJYqEAjBcTOtmJgJJIIcnl5UqhF3+USs2C6aribCs1rsKXH2jJuEpTKS1SpkU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=H1UlPiCv; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=NfCw4Ww/; arc=none smtp.client-ip=64.147.123.156 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="H1UlPiCv"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="NfCw4Ww/" Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailfhigh.west.internal (Postfix) with ESMTP id 018021800082; Mon, 13 May 2024 04:18:27 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Mon, 13 May 2024 04:18:28 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1715588307; x=1715674707; bh=2KSA/rGdWb e7fOkX/TDtw+s0EfN9c/dQ9NUvzIyNLI8=; b=H1UlPiCvTqXP/zHiWB6jEungK1 bdYnG6UwP87bGWHhLaEBB950Fhww3B5/LkmAe4PoG/NFnq8yvAAsZOzzIb3+nPeZ RUqPpQvFujfY3+H7ojgNG/P8Bz+KyNOEvkPC5hNZ4+sTXFl2gvTSz8hOZSh72lH4 L3OPMcir9e+u7vpqSwu+TPBX+QPpycoUhzwcNT4T7yh14zgSDoKUm7WdtUa81yxJ WqeXCfw23Du7e0GTU0tXJkkNkG3x5uWLw1QsY/Y+L+dXX1kcHhRNgwmm+RmkjWg+ GuqyKb9H/k2Mehe2PwZ3VcX3meMRdHU2b0LqUdw/KqsRx5YwMrGiG8WeKMTg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1715588307; x=1715674707; bh=2KSA/rGdWbe7fOkX/TDtw+s0EfN9 c/dQ9NUvzIyNLI8=; b=NfCw4Ww/J0pwOrZrTvzFnoxQXFq1WaLo3DU7fMNZIdeR LiRafLOitgk9sF4K2P1Cmt/eahZ4N83gPt36t+DaSPaLQI18yNKLzKiriGPb2dwK nuo/1d81L9NCXEt22IBZBRYLbr00QeYnkAK7LpfRSywtl4mC+DJO4pinJiD/xqvF obmHKv0dZZSc3SR7aUJGIETPevI5oSnSR4ExrJYjnGSTxCYKM9gfyqEiB9NWc11e l+LsipsmFxmOXnDe2BMQKyXaVv0z0p2BFEc5qc7WtgtvcjgXpahlpXzs0XA8IQTw NY3OgSR29cf/nBWDegd2qBDr3jXCVLvinnMXsa3+ZQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdeggedgtddvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 13 May 2024 04:18:26 -0400 (EDT) Received: by localhost (OpenSMTPD) with ESMTPSA id 8475065c (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 13 May 2024 08:18:07 +0000 (UTC) Date: Mon, 13 May 2024 10:18:23 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Karthik Nayak , Justin Tobler , Junio C Hamano Subject: [PATCH v3 07/11] reftable: use `uint16_t` to track restart interval Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The restart interval can at most be `UINT16_MAX` as specified in the technical documentation of the reftable format. Furthermore, it cannot ever be negative. Regardless of that we use an `int` to track the restart interval. Change the type to use an `uint16_t` instead. Signed-off-by: Patrick Steinhardt --- reftable/block.h | 2 +- reftable/reftable-writer.h | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/reftable/block.h b/reftable/block.h index e91f3d2790..1c8f25ee6e 100644 --- a/reftable/block.h +++ b/reftable/block.h @@ -29,7 +29,7 @@ struct block_writer { uint32_t header_off; /* How often to restart keys. */ - int restart_interval; + uint16_t restart_interval; int hash_size; /* Offset of next uint8_t to write. */ diff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h index 03df3a4963..94804eaa68 100644 --- a/reftable/reftable-writer.h +++ b/reftable/reftable-writer.h @@ -28,7 +28,7 @@ struct reftable_write_options { unsigned skip_index_objects : 1; /* how often to write complete keys in each block. */ - int restart_interval; + uint16_t restart_interval; /* 4-byte identifier ("sha1", "s256") of the hash. * Defaults to SHA1 if unset From patchwork Mon May 13 08:18:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13663176 Received: from wfhigh5-smtp.messagingengine.com (wfhigh5-smtp.messagingengine.com [64.147.123.156]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D24FC147C6D for ; Mon, 13 May 2024 08:18:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=64.147.123.156 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715588314; cv=none; b=XE1Kl1fYEP87ivCy1SLmLhRWZu4qu6Vh/Eq/WKIh7xCWgz9W1u7IV7VApRH4JbsqYTtkjVe2hAWRNB0tOSYECCahpY4FkoRkeSV1rONDQjTb6Un8NR1bsID2OweMQZ5Hz5+vWPxakbADoeCsg/arzr/v+GjScqAHtqo/z1jm1fs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715588314; c=relaxed/simple; bh=e9hUv0eEfvujMa/Fdelfd0KbS5Od3phy2QaFNSYrRKw=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=jzTlOtb3OD67voYEpu828k0QFxI38P/JtJxBwwiBEifxGmLhXbZO5+NH82tW2j+uaWpG+IgGonp7bbswyyx7cUJ8Z4Ehqq4NQJ6dCoDa+I13AL9dTvXUixw6iDih5cW5poKOYUUwK88mIu7XdDsqSzKqwGj2rQxNSskXbAxjpHE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=W0x8XknP; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=bN571ejv; arc=none smtp.client-ip=64.147.123.156 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="W0x8XknP"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="bN571ejv" Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailfhigh.west.internal (Postfix) with ESMTP id 24F711800101; Mon, 13 May 2024 04:18:32 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Mon, 13 May 2024 04:18:32 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1715588311; x=1715674711; bh=V/Z7zdog0o CyriN+gHIoyFw4qbwIj6mQN36ho3mb3gU=; b=W0x8XknP4K27bJkw+uGfo9z647 wtCsr8XrnXtSfwE0DgoD6KN4wYBdRcrRlpnKy5pLPwaQxmG/As+ZCEYhrnSCx8ua R1mwJ1dg0fEOPANceugTSO5zGaJLvhkejuwZ1liKFiW5C/YsY5qEhXUb8lYfSZk4 yaJ1COk8oHEjFtsc/ySCm9yMtLixLViJJDDy2FiQgKz9NkNbHa+1Th4ze6/0kP6F SGHGOB83Ac45OKw7I2Br53Rn+AYQlUG8eUujpasuPnU+pv/NboGMJwU/3O9AvqR4 yerX1ps//SFHkmTcFUV7merx4agnZVTBMIlts5lIaIB/xkjuAvnmj2FMvNrw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1715588311; x=1715674711; bh=V/Z7zdog0oCyriN+gHIoyFw4qbwI j6mQN36ho3mb3gU=; b=bN571ejvNdGL3ozRAV0HrYiSiKRNYxEtlGSK+M91bPb0 xpbYx4cb/M26Zy9lvB2nmwsTfE2TGqjVsO9ShRi9ArYkD1Wb403id/+zdTf92Jyd aAG77aZfMmR0R/PinTVPgXEjCyQYeHITf280L13/7TgRk/YllZqCeG71kKIa1RQ4 coap0VN+zz7I8315aa+YS4zYVGQo+Wa66Je4Md5TL/pcfcwkJItrgnanF7wFb+n0 i9ePha+XMkGWU2m4zW4gPgoszigaz0aMXWvdA/Tlpj/WJFr8KFf7FXnJDJqwVBtz wRqEH/l5JMBi6lZdVw8J38pUyZ50vdUD+emny7Kf+Q== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdeggedgtddvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 13 May 2024 04:18:30 -0400 (EDT) Received: by localhost (OpenSMTPD) with ESMTPSA id c5c6ea7d (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 13 May 2024 08:18:12 +0000 (UTC) Date: Mon, 13 May 2024 10:18:28 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Karthik Nayak , Justin Tobler , Junio C Hamano Subject: [PATCH v3 08/11] refs/reftable: allow configuring restart interval Message-ID: <2b1579570798a325f8a5c97438b1fdeef65aa049.1715587849.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Add a new option `reftable.restartInterval` that allows the user to control the restart interval when writing reftable records used by the reftable library. Signed-off-by: Patrick Steinhardt --- Documentation/config/reftable.txt | 18 +++++++++++++ refs/reftable-backend.c | 5 ++++ t/t0613-reftable-write-options.sh | 43 +++++++++++++++++++++++++++++++ 3 files changed, 66 insertions(+) diff --git a/Documentation/config/reftable.txt b/Documentation/config/reftable.txt index fa7c4be014..2374be71d7 100644 --- a/Documentation/config/reftable.txt +++ b/Documentation/config/reftable.txt @@ -12,3 +12,21 @@ readers during access. + The largest block size is `16777215` bytes (15.99 MiB). The default value is `4096` bytes (4kB). A value of `0` will use the default value. + +reftable.restartInterval:: + The interval at which to create restart points. The reftable backend + determines the restart points at file creation. Every 16 may be + more suitable for smaller block sizes (4k or 8k), every 64 for larger + block sizes (64k). ++ +More frequent restart points reduces prefix compression and increases +space consumed by the restart table, both of which increase file size. ++ +Less frequent restart points makes prefix compression more effective, +decreasing overall file size, with increased penalties for readers +walking through more records after the binary search step. ++ +A maximum of `65535` restart points per block is supported. ++ +The default value is to create restart points every 16 records. A value of `0` +will use the default value. diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c index 8d0ae9e285..a2880aabce 100644 --- a/refs/reftable-backend.c +++ b/refs/reftable-backend.c @@ -240,6 +240,11 @@ static int reftable_be_config(const char *var, const char *value, if (block_size > 16777215) die("reftable block size cannot exceed 16MB"); opts->block_size = block_size; + } else if (!strcmp(var, "reftable.restartinterval")) { + unsigned long restart_interval = git_config_ulong(var, value, ctx->kvi); + if (restart_interval > UINT16_MAX) + die("reftable block size cannot exceed %u", (unsigned)UINT16_MAX); + opts->restart_interval = restart_interval; } return 0; diff --git a/t/t0613-reftable-write-options.sh b/t/t0613-reftable-write-options.sh index 8bdbc6ec70..e0a5b26f58 100755 --- a/t/t0613-reftable-write-options.sh +++ b/t/t0613-reftable-write-options.sh @@ -171,4 +171,47 @@ test_expect_success 'block size exceeding maximum supported size' ' ) ' +test_expect_success 'restart interval at every single record' ' + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + test_commit initial && + for i in $(test_seq 10) + do + printf "update refs/heads/branch-%d HEAD\n" "$i" || + return 1 + done >input && + git update-ref --stdin expect <<-EOF && + header: + block_size: 4096 + ref: + - length: 566 + restarts: 13 + log: + - length: 1393 + restarts: 12 + EOF + test-tool dump-reftable -b .git/reftable/*.ref >actual && + test_cmp expect actual + ) +' + +test_expect_success 'restart interval exceeding maximum supported interval' ' + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + test_commit initial && + cat >expect <<-EOF && + fatal: reftable block size cannot exceed 65535 + EOF + test_must_fail git -c reftable.restartInterval=65536 pack-refs 2>err && + test_cmp expect err + ) +' + test_done From patchwork Mon May 13 08:18:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13663177 Received: from wfout2-smtp.messagingengine.com (wfout2-smtp.messagingengine.com [64.147.123.145]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F2D4B147C7C for ; Mon, 13 May 2024 08:18:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=64.147.123.145 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715588319; cv=none; b=NhHzd8gZGZFEo/WdOxJ+cAHYGb2QKrKV0O7s1Ho9ZG1PG3Rnqj1usTWoFyEIBB6ujCao8n0RMZXnPnFqTnl2Onfq9wF/Pkl4kuJjMMs0swFXakkfpBZPDDqkpGA+C8uypbCkg4Dw5VJ5HKOvXZ4Dm/sqdJNVyShY3Z9P3/PgluQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715588319; c=relaxed/simple; bh=qfUCS3ulBo3iqB2bUF0gX9sSRAs9HT7b/OMhtKgkySs=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=hLnnq8R9pKewt/WWNZu76SP+G+0APquOLXiEF1rhFothK6yTu/7km/nIdeyJan/YJDlVtDyOwpLqP91CF6tCOoVrgYhV6Mh01lvPGk91rfObkIkaXEIA24Ljtz/Iqpnqd/6bOLGNa9J3hhFQ1VZNKHEqm4wfpiZg5zvl9dmfVu4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=Z8REzbyq; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=YVjIpXi7; arc=none smtp.client-ip=64.147.123.145 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="Z8REzbyq"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="YVjIpXi7" Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailfout.west.internal (Postfix) with ESMTP id 452641C000E9; Mon, 13 May 2024 04:18:37 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute5.internal (MEProxy); Mon, 13 May 2024 04:18:37 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1715588316; x=1715674716; bh=pifuyWtq6V VoeaYei6nHtO3GOjMeg7S2kqt6BKzv3IU=; b=Z8REzbyqlftKvXmqo8Md0K4gek Kha6+9LOCxbMHUybt+hVeNkoQWGsHXW2Nzz7SMIQQzBop7B7zmdKEKlHZTL8w1Bb KNik2NAbmqVVXlFMppNwVZSQ73nRsXf9VIfcWIl8GPNUqg8ghGk7o1y0pfogwzpu 3Gu8LWKGSn0XsOxvKhFnH4fKbglJa+Rr4cgwQ6+9Uiomh0DnT6aik/cId7f0D3E4 Er/u2uqnP0MdlA6O/U3U6eDmln1M5HNvP2ePAWoI1GcFlgGgnVSN2TjtnhuI7sUc /NG7L+WGSwxliyhsfAl+wDdsNCx5y0qKJoMlg+Uj6JPETU7pIO/2/k1PuTIQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1715588316; x=1715674716; bh=pifuyWtq6VVoeaYei6nHtO3GOjMe g7S2kqt6BKzv3IU=; b=YVjIpXi7mLNFMdTJWtWgL1ZzLf2uSNJIJp2T+nm9YxJk JujB+HfVJZsRHAtDxGkxLfTLUI/atFLNXZ2fK5++e7RZBdiOB1rogUp2IES8E1+e HpghMwo3jLEF1N8sOzcj7mEy7fj8TRJBpkvl7djBznCa81fuAppC6FPW7vgwKJmE 7iqQghvIaXcdpOeJtaz9tu9Fl+GuFDmGQOUw1UXRejVUsfMKWlvsp5hps6a/3DTp 7CqWNJrT8lMOlRXt5tvKpN38u+vBD3S7Ya6vNb3X3PXRkrrVajo0nDbJmGyIdIy8 t0167bQ/N0HC3zKIXlxEwUDulbpiPOwvJEIOCg9Jfg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdeggedgtdduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepvdevudfhteeiveehheeiffduheevjeffveefieeiudetffeitdejgefgvdfghedu necuffhomhgrihhnpegrlhhlohifrghnhihshhgrudhinhifrghnthdrihhtnecuvehluh hsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphhssehpkhhsrdhi mh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 13 May 2024 04:18:35 -0400 (EDT) Received: by localhost (OpenSMTPD) with ESMTPSA id 0753c547 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 13 May 2024 08:18:17 +0000 (UTC) Date: Mon, 13 May 2024 10:18:33 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Karthik Nayak , Justin Tobler , Junio C Hamano Subject: [PATCH v3 09/11] refs/reftable: allow disabling writing the object index Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Besides the expected "ref" and "log" records, the reftable library also writes "obj" records. These are basically a reverse mapping of object IDs to their respective ref records so that it becomes efficient to figure out which references point to a specific object. The motivation for this data structure is the "uploadpack.allowTipSHA1InWant" config, which allows a client to fetch any object by its hash that has a ref pointing to it. This reverse index is not used by Git at all though, and the expectation is that most hosters nowadays use "uploadpack.allowAnySHA1InWant". It may thus be preferable for many users to disable writing these optional object indices altogether to safe some precious disk space. Add a new config "reftable.indexObjects" that allows the user to disable the object index altogether. Signed-off-by: Patrick Steinhardt --- Documentation/config/reftable.txt | 6 +++ refs/reftable-backend.c | 2 + t/t0613-reftable-write-options.sh | 69 +++++++++++++++++++++++++++++++ 3 files changed, 77 insertions(+) diff --git a/Documentation/config/reftable.txt b/Documentation/config/reftable.txt index 2374be71d7..68083876fa 100644 --- a/Documentation/config/reftable.txt +++ b/Documentation/config/reftable.txt @@ -30,3 +30,9 @@ A maximum of `65535` restart points per block is supported. + The default value is to create restart points every 16 records. A value of `0` will use the default value. + +reftable.indexObjects:: + Whether the reftable backend shall write object blocks. Object blocks + are a reverse mapping of object ID to the references pointing to them. ++ +The default value is `true`. diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c index a2880aabce..5ffb36770a 100644 --- a/refs/reftable-backend.c +++ b/refs/reftable-backend.c @@ -245,6 +245,8 @@ static int reftable_be_config(const char *var, const char *value, if (restart_interval > UINT16_MAX) die("reftable block size cannot exceed %u", (unsigned)UINT16_MAX); opts->restart_interval = restart_interval; + } else if (!strcmp(var, "reftable.indexobjects")) { + opts->skip_index_objects = !git_config_bool(var, value); } return 0; diff --git a/t/t0613-reftable-write-options.sh b/t/t0613-reftable-write-options.sh index e0a5b26f58..e2708e11d5 100755 --- a/t/t0613-reftable-write-options.sh +++ b/t/t0613-reftable-write-options.sh @@ -214,4 +214,73 @@ test_expect_success 'restart interval exceeding maximum supported interval' ' ) ' +test_expect_success 'object index gets written by default with ref index' ' + test_config_global core.logAllRefUpdates false && + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + test_commit initial && + for i in $(test_seq 5) + do + printf "update refs/heads/branch-%d HEAD\n" "$i" || + return 1 + done >input && + git update-ref --stdin expect <<-EOF && + header: + block_size: 100 + ref: + - length: 53 + restarts: 1 + - length: 95 + restarts: 1 + - length: 71 + restarts: 1 + - length: 80 + restarts: 1 + obj: + - length: 11 + restarts: 1 + EOF + test-tool dump-reftable -b .git/reftable/*.ref >actual && + test_cmp expect actual + ) +' + +test_expect_success 'object index can be disabled' ' + test_config_global core.logAllRefUpdates false && + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + test_commit initial && + for i in $(test_seq 5) + do + printf "update refs/heads/branch-%d HEAD\n" "$i" || + return 1 + done >input && + git update-ref --stdin expect <<-EOF && + header: + block_size: 100 + ref: + - length: 53 + restarts: 1 + - length: 95 + restarts: 1 + - length: 71 + restarts: 1 + - length: 80 + restarts: 1 + EOF + test-tool dump-reftable -b .git/reftable/*.ref >actual && + test_cmp expect actual + ) +' + test_done From patchwork Mon May 13 08:18:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13663178 Received: from wfhigh5-smtp.messagingengine.com (wfhigh5-smtp.messagingengine.com [64.147.123.156]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 563A81474AF for ; Mon, 13 May 2024 08:18:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=64.147.123.156 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715588324; cv=none; b=pc1yA/+f6wXRAu9QY0N5mJzMGjYf006m8w2Uiwsu4w+sTl4nE7lkq5g12UnhOQqs2+wXQ/Yboc/jAGLuaNZ7GHRoyRDKHp1JFGU4dIODraHkbDgbklzwBrCjpool7txctM+BO5r1Q7K8+1C2boJAmyfVmGXw78o3Fj/+dM3Fxuc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715588324; c=relaxed/simple; bh=L0GLSaWYBGbvpW2G9xZX//MJe3gHLrKWmKxFtXH05D0=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=PM2ZuAASS7J93KQNgGKinG0A4pPlQQOgY/WCJYdweqhIX0DWvZ8mFIqhSQsDJi7FUW5x3v+wd2832myvw2FZquftUGFf9ftZuc0kOAEBymbWQuIGKWbonvFRL59A/1s1N1wmg8gkdcIQ8qA8W3fv4D54u50TFXUWzpq+s/gNBn0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=UH8zjaCd; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=cEW12YER; arc=none smtp.client-ip=64.147.123.156 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="UH8zjaCd"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="cEW12YER" Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailfhigh.west.internal (Postfix) with ESMTP id 640DC180007D; Mon, 13 May 2024 04:18:42 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute4.internal (MEProxy); Mon, 13 May 2024 04:18:42 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1715588322; x=1715674722; bh=o9XBmHAi3Z KCmBlkAMiA+/yxuXKy0i+3NtEzJQlxuS0=; b=UH8zjaCdqd52nIN1NZKPP1YgYH J2Ch1LfylBR6zfLAQMsL7OWAyzrAHyIstKK9lWO+oXxLOjw/CM0+Hn40yweQi3id az/BQFDMZPKwhf9j2OfDYOTjjEzu60IKVN/Eiyvcc4LVzhzRKg3ViqTJCSunM7wD DU3iFg/YG5Kd985hqaK96UCdY9Sp5nvhIYvnDnwWe9WZ9wg2H417jCxIULa2ALCY YsEsjyZwR76JTUekT2l1erEbc5hAYytedioaNAa60+boTCGSjUa/u25Iod5JPteh E1KymjWBMEAwGF2C/uh8xJ68iSxOmmp4yFclvsKaiJ1Fg9QCaKHSiMlLFhog== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1715588322; x=1715674722; bh=o9XBmHAi3ZKCmBlkAMiA+/yxuXKy 0i+3NtEzJQlxuS0=; b=cEW12YERIcGWkBSCeLSjQLfWCMmryg4kFpMft5VJhcpD O6KfCSCuQSP2IouOBdPLzf7xfi1l1XNqay+/dmlyQ7jCe06MJieIR1SVf7Yd3HKr gcCxm7d8hozZ37Bliw72B/KKJcTORCeS1R955CezO8Rn+e78W7kGS/auxNzLixEN 4SvAH3S50wNxV7jzspmxHaJ9pa2ff5fO0jwNX1WFGU8/ew4QxiNHE8z3EYz3w3WO N51mAoXaIA6IukNAvop8P4V8ldCxv+3BzOU+7yNerdESOK7pJmRHyl2LkYKJ10md iYUcPUOj7LVyOSMgERD0RSlTNEbmFw7Z2z4L9/QgHQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdeggedgtddvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgepvdenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 13 May 2024 04:18:40 -0400 (EDT) Received: by localhost (OpenSMTPD) with ESMTPSA id 262a4cb3 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 13 May 2024 08:18:22 +0000 (UTC) Date: Mon, 13 May 2024 10:18:38 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Karthik Nayak , Justin Tobler , Junio C Hamano Subject: [PATCH v3 10/11] reftable: make the compaction factor configurable Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: When auto-compacting, the reftable library packs references such that the sizes of the tables form a geometric sequence. The factor for this geometric sequence is hardcoded to 2 right now. We're about to expose this as a config option though, so let's expose the factor via write options. Signed-off-by: Patrick Steinhardt --- reftable/constants.h | 1 + reftable/reftable-writer.h | 6 ++++++ reftable/stack.c | 14 ++++++++++---- reftable/stack.h | 3 ++- reftable/stack_test.c | 4 ++-- 5 files changed, 21 insertions(+), 7 deletions(-) diff --git a/reftable/constants.h b/reftable/constants.h index 5eee72c4c1..f6beb843eb 100644 --- a/reftable/constants.h +++ b/reftable/constants.h @@ -17,5 +17,6 @@ license that can be found in the LICENSE file or at #define MAX_RESTARTS ((1 << 16) - 1) #define DEFAULT_BLOCK_SIZE 4096 +#define DEFAULT_GEOMETRIC_FACTOR 2 #endif diff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h index 94804eaa68..189b1f4144 100644 --- a/reftable/reftable-writer.h +++ b/reftable/reftable-writer.h @@ -45,6 +45,12 @@ struct reftable_write_options { /* boolean: Prevent auto-compaction of tables. */ unsigned disable_auto_compact : 1; + + /* + * Geometric sequence factor used by auto-compaction to decide which + * tables to compact. Defaults to 2 if unset. + */ + uint8_t auto_compaction_factor; }; /* reftable_block_stats holds statistics for a single block type */ diff --git a/reftable/stack.c b/reftable/stack.c index d2e68be7e8..0ebe69e81d 100644 --- a/reftable/stack.c +++ b/reftable/stack.c @@ -10,6 +10,7 @@ license that can be found in the LICENSE file or at #include "../write-or-die.h" #include "system.h" +#include "constants.h" #include "merged.h" #include "reader.h" #include "reftable-error.h" @@ -1212,12 +1213,16 @@ static int segment_size(struct segment *s) return s->end - s->start; } -struct segment suggest_compaction_segment(uint64_t *sizes, size_t n) +struct segment suggest_compaction_segment(uint64_t *sizes, size_t n, + uint8_t factor) { struct segment seg = { 0 }; uint64_t bytes; size_t i; + if (!factor) + factor = DEFAULT_GEOMETRIC_FACTOR; + /* * If there are no tables or only a single one then we don't have to * compact anything. The sequence is geometric by definition already. @@ -1249,7 +1254,7 @@ struct segment suggest_compaction_segment(uint64_t *sizes, size_t n) * 64, 32, 16, 8, 4, 3, 1 */ for (i = n - 1; i > 0; i--) { - if (sizes[i - 1] < sizes[i] * 2) { + if (sizes[i - 1] < sizes[i] * factor) { seg.end = i + 1; bytes = sizes[i]; break; @@ -1275,7 +1280,7 @@ struct segment suggest_compaction_segment(uint64_t *sizes, size_t n) uint64_t curr = bytes; bytes += sizes[i - 1]; - if (sizes[i - 1] < curr * 2) { + if (sizes[i - 1] < curr * factor) { seg.start = i - 1; seg.bytes = bytes; } @@ -1301,7 +1306,8 @@ int reftable_stack_auto_compact(struct reftable_stack *st) { uint64_t *sizes = stack_table_sizes_for_compaction(st); struct segment seg = - suggest_compaction_segment(sizes, st->merged->stack_len); + suggest_compaction_segment(sizes, st->merged->stack_len, + st->opts.auto_compaction_factor); reftable_free(sizes); if (segment_size(&seg) > 0) return stack_compact_range_stats(st, seg.start, seg.end - 1, diff --git a/reftable/stack.h b/reftable/stack.h index 97d7ebc043..5b45cff4f7 100644 --- a/reftable/stack.h +++ b/reftable/stack.h @@ -35,6 +35,7 @@ struct segment { uint64_t bytes; }; -struct segment suggest_compaction_segment(uint64_t *sizes, size_t n); +struct segment suggest_compaction_segment(uint64_t *sizes, size_t n, + uint8_t factor); #endif diff --git a/reftable/stack_test.c b/reftable/stack_test.c index d15f11d712..0f7b1453e6 100644 --- a/reftable/stack_test.c +++ b/reftable/stack_test.c @@ -729,7 +729,7 @@ static void test_suggest_compaction_segment(void) { uint64_t sizes[] = { 512, 64, 17, 16, 9, 9, 9, 16, 2, 16 }; struct segment min = - suggest_compaction_segment(sizes, ARRAY_SIZE(sizes)); + suggest_compaction_segment(sizes, ARRAY_SIZE(sizes), 2); EXPECT(min.start == 1); EXPECT(min.end == 10); } @@ -738,7 +738,7 @@ static void test_suggest_compaction_segment_nothing(void) { uint64_t sizes[] = { 64, 32, 16, 8, 4, 2 }; struct segment result = - suggest_compaction_segment(sizes, ARRAY_SIZE(sizes)); + suggest_compaction_segment(sizes, ARRAY_SIZE(sizes), 2); EXPECT(result.start == result.end); } From patchwork Mon May 13 08:18:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13663179 Received: from wfhigh5-smtp.messagingengine.com (wfhigh5-smtp.messagingengine.com [64.147.123.156]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 728791474B2 for ; Mon, 13 May 2024 08:18:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=64.147.123.156 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715588329; cv=none; b=ajXpxAfS4fOtWtplzqlHnGA9TaTjzxMn2hW+n+gwOc5/0Jhn98hnky6EgFiP1lq9eoAdO5S0yI2XXcMutQrqxGwijHLueO5wv2mEPm275mVyPx4RlPwTKBMo1UqNkuEeqcEzSTMHxydkohjg9o9ExTHO6Op0a+8g8WaDgO3M61E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715588329; c=relaxed/simple; bh=k6e/2/IBhxAA7U0D/CthDDygadEBwAGOhuk6/p1zfgs=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=uRSWZFcpZRaD57LARA9Xa83tuZF4QKMoy/YUOK7Hs8vFDGUHNfsLpb2O/bcni7QC2+qDyMev8gUhQizzYD0yyw8aRyx3i+O6CnX82HkPGk1nG9qemw1wzBymue/mHRfIulMyqiVjX0qyCCMmceNWN1kHcnscdz0WaqOSWeO+K7E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=IqXERvLx; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=Udgw9/1f; arc=none smtp.client-ip=64.147.123.156 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="IqXERvLx"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="Udgw9/1f" Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailfhigh.west.internal (Postfix) with ESMTP id 84A3A18000F9; Mon, 13 May 2024 04:18:47 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Mon, 13 May 2024 04:18:47 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1715588327; x=1715674727; bh=t/9n/5hHQ9 ZfixrsQP9qPiFZzutCIWwSTnIdZRNZ3bA=; b=IqXERvLxHG1rOglrDQxmM7ELWz QjnmrhGOqSBf8Ufl28VSkQ7Ro47NrOGPRMO2Xxd5uk99YWmoiCbp6V4+hyeohtKs UZJaVE8xRF6Fj2UX3ZuU5aN3+0tHDle6mYkXOKY7oIYXd/ShLkpYeeJ0eLUIv2ls B4AdxsMTBo7wlJ+61wfh4PYDo7Z3FbDRqzi90sbEIPjhtIKD97dr94ZTsdSi69XZ yojfohk6QXxWT50O0byTlW3rBMkzwoFSkRaws7HNVJFoRWMvidr9XLRGedYWCcZP rtXJUoP3yb+gl0L4umA+ZCxKmhUwM98D/H6svdcF5vPObzkbUBUezy7H++ww== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1715588327; x=1715674727; bh=t/9n/5hHQ9ZfixrsQP9qPiFZzutC IWwSTnIdZRNZ3bA=; b=Udgw9/1fW3eJnu6NU81QyzgpwxZ81eBZzIRFF8WVzTb5 QZagGc7zeE1vn7THM/3Po4q+UH9EpGBXsgrLWkvFd/3NmLP8K59j+UfYT0e8o0Vr PYChNeehoDROxyW9TYK1DLSLZQcEFlqDdRzA3hVCVIRyKwXSqZAWdc5A1SXv1vqq nOWQ8oiT8UEs0Ck69MCK7pmxvWQyzFpfAIE0VZmVc8EMYRZu10i90t8NkGH5XYCd dCeE5e9jxYbTWo49c/nWejhv4L4VfHKBP2v8zB3ixOBj3/6lhbheb/4YclLd+ElI /zap4n0tejz1+5l987g7UkXOWkB0Q5hwnxQhmzCL5A== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdeggedgtddvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 13 May 2024 04:18:45 -0400 (EDT) Received: by localhost (OpenSMTPD) with ESMTPSA id 2ba60a04 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 13 May 2024 08:18:26 +0000 (UTC) Date: Mon, 13 May 2024 10:18:43 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Karthik Nayak , Justin Tobler , Junio C Hamano Subject: [PATCH v3 11/11] refs/reftable: allow configuring geometric factor Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Allow configuring the geometric factor used by the auto-compaction algorithm whenever a new table is appended to the stack of tables. Signed-off-by: Patrick Steinhardt --- Documentation/config/reftable.txt | 10 ++++++++++ refs/reftable-backend.c | 5 +++++ 2 files changed, 15 insertions(+) diff --git a/Documentation/config/reftable.txt b/Documentation/config/reftable.txt index 68083876fa..0515727977 100644 --- a/Documentation/config/reftable.txt +++ b/Documentation/config/reftable.txt @@ -36,3 +36,13 @@ reftable.indexObjects:: are a reverse mapping of object ID to the references pointing to them. + The default value is `true`. + +reftable.geometricFactor:: + Whenever the reftable backend appends a new table to the stack, it + performs auto compaction to ensure that there is only a handful of + tables. The backend does this by ensuring that tables form a geometric + sequence regarding the respective sizes of each table. ++ +By default, the geometric sequence uses a factor of 2, meaning that for any +table, the next-biggest table must at least be twice as big. A maximum factor +of 256 is supported. diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c index 5ffb36770a..da620fd598 100644 --- a/refs/reftable-backend.c +++ b/refs/reftable-backend.c @@ -247,6 +247,11 @@ static int reftable_be_config(const char *var, const char *value, opts->restart_interval = restart_interval; } else if (!strcmp(var, "reftable.indexobjects")) { opts->skip_index_objects = !git_config_bool(var, value); + } else if (!strcmp(var, "reftable.geometricfactor")) { + unsigned long factor = git_config_ulong(var, value, ctx->kvi); + if (factor > UINT8_MAX) + die("reftable geometric factor cannot exceed %u", (unsigned)UINT8_MAX); + opts->auto_compaction_factor = factor; } return 0;