From patchwork Fri May 10 10:29:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13661280 Received: from wfout7-smtp.messagingengine.com (wfout7-smtp.messagingengine.com [64.147.123.150]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 57F1B165FA9 for ; Fri, 10 May 2024 10:30:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=64.147.123.150 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715337001; cv=none; b=bP0M6AVpA0Eup7rEVPWuBHNptWpnlje5Nm6LLNeQGEcCXkkt5X26/EtTKWyLS3cHqgQsAKP0rQfUllu03HniH5Zcjam+cxS+Ud2LE1ZKUfa6pcU1agwMyloNZ+MHEP1Hzqey/WQq2HI0DmzAo+vnD/TXJ03uyWWLgookhKv9RDw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715337001; c=relaxed/simple; bh=VsfLnbHiG2m07njE5cPOYPUZeD+f9MtCqe2SX0WQX2M=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=H7cEaJ/XCJIHDxyjfWMidUhRALyNARnQV7RlEWwKPov0SSPZyElu3mLZWPWqX5nNoqYvRHnq2UtMn3WXpG95EKSe/aHN7hYintO0oINTVmIvuFqGe6+V2bMyj2pgtTEnmp1SkniqKQxw5ZdKLvAbh45mBM4BZam25+D+xJbV+4Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=TXfzBrxW; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=ZMogNNgb; arc=none smtp.client-ip=64.147.123.150 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="TXfzBrxW"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="ZMogNNgb" Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailfout.west.internal (Postfix) with ESMTP id 5A6371C000D3; Fri, 10 May 2024 06:29:59 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Fri, 10 May 2024 06:29:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1715336998; x=1715423398; bh=RYS9Y2gvp0 ExQ3iP7CTzvHI2QaBTTQwBlP92Tc/qzQ0=; b=TXfzBrxW0WveweZk9zAsbAohoE 5/KS+R4l0p4QCeogydjr8wBxqzxYw5xDSdaVkZEPpb/LBPbtv1RodZfWBkJogfts NVR/9tXhoncW4GV7BeyURQUBN+A4qceM3tLTppGL2QmJDnn6f040QfF2bR9RYFM7 3AuvmELz+4Ko08E1M9wSYTsuOoU7MyB6r0PZM3rT7Nq6bhUtQbTC+F2piKptgU3O JTbddxpBkGVH53pVwLMvkFjjuAPH+YXYh2RH1TWlO5Bi4aGe/SE/dPQj79BTPrjs ClkPUS5PV5AYAHkGv1Tw+KkptlA0mnrVp3XzWUBxO0kfC7PQsmKpJ6rAbxKw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1715336998; x=1715423398; bh=RYS9Y2gvp0ExQ3iP7CTzvHI2QaBT TQwBlP92Tc/qzQ0=; b=ZMogNNgb4uvEJ3p+kEnRwJfT4lqVx7DHrx7Q/U4hQxVj 5FXKVTv3qNmcqzMSZYFLXsYi3Zy1xdaEocyKkqswlsDU0U+m3Uzn0Mn1o6uHYIKe 9taG1IbFJuK3EgQ4C/FLmEVbE7PWnjjbHYmJzjTC5MTqhISqTN0UcrBXTJS4l+BQ emBGtGMhK3XSZ+eVw/+pDTCL0mI1/E1k90inBpVgTs03+VjiUxrXcp3+LEM8Hvrd RbM74jVvXAdOEZjfvAFPhTKK3YcagcdQzhYNtBqTJo85ULDY4dyLU4uYu/cufmz5 fiNpkAaAUqPQPY7MDFNQErIaxCNCFylxaoVJcxkMtQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvdefkedgvdekucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkfhggtggujgesghdtreertddtvdenucfhrhhomheprfgrthhr ihgtkhcuufhtvghinhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvg hrnhepueektdevtdffveeljeetgfehheeigeekleduvdeffeeghefgledttdehjeelffet necuvehluhhsthgvrhfuihiivgepgeenucfrrghrrghmpehmrghilhhfrhhomhepphhsse hpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 10 May 2024 06:29:58 -0400 (EDT) Received: by localhost (OpenSMTPD) with ESMTPSA id c11b8a59 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Fri, 10 May 2024 10:29:45 +0000 (UTC) Date: Fri, 10 May 2024 12:29:55 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Karthik Nayak , Justin Tobler Subject: [PATCH v2 08/11] refs/reftable: allow configuring restart interval Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Add a new option `reftable.restartInterval` that allows the user to control the restart interval when writing reftable records used by the reftable library. Signed-off-by: Patrick Steinhardt --- Documentation/config/reftable.txt | 19 ++++++++++++++ refs/reftable-backend.c | 5 ++++ t/t0613-reftable-write-options.sh | 43 +++++++++++++++++++++++++++++++ 3 files changed, 67 insertions(+) diff --git a/Documentation/config/reftable.txt b/Documentation/config/reftable.txt index fa7c4be014..16b915c75e 100644 --- a/Documentation/config/reftable.txt +++ b/Documentation/config/reftable.txt @@ -12,3 +12,22 @@ readers during access. + The largest block size is `16777215` bytes (15.99 MiB). The default value is `4096` bytes (4kB). A value of `0` will use the default value. + +reftable.restartInterval:: + The interval at which to create restart points. The reftable backend + determines the restart points at file creation. The process is + arbitrary, but every 16 or 64 records is recommended. Every 16 may be + more suitable for smaller block sizes (4k or 8k), every 64 for larger + block sizes (64k). ++ +More frequent restart points reduces prefix compression and increases +space consumed by the restart table, both of which increase file size. ++ +Less frequent restart points makes prefix compression more effective, +decreasing overall file size, with increased penalties for readers +walking through more records after the binary search step. ++ +A maximum of `65535` restart points per block is supported. ++ +The default value is to create restart points every 16 records. A value of `0` +will use the default value. diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c index bd9999cefc..9972dfc1a3 100644 --- a/refs/reftable-backend.c +++ b/refs/reftable-backend.c @@ -242,6 +242,11 @@ static int reftable_be_config(const char *var, const char *value, if (block_size > 16777215) die("reftable block size cannot exceed 16MB"); opts->block_size = block_size; + } else if (!strcmp(var, "reftable.restartinterval")) { + unsigned long restart_interval = git_config_ulong(var, value, ctx->kvi); + if (restart_interval > UINT16_MAX) + die("reftable block size cannot exceed %u", (unsigned)UINT16_MAX); + opts->restart_interval = restart_interval; } return 0; diff --git a/t/t0613-reftable-write-options.sh b/t/t0613-reftable-write-options.sh index 8bdbc6ec70..e0a5b26f58 100755 --- a/t/t0613-reftable-write-options.sh +++ b/t/t0613-reftable-write-options.sh @@ -171,4 +171,47 @@ test_expect_success 'block size exceeding maximum supported size' ' ) ' +test_expect_success 'restart interval at every single record' ' + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + test_commit initial && + for i in $(test_seq 10) + do + printf "update refs/heads/branch-%d HEAD\n" "$i" || + return 1 + done >input && + git update-ref --stdin expect <<-EOF && + header: + block_size: 4096 + ref: + - length: 566 + restarts: 13 + log: + - length: 1393 + restarts: 12 + EOF + test-tool dump-reftable -b .git/reftable/*.ref >actual && + test_cmp expect actual + ) +' + +test_expect_success 'restart interval exceeding maximum supported interval' ' + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + test_commit initial && + cat >expect <<-EOF && + fatal: reftable block size cannot exceed 65535 + EOF + test_must_fail git -c reftable.restartInterval=65536 pack-refs 2>err && + test_cmp expect err + ) +' + test_done