From patchwork Thu May 2 06:51:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13651322 Received: from fout4-smtp.messagingengine.com (fout4-smtp.messagingengine.com [103.168.172.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D21C81CAA5 for ; Thu, 2 May 2024 06:51:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.147 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714632694; cv=none; b=MwvwHyoCuIvjan530JADFFx8es88OMQCuZ1GCTrZmA9GUv00HG3TiVt6Rr+17tqKrdpuDmYWOIXS6St/I261f3TszSabzksO3itS4mQ8lzPAOvSbc0N+t354KvfUjvpl66rYA1TxiivsFdQaTUQCeK7+JXBztgAB/m798aZDGsM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714632694; c=relaxed/simple; bh=eRsXI7POe3YDJLkLEcuwC4KqOEa7ya0am3sQDZ5jfcs=; h=Date:From:To:Subject:Message-ID:MIME-Version:Content-Type: Content-Disposition; b=L63p5J+jNLm2cqr2xfVG+2IMsQoAv0AF5gIw4sGt5D4jHx+A7petIAqDi1IL0+b0aIUncaO5YrjHlQbsah4REtadwiPRWA+tWNxPY/u+SibTbpLbl2RcRWAEHPcgD5J1AxQzJ0edgRoV7t1usq/1gv5dIi4Ja/URS/OeVcQP/xM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=W8xIKtGe; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=SOlZv0fp; arc=none smtp.client-ip=103.168.172.147 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="W8xIKtGe"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="SOlZv0fp" Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailfout.nyi.internal (Postfix) with ESMTP id F3F9C13803AC for ; Thu, 2 May 2024 02:51:31 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Thu, 02 May 2024 02:51:31 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc :content-type:content-type:date:date:from:from:in-reply-to :message-id:mime-version:reply-to:subject:subject:to:to; s=fm3; t=1714632691; x=1714719091; bh=0+NhPIM7cRurHU3mgdEnMhvdRzrsqwEO cSnrfYNnu3w=; b=W8xIKtGezojzr2/TT6h9WS3oig3AfsEvRBihGzG4FpMzr8S+ S17Guz6F80EwxsfDSoxqqxhQgxrLlUeaUlOde4zOIl5AUdseWlrNHZsqfIRZgU4k MNVywGQR+P/7N8uTSskIkclq1/pZ27zZtRoQ4gCj15IxxK3xfbuhQzTynq2vIANR MCN7RrvQ/406vMLCpC2cAdEJazfkOtFDpX2ip21LtKQk0BCVwYRYhlipDosavMR9 eFQtxxmmwz5KowMUE3wJJqgUvDc2u6Bdl83IngkUxCsVHENnhpNSIdhaFYIdbi2n R4AHoF0gAX5rTpWeRkZrjNYG7h44KmwanankIg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:message-id :mime-version:reply-to:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t= 1714632691; x=1714719091; bh=0+NhPIM7cRurHU3mgdEnMhvdRzrsqwEOcSn rfYNnu3w=; b=SOlZv0fpe6zLEcvbT7ISInx7WisZ48DrKcWshyV1JSe7e+SgtiQ Lj1fwP6Yvzp9g0I9svV4pWlW31qJd6QUPBK8KwWhVqeysYzXpAQ2q377Ay39VEl9 QqYvizAbfbaQs82ae5Vissth2KwrFHgtPzxKTyEh2MLbnZaC1guNqL+zCcGw1CTK +Siq9aukGUgdnhFkIAjnvmTBWVA5U5Vu6dm6jwLfsOe/UnnkbLi+sONL5LwNe8Lx IY29K3lOZWxaMZXmERVVFTwQBr58cLYon6DD0BJvZEB1YsCpCuxVWYOqbG8kU4/t T4VZlzics7i0g+90DItawwDY+olzG9KWjEg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvddujedguddutdcutefuodetggdotefrod ftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfgh necuuegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfggtggusehgtderre dttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshesphhk shdrihhmqeenucggtffrrghtthgvrhhnpeejieefvdeuleffgfejudffvdeghfeigfejgf dvvdefudevffefveffhffgkeeiffenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgr mhepmhgrihhlfhhrohhmpehpshesphhkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Thu, 2 May 2024 02:51:31 -0400 (EDT) Received: by localhost (OpenSMTPD) with ESMTPSA id 4ccabf0f (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Thu, 2 May 2024 06:51:04 +0000 (UTC) Date: Thu, 2 May 2024 08:51:27 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Subject: [PATCH 00/11] reftable: expose write options as config Message-ID: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline Hi, the reftable format has some flexibility with regards to how exactly it writes the respective tables: - The block size allows you to control how large each block is supposed to be. The bigger the block, the more records you can fit into it. - Restart intervals control how often a restart point is written that breaks prefix compression. The lower the interval, the less disk space savings you get. - Object indices can be enabled or disabled. These are optional and Git doesn't use them right now, so disabling them may be a sensible thing to do if you want to save some disk space. - The geometric factor controls when we compact tables during auto compaction. This patch series exposes all of these via a new set of configs so that they can be tweaked by the user as-needed. It's not expected that those are really of much importance for the "normal" user -- the defaults should be good enough. But for edge cases (huge repos with millions of refs) and for hosting providers these knobs can be helpful. This patch series applies on top of d4cc1ec35f (Start the 2.46 cycle, 2024-04-30). Patrick Patrick Steinhardt (11): reftable: consistently refer to `reftable_write_options` as `opts` reftable: consistently pass write opts as value reftable/writer: drop static variable used to initialize strbuf reftable/writer: improve error when passed an invalid block size reftable/dump: support dumping a table's block structure refs/reftable: allow configuring block size reftable: use `uint16_t` to track restart interval refs/reftable: allow configuring restart interval refs/reftable: allow disabling writing the object index reftable: make the compaction factor configurable refs/reftable: allow configuring geometric factor Documentation/config.txt | 2 + Documentation/config/reftable.txt | 49 +++++ refs/reftable-backend.c | 46 ++++- reftable/block.h | 2 +- reftable/dump.c | 12 +- reftable/merged_test.c | 6 +- reftable/reader.c | 63 +++++++ reftable/readwrite_test.c | 26 +-- reftable/refname_test.c | 2 +- reftable/reftable-reader.h | 2 + reftable/reftable-stack.h | 2 +- reftable/reftable-writer.h | 10 +- reftable/stack.c | 56 +++--- reftable/stack.h | 5 +- reftable/stack_test.c | 118 ++++++------ reftable/writer.c | 20 +-- t/t0613-reftable-write-options.sh | 286 ++++++++++++++++++++++++++++++ 17 files changed, 577 insertions(+), 130 deletions(-) create mode 100644 Documentation/config/reftable.txt create mode 100755 t/t0613-reftable-write-options.sh base-commit: d4cc1ec35f3bcce816b69986ca41943f6ce21377