From patchwork Mon Nov 18 15:33:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13878783 Received: from fhigh-a6-smtp.messagingengine.com (fhigh-a6-smtp.messagingengine.com [103.168.172.157]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4C7381AA1E7 for ; Mon, 18 Nov 2024 15:34:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.157 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731944047; cv=none; b=AA99Tng4MosF3N4bQdRvsaDTHp16FYC1/TpSprsMi/QZnEbwzrTjAAqwlh/9D1QxWXqKuH9J7lMGgr/tLa3eBOz3V81uNHWHFOx/mP59Sq/vWEddyrF3PqUkqB3IXCmX2ulCTjE6NNUz4Q8bL7re9AZLM3+ixGplu36IhMqR0Xw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731944047; c=relaxed/simple; bh=rFIgtVtM1ybnTuv8XbacorfnynVstSBD4mYi543Okxo=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=TuAZcl+/h1XzqzkV9dCLIzQ7kKC6yb06D4t3/g6abWcGwpLHQLrQeltGSq8yyvr/rQryKDKLiZsQFlTzOIR6K2unYKAJco+iXcdRKhxKiguHbChTDW+UrpUDK0rAGe/7VxMVRfIMc+jhE/jgfAjQgWUznEAvWOtVu2oiEicPhyQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=uA07gT25; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=jC1a42sZ; arc=none smtp.client-ip=103.168.172.157 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="uA07gT25"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="jC1a42sZ" Received: from phl-compute-06.internal (phl-compute-06.phl.internal [10.202.2.46]) by mailfhigh.phl.internal (Postfix) with ESMTP id 34D001140230; Mon, 18 Nov 2024 10:34:04 -0500 (EST) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-06.internal (MEProxy); Mon, 18 Nov 2024 10:34:04 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1731944044; x=1732030444; bh=2rBgOxFgHt Y3wZiTsXmtxtg2LgI740Pve9nxzhIh90s=; b=uA07gT258dnT8wIao3eeuzLdpL Ds2OiCJAcJqKl1/3XRfe4jcYScHiPqtgdYPlvOOIj9eEHVcZ0BDT+U/Ok58uR++T CBxbfut1wKi7LDgVVXkyolv5hbuorxMRPrHkwsL5dZReUTKcCc+2IecOy7kdMQY/ OHxloEsRKGNNvBLJnDrd32wS2dkbHXrCHoLo4VtlSgN9v4BV8ZqNYKlOzPhmwtNk Nb8b1Bq0j5QAeiJL1YQNjjC5yxrA6R1vc1rEc+HMTL1T/HXVQmQy2q+65stedy4c KGLybvzxq+CVzPJG5VeriI4p88hA7rIDL0BirOCrKtaRrRgjRjyfj0s/K7Dg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t= 1731944044; x=1732030444; bh=2rBgOxFgHtY3wZiTsXmtxtg2LgI740Pve9n xzhIh90s=; b=jC1a42sZ1fdjwN9YpHDlCuePS27rV91ghwxnM5BAU+yyTJTkvUQ GmtjkyLV9yf2BQUglRsx2+IszRnGfUYTHYylnIfW7mQNXTw9lA70w5CjEciE0qaE zY1SvecOYGUxqwwO/0XJgLRksscTvDwdOxqt3RrwdjHhnh/3CjUPyFSRf4p5TJLm Qgw01GgvNbpZSWbrsUokLEz/NG6Px18OPInC4y2alqzR/a3AW6zuIb91fHJyErU5 yer/RTClUD5UCPOk4oYBufRicRpoBhsxHJtYWeTMubD8eIFBXgVws9pwmGBpZKym l+fhm4oUqGmco0tjSXK6XrEApABMAspfRVA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefuddrfedtgdejiecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdpuffr tefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnth hsucdlqddutddtmdenucfjughrpeffhffvvefukfhfgggtuggjsehttdertddttddvnecu hfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqe enucggtffrrghtthgvrhhnpeevkeekfffhiedtleduiefgjedttedvledvudehgfeugedu gffhueekhfejvdektdenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrih hlfhhrohhmpehpshesphhkshdrihhmpdhnsggprhgtphhtthhopeeipdhmohguvgepshhm thhpohhuthdprhgtphhtthhopehgihhtshhtvghrsehpohgsohigrdgtohhmpdhrtghpth htohepshhunhhshhhinhgvsehsuhhnshhhihhnvggtohdrtghomhdprhgtphhtthhopegv thhhohhmshhonhesvggufigrrhguthhhohhmshhonhdrtghomhdprhgtphhtthhopehjlh htohgslhgvrhesghhmrghilhdrtghomhdprhgtphhtthhopehgihhtsehvghgvrhdrkhgv rhhnvghlrdhorhhgpdhrtghpthhtohepkhgrrhhthhhikhdrudekkeesghhmrghilhdrtg homh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 18 Nov 2024 10:34:02 -0500 (EST) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 14441072 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 18 Nov 2024 15:33:14 +0000 (UTC) Date: Mon, 18 Nov 2024 16:33:52 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Edward Thomson , Eric Sunshine , Justin Tobler , Junio C Hamano , karthik nayak Subject: [PATCH v3 1/7] reftable/system: move "dir.h" to its only user Message-ID: <2b7d4e2852977659752f98b782784eaa6780bbdb.1731943954.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: We still include "dir.h" in "reftable/system.h" even though it is not used by anything but by a single unit test. Move it over into that unit test so that we don't accidentally use any functionality provided by it in the reftable codebase. Signed-off-by: Patrick Steinhardt --- reftable/system.h | 1 - t/unit-tests/t-reftable-stack.c | 1 + 2 files changed, 1 insertion(+), 1 deletion(-) diff --git a/reftable/system.h b/reftable/system.h index 5ec85833434..8564213475e 100644 --- a/reftable/system.h +++ b/reftable/system.h @@ -15,7 +15,6 @@ license that can be found in the LICENSE file or at #include "lockfile.h" #include "tempfile.h" #include "hash.h" /* hash ID, sizes.*/ -#include "dir.h" /* remove_dir_recursively, for tests.*/ int hash_size(uint32_t id); diff --git a/t/unit-tests/t-reftable-stack.c b/t/unit-tests/t-reftable-stack.c index 72f6747064f..1b4363a58fc 100644 --- a/t/unit-tests/t-reftable-stack.c +++ b/t/unit-tests/t-reftable-stack.c @@ -8,6 +8,7 @@ license that can be found in the LICENSE file or at #include "test-lib.h" #include "lib-reftable.h" +#include "dir.h" #include "reftable/merged.h" #include "reftable/reader.h" #include "reftable/reftable-error.h" From patchwork Mon Nov 18 15:33:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13878784 Received: from fhigh-a6-smtp.messagingengine.com (fhigh-a6-smtp.messagingengine.com [103.168.172.157]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9E21F1BD9DA for ; Mon, 18 Nov 2024 15:34:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.157 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731944048; cv=none; b=HqGfJR9jLVit0p7LJ3s6GtzSnqhmq0nYFhn9EwvsiQeNgLoA6E9J0OOQyxK9EzrwsW27O59aTbH1Ay0Cx9sWwua6h3NbhgRYHsPpCGJGhoqsh3UYLjVR4XDKFvSWwBJh3JhESE+MH3Hxxbhdx1Fp2htQud1PeRQhGnA6nxPCJzk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731944048; c=relaxed/simple; bh=5x52Db0w6J+4reS7/8GyZdAdJN85wnSLrjH2wmTPfhw=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=SNjO9EqKN8gpVorj0cSSFnbshl536LG+3LbPO8e1pIZu5Eyn4azfm38DJZI4nUGcd+TZ8FbyOBQLCMj4LHde9FSemwyN1TfISs9VRrtA1hGOJG7XI2sAyYkGzxkCDFup0iCKaaLmcYS7JFuGuuUm/qhh5YM0vWV98bRJI8D2AVI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=NFMd4OVX; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=FHRc05H0; arc=none smtp.client-ip=103.168.172.157 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="NFMd4OVX"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="FHRc05H0" Received: from phl-compute-11.internal (phl-compute-11.phl.internal [10.202.2.51]) by mailfhigh.phl.internal (Postfix) with ESMTP id E5356114022B; Mon, 18 Nov 2024 10:34:05 -0500 (EST) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-11.internal (MEProxy); Mon, 18 Nov 2024 10:34:05 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1731944045; x=1732030445; bh=uebA21VKiI IkAu9Mnh0HKVKcxQKIuZiLf+ZvJ+rN4zA=; b=NFMd4OVX+SVn11Ivu9Ya4JlbES NjQa3Wi/Tc2B+9tVur3hLTjS1kBdEWGX0cA8VacjYtwuqzvbCVPe1cgVCzBbAVFp 5glh6x/6JdFsVBFOXUiWhJ43WJJCY60+Yqt4Xr93+xeSN8ta1ipfc/XC55u4Ez8y 1NHzgeIOdw4fcy7x7PjLdKl2G8AgjKFF3SMUvoxBvQHR1pvhINvUILJBSavcAMET yIFRRCyU1RK/l/oVn7Tp98fIaXIcQYCLEd/qJDoJUWv21hohuuFC5WfxbOAasMti M/trQTMND/pD6uG6HwH+DrJBiP+L0pwBcmno9Q5g53G7FFavpJdEpQBtrH5Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t= 1731944045; x=1732030445; bh=uebA21VKiIIkAu9Mnh0HKVKcxQKIuZiLf+Z vJ+rN4zA=; b=FHRc05H0SUXlbHySNeYjhnGTUXjpQSMaacgq6AeBccf2l7abbGM IVvI1p/8hJSkrV7sMCCb3jhu2MEQguoRCnyvgD0PgZ2ZD5KJOUFyIVKF02YvjQbW vsIJ2q183xHm5wLHXwH3iFbLqq78V2c19RbFBAsRd5L00QUrNJpaOvCOozEblo8P 9YwvnFUja2ZlSMXL4WXpvAJDf2NzDszDbCjwq5iVd3P6i7ggTTFfhEAivpUAShqH Y3qSPChBLRyWQTzFUaupiL+dUOPzboIZ3lTzZdb2hLYTJ/fuLuOPlPb++sS8+Eh5 yjLC1DQx3dLWutPwE75PTBl44GTfPIPyAGA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefuddrfedtgdejiecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdpuffr tefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnth hsucdlqddutddtmdenucfjughrpeffhffvvefukfhfgggtuggjsehttdertddttddvnecu hfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqe enucggtffrrghtthgvrhhnpeevkeekfffhiedtleduiefgjedttedvledvudehgfeugedu gffhueekhfejvdektdenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrih hlfhhrohhmpehpshesphhkshdrihhmpdhnsggprhgtphhtthhopeeipdhmohguvgepshhm thhpohhuthdprhgtphhtthhopehkrghrthhhihhkrddukeeksehgmhgrihhlrdgtohhmpd hrtghpthhtohepshhunhhshhhinhgvsehsuhhnshhhihhnvggtohdrtghomhdprhgtphht thhopehgihhtsehvghgvrhdrkhgvrhhnvghlrdhorhhgpdhrtghpthhtohepjhhlthhosg hlvghrsehgmhgrihhlrdgtohhmpdhrtghpthhtohepvghthhhomhhsohhnsegvugifrghr ughthhhomhhsohhnrdgtohhmpdhrtghpthhtohepghhithhsthgvrhesphhosghogidrtg homh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 18 Nov 2024 10:34:04 -0500 (EST) Received: by vm-mail (OpenSMTPD) with ESMTPSA id c6515377 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 18 Nov 2024 15:33:16 +0000 (UTC) Date: Mon, 18 Nov 2024 16:33:55 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Edward Thomson , Eric Sunshine , Justin Tobler , Junio C Hamano , karthik nayak Subject: [PATCH v3 2/7] reftable: explicitly handle hash format IDs Message-ID: <38cfe85bf5b82c70848e4b295ba6cae33dcfd667.1731943954.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The hash format IDs are used for two different things across the reftable codebase: - They are used as a 32 bit unsigned integer when reading and writing the header in order to identify the hash function. - They are used internally to identify which hash function is in use. When one only considers the second usecase one might think that one can easily change the representation of those hash IDs. But because those IDs end up in the reftable header and footer on disk it is important that those never change. Create separate constants `REFTABLE_FORMAT_ID_*` and use them in contexts where we read or write reftable headers. This serves multiple purposes: - It allows us to more easily discern cases where we actually use those constants for the on-disk format. - It detangles us from the same constants that are defined in libgit.a, which is another required step to convert the reftable library to become standalone. - It makes the next step easier where we stop using `GIT_*_FORMAT_ID` constants in favor of a custom enum. Signed-off-by: Patrick Steinhardt --- reftable/basics.h | 8 ++++++++ reftable/reader.c | 10 ++++++---- reftable/writer.c | 16 +++++++++++++++- 3 files changed, 29 insertions(+), 5 deletions(-) diff --git a/reftable/basics.h b/reftable/basics.h index 7aa46d7c30d..bcab0b529b0 100644 --- a/reftable/basics.h +++ b/reftable/basics.h @@ -150,4 +150,12 @@ int common_prefix_size(struct reftable_buf *a, struct reftable_buf *b); int hash_size(uint32_t id); +/* + * Format IDs that identify the hash function used by a reftable. Note that + * these constants end up on disk and thus mustn't change. The format IDs are + * "sha1" and "s256" in big endian, respectively. + */ +#define REFTABLE_FORMAT_ID_SHA1 ((uint32_t) 0x73686131) +#define REFTABLE_FORMAT_ID_SHA256 ((uint32_t) 0x73323536) + #endif diff --git a/reftable/reader.c b/reftable/reader.c index 90dc950b577..64eb6938efe 100644 --- a/reftable/reader.c +++ b/reftable/reader.c @@ -109,16 +109,18 @@ static int parse_footer(struct reftable_reader *r, uint8_t *footer, if (r->version == 1) { r->hash_id = GIT_SHA1_FORMAT_ID; } else { - r->hash_id = get_be32(f); - switch (r->hash_id) { - case GIT_SHA1_FORMAT_ID: + switch (get_be32(f)) { + case REFTABLE_FORMAT_ID_SHA1: + r->hash_id = GIT_SHA1_FORMAT_ID; break; - case GIT_SHA256_FORMAT_ID: + case REFTABLE_FORMAT_ID_SHA256: + r->hash_id = GIT_SHA256_FORMAT_ID; break; default: err = REFTABLE_FORMAT_ERROR; goto done; } + f += 4; } diff --git a/reftable/writer.c b/reftable/writer.c index fd136794d5a..9aa45de6340 100644 --- a/reftable/writer.c +++ b/reftable/writer.c @@ -103,8 +103,22 @@ static int writer_write_header(struct reftable_writer *w, uint8_t *dest) put_be64(dest + 8, w->min_update_index); put_be64(dest + 16, w->max_update_index); if (writer_version(w) == 2) { - put_be32(dest + 24, w->opts.hash_id); + uint32_t hash_id; + + switch (w->opts.hash_id) { + case GIT_SHA1_FORMAT_ID: + hash_id = REFTABLE_FORMAT_ID_SHA1; + break; + case GIT_SHA256_FORMAT_ID: + hash_id = REFTABLE_FORMAT_ID_SHA256; + break; + default: + return -1; + } + + put_be32(dest + 24, hash_id); } + return header_size(writer_version(w)); } From patchwork Mon Nov 18 15:33:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13878786 Received: from fhigh-a6-smtp.messagingengine.com (fhigh-a6-smtp.messagingengine.com [103.168.172.157]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9FE371BDAA0 for ; Mon, 18 Nov 2024 15:34:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.157 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731944052; cv=none; b=kkWnP9MTKVST58RuKw5AhZ0naEJS2WHlirm++ew2nS3fh0pF8PznWeNnvOuESgEMueMZ7KTZSpEmIo12fijo7BbUhcvAIvV0wXSLLarrREMP5gemibFm6E3oKEj1p/Wo1PsPAQxIAZs9fyQTVOux+0GIurarpbz1c1pJeD2PxMM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731944052; c=relaxed/simple; bh=plFD16D5GkVkiZ1l+YdYLHVVb/4r8RqjS8bfa/q+6+A=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=rB+TicOdtFa/+uD/PXGwuq9BOcbG9uHHHJodF5FKLL348kdtyPOeQ1Uw9ilr50NmtqoGrLUPGch8Ki3cNuTshrWYeEyzCaDiUQvp+osz83XBl0GN/8YHl2pHdbBwbKi8DYsewEGBo0pyPfE8/XlqqORvYb8l1trLDWsg4H7tW/Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=Q3Mu/cJi; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=cIZI4YxM; arc=none smtp.client-ip=103.168.172.157 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="Q3Mu/cJi"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="cIZI4YxM" Received: from phl-compute-05.internal (phl-compute-05.phl.internal [10.202.2.45]) by mailfhigh.phl.internal (Postfix) with ESMTP id C2096114022B; Mon, 18 Nov 2024 10:34:08 -0500 (EST) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-05.internal (MEProxy); Mon, 18 Nov 2024 10:34:08 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1731944048; x=1732030448; bh=ui8jAB6WCM Q7sddNw8VrAi4jDVkKjrdCCtA352xzEWo=; b=Q3Mu/cJip3BH1z6IBGO+JnipKh 1l8lnil/xp2G8Dg+04kaTUC5h8AMPLd7erXfuj2JLaoJiruTf95urVMkmlxQ1RJe x3xPsv+FH8o3+5OIeH1ony+Z4CjqSgNdoSZyD/rBKaQ73Dsm5QqsUtk4gMrUu3LT QK5NZ/Xmhe3Z8Vy7aw7tNl8tglqykTUoRT4u9NJId48XsF70fWLI1hqi6g3ptXFo 41Cg7dbjVSryXAG56bytdW7NZSoQhT/xskQLFbyXEOuB8YOrQHAVsCs64fzmR61o YyxolXQv8Zjhy2DyfAw1wT75T0JBASHLT3k2fZ+mqPopwTHZQlzObB030YjQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t= 1731944048; x=1732030448; bh=ui8jAB6WCMQ7sddNw8VrAi4jDVkKjrdCCtA 352xzEWo=; b=cIZI4YxMYRJdI8evOZgQE9TWN/jdx5udRGWPpGo0MC3SKAll2Zj dSUcQ6TaNGvAfc0zG5UZRmPBvBL1aByPcZpet37CnrtpBe5XxZaa+XZdplB0SJpi zK8rsYqiKJr36S/8VinFe6a9Sv/CCwKZ6M+6VpjR/H7TMRASdiZxnRH04w1RGulC jKC/Xplbm3NC4LT24Q3suOsg5mSYE+vGaETtK03KvYG1Vc3x/8UZ9iYCs1Bz2BEu f8zSLGPhZ9RfEDUGozAA3Ctm0TOyJsb+q6SqO+mubM5qjdF4GJPLOtpf5zyzBYvN COd0tfiYSj9Gi3j5P2vOgkGgpl681z9jpFQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefuddrfedtgdejiecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdpuffr tefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnth hsucdlqddutddtmdenucfjughrpeffhffvvefukfhfgggtuggjsehttdertddttddvnecu hfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqe enucggtffrrghtthgvrhhnpeegtdfgfefhtdfftedtveejieetkedtffekfeehueekteef leethfffvdeijefgkeenucffohhmrghinhepuhhpuggrthgvrdhnvgifpdhvrghlvddrth grrhhgvghtpdhuphgurghtvgdrvghmrghilhenucevlhhushhtvghrufhiiigvpedtnecu rfgrrhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhmpdhnsggprhgtphhtthhope eipdhmohguvgepshhmthhpohhuthdprhgtphhtthhopehgihhtshhtvghrsehpohgsohig rdgtohhmpdhrtghpthhtohepghhithesvhhgvghrrdhkvghrnhgvlhdrohhrghdprhgtph htthhopehkrghrthhhihhkrddukeeksehgmhgrihhlrdgtohhmpdhrtghpthhtohepvght hhhomhhsohhnsegvugifrghrughthhhomhhsohhnrdgtohhmpdhrtghpthhtohepshhunh hshhhinhgvsehsuhhnshhhihhnvggtohdrtghomhdprhgtphhtthhopehjlhhtohgslhgv rhesghhmrghilhdrtghomh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 18 Nov 2024 10:34:07 -0500 (EST) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 65acec80 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 18 Nov 2024 15:33:19 +0000 (UTC) Date: Mon, 18 Nov 2024 16:33:57 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Edward Thomson , Eric Sunshine , Justin Tobler , Junio C Hamano , karthik nayak Subject: [PATCH v3 3/7] reftable/system: stop depending on "hash.h" Message-ID: <745c1a070ddd7180f0ee89c6a1d057a256894599.1731943954.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: We include "hash.h" in "reftable/system.h" such that we can use hash format IDs as well as the raw size of SHA1 and SHA256. As we are in the process of converting the reftable library to become standalone we of course cannot rely on those constants anymore. Introduce a new `enum reftable_hash` to replace internal uses of the hash format IDs and new constants that replace internal uses of the hash size. Adapt the reftable backend to set up the correct hash function. Signed-off-by: Patrick Steinhardt --- refs/reftable-backend.c | 12 +++++- reftable/basics.c | 13 ++++--- reftable/basics.h | 2 +- reftable/merged.c | 4 +- reftable/merged.h | 3 +- reftable/reader.c | 8 ++-- reftable/reader.h | 4 +- reftable/reftable-basics.h | 13 +++++++ reftable/reftable-merged.h | 4 +- reftable/reftable-reader.h | 2 +- reftable/reftable-record.h | 12 +++--- reftable/reftable-writer.h | 2 +- reftable/stack.c | 4 +- reftable/system.h | 3 -- reftable/writer.c | 8 ++-- t/helper/test-reftable.c | 4 +- t/unit-tests/lib-reftable.c | 4 +- t/unit-tests/lib-reftable.h | 2 +- t/unit-tests/t-reftable-block.c | 40 +++++++++---------- t/unit-tests/t-reftable-merged.c | 26 ++++++------- t/unit-tests/t-reftable-pq.c | 2 +- t/unit-tests/t-reftable-reader.c | 4 +- t/unit-tests/t-reftable-readwrite.c | 40 +++++++++---------- t/unit-tests/t-reftable-record.c | 59 +++++++++++++++-------------- t/unit-tests/t-reftable-stack.c | 34 ++++++++--------- 25 files changed, 166 insertions(+), 143 deletions(-) diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c index 3c6107c7ce5..7d86d920970 100644 --- a/refs/reftable-backend.c +++ b/refs/reftable-backend.c @@ -15,6 +15,7 @@ #include "../object.h" #include "../path.h" #include "../refs.h" +#include "../reftable/reftable-basics.h" #include "../reftable/reftable-stack.h" #include "../reftable/reftable-record.h" #include "../reftable/reftable-error.h" @@ -289,7 +290,16 @@ static struct ref_store *reftable_be_init(struct repository *repo, refs->store_flags = store_flags; refs->log_all_ref_updates = repo_settings_get_log_all_ref_updates(repo); - refs->write_options.hash_id = repo->hash_algo->format_id; + switch (repo->hash_algo->format_id) { + case GIT_SHA1_FORMAT_ID: + refs->write_options.hash_id = REFTABLE_HASH_SHA1; + break; + case GIT_SHA256_FORMAT_ID: + refs->write_options.hash_id = REFTABLE_HASH_SHA256; + break; + default: + BUG("unknown hash algorithm %d", repo->hash_algo->format_id); + } refs->write_options.default_permissions = calc_shared_perm(0666 & ~mask); refs->write_options.disable_auto_compact = !git_env_bool("GIT_TEST_REFTABLE_AUTOCOMPACTION", 1); diff --git a/reftable/basics.c b/reftable/basics.c index bc4fcc91446..7d84a5d62de 100644 --- a/reftable/basics.c +++ b/reftable/basics.c @@ -271,14 +271,15 @@ int common_prefix_size(struct reftable_buf *a, struct reftable_buf *b) return p; } -int hash_size(uint32_t id) +int hash_size(enum reftable_hash id) { + if (!id) + return REFTABLE_HASH_SIZE_SHA1; switch (id) { - case 0: - case GIT_SHA1_FORMAT_ID: - return GIT_SHA1_RAWSZ; - case GIT_SHA256_FORMAT_ID: - return GIT_SHA256_RAWSZ; + case REFTABLE_HASH_SHA1: + return REFTABLE_HASH_SIZE_SHA1; + case REFTABLE_HASH_SHA256: + return REFTABLE_HASH_SIZE_SHA256; } abort(); } diff --git a/reftable/basics.h b/reftable/basics.h index bcab0b529b0..36beda2c25a 100644 --- a/reftable/basics.h +++ b/reftable/basics.h @@ -148,7 +148,7 @@ char *reftable_strdup(const char *str); /* Find the longest shared prefix size of `a` and `b` */ int common_prefix_size(struct reftable_buf *a, struct reftable_buf *b); -int hash_size(uint32_t id); +int hash_size(enum reftable_hash id); /* * Format IDs that identify the hash function used by a reftable. Note that diff --git a/reftable/merged.c b/reftable/merged.c index 514d6facf45..5b93e20f429 100644 --- a/reftable/merged.c +++ b/reftable/merged.c @@ -181,7 +181,7 @@ static void iterator_from_merged_iter(struct reftable_iterator *it, int reftable_merged_table_new(struct reftable_merged_table **dest, struct reftable_reader **readers, size_t n, - uint32_t hash_id) + enum reftable_hash hash_id) { struct reftable_merged_table *m = NULL; uint64_t last_max = 0; @@ -293,7 +293,7 @@ int reftable_merged_table_init_log_iterator(struct reftable_merged_table *mt, return merged_table_init_iter(mt, it, BLOCK_TYPE_LOG); } -uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *mt) +enum reftable_hash reftable_merged_table_hash_id(struct reftable_merged_table *mt) { return mt->hash_id; } diff --git a/reftable/merged.h b/reftable/merged.h index 89bd0c4b35b..0b7d939e92b 100644 --- a/reftable/merged.h +++ b/reftable/merged.h @@ -10,11 +10,12 @@ license that can be found in the LICENSE file or at #define MERGED_H #include "system.h" +#include "reftable-basics.h" struct reftable_merged_table { struct reftable_reader **readers; size_t readers_len; - uint32_t hash_id; + enum reftable_hash hash_id; /* If unset, produce deletions. This is useful for compaction. For the * full stack, deletions should be produced. */ diff --git a/reftable/reader.c b/reftable/reader.c index 64eb6938efe..ea82955c9bc 100644 --- a/reftable/reader.c +++ b/reftable/reader.c @@ -67,7 +67,7 @@ static int reader_get_block(struct reftable_reader *r, return block_source_read_block(&r->source, dest, off, sz); } -uint32_t reftable_reader_hash_id(struct reftable_reader *r) +enum reftable_hash reftable_reader_hash_id(struct reftable_reader *r) { return r->hash_id; } @@ -107,14 +107,14 @@ static int parse_footer(struct reftable_reader *r, uint8_t *footer, f += 8; if (r->version == 1) { - r->hash_id = GIT_SHA1_FORMAT_ID; + r->hash_id = REFTABLE_HASH_SHA1; } else { switch (get_be32(f)) { case REFTABLE_FORMAT_ID_SHA1: - r->hash_id = GIT_SHA1_FORMAT_ID; + r->hash_id = REFTABLE_HASH_SHA1; break; case REFTABLE_FORMAT_ID_SHA256: - r->hash_id = GIT_SHA256_FORMAT_ID; + r->hash_id = REFTABLE_HASH_SHA256; break; default: err = REFTABLE_FORMAT_ERROR; diff --git a/reftable/reader.h b/reftable/reader.h index 010fbfe8511..d2b48a48499 100644 --- a/reftable/reader.h +++ b/reftable/reader.h @@ -37,8 +37,8 @@ struct reftable_reader { /* Size of the file, excluding the footer. */ uint64_t size; - /* 'sha1' for SHA1, 's256' for SHA-256 */ - uint32_t hash_id; + /* The hash function used for ref records. */ + enum reftable_hash hash_id; uint32_t block_size; uint64_t min_update_index; diff --git a/reftable/reftable-basics.h b/reftable/reftable-basics.h index 6e8e636b716..e0397ed5836 100644 --- a/reftable/reftable-basics.h +++ b/reftable/reftable-basics.h @@ -11,6 +11,19 @@ #include +/* + * Hash functions understood by the reftable library. Note that the values are + * arbitrary and somewhat random such that we can easily detect cases where the + * hash hasn't been properly set up. + */ +enum reftable_hash { + REFTABLE_HASH_SHA1 = 89, + REFTABLE_HASH_SHA256 = 247, +}; +#define REFTABLE_HASH_SIZE_SHA1 20 +#define REFTABLE_HASH_SIZE_SHA256 32 +#define REFTABLE_HASH_SIZE_MAX REFTABLE_HASH_SIZE_SHA256 + /* Overrides the functions to use for memory management. */ void reftable_set_alloc(void *(*malloc)(size_t), void *(*realloc)(void *, size_t), void (*free)(void *)); diff --git a/reftable/reftable-merged.h b/reftable/reftable-merged.h index a970d5dd89a..f2d01c3ef82 100644 --- a/reftable/reftable-merged.h +++ b/reftable/reftable-merged.h @@ -34,7 +34,7 @@ struct reftable_reader; */ int reftable_merged_table_new(struct reftable_merged_table **dest, struct reftable_reader **readers, size_t n, - uint32_t hash_id); + enum reftable_hash hash_id); /* Initialize a merged table iterator for reading refs. */ int reftable_merged_table_init_ref_iterator(struct reftable_merged_table *mt, @@ -56,6 +56,6 @@ reftable_merged_table_min_update_index(struct reftable_merged_table *mt); void reftable_merged_table_free(struct reftable_merged_table *m); /* return the hash ID of the merged table. */ -uint32_t reftable_merged_table_hash_id(struct reftable_merged_table *m); +enum reftable_hash reftable_merged_table_hash_id(struct reftable_merged_table *m); #endif diff --git a/reftable/reftable-reader.h b/reftable/reftable-reader.h index 6a2d0b693f5..0085fbb9032 100644 --- a/reftable/reftable-reader.h +++ b/reftable/reftable-reader.h @@ -54,7 +54,7 @@ int reftable_reader_init_log_iterator(struct reftable_reader *r, struct reftable_iterator *it); /* returns the hash ID used in this table. */ -uint32_t reftable_reader_hash_id(struct reftable_reader *r); +enum reftable_hash reftable_reader_hash_id(struct reftable_reader *r); /* return an iterator for the refs pointing to `oid`. */ int reftable_reader_refs_for(struct reftable_reader *r, diff --git a/reftable/reftable-record.h b/reftable/reftable-record.h index 2d42463c581..ddd48eb5798 100644 --- a/reftable/reftable-record.h +++ b/reftable/reftable-record.h @@ -9,7 +9,7 @@ license that can be found in the LICENSE file or at #ifndef REFTABLE_RECORD_H #define REFTABLE_RECORD_H -#include "hash.h" +#include "reftable-basics.h" #include /* @@ -40,10 +40,10 @@ struct reftable_ref_record { #define REFTABLE_NR_REF_VALUETYPES 4 } value_type; union { - unsigned char val1[GIT_MAX_RAWSZ]; + unsigned char val1[REFTABLE_HASH_SIZE_MAX]; struct { - unsigned char value[GIT_MAX_RAWSZ]; /* first hash */ - unsigned char target_value[GIT_MAX_RAWSZ]; /* second hash */ + unsigned char value[REFTABLE_HASH_SIZE_MAX]; /* first hash */ + unsigned char target_value[REFTABLE_HASH_SIZE_MAX]; /* second hash */ } val2; char *symref; /* referent, malloced 0-terminated string */ } value; @@ -85,8 +85,8 @@ struct reftable_log_record { union { struct { - unsigned char new_hash[GIT_MAX_RAWSZ]; - unsigned char old_hash[GIT_MAX_RAWSZ]; + unsigned char new_hash[REFTABLE_HASH_SIZE_MAX]; + unsigned char old_hash[REFTABLE_HASH_SIZE_MAX]; char *name; char *email; uint64_t time; diff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h index e4fc9537883..211860d08a4 100644 --- a/reftable/reftable-writer.h +++ b/reftable/reftable-writer.h @@ -33,7 +33,7 @@ struct reftable_write_options { /* 4-byte identifier ("sha1", "s256") of the hash. * Defaults to SHA1 if unset */ - uint32_t hash_id; + enum reftable_hash hash_id; /* Default mode for creating files. If unset, use 0666 (+umask) */ unsigned int default_permissions; diff --git a/reftable/stack.c b/reftable/stack.c index c33979536ef..9ae716ff375 100644 --- a/reftable/stack.c +++ b/reftable/stack.c @@ -73,7 +73,7 @@ int reftable_new_stack(struct reftable_stack **dest, const char *dir, if (_opts) opts = *_opts; if (opts.hash_id == 0) - opts.hash_id = GIT_SHA1_FORMAT_ID; + opts.hash_id = REFTABLE_HASH_SHA1; *dest = NULL; @@ -1603,7 +1603,7 @@ struct segment suggest_compaction_segment(uint64_t *sizes, size_t n, static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st) { - int version = (st->opts.hash_id == GIT_SHA1_FORMAT_ID) ? 1 : 2; + int version = (st->opts.hash_id == REFTABLE_HASH_SHA1) ? 1 : 2; int overhead = header_size(version) - 1; uint64_t *sizes; diff --git a/reftable/system.h b/reftable/system.h index 8564213475e..38d3534620e 100644 --- a/reftable/system.h +++ b/reftable/system.h @@ -14,8 +14,5 @@ license that can be found in the LICENSE file or at #include "git-compat-util.h" #include "lockfile.h" #include "tempfile.h" -#include "hash.h" /* hash ID, sizes.*/ - -int hash_size(uint32_t id); #endif diff --git a/reftable/writer.c b/reftable/writer.c index 9aa45de6340..ea2f831fc58 100644 --- a/reftable/writer.c +++ b/reftable/writer.c @@ -79,7 +79,7 @@ static void options_set_defaults(struct reftable_write_options *opts) } if (opts->hash_id == 0) { - opts->hash_id = GIT_SHA1_FORMAT_ID; + opts->hash_id = REFTABLE_HASH_SHA1; } if (opts->block_size == 0) { opts->block_size = DEFAULT_BLOCK_SIZE; @@ -88,7 +88,7 @@ static void options_set_defaults(struct reftable_write_options *opts) static int writer_version(struct reftable_writer *w) { - return (w->opts.hash_id == 0 || w->opts.hash_id == GIT_SHA1_FORMAT_ID) ? + return (w->opts.hash_id == 0 || w->opts.hash_id == REFTABLE_HASH_SHA1) ? 1 : 2; } @@ -106,10 +106,10 @@ static int writer_write_header(struct reftable_writer *w, uint8_t *dest) uint32_t hash_id; switch (w->opts.hash_id) { - case GIT_SHA1_FORMAT_ID: + case REFTABLE_HASH_SHA1: hash_id = REFTABLE_FORMAT_ID_SHA1; break; - case GIT_SHA256_FORMAT_ID: + case REFTABLE_HASH_SHA256: hash_id = REFTABLE_FORMAT_ID_SHA256; break; default: diff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c index 5c8849d115b..3c72ed985b3 100644 --- a/t/helper/test-reftable.c +++ b/t/helper/test-reftable.c @@ -156,7 +156,7 @@ int cmd__dump_reftable(int argc, const char **argv) int opt_dump_blocks = 0; int opt_dump_table = 0; int opt_dump_stack = 0; - uint32_t opt_hash_id = GIT_SHA1_FORMAT_ID; + uint32_t opt_hash_id = REFTABLE_HASH_SHA1; const char *arg = NULL, *argv0 = argv[0]; for (; argc > 1; argv++, argc--) @@ -167,7 +167,7 @@ int cmd__dump_reftable(int argc, const char **argv) else if (!strcmp("-t", argv[1])) opt_dump_table = 1; else if (!strcmp("-6", argv[1])) - opt_hash_id = GIT_SHA256_FORMAT_ID; + opt_hash_id = REFTABLE_HASH_SHA256; else if (!strcmp("-s", argv[1])) opt_dump_stack = 1; else if (!strcmp("-?", argv[1]) || !strcmp("-h", argv[1])) { diff --git a/t/unit-tests/lib-reftable.c b/t/unit-tests/lib-reftable.c index 2ddf480588d..c1631f45275 100644 --- a/t/unit-tests/lib-reftable.c +++ b/t/unit-tests/lib-reftable.c @@ -3,7 +3,7 @@ #include "reftable/constants.h" #include "reftable/writer.h" -void t_reftable_set_hash(uint8_t *p, int i, uint32_t id) +void t_reftable_set_hash(uint8_t *p, int i, enum reftable_hash id) { memset(p, (uint8_t)i, hash_size(id)); } @@ -82,7 +82,7 @@ void t_reftable_write_to_buf(struct reftable_buf *buf, size_t off = i * (opts.block_size ? opts.block_size : DEFAULT_BLOCK_SIZE); if (!off) - off = header_size(opts.hash_id == GIT_SHA256_FORMAT_ID ? 2 : 1); + off = header_size(opts.hash_id == REFTABLE_HASH_SHA256 ? 2 : 1); check_char(buf->buf[off], ==, 'r'); } diff --git a/t/unit-tests/lib-reftable.h b/t/unit-tests/lib-reftable.h index d4950fed3da..e4c360fa7ee 100644 --- a/t/unit-tests/lib-reftable.h +++ b/t/unit-tests/lib-reftable.h @@ -6,7 +6,7 @@ struct reftable_buf; -void t_reftable_set_hash(uint8_t *p, int i, uint32_t id); +void t_reftable_set_hash(uint8_t *p, int i, enum reftable_hash id); struct reftable_writer *t_reftable_strbuf_writer(struct reftable_buf *buf, struct reftable_write_options *opts); diff --git a/t/unit-tests/t-reftable-block.c b/t/unit-tests/t-reftable-block.c index f9af907117b..13e10807dae 100644 --- a/t/unit-tests/t-reftable-block.c +++ b/t/unit-tests/t-reftable-block.c @@ -36,7 +36,7 @@ static void t_ref_block_read_write(void) block.len = block_size; block_source_from_buf(&block.source ,&buf); ret = block_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size, - header_off, hash_size(GIT_SHA1_FORMAT_ID)); + header_off, hash_size(REFTABLE_HASH_SHA1)); check(!ret); rec.u.ref.refname = (char *) ""; @@ -47,7 +47,7 @@ static void t_ref_block_read_write(void) for (i = 0; i < N; i++) { rec.u.ref.refname = xstrfmt("branch%02"PRIuMAX, (uintmax_t)i); rec.u.ref.value_type = REFTABLE_REF_VAL1; - memset(rec.u.ref.value.val1, i, GIT_SHA1_RAWSZ); + memset(rec.u.ref.value.val1, i, REFTABLE_HASH_SIZE_SHA1); recs[i] = rec; ret = block_writer_add(&bw, &rec); @@ -61,7 +61,7 @@ static void t_ref_block_read_write(void) block_writer_release(&bw); - block_reader_init(&br, &block, header_off, block_size, GIT_SHA1_RAWSZ); + block_reader_init(&br, &block, header_off, block_size, REFTABLE_HASH_SIZE_SHA1); block_iter_seek_start(&it, &br); @@ -72,7 +72,7 @@ static void t_ref_block_read_write(void) check_int(i, ==, N); break; } - check(reftable_record_equal(&recs[i], &rec, GIT_SHA1_RAWSZ)); + check(reftable_record_equal(&recs[i], &rec, REFTABLE_HASH_SIZE_SHA1)); } for (i = 0; i < N; i++) { @@ -85,7 +85,7 @@ static void t_ref_block_read_write(void) ret = block_iter_next(&it, &rec); check_int(ret, ==, 0); - check(reftable_record_equal(&recs[i], &rec, GIT_SHA1_RAWSZ)); + check(reftable_record_equal(&recs[i], &rec, REFTABLE_HASH_SIZE_SHA1)); want.len--; ret = block_iter_seek_key(&it, &br, &want); @@ -93,7 +93,7 @@ static void t_ref_block_read_write(void) ret = block_iter_next(&it, &rec); check_int(ret, ==, 0); - check(reftable_record_equal(&recs[10 * (i / 10)], &rec, GIT_SHA1_RAWSZ)); + check(reftable_record_equal(&recs[10 * (i / 10)], &rec, REFTABLE_HASH_SIZE_SHA1)); } block_reader_release(&br); @@ -130,7 +130,7 @@ static void t_log_block_read_write(void) block.len = block_size; block_source_from_buf(&block.source ,&buf); ret = block_writer_init(&bw, BLOCK_TYPE_LOG, block.data, block_size, - header_off, hash_size(GIT_SHA1_FORMAT_ID)); + header_off, hash_size(REFTABLE_HASH_SHA1)); check(!ret); for (i = 0; i < N; i++) { @@ -150,7 +150,7 @@ static void t_log_block_read_write(void) block_writer_release(&bw); - block_reader_init(&br, &block, header_off, block_size, GIT_SHA1_RAWSZ); + block_reader_init(&br, &block, header_off, block_size, REFTABLE_HASH_SIZE_SHA1); block_iter_seek_start(&it, &br); @@ -161,7 +161,7 @@ static void t_log_block_read_write(void) check_int(i, ==, N); break; } - check(reftable_record_equal(&recs[i], &rec, GIT_SHA1_RAWSZ)); + check(reftable_record_equal(&recs[i], &rec, REFTABLE_HASH_SIZE_SHA1)); } for (i = 0; i < N; i++) { @@ -175,7 +175,7 @@ static void t_log_block_read_write(void) ret = block_iter_next(&it, &rec); check_int(ret, ==, 0); - check(reftable_record_equal(&recs[i], &rec, GIT_SHA1_RAWSZ)); + check(reftable_record_equal(&recs[i], &rec, REFTABLE_HASH_SIZE_SHA1)); want.len--; ret = block_iter_seek_key(&it, &br, &want); @@ -183,7 +183,7 @@ static void t_log_block_read_write(void) ret = block_iter_next(&it, &rec); check_int(ret, ==, 0); - check(reftable_record_equal(&recs[10 * (i / 10)], &rec, GIT_SHA1_RAWSZ)); + check(reftable_record_equal(&recs[10 * (i / 10)], &rec, REFTABLE_HASH_SIZE_SHA1)); } block_reader_release(&br); @@ -220,7 +220,7 @@ static void t_obj_block_read_write(void) block.len = block_size; block_source_from_buf(&block.source, &buf); ret = block_writer_init(&bw, BLOCK_TYPE_OBJ, block.data, block_size, - header_off, hash_size(GIT_SHA1_FORMAT_ID)); + header_off, hash_size(REFTABLE_HASH_SHA1)); check(!ret); for (i = 0; i < N; i++) { @@ -242,7 +242,7 @@ static void t_obj_block_read_write(void) block_writer_release(&bw); - block_reader_init(&br, &block, header_off, block_size, GIT_SHA1_RAWSZ); + block_reader_init(&br, &block, header_off, block_size, REFTABLE_HASH_SIZE_SHA1); block_iter_seek_start(&it, &br); @@ -253,7 +253,7 @@ static void t_obj_block_read_write(void) check_int(i, ==, N); break; } - check(reftable_record_equal(&recs[i], &rec, GIT_SHA1_RAWSZ)); + check(reftable_record_equal(&recs[i], &rec, REFTABLE_HASH_SIZE_SHA1)); } for (i = 0; i < N; i++) { @@ -266,7 +266,7 @@ static void t_obj_block_read_write(void) ret = block_iter_next(&it, &rec); check_int(ret, ==, 0); - check(reftable_record_equal(&recs[i], &rec, GIT_SHA1_RAWSZ)); + check(reftable_record_equal(&recs[i], &rec, REFTABLE_HASH_SIZE_SHA1)); } block_reader_release(&br); @@ -304,7 +304,7 @@ static void t_index_block_read_write(void) block.len = block_size; block_source_from_buf(&block.source, &buf); ret = block_writer_init(&bw, BLOCK_TYPE_INDEX, block.data, block_size, - header_off, hash_size(GIT_SHA1_FORMAT_ID)); + header_off, hash_size(REFTABLE_HASH_SHA1)); check(!ret); for (i = 0; i < N; i++) { @@ -326,7 +326,7 @@ static void t_index_block_read_write(void) block_writer_release(&bw); - block_reader_init(&br, &block, header_off, block_size, GIT_SHA1_RAWSZ); + block_reader_init(&br, &block, header_off, block_size, REFTABLE_HASH_SIZE_SHA1); block_iter_seek_start(&it, &br); @@ -337,7 +337,7 @@ static void t_index_block_read_write(void) check_int(i, ==, N); break; } - check(reftable_record_equal(&recs[i], &rec, GIT_SHA1_RAWSZ)); + check(reftable_record_equal(&recs[i], &rec, REFTABLE_HASH_SIZE_SHA1)); } for (i = 0; i < N; i++) { @@ -350,7 +350,7 @@ static void t_index_block_read_write(void) ret = block_iter_next(&it, &rec); check_int(ret, ==, 0); - check(reftable_record_equal(&recs[i], &rec, GIT_SHA1_RAWSZ)); + check(reftable_record_equal(&recs[i], &rec, REFTABLE_HASH_SIZE_SHA1)); want.len--; ret = block_iter_seek_key(&it, &br, &want); @@ -358,7 +358,7 @@ static void t_index_block_read_write(void) ret = block_iter_next(&it, &rec); check_int(ret, ==, 0); - check(reftable_record_equal(&recs[10 * (i / 10)], &rec, GIT_SHA1_RAWSZ)); + check(reftable_record_equal(&recs[10 * (i / 10)], &rec, REFTABLE_HASH_SIZE_SHA1)); } block_reader_release(&br); diff --git a/t/unit-tests/t-reftable-merged.c b/t/unit-tests/t-reftable-merged.c index 484c18251f3..0573d9470a6 100644 --- a/t/unit-tests/t-reftable-merged.c +++ b/t/unit-tests/t-reftable-merged.c @@ -42,7 +42,7 @@ merged_table_from_records(struct reftable_ref_record **refs, check(!err); } - err = reftable_merged_table_new(&mt, *readers, n, GIT_SHA1_FORMAT_ID); + err = reftable_merged_table_new(&mt, *readers, n, REFTABLE_HASH_SHA1); check(!err); return mt; } @@ -91,7 +91,7 @@ static void t_merged_single_record(void) err = reftable_iterator_next_ref(&it, &ref); check(!err); - check(reftable_ref_record_equal(&r2[0], &ref, GIT_SHA1_RAWSZ)); + check(reftable_ref_record_equal(&r2[0], &ref, REFTABLE_HASH_SIZE_SHA1)); reftable_ref_record_release(&ref); reftable_iterator_destroy(&it); readers_destroy(readers, 3); @@ -168,7 +168,7 @@ static void t_merged_refs(void) check(!err); err = reftable_iterator_seek_ref(&it, "a"); check(!err); - check_int(reftable_merged_table_hash_id(mt), ==, GIT_SHA1_FORMAT_ID); + check_int(reftable_merged_table_hash_id(mt), ==, REFTABLE_HASH_SHA1); check_int(reftable_merged_table_min_update_index(mt), ==, 1); check_int(reftable_merged_table_max_update_index(mt), ==, 3); @@ -186,7 +186,7 @@ static void t_merged_refs(void) check_int(ARRAY_SIZE(want), ==, len); for (i = 0; i < len; i++) check(reftable_ref_record_equal(want[i], &out[i], - GIT_SHA1_RAWSZ)); + REFTABLE_HASH_SIZE_SHA1)); for (i = 0; i < len; i++) reftable_ref_record_release(&out[i]); reftable_free(out); @@ -252,12 +252,12 @@ static void t_merged_seek_multiple_times(void) err = reftable_iterator_next_ref(&it, &rec); check(!err); - err = reftable_ref_record_equal(&rec, &r1[1], GIT_SHA1_RAWSZ); + err = reftable_ref_record_equal(&rec, &r1[1], REFTABLE_HASH_SIZE_SHA1); check(err == 1); err = reftable_iterator_next_ref(&it, &rec); check(!err); - err = reftable_ref_record_equal(&rec, &r2[1], GIT_SHA1_RAWSZ); + err = reftable_ref_record_equal(&rec, &r2[1], REFTABLE_HASH_SIZE_SHA1); check(err == 1); err = reftable_iterator_next_ref(&it, &rec); @@ -300,7 +300,7 @@ merged_table_from_log_records(struct reftable_log_record **logs, check(!err); } - err = reftable_merged_table_new(&mt, *readers, n, GIT_SHA1_FORMAT_ID); + err = reftable_merged_table_new(&mt, *readers, n, REFTABLE_HASH_SHA1); check(!err); return mt; } @@ -377,7 +377,7 @@ static void t_merged_logs(void) check(!err); err = reftable_iterator_seek_log(&it, "a"); check(!err); - check_int(reftable_merged_table_hash_id(mt), ==, GIT_SHA1_FORMAT_ID); + check_int(reftable_merged_table_hash_id(mt), ==, REFTABLE_HASH_SHA1); check_int(reftable_merged_table_min_update_index(mt), ==, 1); check_int(reftable_merged_table_max_update_index(mt), ==, 3); @@ -395,7 +395,7 @@ static void t_merged_logs(void) check_int(ARRAY_SIZE(want), ==, len); for (i = 0; i < len; i++) check(reftable_log_record_equal(want[i], &out[i], - GIT_SHA1_RAWSZ)); + REFTABLE_HASH_SIZE_SHA1)); err = merged_table_init_iter(mt, &it, BLOCK_TYPE_LOG); check(!err); @@ -404,7 +404,7 @@ static void t_merged_logs(void) reftable_log_record_release(&out[0]); err = reftable_iterator_next_log(&it, &out[0]); check(!err); - check(reftable_log_record_equal(&out[0], &r3[0], GIT_SHA1_RAWSZ)); + check(reftable_log_record_equal(&out[0], &r3[0], REFTABLE_HASH_SIZE_SHA1)); reftable_iterator_destroy(&it); for (i = 0; i < len; i++) @@ -448,11 +448,11 @@ static void t_default_write_opts(void) check(!err); hash_id = reftable_reader_hash_id(rd); - check_int(hash_id, ==, GIT_SHA1_FORMAT_ID); + check_int(hash_id, ==, REFTABLE_HASH_SHA1); - err = reftable_merged_table_new(&merged, &rd, 1, GIT_SHA256_FORMAT_ID); + err = reftable_merged_table_new(&merged, &rd, 1, REFTABLE_HASH_SHA256); check_int(err, ==, REFTABLE_FORMAT_ERROR); - err = reftable_merged_table_new(&merged, &rd, 1, GIT_SHA1_FORMAT_ID); + err = reftable_merged_table_new(&merged, &rd, 1, REFTABLE_HASH_SHA1); check(!err); reftable_reader_decref(rd); diff --git a/t/unit-tests/t-reftable-pq.c b/t/unit-tests/t-reftable-pq.c index ada4c19f18a..272da05bea6 100644 --- a/t/unit-tests/t-reftable-pq.c +++ b/t/unit-tests/t-reftable-pq.c @@ -132,7 +132,7 @@ static void t_merged_iter_pqueue_top(void) merged_iter_pqueue_check(&pq); check(pq_entry_equal(&top, &e)); - check(reftable_record_equal(top.rec, &recs[i], GIT_SHA1_RAWSZ)); + check(reftable_record_equal(top.rec, &recs[i], REFTABLE_HASH_SIZE_SHA1)); for (size_t j = 0; i < pq.len; j++) { check(pq_less(&top, &pq.heap[j])); check_int(top.index, >, j); diff --git a/t/unit-tests/t-reftable-reader.c b/t/unit-tests/t-reftable-reader.c index 19cb53b6415..546df6005e4 100644 --- a/t/unit-tests/t-reftable-reader.c +++ b/t/unit-tests/t-reftable-reader.c @@ -31,7 +31,7 @@ static int t_reader_seek_once(void) ret = reftable_iterator_next_ref(&it, &ref); check(!ret); - ret = reftable_ref_record_equal(&ref, &records[0], GIT_SHA1_RAWSZ); + ret = reftable_ref_record_equal(&ref, &records[0], REFTABLE_HASH_SIZE_SHA1); check_int(ret, ==, 1); ret = reftable_iterator_next_ref(&it, &ref); @@ -74,7 +74,7 @@ static int t_reader_reseek(void) ret = reftable_iterator_next_ref(&it, &ref); check(!ret); - ret = reftable_ref_record_equal(&ref, &records[0], GIT_SHA1_RAWSZ); + ret = reftable_ref_record_equal(&ref, &records[0], REFTABLE_HASH_SIZE_SHA1); check_int(ret, ==, 1); ret = reftable_iterator_next_ref(&it, &ref); diff --git a/t/unit-tests/t-reftable-readwrite.c b/t/unit-tests/t-reftable-readwrite.c index d279b86df0a..57896922eb1 100644 --- a/t/unit-tests/t-reftable-readwrite.c +++ b/t/unit-tests/t-reftable-readwrite.c @@ -41,7 +41,7 @@ static void t_buffer(void) } static void write_table(char ***names, struct reftable_buf *buf, int N, - int block_size, uint32_t hash_id) + int block_size, enum reftable_hash hash_id) { struct reftable_write_options opts = { .block_size = block_size, @@ -62,7 +62,7 @@ static void write_table(char ***names, struct reftable_buf *buf, int N, refs[i].refname = (*names)[i] = xstrfmt("refs/heads/branch%02d", i); refs[i].update_index = update_index; refs[i].value_type = REFTABLE_REF_VAL1; - t_reftable_set_hash(refs[i].value.val1, i, GIT_SHA1_FORMAT_ID); + t_reftable_set_hash(refs[i].value.val1, i, REFTABLE_HASH_SHA1); } for (i = 0; i < N; i++) { @@ -70,7 +70,7 @@ static void write_table(char ***names, struct reftable_buf *buf, int N, logs[i].update_index = update_index; logs[i].value_type = REFTABLE_LOG_UPDATE; t_reftable_set_hash(logs[i].value.update.new_hash, i, - GIT_SHA1_FORMAT_ID); + REFTABLE_HASH_SHA1); logs[i].value.update.message = (char *) "message"; } @@ -104,7 +104,7 @@ static void t_log_buffer_size(void) /* This tests buffer extension for log compression. Must use a random hash, to ensure that the compressed part is larger than the original. */ - for (i = 0; i < GIT_SHA1_RAWSZ; i++) { + for (i = 0; i < REFTABLE_HASH_SIZE_SHA1; i++) { log.value.update.old_hash[i] = (uint8_t)(git_rand() % 256); log.value.update.new_hash[i] = (uint8_t)(git_rand() % 256); } @@ -191,9 +191,9 @@ static void t_log_write_read(void) log.update_index = i; log.value_type = REFTABLE_LOG_UPDATE; t_reftable_set_hash(log.value.update.old_hash, i, - GIT_SHA1_FORMAT_ID); + REFTABLE_HASH_SHA1); t_reftable_set_hash(log.value.update.new_hash, i + 1, - GIT_SHA1_FORMAT_ID); + REFTABLE_HASH_SHA1); err = reftable_writer_add_log(w, &log); check(!err); @@ -326,7 +326,7 @@ static void t_table_read_write_sequential(void) int err = 0; int j = 0; - write_table(&names, &buf, N, 256, GIT_SHA1_FORMAT_ID); + write_table(&names, &buf, N, 256, REFTABLE_HASH_SHA1); block_source_from_buf(&source, &buf); @@ -361,7 +361,7 @@ static void t_table_write_small_table(void) char **names; struct reftable_buf buf = REFTABLE_BUF_INIT; int N = 1; - write_table(&names, &buf, N, 4096, GIT_SHA1_FORMAT_ID); + write_table(&names, &buf, N, 4096, REFTABLE_HASH_SHA1); check_int(buf.len, <, 200); reftable_buf_release(&buf); free_names(names); @@ -378,7 +378,7 @@ static void t_table_read_api(void) struct reftable_log_record log = { 0 }; struct reftable_iterator it = { 0 }; - write_table(&names, &buf, N, 256, GIT_SHA1_FORMAT_ID); + write_table(&names, &buf, N, 256, REFTABLE_HASH_SHA1); block_source_from_buf(&source, &buf); @@ -400,7 +400,7 @@ static void t_table_read_api(void) reftable_buf_release(&buf); } -static void t_table_read_write_seek(int index, int hash_id) +static void t_table_read_write_seek(int index, enum reftable_hash hash_id) { char **names; struct reftable_buf buf = REFTABLE_BUF_INIT; @@ -467,24 +467,24 @@ static void t_table_read_write_seek(int index, int hash_id) static void t_table_read_write_seek_linear(void) { - t_table_read_write_seek(0, GIT_SHA1_FORMAT_ID); + t_table_read_write_seek(0, REFTABLE_HASH_SHA1); } static void t_table_read_write_seek_linear_sha256(void) { - t_table_read_write_seek(0, GIT_SHA256_FORMAT_ID); + t_table_read_write_seek(0, REFTABLE_HASH_SHA256); } static void t_table_read_write_seek_index(void) { - t_table_read_write_seek(1, GIT_SHA1_FORMAT_ID); + t_table_read_write_seek(1, REFTABLE_HASH_SHA1); } static void t_table_refs_for(int indexed) { char **want_names; int want_names_len = 0; - uint8_t want_hash[GIT_SHA1_RAWSZ]; + uint8_t want_hash[REFTABLE_HASH_SIZE_SHA1]; struct reftable_write_options opts = { .block_size = 256, @@ -500,10 +500,10 @@ static void t_table_refs_for(int indexed) want_names = reftable_calloc(N + 1, sizeof(*want_names)); check(want_names != NULL); - t_reftable_set_hash(want_hash, 4, GIT_SHA1_FORMAT_ID); + t_reftable_set_hash(want_hash, 4, REFTABLE_HASH_SHA1); for (i = 0; i < N; i++) { - uint8_t hash[GIT_SHA1_RAWSZ]; + uint8_t hash[REFTABLE_HASH_SIZE_SHA1]; char fill[51] = { 0 }; char name[100]; struct reftable_ref_record ref = { 0 }; @@ -517,9 +517,9 @@ static void t_table_refs_for(int indexed) ref.value_type = REFTABLE_REF_VAL2; t_reftable_set_hash(ref.value.val2.value, i / 4, - GIT_SHA1_FORMAT_ID); + REFTABLE_HASH_SHA1); t_reftable_set_hash(ref.value.val2.target_value, 3 + i / 4, - GIT_SHA1_FORMAT_ID); + REFTABLE_HASH_SHA1); /* 80 bytes / entry, so 3 entries per block. Yields 17 */ @@ -527,8 +527,8 @@ static void t_table_refs_for(int indexed) n = reftable_writer_add_ref(w, &ref); check_int(n, ==, 0); - if (!memcmp(ref.value.val2.value, want_hash, GIT_SHA1_RAWSZ) || - !memcmp(ref.value.val2.target_value, want_hash, GIT_SHA1_RAWSZ)) + if (!memcmp(ref.value.val2.value, want_hash, REFTABLE_HASH_SIZE_SHA1) || + !memcmp(ref.value.val2.target_value, want_hash, REFTABLE_HASH_SIZE_SHA1)) want_names[want_names_len++] = xstrdup(name); } diff --git a/t/unit-tests/t-reftable-record.c b/t/unit-tests/t-reftable-record.c index eb98bf2da91..42bc64cec87 100644 --- a/t/unit-tests/t-reftable-record.c +++ b/t/unit-tests/t-reftable-record.c @@ -7,6 +7,7 @@ */ #include "test-lib.h" +#include "reftable/basics.h" #include "reftable/constants.h" #include "reftable/record.h" @@ -17,10 +18,10 @@ static void t_copy(struct reftable_record *rec) typ = reftable_record_type(rec); reftable_record_init(©, typ); - reftable_record_copy_from(©, rec, GIT_SHA1_RAWSZ); + reftable_record_copy_from(©, rec, REFTABLE_HASH_SIZE_SHA1); /* do it twice to catch memory leaks */ - reftable_record_copy_from(©, rec, GIT_SHA1_RAWSZ); - check(reftable_record_equal(rec, ©, GIT_SHA1_RAWSZ)); + reftable_record_copy_from(©, rec, REFTABLE_HASH_SIZE_SHA1); + check(reftable_record_equal(rec, ©, REFTABLE_HASH_SIZE_SHA1)); reftable_record_release(©); } @@ -59,7 +60,7 @@ static void t_varint_roundtrip(void) static void set_hash(uint8_t *h, int j) { - for (int i = 0; i < hash_size(GIT_SHA1_FORMAT_ID); i++) + for (int i = 0; i < hash_size(REFTABLE_HASH_SHA1); i++) h[i] = (j >> i) & 0xff; } @@ -84,14 +85,14 @@ static void t_reftable_ref_record_comparison(void) }, }; - check(!reftable_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ)); + check(!reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1)); check(!reftable_record_cmp(&in[0], &in[1])); - check(!reftable_record_equal(&in[1], &in[2], GIT_SHA1_RAWSZ)); + check(!reftable_record_equal(&in[1], &in[2], REFTABLE_HASH_SIZE_SHA1)); check_int(reftable_record_cmp(&in[1], &in[2]), >, 0); in[1].u.ref.value_type = in[0].u.ref.value_type; - check(reftable_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ)); + check(reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1)); check(!reftable_record_cmp(&in[0], &in[1])); } @@ -155,15 +156,15 @@ static void t_reftable_ref_record_roundtrip(void) check_int(reftable_record_is_deletion(&in), ==, i == REFTABLE_REF_DELETION); reftable_record_key(&in, &key); - n = reftable_record_encode(&in, dest, GIT_SHA1_RAWSZ); + n = reftable_record_encode(&in, dest, REFTABLE_HASH_SIZE_SHA1); check_int(n, >, 0); /* decode into a non-zero reftable_record to test for leaks. */ - m = reftable_record_decode(&out, key, i, dest, GIT_SHA1_RAWSZ, &scratch); + m = reftable_record_decode(&out, key, i, dest, REFTABLE_HASH_SIZE_SHA1, &scratch); check_int(n, ==, m); check(reftable_ref_record_equal(&in.u.ref, &out.u.ref, - GIT_SHA1_RAWSZ)); + REFTABLE_HASH_SIZE_SHA1)); reftable_record_release(&in); reftable_buf_release(&key); @@ -193,15 +194,15 @@ static void t_reftable_log_record_comparison(void) }, }; - check(!reftable_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ)); - check(!reftable_record_equal(&in[1], &in[2], GIT_SHA1_RAWSZ)); + check(!reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1)); + check(!reftable_record_equal(&in[1], &in[2], REFTABLE_HASH_SIZE_SHA1)); check_int(reftable_record_cmp(&in[1], &in[2]), >, 0); /* comparison should be reversed for equal keys, because * comparison is now performed on the basis of update indices */ check_int(reftable_record_cmp(&in[0], &in[1]), <, 0); in[1].u.log.update_index = in[0].u.log.update_index; - check(reftable_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ)); + check(reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1)); check(!reftable_record_cmp(&in[0], &in[1])); } @@ -303,15 +304,15 @@ static void t_reftable_log_record_roundtrip(void) reftable_record_key(&rec, &key); - n = reftable_record_encode(&rec, dest, GIT_SHA1_RAWSZ); + n = reftable_record_encode(&rec, dest, REFTABLE_HASH_SIZE_SHA1); check_int(n, >=, 0); valtype = reftable_record_val_type(&rec); m = reftable_record_decode(&out, key, valtype, dest, - GIT_SHA1_RAWSZ, &scratch); + REFTABLE_HASH_SIZE_SHA1, &scratch); check_int(n, ==, m); check(reftable_log_record_equal(&in[i], &out.u.log, - GIT_SHA1_RAWSZ)); + REFTABLE_HASH_SIZE_SHA1)); reftable_log_record_release(&in[i]); reftable_buf_release(&key); reftable_record_release(&out); @@ -380,20 +381,20 @@ static void t_reftable_obj_record_comparison(void) }, }; - check(!reftable_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ)); + check(!reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1)); check(!reftable_record_cmp(&in[0], &in[1])); - check(!reftable_record_equal(&in[1], &in[2], GIT_SHA1_RAWSZ)); + check(!reftable_record_equal(&in[1], &in[2], REFTABLE_HASH_SIZE_SHA1)); check_int(reftable_record_cmp(&in[1], &in[2]), >, 0); in[1].u.obj.offset_len = in[0].u.obj.offset_len; - check(reftable_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ)); + check(reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1)); check(!reftable_record_cmp(&in[0], &in[1])); } static void t_reftable_obj_record_roundtrip(void) { - uint8_t testHash1[GIT_SHA1_RAWSZ] = { 1, 2, 3, 4, 0 }; + uint8_t testHash1[REFTABLE_HASH_SIZE_SHA1] = { 1, 2, 3, 4, 0 }; uint64_t till9[] = { 1, 2, 3, 4, 500, 600, 700, 800, 9000 }; struct reftable_obj_record recs[3] = { { @@ -435,14 +436,14 @@ static void t_reftable_obj_record_roundtrip(void) check(!reftable_record_is_deletion(&in)); t_copy(&in); reftable_record_key(&in, &key); - n = reftable_record_encode(&in, dest, GIT_SHA1_RAWSZ); + n = reftable_record_encode(&in, dest, REFTABLE_HASH_SIZE_SHA1); check_int(n, >, 0); extra = reftable_record_val_type(&in); m = reftable_record_decode(&out, key, extra, dest, - GIT_SHA1_RAWSZ, &scratch); + REFTABLE_HASH_SIZE_SHA1, &scratch); check_int(n, ==, m); - check(reftable_record_equal(&in, &out, GIT_SHA1_RAWSZ)); + check(reftable_record_equal(&in, &out, REFTABLE_HASH_SIZE_SHA1)); reftable_buf_release(&key); reftable_record_release(&out); } @@ -473,14 +474,14 @@ static void t_reftable_index_record_comparison(void) check(!reftable_buf_addstr(&in[1].u.idx.last_key, "refs/heads/master")); check(!reftable_buf_addstr(&in[2].u.idx.last_key, "refs/heads/branch")); - check(!reftable_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ)); + check(!reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1)); check(!reftable_record_cmp(&in[0], &in[1])); - check(!reftable_record_equal(&in[1], &in[2], GIT_SHA1_RAWSZ)); + check(!reftable_record_equal(&in[1], &in[2], REFTABLE_HASH_SIZE_SHA1)); check_int(reftable_record_cmp(&in[1], &in[2]), >, 0); in[1].u.idx.offset = in[0].u.idx.offset; - check(reftable_record_equal(&in[0], &in[1], GIT_SHA1_RAWSZ)); + check(reftable_record_equal(&in[0], &in[1], REFTABLE_HASH_SIZE_SHA1)); check(!reftable_record_cmp(&in[0], &in[1])); for (size_t i = 0; i < ARRAY_SIZE(in); i++) @@ -516,15 +517,15 @@ static void t_reftable_index_record_roundtrip(void) check(!reftable_record_is_deletion(&in)); check(!reftable_buf_cmp(&key, &in.u.idx.last_key)); - n = reftable_record_encode(&in, dest, GIT_SHA1_RAWSZ); + n = reftable_record_encode(&in, dest, REFTABLE_HASH_SIZE_SHA1); check_int(n, >, 0); extra = reftable_record_val_type(&in); - m = reftable_record_decode(&out, key, extra, dest, GIT_SHA1_RAWSZ, + m = reftable_record_decode(&out, key, extra, dest, REFTABLE_HASH_SIZE_SHA1, &scratch); check_int(m, ==, n); - check(reftable_record_equal(&in, &out, GIT_SHA1_RAWSZ)); + check(reftable_record_equal(&in, &out, REFTABLE_HASH_SIZE_SHA1)); reftable_record_release(&out); reftable_buf_release(&key); diff --git a/t/unit-tests/t-reftable-stack.c b/t/unit-tests/t-reftable-stack.c index 1b4363a58fc..13fd8d8f941 100644 --- a/t/unit-tests/t-reftable-stack.c +++ b/t/unit-tests/t-reftable-stack.c @@ -121,7 +121,7 @@ static void write_n_ref_tables(struct reftable_stack *st, snprintf(buf, sizeof(buf), "refs/heads/branch-%04"PRIuMAX, (uintmax_t)i); ref.refname = buf; - t_reftable_set_hash(ref.value.val1, i, GIT_SHA1_FORMAT_ID); + t_reftable_set_hash(ref.value.val1, i, REFTABLE_HASH_SHA1); err = reftable_stack_add(st, &write_test_ref, &ref); check(!err); @@ -169,7 +169,7 @@ static void t_reftable_stack_add_one(void) err = reftable_stack_read_ref(st, ref.refname, &dest); check(!err); - check(reftable_ref_record_equal(&ref, &dest, GIT_SHA1_RAWSZ)); + check(reftable_ref_record_equal(&ref, &dest, REFTABLE_HASH_SIZE_SHA1)); check_int(st->readers_len, >, 0); #ifndef GIT_WINDOWS_NATIVE @@ -280,7 +280,7 @@ static void t_reftable_stack_transaction_api(void) err = reftable_stack_read_ref(st, ref.refname, &dest); check(!err); check_int(REFTABLE_REF_SYMREF, ==, dest.value_type); - check(reftable_ref_record_equal(&ref, &dest, GIT_SHA1_RAWSZ)); + check(reftable_ref_record_equal(&ref, &dest, REFTABLE_HASH_SIZE_SHA1)); reftable_ref_record_release(&dest); reftable_stack_destroy(st); @@ -340,7 +340,7 @@ static void t_reftable_stack_transaction_with_reload(void) for (size_t i = 0; i < ARRAY_SIZE(refs); i++) { err = reftable_stack_read_ref(st2, refs[i].refname, &ref); check(!err); - check(reftable_ref_record_equal(&refs[i], &ref, GIT_SHA1_RAWSZ)); + check(reftable_ref_record_equal(&refs[i], &ref, REFTABLE_HASH_SIZE_SHA1)); } reftable_ref_record_release(&ref); @@ -530,13 +530,13 @@ static void t_reftable_stack_add(void) refs[i].refname = xstrdup(buf); refs[i].update_index = i + 1; refs[i].value_type = REFTABLE_REF_VAL1; - t_reftable_set_hash(refs[i].value.val1, i, GIT_SHA1_FORMAT_ID); + t_reftable_set_hash(refs[i].value.val1, i, REFTABLE_HASH_SHA1); logs[i].refname = xstrdup(buf); logs[i].update_index = N + i + 1; logs[i].value_type = REFTABLE_LOG_UPDATE; logs[i].value.update.email = xstrdup("identity@invalid"); - t_reftable_set_hash(logs[i].value.update.new_hash, i, GIT_SHA1_FORMAT_ID); + t_reftable_set_hash(logs[i].value.update.new_hash, i, REFTABLE_HASH_SHA1); } for (i = 0; i < N; i++) { @@ -562,7 +562,7 @@ static void t_reftable_stack_add(void) int err = reftable_stack_read_ref(st, refs[i].refname, &dest); check(!err); check(reftable_ref_record_equal(&dest, refs + i, - GIT_SHA1_RAWSZ)); + REFTABLE_HASH_SIZE_SHA1)); reftable_ref_record_release(&dest); } @@ -571,7 +571,7 @@ static void t_reftable_stack_add(void) int err = reftable_stack_read_log(st, refs[i].refname, &dest); check(!err); check(reftable_log_record_equal(&dest, logs + i, - GIT_SHA1_RAWSZ)); + REFTABLE_HASH_SIZE_SHA1)); reftable_log_record_release(&dest); } @@ -622,14 +622,14 @@ static void t_reftable_stack_iterator(void) refs[i].refname = xstrfmt("branch%02"PRIuMAX, (uintmax_t)i); refs[i].update_index = i + 1; refs[i].value_type = REFTABLE_REF_VAL1; - t_reftable_set_hash(refs[i].value.val1, i, GIT_SHA1_FORMAT_ID); + t_reftable_set_hash(refs[i].value.val1, i, REFTABLE_HASH_SHA1); logs[i].refname = xstrfmt("branch%02"PRIuMAX, (uintmax_t)i); logs[i].update_index = i + 1; logs[i].value_type = REFTABLE_LOG_UPDATE; logs[i].value.update.email = xstrdup("johndoe@invalid"); logs[i].value.update.message = xstrdup("commit\n"); - t_reftable_set_hash(logs[i].value.update.new_hash, i, GIT_SHA1_FORMAT_ID); + t_reftable_set_hash(logs[i].value.update.new_hash, i, REFTABLE_HASH_SHA1); } for (i = 0; i < N; i++) { @@ -656,7 +656,7 @@ static void t_reftable_stack_iterator(void) if (err > 0) break; check(!err); - check(reftable_ref_record_equal(&ref, &refs[i], GIT_SHA1_RAWSZ)); + check(reftable_ref_record_equal(&ref, &refs[i], REFTABLE_HASH_SIZE_SHA1)); reftable_ref_record_release(&ref); } check_int(i, ==, N); @@ -674,7 +674,7 @@ static void t_reftable_stack_iterator(void) if (err > 0) break; check(!err); - check(reftable_log_record_equal(&log, &logs[i], GIT_SHA1_RAWSZ)); + check(reftable_log_record_equal(&log, &logs[i], REFTABLE_HASH_SIZE_SHA1)); reftable_log_record_release(&log); } check_int(i, ==, N); @@ -767,7 +767,7 @@ static void t_reftable_stack_tombstone(void) if (i % 2 == 0) { refs[i].value_type = REFTABLE_REF_VAL1; t_reftable_set_hash(refs[i].value.val1, i, - GIT_SHA1_FORMAT_ID); + REFTABLE_HASH_SHA1); } logs[i].refname = xstrdup(buf); @@ -776,7 +776,7 @@ static void t_reftable_stack_tombstone(void) if (i % 2 == 0) { logs[i].value_type = REFTABLE_LOG_UPDATE; t_reftable_set_hash(logs[i].value.update.new_hash, i, - GIT_SHA1_FORMAT_ID); + REFTABLE_HASH_SHA1); logs[i].value.update.email = xstrdup("identity@invalid"); } @@ -836,7 +836,7 @@ static void t_reftable_stack_hash_id(void) .value.symref = (char *) "target", .update_index = 1, }; - struct reftable_write_options opts32 = { .hash_id = GIT_SHA256_FORMAT_ID }; + struct reftable_write_options opts32 = { .hash_id = REFTABLE_HASH_SHA256 }; struct reftable_stack *st32 = NULL; struct reftable_write_options opts_default = { 0 }; struct reftable_stack *st_default = NULL; @@ -859,7 +859,7 @@ static void t_reftable_stack_hash_id(void) err = reftable_stack_read_ref(st_default, "master", &dest); check(!err); - check(reftable_ref_record_equal(&ref, &dest, GIT_SHA1_RAWSZ)); + check(reftable_ref_record_equal(&ref, &dest, REFTABLE_HASH_SIZE_SHA1)); reftable_ref_record_release(&dest); reftable_stack_destroy(st); reftable_stack_destroy(st_default); @@ -909,7 +909,7 @@ static void t_reflog_expire(void) logs[i].value.update.time = i; logs[i].value.update.email = xstrdup("identity@invalid"); t_reftable_set_hash(logs[i].value.update.new_hash, i, - GIT_SHA1_FORMAT_ID); + REFTABLE_HASH_SHA1); } for (i = 1; i <= N; i++) { From patchwork Mon Nov 18 15:34:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13878787 Received: from fhigh-a6-smtp.messagingengine.com (fhigh-a6-smtp.messagingengine.com [103.168.172.157]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 673DC1B6D0A for ; Mon, 18 Nov 2024 15:34:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.157 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731944054; cv=none; b=QW4uidHsN1oiOJQZIryTXtkRBhfTv5b/5gGkDbbGcMmnTXrFnsX2RY87NA8rxGLqzwBVZcICuxIeyhx7Hgf8R5IfFluxhoviHl932nCB/GfOb2HbcF2mui1hWrCmo+Pcw68uuek7uGEBD8WOQ8/bE7ar/YLQ7Ut9SmO/hd/mrcM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731944054; c=relaxed/simple; bh=9XE2Au5AbUcXSqiGR7fuNx1kxJy6M93xzRSgUD/rCwA=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=LH7NuWZX2paZd/7dju2MZXIW0ZbGZOGEo/JCRjps3JSjfx+7OJewl7b9PpDBXOahgCgA62YPPyZUZGRpkXxnmR8WR2/QgcJSz5sKHwlOnjhVcHl0kz1/TkTnDQH35moFqCEiCU8P4G3mgIQsOZD4SmTwdVYPAJ8/lZfG0cvyM4Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=4OZ202bb; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=lAuVcEb8; arc=none smtp.client-ip=103.168.172.157 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="4OZ202bb"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="lAuVcEb8" Received: from phl-compute-07.internal (phl-compute-07.phl.internal [10.202.2.47]) by mailfhigh.phl.internal (Postfix) with ESMTP id 84B76114022C; Mon, 18 Nov 2024 10:34:11 -0500 (EST) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-07.internal (MEProxy); Mon, 18 Nov 2024 10:34:11 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1731944051; x=1732030451; bh=ApxP+lZ2lO KFRtbWchhaxSoWLiKJPCPJs/CCInpTMxg=; b=4OZ202bbTeA8bBxQLdZov10cSi DPMV+NH4uGJ5120EXqohtxECIeTyh5Tx7rEL7sWdgRW7veqmQNskfbuA9bEbA9nY d6ZXtkagFHeYbcu9Uvq35TZIXL40lxc3eqXLZyr4vbdxk9lNfbEpQ6adqVs8mE2D r+U3YP0MD6jKIA2LjcGX0/2JU8WeWhHbzAwm1QM6q7KaSZIwo9ZnGKUu/b2XFYjE 6izzQ/GUSqEUDMmvDQlR9Y3HVlRqrf68CX2xEosGG9iCi3xodV4N2KxUm4d6ih2h m0OKju7wO7YD4Vn6GezVoDGGFICtJACzPawf0aq3aWF8SZpJG3j7i8HFZ1Xg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t= 1731944051; x=1732030451; bh=ApxP+lZ2lOKFRtbWchhaxSoWLiKJPCPJs/C CInpTMxg=; b=lAuVcEb80qMuMdaxJ9dgTQOIjJU2cU6RQwwcS3AG7CXpa5/8Mmd 7WNFqsl+YQGrMJbvDm6PzprhsEmB0TO+KCjKIT3OqwlzvyiQqAcY2HrZ5iA75tZ1 eEHCGixqX504dKGXbIUXNOnWE8V0I+27peJRohTiRze1knvULxbLxu04WII96yg8 0n3yyqIWpnB+aTfng/vgVaUaHja0fwvv32VRD6RmstpighN2twz9lWJjCvMFctTu tQWcc++7VZrlpQbpaoWfOTEbGtBTkrPEsgCXu5trZJJdJ8nKmIIbrPFaLA4mbPk8 8+sc7Zf5dWONtUFhm8LztWpQF+pJSj5zIlw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefuddrfedtgdejiecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdpuffr tefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnth hsucdlqddutddtmdenucfjughrpeffhffvvefukfhfgggtuggjsehttdertddttddvnecu hfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqe enucggtffrrghtthgvrhhnpeevkeekfffhiedtleduiefgjedttedvledvudehgfeugedu gffhueekhfejvdektdenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrih hlfhhrohhmpehpshesphhkshdrihhmpdhnsggprhgtphhtthhopeeipdhmohguvgepshhm thhpohhuthdprhgtphhtthhopehkrghrthhhihhkrddukeeksehgmhgrihhlrdgtohhmpd hrtghpthhtohepvghthhhomhhsohhnsegvugifrghrughthhhomhhsohhnrdgtohhmpdhr tghpthhtohepghhithesvhhgvghrrdhkvghrnhgvlhdrohhrghdprhgtphhtthhopehgih htshhtvghrsehpohgsohigrdgtohhmpdhrtghpthhtohepshhunhhshhhinhgvsehsuhhn shhhihhnvggtohdrtghomhdprhgtphhtthhopehjlhhtohgslhgvrhesghhmrghilhdrtg homh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 18 Nov 2024 10:34:09 -0500 (EST) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 8d2961b7 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 18 Nov 2024 15:33:22 +0000 (UTC) Date: Mon, 18 Nov 2024 16:34:00 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Edward Thomson , Eric Sunshine , Justin Tobler , Junio C Hamano , karthik nayak Subject: [PATCH v3 4/7] reftable/stack: stop using `fsync_component()` directly Message-ID: <7782652b97545f95c83aef08786c70384e5cfa52.1731943954.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: We're executing `fsync_component()` directly in the reftable library so that we can fsync data to disk depending on "core.fsync". But as we're in the process of converting the reftable library to become standalone we cannot use that function in the library anymore. Refactor the code such that users of the library can inject a custom fsync function via the write options. This allows us to get rid of the dependency on "write-or-die.h". Signed-off-by: Patrick Steinhardt --- refs/reftable-backend.c | 7 ++++++ reftable/reftable-writer.h | 6 +++++ reftable/stack.c | 49 +++++++++++++++++++++++++------------- 3 files changed, 45 insertions(+), 17 deletions(-) diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c index 7d86d920970..2e774176eda 100644 --- a/refs/reftable-backend.c +++ b/refs/reftable-backend.c @@ -24,6 +24,7 @@ #include "../setup.h" #include "../strmap.h" #include "../trace2.h" +#include "../write-or-die.h" #include "parse.h" #include "refs-internal.h" @@ -273,6 +274,11 @@ static int reftable_be_config(const char *var, const char *value, return 0; } +static int reftable_be_fsync(int fd) +{ + return fsync_component(FSYNC_COMPONENT_REFERENCE, fd); +} + static struct ref_store *reftable_be_init(struct repository *repo, const char *gitdir, unsigned int store_flags) @@ -304,6 +310,7 @@ static struct ref_store *reftable_be_init(struct repository *repo, refs->write_options.disable_auto_compact = !git_env_bool("GIT_TEST_REFTABLE_AUTOCOMPACTION", 1); refs->write_options.lock_timeout_ms = 100; + refs->write_options.fsync = reftable_be_fsync; git_config(reftable_be_config, &refs->write_options); diff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h index 211860d08a4..c85ef5a5bd1 100644 --- a/reftable/reftable-writer.h +++ b/reftable/reftable-writer.h @@ -62,6 +62,12 @@ struct reftable_write_options { * negative value will cause us to block indefinitely. */ long lock_timeout_ms; + + /* + * Optional callback used to fsync files to disk. Falls back to using + * fsync(3P) when unset. + */ + int (*fsync)(int fd); }; /* reftable_block_stats holds statistics for a single block type */ diff --git a/reftable/stack.c b/reftable/stack.c index 9ae716ff375..c67bdd952ca 100644 --- a/reftable/stack.c +++ b/reftable/stack.c @@ -8,7 +8,6 @@ license that can be found in the LICENSE file or at #include "stack.h" -#include "../write-or-die.h" #include "system.h" #include "constants.h" #include "merged.h" @@ -43,17 +42,28 @@ static int stack_filename(struct reftable_buf *dest, struct reftable_stack *st, return 0; } -static ssize_t reftable_fd_write(void *arg, const void *data, size_t sz) +static int stack_fsync(const struct reftable_write_options *opts, int fd) { - int *fdp = (int *)arg; - return write_in_full(*fdp, data, sz); + if (opts->fsync) + return opts->fsync(fd); + return fsync(fd); } -static int reftable_fd_flush(void *arg) +struct fd_writer { + const struct reftable_write_options *opts; + int fd; +}; + +static ssize_t fd_writer_write(void *arg, const void *data, size_t sz) { - int *fdp = (int *)arg; + struct fd_writer *writer = arg; + return write_in_full(writer->fd, data, sz); +} - return fsync_component(FSYNC_COMPONENT_REFERENCE, *fdp); +static int fd_writer_flush(void *arg) +{ + struct fd_writer *writer = arg; + return stack_fsync(writer->opts, writer->fd); } int reftable_new_stack(struct reftable_stack **dest, const char *dir, @@ -765,7 +775,7 @@ int reftable_addition_commit(struct reftable_addition *add) goto done; } - err = fsync_component(FSYNC_COMPONENT_REFERENCE, lock_file_fd); + err = stack_fsync(&add->stack->opts, lock_file_fd); if (err < 0) { err = REFTABLE_IO_ERROR; goto done; @@ -858,8 +868,10 @@ int reftable_addition_add(struct reftable_addition *add, struct reftable_buf next_name = REFTABLE_BUF_INIT; struct reftable_writer *wr = NULL; struct tempfile *tab_file = NULL; + struct fd_writer writer = { + .opts = &add->stack->opts, + }; int err = 0; - int tab_fd; reftable_buf_reset(&next_name); @@ -887,10 +899,10 @@ int reftable_addition_add(struct reftable_addition *add, goto done; } } - tab_fd = get_tempfile_fd(tab_file); - err = reftable_writer_new(&wr, reftable_fd_write, reftable_fd_flush, - &tab_fd, &add->stack->opts); + writer.fd = get_tempfile_fd(tab_file); + err = reftable_writer_new(&wr, fd_writer_write, fd_writer_flush, + &writer, &add->stack->opts); if (err < 0) goto done; @@ -973,8 +985,11 @@ static int stack_compact_locked(struct reftable_stack *st, struct reftable_buf next_name = REFTABLE_BUF_INIT; struct reftable_buf tab_file_path = REFTABLE_BUF_INIT; struct reftable_writer *wr = NULL; + struct fd_writer writer= { + .opts = &st->opts, + }; struct tempfile *tab_file; - int tab_fd, err = 0; + int err = 0; err = format_name(&next_name, reftable_reader_min_update_index(st->readers[first]), reftable_reader_max_update_index(st->readers[last])); @@ -994,7 +1009,6 @@ static int stack_compact_locked(struct reftable_stack *st, err = REFTABLE_IO_ERROR; goto done; } - tab_fd = get_tempfile_fd(tab_file); if (st->opts.default_permissions && chmod(get_tempfile_path(tab_file), st->opts.default_permissions) < 0) { @@ -1002,8 +1016,9 @@ static int stack_compact_locked(struct reftable_stack *st, goto done; } - err = reftable_writer_new(&wr, reftable_fd_write, reftable_fd_flush, - &tab_fd, &st->opts); + writer.fd = get_tempfile_fd(tab_file); + err = reftable_writer_new(&wr, fd_writer_write, fd_writer_flush, + &writer, &st->opts); if (err < 0) goto done; @@ -1460,7 +1475,7 @@ static int stack_compact_range(struct reftable_stack *st, goto done; } - err = fsync_component(FSYNC_COMPONENT_REFERENCE, get_lock_file_fd(&tables_list_lock)); + err = stack_fsync(&st->opts, get_lock_file_fd(&tables_list_lock)); if (err < 0) { err = REFTABLE_IO_ERROR; unlink(new_table_path.buf); From patchwork Mon Nov 18 15:34:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13878788 Received: from fhigh-a6-smtp.messagingengine.com (fhigh-a6-smtp.messagingengine.com [103.168.172.157]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C7F9A1BDAA0 for ; Mon, 18 Nov 2024 15:34:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.157 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731944056; cv=none; b=Pq2uOGfJKXOZ8wgBQW3ONDyTtYAEFZCP5nSojgYEkV4wjOsDegz/n0lmBHEzopfiE6sjUCooB/lftxFjTgSbaa9YGf0PijItfapEuHuYgpJFli07p74b7CRYz9ct1T7YDB+wCwEfafrC0QRyIP+5JkQyDpB62RI3HxsYXzT5BZo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731944056; c=relaxed/simple; bh=bELkNVfNB8JPG78groyTcXBE/23JDYT/gfif+4oh8Co=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=UuIKTdXH/7/ijWiFDxb8K8x8UTRK31BXeNFMRCvBMstaikWDwCcuaX84rWXmNhm44bfBkX19HD9JwI97HfNz+uaUBHSaPtbt3hZ1mz+Dz4f7VhYrCa19BpF/KKj666FB8qnF/e0gulFtVneASkqfEy84ZI6mJhQ86R3iBiDGezQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=UNrEbWMr; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=KUxUds11; arc=none smtp.client-ip=103.168.172.157 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="UNrEbWMr"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="KUxUds11" Received: from phl-compute-04.internal (phl-compute-04.phl.internal [10.202.2.44]) by mailfhigh.phl.internal (Postfix) with ESMTP id 31F57114022C; Mon, 18 Nov 2024 10:34:14 -0500 (EST) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-04.internal (MEProxy); Mon, 18 Nov 2024 10:34:14 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1731944054; x=1732030454; bh=ejuqnsYRrN 0wduGIZ9nwT0GInZCpwtkaeHA22CtSnmk=; b=UNrEbWMrrIyiA2R4wAro2fNUdA v4zo9BKKXL6KtZCsNoNxu2Px1g3XQY+QltRCLBhfRuq27VYlxNP8J+Arfd3FvWIp UOAzIKGV3Tew1LnIqpA03eU9XZ5qr0GUoZ89lrXxSTR8k9qdQXCLO7ZtQ+F+x8sj 5m8MAkbK7haGtowfwYvr2L3AQ1SzHX03gWjj9NYf/jD9kahzf9u0vrxbm6lLw5ID UEAt0rEFSBEO9gJnQKhagagkA8pfxuaDLDZoSHF/x7cehCeJgmCPHOOMtg60L+Gm AvQkZN9KK7naY7uxl69cjBhAbvOst7LQ4P2X0O9MRa34Q0fsrvi1qg9UaZIA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t= 1731944054; x=1732030454; bh=ejuqnsYRrN0wduGIZ9nwT0GInZCpwtkaeHA 22CtSnmk=; b=KUxUds11TP/539/rwe0k1uu0IQistceM3oJkZVIbPi+cSalX0tC /cYUxoh3hpmM93Idbkf9sO55oxki/PsWmCNhT7EkLVEookH6LRZE83MjeVZ8Z/NQ usm0hSrnxQZieq5hoFK5BJPltk5ozI/KP2S6QJoinxkmIJuhC7SBrGCrShRkgmw3 ud+9ROdOgEiOgdojnX9UKT30VAbxofTxN2IRzPXgKiloPmBECuRCTrWnTq++UCAy Ky+iNUC6HWpS2EW4N0CiC9UEqIzQdMzfVikYgwowmAU0eswwtu+KBjl4/ykytyA/ wfyqYID8OcTN2wx9/zyO+WEFVGJQ1fQqJmQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefuddrfedtgdejjecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdpuffr tefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnth hsucdlqddutddtmdenucfjughrpeffhffvvefukfhfgggtuggjsehttdertddttddvnecu hfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqe enucggtffrrghtthgvrhhnpeevkeekfffhiedtleduiefgjedttedvledvudehgfeugedu gffhueekhfejvdektdenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrih hlfhhrohhmpehpshesphhkshdrihhmpdhnsggprhgtphhtthhopeeipdhmohguvgepshhm thhpohhuthdprhgtphhtthhopehsuhhnshhhihhnvgesshhunhhshhhinhgvtghordgtoh hmpdhrtghpthhtohepghhithhsthgvrhesphhosghogidrtghomhdprhgtphhtthhopehk rghrthhhihhkrddukeeksehgmhgrihhlrdgtohhmpdhrtghpthhtohepjhhlthhosghlvg hrsehgmhgrihhlrdgtohhmpdhrtghpthhtohepvghthhhomhhsohhnsegvugifrghrught hhhomhhsohhnrdgtohhmpdhrtghpthhtohepghhithesvhhgvghrrdhkvghrnhgvlhdroh hrgh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 18 Nov 2024 10:34:12 -0500 (EST) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 65009d62 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 18 Nov 2024 15:33:24 +0000 (UTC) Date: Mon, 18 Nov 2024 16:34:03 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Edward Thomson , Eric Sunshine , Justin Tobler , Junio C Hamano , karthik nayak Subject: [PATCH v3 5/7] reftable/system: provide thin wrapper for tempfile subsystem Message-ID: <430be1045d6b9728748ed9dcabe0aab3a9b67859.1731943954.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: We use the tempfile subsystem to write temporary tables, but given that we're in the process of converting the reftable library to become standalone we cannot use this subsystem directly anymore. While we could in theory convert the code to use mkstemp(3p) instead, we'd lose access to our infrastructure that automatically prunes tempfiles via atexit(3p) or signal handlers. Provide a thin wrapper for the tempfile subsystem instead. Like this, the compatibility shim is fully self-contained in "reftable/system.c". Downstream users of the reftable library would have to implement their own tempfile shims by replacing "system.c" with a custom version. Signed-off-by: Patrick Steinhardt --- Makefile | 1 + reftable/stack.c | 57 +++++++++++++++++++---------------------------- reftable/system.c | 49 ++++++++++++++++++++++++++++++++++++++++ reftable/system.h | 42 +++++++++++++++++++++++++++++++++- 4 files changed, 114 insertions(+), 35 deletions(-) create mode 100644 reftable/system.c diff --git a/Makefile b/Makefile index feeed6f9321..50a79ad83fc 100644 --- a/Makefile +++ b/Makefile @@ -2722,6 +2722,7 @@ REFTABLE_OBJS += reftable/pq.o REFTABLE_OBJS += reftable/reader.o REFTABLE_OBJS += reftable/record.o REFTABLE_OBJS += reftable/stack.o +REFTABLE_OBJS += reftable/system.o REFTABLE_OBJS += reftable/tree.o REFTABLE_OBJS += reftable/writer.o diff --git a/reftable/stack.c b/reftable/stack.c index c67bdd952ca..2ac6a371516 100644 --- a/reftable/stack.c +++ b/reftable/stack.c @@ -16,7 +16,6 @@ license that can be found in the LICENSE file or at #include "reftable-record.h" #include "reftable-merged.h" #include "writer.h" -#include "tempfile.h" static int stack_try_add(struct reftable_stack *st, int (*write_table)(struct reftable_writer *wr, @@ -867,7 +866,7 @@ int reftable_addition_add(struct reftable_addition *add, struct reftable_buf tab_file_name = REFTABLE_BUF_INIT; struct reftable_buf next_name = REFTABLE_BUF_INIT; struct reftable_writer *wr = NULL; - struct tempfile *tab_file = NULL; + struct reftable_tmpfile tab_file = REFTABLE_TMPFILE_INIT; struct fd_writer writer = { .opts = &add->stack->opts, }; @@ -887,20 +886,18 @@ int reftable_addition_add(struct reftable_addition *add, if (err < 0) goto done; - tab_file = mks_tempfile(temp_tab_file_name.buf); - if (!tab_file) { - err = REFTABLE_IO_ERROR; + err = tmpfile_from_pattern(&tab_file, temp_tab_file_name.buf); + if (err < 0) goto done; - } if (add->stack->opts.default_permissions) { - if (chmod(get_tempfile_path(tab_file), + if (chmod(tab_file.path, add->stack->opts.default_permissions)) { err = REFTABLE_IO_ERROR; goto done; } } - writer.fd = get_tempfile_fd(tab_file); + writer.fd = tab_file.fd; err = reftable_writer_new(&wr, fd_writer_write, fd_writer_flush, &writer, &add->stack->opts); if (err < 0) @@ -918,11 +915,9 @@ int reftable_addition_add(struct reftable_addition *add, if (err < 0) goto done; - err = close_tempfile_gently(tab_file); - if (err < 0) { - err = REFTABLE_IO_ERROR; + err = tmpfile_close(&tab_file); + if (err < 0) goto done; - } if (wr->min_update_index < add->next_update_index) { err = REFTABLE_API_ERROR; @@ -945,11 +940,9 @@ int reftable_addition_add(struct reftable_addition *add, On windows, this relies on rand() picking a unique destination name. Maybe we should do retry loop as well? */ - err = rename_tempfile(&tab_file, tab_file_name.buf); - if (err < 0) { - err = REFTABLE_IO_ERROR; + err = tmpfile_rename(&tab_file, tab_file_name.buf); + if (err < 0) goto done; - } REFTABLE_ALLOC_GROW(add->new_tables, add->new_tables_len + 1, add->new_tables_cap); @@ -960,7 +953,7 @@ int reftable_addition_add(struct reftable_addition *add, add->new_tables[add->new_tables_len++] = reftable_buf_detach(&next_name); done: - delete_tempfile(&tab_file); + tmpfile_delete(&tab_file); reftable_buf_release(&temp_tab_file_name); reftable_buf_release(&tab_file_name); reftable_buf_release(&next_name); @@ -980,7 +973,7 @@ uint64_t reftable_stack_next_update_index(struct reftable_stack *st) static int stack_compact_locked(struct reftable_stack *st, size_t first, size_t last, struct reftable_log_expiry_config *config, - struct tempfile **tab_file_out) + struct reftable_tmpfile *tab_file_out) { struct reftable_buf next_name = REFTABLE_BUF_INIT; struct reftable_buf tab_file_path = REFTABLE_BUF_INIT; @@ -988,7 +981,7 @@ static int stack_compact_locked(struct reftable_stack *st, struct fd_writer writer= { .opts = &st->opts, }; - struct tempfile *tab_file; + struct reftable_tmpfile tab_file = REFTABLE_TMPFILE_INIT; int err = 0; err = format_name(&next_name, reftable_reader_min_update_index(st->readers[first]), @@ -1004,19 +997,17 @@ static int stack_compact_locked(struct reftable_stack *st, if (err < 0) goto done; - tab_file = mks_tempfile(tab_file_path.buf); - if (!tab_file) { - err = REFTABLE_IO_ERROR; + err = tmpfile_from_pattern(&tab_file, tab_file_path.buf); + if (err < 0) goto done; - } if (st->opts.default_permissions && - chmod(get_tempfile_path(tab_file), st->opts.default_permissions) < 0) { + chmod(tab_file.path, st->opts.default_permissions) < 0) { err = REFTABLE_IO_ERROR; goto done; } - writer.fd = get_tempfile_fd(tab_file); + writer.fd = tab_file.fd; err = reftable_writer_new(&wr, fd_writer_write, fd_writer_flush, &writer, &st->opts); if (err < 0) @@ -1030,15 +1021,15 @@ static int stack_compact_locked(struct reftable_stack *st, if (err < 0) goto done; - err = close_tempfile_gently(tab_file); + err = tmpfile_close(&tab_file); if (err < 0) goto done; *tab_file_out = tab_file; - tab_file = NULL; + tab_file = REFTABLE_TMPFILE_INIT; done: - delete_tempfile(&tab_file); + tmpfile_delete(&tab_file); reftable_writer_free(wr); reftable_buf_release(&next_name); reftable_buf_release(&tab_file_path); @@ -1171,7 +1162,7 @@ static int stack_compact_range(struct reftable_stack *st, struct reftable_buf table_name = REFTABLE_BUF_INIT; struct lock_file tables_list_lock = LOCK_INIT; struct lock_file *table_locks = NULL; - struct tempfile *new_table = NULL; + struct reftable_tmpfile new_table = REFTABLE_TMPFILE_INIT; int is_empty_table = 0, err = 0; size_t first_to_replace, last_to_replace; size_t i, nlocks = 0; @@ -1439,11 +1430,9 @@ static int stack_compact_range(struct reftable_stack *st, if (err < 0) goto done; - err = rename_tempfile(&new_table, new_table_path.buf); - if (err < 0) { - err = REFTABLE_IO_ERROR; + err = tmpfile_rename(&new_table, new_table_path.buf); + if (err < 0) goto done; - } } /* @@ -1515,7 +1504,7 @@ static int stack_compact_range(struct reftable_stack *st, rollback_lock_file(&table_locks[i]); reftable_free(table_locks); - delete_tempfile(&new_table); + tmpfile_delete(&new_table); reftable_buf_release(&new_table_name); reftable_buf_release(&new_table_path); reftable_buf_release(&tables_list_buf); diff --git a/reftable/system.c b/reftable/system.c new file mode 100644 index 00000000000..01f96f03d84 --- /dev/null +++ b/reftable/system.c @@ -0,0 +1,49 @@ +#include "system.h" +#include "basics.h" +#include "reftable-error.h" +#include "../tempfile.h" + +int tmpfile_from_pattern(struct reftable_tmpfile *out, const char *pattern) +{ + struct tempfile *tempfile; + + tempfile = mks_tempfile(pattern); + if (!tempfile) + return REFTABLE_IO_ERROR; + + out->path = tempfile->filename.buf; + out->fd = tempfile->fd; + out->priv = tempfile; + + return 0; +} + +int tmpfile_close(struct reftable_tmpfile *t) +{ + struct tempfile *tempfile = t->priv; + int ret = close_tempfile_gently(tempfile); + t->fd = -1; + if (ret < 0) + return REFTABLE_IO_ERROR; + return 0; +} + +int tmpfile_delete(struct reftable_tmpfile *t) +{ + struct tempfile *tempfile = t->priv; + int ret = delete_tempfile(&tempfile); + *t = REFTABLE_TMPFILE_INIT; + if (ret < 0) + return REFTABLE_IO_ERROR; + return 0; +} + +int tmpfile_rename(struct reftable_tmpfile *t, const char *path) +{ + struct tempfile *tempfile = t->priv; + int ret = rename_tempfile(&tempfile, path); + *t = REFTABLE_TMPFILE_INIT; + if (ret < 0) + return REFTABLE_IO_ERROR; + return 0; +} diff --git a/reftable/system.h b/reftable/system.h index 38d3534620e..858189fd55d 100644 --- a/reftable/system.h +++ b/reftable/system.h @@ -13,6 +13,46 @@ license that can be found in the LICENSE file or at #include "git-compat-util.h" #include "lockfile.h" -#include "tempfile.h" + +/* + * An implementation-specific temporary file. By making this specific to the + * implementation it becomes possible to tie temporary files into any kind of + * signal or atexit handlers for cleanup on abnormal situations. + */ +struct reftable_tmpfile { + const char *path; + int fd; + void *priv; +}; +#define REFTABLE_TMPFILE_INIT ((struct reftable_tmpfile) { .fd = -1, }) + +/* + * Create a temporary file from a pattern similar to how mkstemp(3p) would. + * The `pattern` shall not be modified. On success, the structure at `out` has + * been initialized such that it is ready for use. Returns 0 on success, a + * reftable error code on error. + */ +int tmpfile_from_pattern(struct reftable_tmpfile *out, const char *pattern); + +/* + * Close the temporary file's file descriptor without removing the file itself. + * This is a no-op in case the file has already been closed beforehand. Returns + * 0 on success, a reftable error code on error. + */ +int tmpfile_close(struct reftable_tmpfile *t); + +/* + * Close the temporary file and delete it. This is a no-op in case the file has + * already been deleted or renamed beforehand. Returns 0 on success, a reftable + * error code on error. + */ +int tmpfile_delete(struct reftable_tmpfile *t); + +/* + * Rename the temporary file to the provided path. The temporary file must be + * active. Return 0 on success, a reftable error code on error. Deactivates the + * temporary file. + */ +int tmpfile_rename(struct reftable_tmpfile *t, const char *path); #endif From patchwork Mon Nov 18 15:34:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13878789 Received: from fhigh-a6-smtp.messagingengine.com (fhigh-a6-smtp.messagingengine.com [103.168.172.157]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 815CC1C07D9 for ; Mon, 18 Nov 2024 15:34:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.157 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731944059; cv=none; b=M3IkdVHmsnIVzcdE4thIyC9+XCEJaNMwIYniKgLcL/ipH5o/5uH3G+7bB7FCdd5NukhCkPH3JnrbD0sAHCa0uD4tLcLhgxcBrPv+m4b/1BVimiJw0Z16RuxI3x+5tsad07rQQ+uLi3xeWSlU5i+CwcqUVtHvFLW0SNNJJCXASCE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731944059; c=relaxed/simple; bh=MqmRnsZu+3+hUMVTy5yh0Jrcvus3SnW1rHSyx+lQgkk=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=RgblI/bLQ3s/ofrPSQ1vFKoqNerDqoRb0RzEgOrlQ7ZAz/S/scng+O3WndrF5318/C86UyiiMckyACCEXClgjia9ZP532MD6hrANw+yjbgXjToE7K9X7vbjq/UVoINhIJW8UMmdvFVModbFnKdxvoJcXTXdTu7+GTX2rtlvya0Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=iZXCL0bk; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=QiaVM5wb; arc=none smtp.client-ip=103.168.172.157 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="iZXCL0bk"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="QiaVM5wb" Received: from phl-compute-10.internal (phl-compute-10.phl.internal [10.202.2.50]) by mailfhigh.phl.internal (Postfix) with ESMTP id CC8D5114022D; Mon, 18 Nov 2024 10:34:16 -0500 (EST) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-10.internal (MEProxy); Mon, 18 Nov 2024 10:34:16 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1731944056; x=1732030456; bh=+VNWPqLF6N LI/to6uLo4QfXE4JHBqq9EgGE4AuHO57U=; b=iZXCL0bkyTc4w1OlD0QriSJCsh EpNRKmtTHoHvT1mtSwPFZfzx0c7h6T4A7kQKLh3qC/wT0R4jsog0/qCiq1RxAZuy zcJpnY2ZGP/hxZPwdIeUatRDK7Cpn7/8aMEmtT8AdvXtA9sWVIIyOYImnGjQCKaH I5fPug4Lhhd+vP051Btn2vXHP8BtWdb3DYP8F/7KsnzhLj5P5ZhOJFcbPms3dgL2 +7zW6P9Gm2wh5jdjpjH+kT/dOYQwzm7iCh5ZVYciafkyky9F9F3QW+KHhAWWXrtH p7froVvqED2SN53rTPidV4+t3ltY4q1ghI6zLMrjrLJi6L+oxvoTeUyxyNqA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t= 1731944056; x=1732030456; bh=+VNWPqLF6NLI/to6uLo4QfXE4JHBqq9EgGE 4AuHO57U=; b=QiaVM5wbUIGWFiHXa01LZ+axtwrwyRUpA5i4iya1MlBQs4s5kno hQIJtcttCiZeZqiU9Iz9Eft0o8fwCOb/mdbhWzy+JSGnbn5c/OnyteaPK2W/FHmK C8sH5CHNSyX6xoVAnAdJHyX+l/gAv2nHvxRisEcA88BmA5/clU6EzY4G7Ks7taKR yqDm+vsS4LkSpIlEGuFpAmzIty2gZoMZG8TKvbEtscBmnTm/5Osz+LYSS1+QVKDE Dgf2tVYEtjf6PXJpj5ia2hBPosPCI2okzjvYKeoemrPYQWvEwxsLu2stobTd/6av mtQg3XshLVXISQ3i/Xh7KRfGTLvivjWhaqA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefuddrfedtgdejiecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdpuffr tefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnth hsucdlqddutddtmdenucfjughrpeffhffvvefukfhfgggtuggjsehttdertddttddvnecu hfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqe enucggtffrrghtthgvrhhnpeevkeekfffhiedtleduiefgjedttedvledvudehgfeugedu gffhueekhfejvdektdenucevlhhushhtvghrufhiiigvpedunecurfgrrhgrmhepmhgrih hlfhhrohhmpehpshesphhkshdrihhmpdhnsggprhgtphhtthhopeeipdhmohguvgepshhm thhpohhuthdprhgtphhtthhopehsuhhnshhhihhnvgesshhunhhshhhinhgvtghordgtoh hmpdhrtghpthhtohepkhgrrhhthhhikhdrudekkeesghhmrghilhdrtghomhdprhgtphht thhopehjlhhtohgslhgvrhesghhmrghilhdrtghomhdprhgtphhtthhopehgihhtsehvgh gvrhdrkhgvrhhnvghlrdhorhhgpdhrtghpthhtohepvghthhhomhhsohhnsegvugifrghr ughthhhomhhsohhnrdgtohhmpdhrtghpthhtohepghhithhsthgvrhesphhosghogidrtg homh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 18 Nov 2024 10:34:15 -0500 (EST) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 9ea93160 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 18 Nov 2024 15:33:27 +0000 (UTC) Date: Mon, 18 Nov 2024 16:34:05 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Edward Thomson , Eric Sunshine , Justin Tobler , Junio C Hamano , karthik nayak Subject: [PATCH v3 6/7] reftable/stack: drop only use of `get_locked_file_path()` Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: We've got a single callsite where we call `get_locked_file_path()`. As we're about to convert our usage of the lockfile subsystem to instead be used via a compatibility shim we'd have to implement more logic for this single callsite. While that would be okay if Git was the only supposed user of the reftable library, it's a bit more awkward when considering that we have to reimplement this functionality for every user of the library eventually. Refactor the code such that we don't call `get_locked_file_path()` anymore. Signed-off-by: Patrick Steinhardt --- reftable/stack.c | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/reftable/stack.c b/reftable/stack.c index 2ac6a371516..223d7c622d9 100644 --- a/reftable/stack.c +++ b/reftable/stack.c @@ -1493,9 +1493,15 @@ static int stack_compact_range(struct reftable_stack *st, */ for (i = 0; i < nlocks; i++) { struct lock_file *table_lock = &table_locks[i]; - char *table_path = get_locked_file_path(table_lock); - unlink(table_path); - reftable_free(table_path); + const char *lock_path = get_lock_file_path(table_lock); + + reftable_buf_reset(&table_name); + err = reftable_buf_add(&table_name, lock_path, + strlen(lock_path) - strlen(".lock")); + if (err) + continue; + + unlink(table_name.buf); } done: From patchwork Mon Nov 18 15:34:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13878791 Received: from fout-a3-smtp.messagingengine.com (fout-a3-smtp.messagingengine.com [103.168.172.146]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8A09D1C07EE for ; Mon, 18 Nov 2024 15:34:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.146 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731944062; cv=none; b=ACC2xagsb5qfyGpE6VW7h2Jfr26bMe2YSNJNNXKnr4bPNaDYPRpCzH+ec1Ok+OKj4bBzJ7TmrGctwRLU1DS+NYEavGMi6MYtvCMXWAOS4LKX4HIgqSOJZR/1zAlvpVaccTs/8aNTTTwQL2vesx7y8doJpbILWllQHLLFsv9/t2I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731944062; c=relaxed/simple; bh=eu3WG6u6M+17btjeZu9/w+FpQqIgll4Tl2SZNbUMjJE=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=JF5kL/c11YRvXBRyYMv3gGA+Zsur1NDfiMuKtUiQ5tjMafzV+kn8osR6uByZHi9GQFujQjk62xHJjrKBFoSUBvfqfTefy14rANXjjnut5yTuF1wo+rOtoYTeXNeV3s20l693G7xlJWy7jv7ZHcKAfNRf/vRG2PCGaHWc7X+huBQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=SMAuHfeB; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=RB11zRtB; arc=none smtp.client-ip=103.168.172.146 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="SMAuHfeB"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="RB11zRtB" Received: from phl-compute-03.internal (phl-compute-03.phl.internal [10.202.2.43]) by mailfout.phl.internal (Postfix) with ESMTP id 8043413805D0; Mon, 18 Nov 2024 10:34:19 -0500 (EST) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-03.internal (MEProxy); Mon, 18 Nov 2024 10:34:19 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1731944059; x=1732030459; bh=Fb+r1ZGTT9 An23xD91iurwxlU7+WaRITYVfg9k1XSlo=; b=SMAuHfeBDyFGoXK+Y9jeYxlObA KbGzJPbCacCGIrhM8e3MDRabxgd6Z+7QVjR/VfWWQfms8AG1cfCdv7HQ6DY/zFTW +5N04SMngoXpvCppuYXSmk4zOWdxmqzXOISbSaKWhxQlmkzpPJLZ9TP5unLEE7na /PKovqkMxZU3qf5o+hGghGA25t6UiVNDfTBTtE+QCafID1ut6yCYvuxFjM+MXJCw CelcsYBDqNOJZPee78IQxkX7Yufbr0pDDxhshzrgLcXfbJi9bWs7KRnecw4yg0PO eqN5LqFxoqfmV2uN2/mnAlj92NzuSU+ILv9Y3E6Fbn2CxTNGwq8NlmS4j5bA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t= 1731944059; x=1732030459; bh=Fb+r1ZGTT9An23xD91iurwxlU7+WaRITYVf g9k1XSlo=; b=RB11zRtBg9fcQzBOkNsHKIiKuXl6O7LtMOtxakaJwDPT7gtJvAG ytQuqs8T4Zc6STmt4Yh7INe8KoQNKiCV1CtGpk8su9I/shvvFgfeBj5PfjJMZGEt /fFlzSdZWDMBzd4A8Q7I8URFFMUfUARWbTQyK/IiKCG40r86QWOMNEteypXxP3Sr 87JBiIronf3epHYLFKE+XD4Zxq7Ugzu8htJXlovzsDKiybW52vE/00dEqYKxYMq3 a7TxaLJ69O64FpgyF3PxGdpTJwVa2a0vfMA2SfPgPzw8cM+eFdW3H8Q0Wv1g80A8 Csh+H1ev5xJT1sOv0cTVjYprRf6I6X96y1w== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeefuddrfedtgdejiecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdpuffr tefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnth hsucdlqddutddtmdenucfjughrpeffhffvvefukfhfgggtuggjsehttdertddttddvnecu hfhrohhmpefrrghtrhhitghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqe enucggtffrrghtthgvrhhnpefhveehhfdtvdeuueegueeuudffueeigeeujeelhedvgfeh udfhfeevjeetjeeufeenucffohhmrghinheplhhishhtrdgrshenucevlhhushhtvghruf hiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhmpdhnsggp rhgtphhtthhopeeipdhmohguvgepshhmthhpohhuthdprhgtphhtthhopehgihhtsehvgh gvrhdrkhgvrhhnvghlrdhorhhgpdhrtghpthhtohepghhithhsthgvrhesphhosghogidr tghomhdprhgtphhtthhopehkrghrthhhihhkrddukeeksehgmhgrihhlrdgtohhmpdhrtg hpthhtohepshhunhhshhhinhgvsehsuhhnshhhihhnvggtohdrtghomhdprhgtphhtthho pehjlhhtohgslhgvrhesghhmrghilhdrtghomhdprhgtphhtthhopegvthhhohhmshhonh esvggufigrrhguthhhohhmshhonhdrtghomh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 18 Nov 2024 10:34:17 -0500 (EST) Received: by vm-mail (OpenSMTPD) with ESMTPSA id c6a40eb3 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 18 Nov 2024 15:33:30 +0000 (UTC) Date: Mon, 18 Nov 2024 16:34:08 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Edward Thomson , Eric Sunshine , Justin Tobler , Junio C Hamano , karthik nayak Subject: [PATCH v3 7/7] reftable/system: provide thin wrapper for lockfile subsystem Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: We use the lockfile subsystem to write lockfiles for "tables.list". As with the tempfile subsystem, the lockfile subsystem also hooks into our infrastructure to prune stale locks via atexit(3p) or signal handlers. Furthermore, the lockfile subsystem also handles locking timeouts, which do add quite a bit of logic. Having to reimplement that in the context of Git wouldn't make a whole lot of sense, and it is quite likely that downstream users of the reftable library may have a better idea for how exactly to implement timeouts. So again, provide a thin wrapper for the lockfile subsystem instead such that the compatibility shim is fully self-contained. Signed-off-by: Patrick Steinhardt --- reftable/stack.c | 63 ++++++++++------------- reftable/system.c | 77 +++++++++++++++++++++++++++++ reftable/system.h | 45 ++++++++++++++++- t/unit-tests/lib-reftable.c | 1 + t/unit-tests/t-reftable-block.c | 1 + t/unit-tests/t-reftable-pq.c | 1 + t/unit-tests/t-reftable-readwrite.c | 1 + t/unit-tests/t-reftable-stack.c | 2 + 8 files changed, 154 insertions(+), 37 deletions(-) diff --git a/reftable/stack.c b/reftable/stack.c index 223d7c622d9..10d45e89d00 100644 --- a/reftable/stack.c +++ b/reftable/stack.c @@ -657,7 +657,7 @@ static int format_name(struct reftable_buf *dest, uint64_t min, uint64_t max) } struct reftable_addition { - struct lock_file tables_list_lock; + struct reftable_flock tables_list_lock; struct reftable_stack *stack; char **new_tables; @@ -676,10 +676,8 @@ static int reftable_stack_init_addition(struct reftable_addition *add, add->stack = st; - err = hold_lock_file_for_update_timeout(&add->tables_list_lock, - st->list_file, - LOCK_NO_DEREF, - st->opts.lock_timeout_ms); + err = flock_acquire(&add->tables_list_lock, st->list_file, + st->opts.lock_timeout_ms); if (err < 0) { if (errno == EEXIST) { err = REFTABLE_LOCK_ERROR; @@ -689,7 +687,7 @@ static int reftable_stack_init_addition(struct reftable_addition *add, goto done; } if (st->opts.default_permissions) { - if (chmod(get_lock_file_path(&add->tables_list_lock), + if (chmod(add->tables_list_lock.path, st->opts.default_permissions) < 0) { err = REFTABLE_IO_ERROR; goto done; @@ -733,7 +731,7 @@ static void reftable_addition_close(struct reftable_addition *add) add->new_tables_len = 0; add->new_tables_cap = 0; - rollback_lock_file(&add->tables_list_lock); + flock_release(&add->tables_list_lock); reftable_buf_release(&nm); } @@ -749,7 +747,6 @@ void reftable_addition_destroy(struct reftable_addition *add) int reftable_addition_commit(struct reftable_addition *add) { struct reftable_buf table_list = REFTABLE_BUF_INIT; - int lock_file_fd = get_lock_file_fd(&add->tables_list_lock); int err = 0; size_t i; @@ -767,20 +764,20 @@ int reftable_addition_commit(struct reftable_addition *add) goto done; } - err = write_in_full(lock_file_fd, table_list.buf, table_list.len); + err = write_in_full(add->tables_list_lock.fd, table_list.buf, table_list.len); reftable_buf_release(&table_list); if (err < 0) { err = REFTABLE_IO_ERROR; goto done; } - err = stack_fsync(&add->stack->opts, lock_file_fd); + err = stack_fsync(&add->stack->opts, add->tables_list_lock.fd); if (err < 0) { err = REFTABLE_IO_ERROR; goto done; } - err = commit_lock_file(&add->tables_list_lock); + err = flock_commit(&add->tables_list_lock); if (err < 0) { err = REFTABLE_IO_ERROR; goto done; @@ -1160,8 +1157,8 @@ static int stack_compact_range(struct reftable_stack *st, struct reftable_buf new_table_name = REFTABLE_BUF_INIT; struct reftable_buf new_table_path = REFTABLE_BUF_INIT; struct reftable_buf table_name = REFTABLE_BUF_INIT; - struct lock_file tables_list_lock = LOCK_INIT; - struct lock_file *table_locks = NULL; + struct reftable_flock tables_list_lock = REFTABLE_FLOCK_INIT; + struct reftable_flock *table_locks = NULL; struct reftable_tmpfile new_table = REFTABLE_TMPFILE_INIT; int is_empty_table = 0, err = 0; size_t first_to_replace, last_to_replace; @@ -1179,10 +1176,7 @@ static int stack_compact_range(struct reftable_stack *st, * Hold the lock so that we can read "tables.list" and lock all tables * which are part of the user-specified range. */ - err = hold_lock_file_for_update_timeout(&tables_list_lock, - st->list_file, - LOCK_NO_DEREF, - st->opts.lock_timeout_ms); + err = flock_acquire(&tables_list_lock, st->list_file, st->opts.lock_timeout_ms); if (err < 0) { if (errno == EEXIST) err = REFTABLE_LOCK_ERROR; @@ -1205,19 +1199,20 @@ static int stack_compact_range(struct reftable_stack *st, * older process is still busy compacting tables which are preexisting * from the point of view of the newer process. */ - REFTABLE_CALLOC_ARRAY(table_locks, last - first + 1); + REFTABLE_ALLOC_ARRAY(table_locks, last - first + 1); if (!table_locks) { err = REFTABLE_OUT_OF_MEMORY_ERROR; goto done; } + for (i = 0; i < last - first + 1; i++) + table_locks[i] = REFTABLE_FLOCK_INIT; for (i = last + 1; i > first; i--) { err = stack_filename(&table_name, st, reader_name(st->readers[i - 1])); if (err < 0) goto done; - err = hold_lock_file_for_update(&table_locks[nlocks], - table_name.buf, LOCK_NO_DEREF); + err = flock_acquire(&table_locks[nlocks], table_name.buf, 0); if (err < 0) { /* * When the table is locked already we may do a @@ -1253,7 +1248,7 @@ static int stack_compact_range(struct reftable_stack *st, * run into file descriptor exhaustion when we compress a lot * of tables. */ - err = close_lock_file_gently(&table_locks[nlocks++]); + err = flock_close(&table_locks[nlocks++]); if (err < 0) { err = REFTABLE_IO_ERROR; goto done; @@ -1265,7 +1260,7 @@ static int stack_compact_range(struct reftable_stack *st, * "tables.list" lock while compacting the locked tables. This allows * concurrent updates to the stack to proceed. */ - err = rollback_lock_file(&tables_list_lock); + err = flock_release(&tables_list_lock); if (err < 0) { err = REFTABLE_IO_ERROR; goto done; @@ -1288,10 +1283,7 @@ static int stack_compact_range(struct reftable_stack *st, * "tables.list". We'll then replace the compacted range of tables with * the new table. */ - err = hold_lock_file_for_update_timeout(&tables_list_lock, - st->list_file, - LOCK_NO_DEREF, - st->opts.lock_timeout_ms); + err = flock_acquire(&tables_list_lock, st->list_file, st->opts.lock_timeout_ms); if (err < 0) { if (errno == EEXIST) err = REFTABLE_LOCK_ERROR; @@ -1301,7 +1293,7 @@ static int stack_compact_range(struct reftable_stack *st, } if (st->opts.default_permissions) { - if (chmod(get_lock_file_path(&tables_list_lock), + if (chmod(tables_list_lock.path, st->opts.default_permissions) < 0) { err = REFTABLE_IO_ERROR; goto done; @@ -1456,7 +1448,7 @@ static int stack_compact_range(struct reftable_stack *st, goto done; } - err = write_in_full(get_lock_file_fd(&tables_list_lock), + err = write_in_full(tables_list_lock.fd, tables_list_buf.buf, tables_list_buf.len); if (err < 0) { err = REFTABLE_IO_ERROR; @@ -1464,14 +1456,14 @@ static int stack_compact_range(struct reftable_stack *st, goto done; } - err = stack_fsync(&st->opts, get_lock_file_fd(&tables_list_lock)); + err = stack_fsync(&st->opts, tables_list_lock.fd); if (err < 0) { err = REFTABLE_IO_ERROR; unlink(new_table_path.buf); goto done; } - err = commit_lock_file(&tables_list_lock); + err = flock_commit(&tables_list_lock); if (err < 0) { err = REFTABLE_IO_ERROR; unlink(new_table_path.buf); @@ -1492,12 +1484,11 @@ static int stack_compact_range(struct reftable_stack *st, * readers, so it is expected that unlinking tables may fail. */ for (i = 0; i < nlocks; i++) { - struct lock_file *table_lock = &table_locks[i]; - const char *lock_path = get_lock_file_path(table_lock); + struct reftable_flock *table_lock = &table_locks[i]; reftable_buf_reset(&table_name); - err = reftable_buf_add(&table_name, lock_path, - strlen(lock_path) - strlen(".lock")); + err = reftable_buf_add(&table_name, table_lock->path, + strlen(table_lock->path) - strlen(".lock")); if (err) continue; @@ -1505,9 +1496,9 @@ static int stack_compact_range(struct reftable_stack *st, } done: - rollback_lock_file(&tables_list_lock); + flock_release(&tables_list_lock); for (i = 0; table_locks && i < nlocks; i++) - rollback_lock_file(&table_locks[i]); + flock_release(&table_locks[i]); reftable_free(table_locks); tmpfile_delete(&new_table); diff --git a/reftable/system.c b/reftable/system.c index 01f96f03d84..adf8e4d30b8 100644 --- a/reftable/system.c +++ b/reftable/system.c @@ -1,6 +1,7 @@ #include "system.h" #include "basics.h" #include "reftable-error.h" +#include "../lockfile.h" #include "../tempfile.h" int tmpfile_from_pattern(struct reftable_tmpfile *out, const char *pattern) @@ -47,3 +48,79 @@ int tmpfile_rename(struct reftable_tmpfile *t, const char *path) return REFTABLE_IO_ERROR; return 0; } + +int flock_acquire(struct reftable_flock *l, const char *target_path, + long timeout_ms) +{ + struct lock_file *lockfile; + int err; + + lockfile = reftable_malloc(sizeof(*lockfile)); + if (!lockfile) + return REFTABLE_OUT_OF_MEMORY_ERROR; + + err = hold_lock_file_for_update_timeout(lockfile, target_path, LOCK_NO_DEREF, + timeout_ms); + if (err < 0) { + reftable_free(lockfile); + if (errno == EEXIST) + return REFTABLE_LOCK_ERROR; + return -1; + } + + l->fd = get_lock_file_fd(lockfile); + l->path = get_lock_file_path(lockfile); + l->priv = lockfile; + + return 0; +} + +int flock_close(struct reftable_flock *l) +{ + struct lock_file *lockfile = l->priv; + int ret; + + if (!lockfile) + return REFTABLE_API_ERROR; + + ret = close_lock_file_gently(lockfile); + l->fd = -1; + if (ret < 0) + return REFTABLE_IO_ERROR; + + return 0; +} + +int flock_release(struct reftable_flock *l) +{ + struct lock_file *lockfile = l->priv; + int ret; + + if (!lockfile) + return 0; + + ret = rollback_lock_file(lockfile); + reftable_free(lockfile); + *l = REFTABLE_FLOCK_INIT; + if (ret < 0) + return REFTABLE_IO_ERROR; + + return 0; +} + +int flock_commit(struct reftable_flock *l) +{ + struct lock_file *lockfile = l->priv; + int ret; + + if (!lockfile) + return REFTABLE_API_ERROR; + + ret = commit_lock_file(lockfile); + reftable_free(lockfile); + *l = REFTABLE_FLOCK_INIT; + if (ret < 0) + return REFTABLE_IO_ERROR; + + return 0; +} diff --git a/reftable/system.h b/reftable/system.h index 858189fd55d..7d5f803eeb1 100644 --- a/reftable/system.h +++ b/reftable/system.h @@ -12,7 +12,6 @@ license that can be found in the LICENSE file or at /* This header glues the reftable library to the rest of Git */ #include "git-compat-util.h" -#include "lockfile.h" /* * An implementation-specific temporary file. By making this specific to the @@ -55,4 +54,48 @@ int tmpfile_delete(struct reftable_tmpfile *t); */ int tmpfile_rename(struct reftable_tmpfile *t, const char *path); +/* + * An implementation-specific file lock. Same as with `reftable_tmpfile`, + * making this specific to the implementation makes it possible to tie this + * into signal or atexit handlers such that we know to clean up stale locks on + * abnormal exits. + */ +struct reftable_flock { + const char *path; + int fd; + void *priv; +}; +#define REFTABLE_FLOCK_INIT ((struct reftable_flock){ .fd = -1, }) + +/* + * Acquire the lock for the given target path by exclusively creating a file + * with ".lock" appended to it. If that lock exists, we wait up to `timeout_ms` + * to acquire the lock. If `timeout_ms` is 0 we don't wait, if it is negative + * we block indefinitely. + * + * Retrun 0 on success, a reftable error code on error. + */ +int flock_acquire(struct reftable_flock *l, const char *target_path, + long timeout_ms); + +/* + * Close the lockfile's file descriptor without removing the lock itself. This + * is a no-op in case the lockfile has already been closed beforehand. Returns + * 0 on success, a reftable error code on error. + */ +int flock_close(struct reftable_flock *l); + +/* + * Release the lock by unlinking the lockfile. This is a no-op in case the + * lockfile has already been released or committed beforehand. Returns 0 on + * success, a reftable error code on error. + */ +int flock_release(struct reftable_flock *l); + +/* + * Commit the lock by renaming the lockfile into place. Returns 0 on success, a + * reftable error code on error. + */ +int flock_commit(struct reftable_flock *l); + #endif diff --git a/t/unit-tests/lib-reftable.c b/t/unit-tests/lib-reftable.c index c1631f45275..d795dfb7c99 100644 --- a/t/unit-tests/lib-reftable.c +++ b/t/unit-tests/lib-reftable.c @@ -2,6 +2,7 @@ #include "test-lib.h" #include "reftable/constants.h" #include "reftable/writer.h" +#include "strbuf.h" void t_reftable_set_hash(uint8_t *p, int i, enum reftable_hash id) { diff --git a/t/unit-tests/t-reftable-block.c b/t/unit-tests/t-reftable-block.c index 13e10807dae..22040aeefa5 100644 --- a/t/unit-tests/t-reftable-block.c +++ b/t/unit-tests/t-reftable-block.c @@ -11,6 +11,7 @@ license that can be found in the LICENSE file or at #include "reftable/blocksource.h" #include "reftable/constants.h" #include "reftable/reftable-error.h" +#include "strbuf.h" static void t_ref_block_read_write(void) { diff --git a/t/unit-tests/t-reftable-pq.c b/t/unit-tests/t-reftable-pq.c index 272da05bea6..f3f8a0cdf38 100644 --- a/t/unit-tests/t-reftable-pq.c +++ b/t/unit-tests/t-reftable-pq.c @@ -9,6 +9,7 @@ license that can be found in the LICENSE file or at #include "test-lib.h" #include "reftable/constants.h" #include "reftable/pq.h" +#include "strbuf.h" static void merged_iter_pqueue_check(const struct merged_iter_pqueue *pq) { diff --git a/t/unit-tests/t-reftable-readwrite.c b/t/unit-tests/t-reftable-readwrite.c index 57896922eb1..91c881aedfa 100644 --- a/t/unit-tests/t-reftable-readwrite.c +++ b/t/unit-tests/t-reftable-readwrite.c @@ -13,6 +13,7 @@ license that can be found in the LICENSE file or at #include "reftable/reader.h" #include "reftable/reftable-error.h" #include "reftable/reftable-writer.h" +#include "strbuf.h" static const int update_index = 5; diff --git a/t/unit-tests/t-reftable-stack.c b/t/unit-tests/t-reftable-stack.c index 13fd8d8f941..b2f6c1c37e9 100644 --- a/t/unit-tests/t-reftable-stack.c +++ b/t/unit-tests/t-reftable-stack.c @@ -13,6 +13,8 @@ license that can be found in the LICENSE file or at #include "reftable/reader.h" #include "reftable/reftable-error.h" #include "reftable/stack.h" +#include "strbuf.h" +#include "tempfile.h" #include static void clear_dir(const char *dirname)