From patchwork Mon Aug 19 15:39:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13768506 Received: from fhigh2-smtp.messagingengine.com (fhigh2-smtp.messagingengine.com [103.168.172.153]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9B6127E0E4 for ; Mon, 19 Aug 2024 15:39:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.153 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724081987; cv=none; b=eirSNVqXGv6WN1IntY0gLU883BKeUOTuEIbG9r/wvCnp/pbKPNObsDNfQQlNr9KCvUVdo5nnUlBSip5TjdymQkG7mmkdJLoL6Tde4SA63qoyPBvTXtmQXdoneA2BSZ9LQrTwuj5h23RlfdxPVJNYjljxOGl3jwbqVMMYEMebpnY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724081987; c=relaxed/simple; bh=zZg84Pq3vFyO6otM3XyedC/wxEFKc58/6LQJeyGrhHc=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=nwIf4dpMaEBrlkB/DLztkBLTAbbbxFQ4M3mz2dDnFeVE8U5x2D7wjrDG1hum/jJjZrH1G2TqFe4SaaWc3n8Wxb7EAafvq138zSXyKrAkQQx4bT9w72DffAT9k3oYXLY5VYfbLHf0I/QAQfhP3w0EwwBzV1qyg+xcovqkJjajWho= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=KeIApdma; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=LQyUQtuo; arc=none smtp.client-ip=103.168.172.153 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="KeIApdma"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="LQyUQtuo" Received: from phl-compute-08.internal (phl-compute-08.nyi.internal [10.202.2.48]) by mailfhigh.nyi.internal (Postfix) with ESMTP id ACAA01151ACD; Mon, 19 Aug 2024 11:39:43 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-08.internal (MEProxy); Mon, 19 Aug 2024 11:39:43 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm1; t=1724081983; x=1724168383; bh=qcEV6S2ywj eZc2wSaO+enL19Q6SnAHCDwYLUAgcg1p4=; b=KeIApdmad/UqrcEIuuGHsPHXyt TyWDjk32dvSWucVGS9WYtr8inAm3Zgj1upyCfBvfAX9IdfjXeWbUfLTVrbWzywVJ kkFeoKBxUNqKy+jinSlolk1ma8LQ/4jzyFpfSFRWGcXZfIScg37G5Z8hunf7YLSe 158yNKA3QXIhSAqBpIfJ+AN7HDZmTk66LjPajquvBKlwr/SunT/D15wm9CI8bT9W 2FxSS+bcWnOF6my58iZ8kb9mOupQiGfYp1UhSxzXCL+2/EeIgFKFaIHhkvrPenfm PEhJtSE49aMiuIPfX9bskV/llKhYOCC3tqQrh3+Ubrc7zI5PEv0Dj83j6UEg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; t=1724081983; x=1724168383; bh=qcEV6S2ywjeZc2wSaO+enL19Q6Sn AHCDwYLUAgcg1p4=; b=LQyUQtuos640sxA+Ar4V/VZg3aYBhYOtPc1mNrp7Ya36 X0sRHYhgXx6dNCO/igNjRZpPSGuTfBGV7lqK52ZiSSrdkAjkd0IyJqOmEL5wkoHd ERwoikaUu7GnEQ0PtBirSgdd3hvEzSBE2tm9Nl2z7/3P/OB2NUEE4T9s6tmrQE2s inmM06GW/D9VZ9sYpsdsqfZ3KgFel9kwWeWAii7HNBhRDqEWDm93M3GnRof1Etpa jlsgCkcbrd8xbRPlqg3C6iOUNVafT08IyQj4+llOIEAdkFMZ0ifMGmSh0zNifjpu Q0Vy0S9RAOrkXZRf4KH2Uddvpiohf9GqeYkROFQjdA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddruddugedgleduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvve fukfhfgggtuggjsehttdertddttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhn hhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrhhnpeevkeekfffhie dtleduiefgjedttedvledvudehgfeugedugffhueekhfejvdektdenucevlhhushhtvghr ufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhmpdhnsg gprhgtphhtthhopedvpdhmohguvgepshhmthhpohhuthdprhgtphhtthhopehgihhtsehv ghgvrhdrkhgvrhhnvghlrdhorhhgpdhrtghpthhtohepphgvfhhfsehpvghffhdrnhgvth X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 19 Aug 2024 11:39:42 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id e0dd4b4b (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 19 Aug 2024 15:39:13 +0000 (UTC) Date: Mon, 19 Aug 2024 17:39:41 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Jeff King Subject: [PATCH 01/10] reftable/blocksource: drop malloc block source Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The reftable blocksource provides a generic interface to read blocks via different sources, e.g. from disk or from memory. One of the block sources is the malloc block source, which can in theory read data from memory. We nowadays also have a strbuf block source though, which provides essentially the same functionality with better ergonomics. Adapt the only remaining user of the malloc block source in our tests to use the strbuf block source, instead, and remove the now-unused malloc block source. Signed-off-by: Patrick Steinhardt --- reftable/block_test.c | 3 ++- reftable/blocksource.c | 20 -------------------- reftable/blocksource.h | 2 -- 3 files changed, 2 insertions(+), 23 deletions(-) diff --git a/reftable/block_test.c b/reftable/block_test.c index 90aecd5a7c..de8f426a42 100644 --- a/reftable/block_test.c +++ b/reftable/block_test.c @@ -34,11 +34,12 @@ static void test_block_read_write(void) struct block_reader br = { 0 }; struct block_iter it = BLOCK_ITER_INIT; int j = 0; + struct strbuf block_data = STRBUF_INIT; struct strbuf want = STRBUF_INIT; REFTABLE_CALLOC_ARRAY(block.data, block_size); block.len = block_size; - block.source = malloc_block_source(); + block_source_from_strbuf(&block.source, &block_data); block_writer_init(&bw, BLOCK_TYPE_REF, block.data, block_size, header_off, hash_size(GIT_SHA1_FORMAT_ID)); diff --git a/reftable/blocksource.c b/reftable/blocksource.c index eeed254ba9..1774853011 100644 --- a/reftable/blocksource.c +++ b/reftable/blocksource.c @@ -55,26 +55,6 @@ void block_source_from_strbuf(struct reftable_block_source *bs, bs->arg = buf; } -static void malloc_return_block(void *b, struct reftable_block *dest) -{ - if (dest->len) - memset(dest->data, 0xff, dest->len); - reftable_free(dest->data); -} - -static struct reftable_block_source_vtable malloc_vtable = { - .return_block = &malloc_return_block, -}; - -static struct reftable_block_source malloc_block_source_instance = { - .ops = &malloc_vtable, -}; - -struct reftable_block_source malloc_block_source(void) -{ - return malloc_block_source_instance; -} - struct file_block_source { uint64_t size; unsigned char *data; diff --git a/reftable/blocksource.h b/reftable/blocksource.h index 072e2727ad..659a27b406 100644 --- a/reftable/blocksource.h +++ b/reftable/blocksource.h @@ -17,6 +17,4 @@ struct reftable_block_source; void block_source_from_strbuf(struct reftable_block_source *bs, struct strbuf *buf); -struct reftable_block_source malloc_block_source(void); - #endif From patchwork Mon Aug 19 15:39:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13768507 Received: from fout7-smtp.messagingengine.com (fout7-smtp.messagingengine.com [103.168.172.150]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CBB8E85956 for ; Mon, 19 Aug 2024 15:39:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.150 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724081989; cv=none; b=A2EMmF3oaZn211rCyN8thgdxK4ct5QcsReGJYX7H6+TMuffX5OqshUX9DoKKUcQ9yHosp2aCwsOOM3M31uOZqkP8DDbyQ5H5NdZQtXbNAjwKM5UGd3Nri6Vikq/zdTY7ZzX6m8AHS9DQ+RGpBMiM8nUsLWP7hUgvtO4+FCN0CKQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724081989; c=relaxed/simple; bh=821QLf6UPKXLEbNF4c5/xuGbilO+2kvNGcVIa8jKX4M=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=Qtc1J4sAJIQFx8bV1VWEzjYHLwbHrf0NgIMGLZVMQLr1dzQYKiS9xy3T8kQSJ/HeMzwsGGp2xtsqQcR/vGpGLkvLED+D2UpYhvYnDJT1PrI8mXkpkNX5uKrxzzoDM1TPOXpS/YtCgZN7Y88l4AfvcuB31M8fSTojvdTrJzDq/YM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=inRV3Wmw; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=fx30TRgD; arc=none smtp.client-ip=103.168.172.150 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="inRV3Wmw"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="fx30TRgD" Received: from phl-compute-06.internal (phl-compute-06.nyi.internal [10.202.2.46]) by mailfout.nyi.internal (Postfix) with ESMTP id B1EBE138FF3A; Mon, 19 Aug 2024 11:39:46 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-06.internal (MEProxy); Mon, 19 Aug 2024 11:39:46 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm1; t=1724081986; x=1724168386; bh=LX7Ry5Q2vW W7NbPfwwLw+yndxqhGNGCGEHGEx1nhxYk=; b=inRV3WmwZsaIe3XVWL9dAhWyKT m95H8nsACsLy0aOzXaWOQsx4STFqZJVYBzmWaGCS2qWjD1YOBMEoYfEv8GPuuvbo kyh79O9H8LMN8eL8N47NGM9bVToes1/2axh/K9CaHJ1M56MFLB8Uyh5AohjEjB/q Z+lrvl3cEJHZCk799LRmDA4kT2bGij5mrzgUIR0bAM/MdPVC6B/4pf1fkNnB/UEg c/U5jr7ymjq8crWVSClNj4QDynhp0ZmbiZ4SgRfs0QajDxt4B29rqK3K3SiriDX9 1l/ggrZWj5aTgvnKCKZXG20X0bDesn4csgW8oMg4jSS0/JPz4ACItYIn+ONQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; t=1724081986; x=1724168386; bh=LX7Ry5Q2vWW7NbPfwwLw+yndxqhG NGCGEHGEx1nhxYk=; b=fx30TRgD9j6v8KQkW/BWcps9lPlGXdapdyTrakQ0dp9t GLgtEaeKjIKiChqHbIgZeDqUnhlE2MGCE9utXaO4mmGq6qWJHBkk086K2WCDeByw eBAnrsaYoAtrgJRBEYr3jdxwbT/PrrWUsCbFJ7PX4+sWQICjJgSYwu5+RVJGUq+2 Q9ZZ1ENOFd1oamR/BfdUTWuyEJd61xHT2S1YgcjTlUGPuhC3A+Uh0RhofG1oZEW6 Yx8j06Fy0B6bH0y9kQbTv2qET9ctA4LfrH9Q1dKaxPfBCi5oAXRgNiDjQMVFCgM4 kW0NVY78gFaGjyD1p2Q215yfpkiGFRfLPpwSv10j0w== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddruddugedgledvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvve fukfhfgggtuggjsehttdertddttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhn hhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrhhnpeevkeekfffhie dtleduiefgjedttedvledvudehgfeugedugffhueekhfejvdektdenucevlhhushhtvghr ufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhmpdhnsg gprhgtphhtthhopedvpdhmohguvgepshhmthhpohhuthdprhgtphhtthhopehpvghffhes phgvfhhfrdhnvghtpdhrtghpthhtohepghhithesvhhgvghrrdhkvghrnhgvlhdrohhrgh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 19 Aug 2024 11:39:45 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id d3bc7aa6 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 19 Aug 2024 15:39:16 +0000 (UTC) Date: Mon, 19 Aug 2024 17:39:44 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Jeff King Subject: [PATCH 02/10] reftable/stack: inline `stack_compact_range_stats()` Message-ID: <3c0cf2bf46f4106d7c0b30c863946262639b2351.1724080006.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The only difference between `stack_compact_range_stats()` and `stack_compact_range()` is that the former updates stats on failure, whereas the latter doesn't. There are no callers anymore that do not want their stats updated though, making the indirection unnecessary. Inline the stat updates into `stack_compact_range()`. Signed-off-by: Patrick Steinhardt --- reftable/stack.c | 16 ++++------------ 1 file changed, 4 insertions(+), 12 deletions(-) diff --git a/reftable/stack.c b/reftable/stack.c index d3a95d2f1d..891ea971b7 100644 --- a/reftable/stack.c +++ b/reftable/stack.c @@ -1328,17 +1328,9 @@ static int stack_compact_range(struct reftable_stack *st, strbuf_release(&table_name); free_names(names); - return err; -} - -static int stack_compact_range_stats(struct reftable_stack *st, - size_t first, size_t last, - struct reftable_log_expiry_config *config, - unsigned int flags) -{ - int err = stack_compact_range(st, first, last, config, flags); if (err == REFTABLE_LOCK_ERROR) st->stats.failures++; + return err; } @@ -1346,7 +1338,7 @@ int reftable_stack_compact_all(struct reftable_stack *st, struct reftable_log_expiry_config *config) { size_t last = st->merged->readers_len ? st->merged->readers_len - 1 : 0; - return stack_compact_range_stats(st, 0, last, config, 0); + return stack_compact_range(st, 0, last, config, 0); } static int segment_size(struct segment *s) @@ -1452,8 +1444,8 @@ int reftable_stack_auto_compact(struct reftable_stack *st) st->opts.auto_compaction_factor); reftable_free(sizes); if (segment_size(&seg) > 0) - return stack_compact_range_stats(st, seg.start, seg.end - 1, - NULL, STACK_COMPACT_RANGE_BEST_EFFORT); + return stack_compact_range(st, seg.start, seg.end - 1, + NULL, STACK_COMPACT_RANGE_BEST_EFFORT); return 0; } From patchwork Mon Aug 19 15:39:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13768508 Received: from fhigh2-smtp.messagingengine.com (fhigh2-smtp.messagingengine.com [103.168.172.153]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A57E961FD7 for ; Mon, 19 Aug 2024 15:39:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.153 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724081993; cv=none; b=GNtVrHZEhHxDTFJphw3uJYVmZxyaHBYmVWJMgUxHvJIna1vsQpKOCR44c4Ke6YZB2j5NLbqyR0kGDAxjwXilP5x1vb641BnHjqVSi83t2D61HTwuYXxTH0ELPaVeCXE2K3NVAZy1clj9EHcOUAerdgQJzJErZ20+FUnADir/B7M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724081993; c=relaxed/simple; bh=tCg52VtFRQWbunFi4Tsofyz5VMuGgGTX6wi1vXBvagg=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=XFS1ynXEjtZY5e528WSx+2cwbozmNahfIDgbhrD5Co/HyDgSAThTokDsFQN+rz5VC/VQjQVDUWCBx88S3ok/ralQ1CgMQ3aW4Q7xYZaYFCja29SglLJzgqk70aUdF2byd8PS/lUYDuXatLX7uufl+m/9x+LmRatkSjfBRs2p7ro= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=CaEHs6/o; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=ORwJEpyG; arc=none smtp.client-ip=103.168.172.153 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="CaEHs6/o"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="ORwJEpyG" Received: from phl-compute-01.internal (phl-compute-01.nyi.internal [10.202.2.41]) by mailfhigh.nyi.internal (Postfix) with ESMTP id AF9951151ACE; Mon, 19 Aug 2024 11:39:50 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-01.internal (MEProxy); Mon, 19 Aug 2024 11:39:50 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm1; t=1724081990; x=1724168390; bh=II6yM6lotm hTAc21eui0N04Vo0NJEltb23pCtMjczwE=; b=CaEHs6/oSBih3/HsD2v5siRUTg zDhR68ZMjLYeeFu8RKjcLol4eDsbZUfHkl8ZXc1v2O4Zc7aXvRnE8iFA0XcRFUzx I18Et+YSrEF3sPlKk18WeepnuovSF0EJiw7KVTLm8RA6TBqu2iW7VrGHZoy7uyl5 mhhQwI+fqlmaJ3NeD+yOigN9qnWgRhtP0XqmHmbsIK/jzDzDxXkUTvIfXg0o6+hQ 11I7ScjtfjKqKSeUpWNB3FKjFvOuj4ej1zPji0CjwYXujQlqZA633uN8KKvBasiT /taOxYR2TCGKQ4Vby5sZeVN60diw05YM1cQZlj7p7Klb78nX8XDKlzAmnovw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; t=1724081990; x=1724168390; bh=II6yM6lotmhTAc21eui0N04Vo0NJ Eltb23pCtMjczwE=; b=ORwJEpyGT1f3aS36K/XKBlYWxLs98C9mPH6/L/FX8gyY a89MjFjbVw3QYonRvhyFHvNenQGP1TLYj086uNa86Y2hFiKZNBH+pCH6yCB3PyNP gAg/QBd9A7f1RFzitcf34K+z6BUBwbvKG8VQ/M9igYhanXOdSIkvrZSW+gLbO9VR OpRKJIYm+DBTEf+5/y2w7J2M8n228AkiC8+AFpwlzTN9afQdkI9Pc7Eg+eSZdxil HiDohaUNLBv4mf1vdL2PFihzWumDW1fXHlFltRuElFaEYeOTMRs98bHXe8Byegs/ Ei3oeLjL6IDLJTot7Aub+oVtwfK3u5dgylr5OU75Fw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddruddugedgledvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvve fukfhfgggtuggjsehttdertddttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhn hhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrhhnpeevkeekfffhie dtleduiefgjedttedvledvudehgfeugedugffhueekhfejvdektdenucevlhhushhtvghr ufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhmpdhnsg gprhgtphhtthhopedvpdhmohguvgepshhmthhpohhuthdprhgtphhtthhopehgihhtsehv ghgvrhdrkhgvrhhnvghlrdhorhhgpdhrtghpthhtohepphgvfhhfsehpvghffhdrnhgvth X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 19 Aug 2024 11:39:49 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 541d6804 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 19 Aug 2024 15:39:20 +0000 (UTC) Date: Mon, 19 Aug 2024 17:39:47 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Jeff King Subject: [PATCH 03/10] reftable/reader: rename `reftable_new_reader()` Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Rename the `reftable_new_reader()` function to `reftable_reader_new()` to match our coding guidelines. Signed-off-by: Patrick Steinhardt --- reftable/reader.c | 4 ++-- reftable/reader.h | 2 +- reftable/readwrite_test.c | 6 +++--- reftable/reftable-reader.h | 4 ++-- reftable/stack.c | 4 ++-- t/helper/test-reftable.c | 2 +- t/unit-tests/t-reftable-merged.c | 6 +++--- 7 files changed, 14 insertions(+), 14 deletions(-) diff --git a/reftable/reader.c b/reftable/reader.c index 082cf00b60..ea4fdea90b 100644 --- a/reftable/reader.c +++ b/reftable/reader.c @@ -637,7 +637,7 @@ void reader_close(struct reftable_reader *r) FREE_AND_NULL(r->name); } -int reftable_new_reader(struct reftable_reader **p, +int reftable_reader_new(struct reftable_reader **p, struct reftable_block_source *src, char const *name) { struct reftable_reader *rd = reftable_calloc(1, sizeof(*rd)); @@ -786,7 +786,7 @@ int reftable_reader_print_blocks(const char *tablename) if (err < 0) goto done; - err = reftable_new_reader(&r, &src, tablename); + err = reftable_reader_new(&r, &src, tablename); if (err < 0) goto done; diff --git a/reftable/reader.h b/reftable/reader.h index a2c204d523..d8cbfd6404 100644 --- a/reftable/reader.h +++ b/reftable/reader.h @@ -52,7 +52,7 @@ struct reftable_reader { struct reftable_reader_offsets log_offsets; }; -int init_reader(struct reftable_reader *r, struct reftable_block_source *source, +int reader_init(struct reftable_reader *r, struct reftable_block_source *source, const char *name); void reader_close(struct reftable_reader *r); const char *reader_name(struct reftable_reader *r); diff --git a/reftable/readwrite_test.c b/reftable/readwrite_test.c index f411abfe9c..101cdb5cd6 100644 --- a/reftable/readwrite_test.c +++ b/reftable/readwrite_test.c @@ -646,7 +646,7 @@ static void test_write_empty_table(void) block_source_from_strbuf(&source, &buf); - err = reftable_new_reader(&rd, &source, "filename"); + err = reftable_reader_new(&rd, &source, "filename"); EXPECT_ERR(err); reftable_reader_init_ref_iterator(rd, &it); @@ -850,7 +850,7 @@ static void test_write_multiple_indices(void) EXPECT(stats->log_stats.index_offset > 0); block_source_from_strbuf(&source, &writer_buf); - err = reftable_new_reader(&reader, &source, "filename"); + err = reftable_reader_new(&reader, &source, "filename"); EXPECT_ERR(err); /* @@ -907,7 +907,7 @@ static void test_write_multi_level_index(void) EXPECT(stats->ref_stats.max_index_level == 2); block_source_from_strbuf(&source, &writer_buf); - err = reftable_new_reader(&reader, &source, "filename"); + err = reftable_reader_new(&reader, &source, "filename"); EXPECT_ERR(err); /* diff --git a/reftable/reftable-reader.h b/reftable/reftable-reader.h index 69621c5b0f..8a05c84685 100644 --- a/reftable/reftable-reader.h +++ b/reftable/reftable-reader.h @@ -23,14 +23,14 @@ /* The reader struct is a handle to an open reftable file. */ struct reftable_reader; -/* reftable_new_reader opens a reftable for reading. If successful, +/* reftable_reader_new opens a reftable for reading. If successful, * returns 0 code and sets pp. The name is used for creating a * stack. Typically, it is the basename of the file. The block source * `src` is owned by the reader, and is closed on calling * reftable_reader_destroy(). On error, the block source `src` is * closed as well. */ -int reftable_new_reader(struct reftable_reader **pp, +int reftable_reader_new(struct reftable_reader **pp, struct reftable_block_source *src, const char *name); /* Initialize a reftable iterator for reading refs. */ diff --git a/reftable/stack.c b/reftable/stack.c index 891ea971b7..c72435b059 100644 --- a/reftable/stack.c +++ b/reftable/stack.c @@ -258,7 +258,7 @@ static int reftable_stack_reload_once(struct reftable_stack *st, if (err < 0) goto done; - err = reftable_new_reader(&rd, &src, name); + err = reftable_reader_new(&rd, &src, name); if (err < 0) goto done; } @@ -1532,7 +1532,7 @@ static void remove_maybe_stale_table(struct reftable_stack *st, uint64_t max, if (err < 0) goto done; - err = reftable_new_reader(&rd, &src, name); + err = reftable_reader_new(&rd, &src, name); if (err < 0) goto done; diff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c index c1942156b5..87c2f42aae 100644 --- a/t/helper/test-reftable.c +++ b/t/helper/test-reftable.c @@ -139,7 +139,7 @@ static int dump_reftable(const char *tablename) if (err < 0) goto done; - err = reftable_new_reader(&r, &src, tablename); + err = reftable_reader_new(&r, &src, tablename); if (err < 0) goto done; diff --git a/t/unit-tests/t-reftable-merged.c b/t/unit-tests/t-reftable-merged.c index 93345c6c8b..8f51aca1b6 100644 --- a/t/unit-tests/t-reftable-merged.c +++ b/t/unit-tests/t-reftable-merged.c @@ -102,7 +102,7 @@ merged_table_from_records(struct reftable_ref_record **refs, write_test_table(&buf[i], refs[i], sizes[i]); block_source_from_strbuf(&(*source)[i], &buf[i]); - err = reftable_new_reader(&(*readers)[i], &(*source)[i], + err = reftable_reader_new(&(*readers)[i], &(*source)[i], "name"); check(!err); } @@ -277,7 +277,7 @@ merged_table_from_log_records(struct reftable_log_record **logs, write_test_log_table(&buf[i], logs[i], sizes[i], i + 1); block_source_from_strbuf(&(*source)[i], &buf[i]); - err = reftable_new_reader(&(*readers)[i], &(*source)[i], + err = reftable_reader_new(&(*readers)[i], &(*source)[i], "name"); check(!err); } @@ -426,7 +426,7 @@ static void t_default_write_opts(void) block_source_from_strbuf(&source, &buf); - err = reftable_new_reader(&rd, &source, "filename"); + err = reftable_reader_new(&rd, &source, "filename"); check(!err); hash_id = reftable_reader_hash_id(rd); From patchwork Mon Aug 19 15:39:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13768509 Received: from fhigh2-smtp.messagingengine.com (fhigh2-smtp.messagingengine.com [103.168.172.153]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A83E961FD7 for ; Mon, 19 Aug 2024 15:39:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.153 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724081999; cv=none; b=INbRADpIQjs38jBQv4e74Hka8feL5auJ4VPEhtp438Z3iqflmM1fpBCTDjNlsFNW3H34DLlx6GOGt3WGmnFrncVTjF6gniaaLx4Z9Vq0AloONnSy1ugUoxbZ+vYLDeG/QXAL9v0CVwNvJhdb0jKHw/UeNRkSnk6GFzqVxSmJowc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724081999; c=relaxed/simple; bh=+HcxEy4aoEU84kakcmd8xbN5ovoPRYckyTfkiwvd6nk=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=CEhCfqFIFDQaX8Fk/TLq0282fi6tr1EVEHfIVz4zlpk6E6C3mtr61rmsWsiwhgknvSBbXqCYpTjciAat+o+CqQVsWgpP4VzD5ADmju9Ffc9HPxJtE0pHdmMnfq9UIpRQ/i5+O9Jn7/JPJ1Lee0FdtpQp8GwHpy3rHXjPfyqMMbE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=hn+aANIy; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=GSYvWVR6; arc=none smtp.client-ip=103.168.172.153 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="hn+aANIy"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="GSYvWVR6" Received: from phl-compute-02.internal (phl-compute-02.nyi.internal [10.202.2.42]) by mailfhigh.nyi.internal (Postfix) with ESMTP id B364F1151A97; Mon, 19 Aug 2024 11:39:56 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-02.internal (MEProxy); Mon, 19 Aug 2024 11:39:56 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm1; t=1724081996; x=1724168396; bh=hO795mKdJJ qsrWeMFzNh+llv07v5LW2bwqpGVsA7mPU=; b=hn+aANIysp+S0nmYXN1RxTpCmn 6ybx9JbOfNbukGReztlZzwYcZ6bCV+xuaHZFwjTQ9iP6Gjb5cIAYCWYG79ztNSJJ dLbWLq3RneMXLqhrYIl8m3DZeUCoO1OF3XQaUgj55HWclclpM3x1WM38W2I+4t0b TEWda5tmcp8FWDDqhAwAcRSqUW3APAhrl7BcQaqSOPCjj8wDmUk+bpP+T08gc3oT LBO6xPybObv7xgkHz7CA4KSdAXe0TQMPUZddr6XBDDTga9VRXePne6sukKz/YOmH oNiiMEvI0inGPZxydmOBqh/2mf839+h64GJjXIa4tS206LGJQmU0Wuqmwy5Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; t=1724081996; x=1724168396; bh=hO795mKdJJqsrWeMFzNh+llv07v5 LW2bwqpGVsA7mPU=; b=GSYvWVR6r1XJw363R/f+Kuiutl+zPhpn0aUy0/8QYgNA UC6X+++JZs2cFliK/AlijLk64DzVAqZ38DbWnEF4ccKMsOI+qMT8bIhctEGC7YCO 80xjxXwk95vIsPjMF/UGHz9XPLvPXIPWYLVI75dXWbF97vrijU5HVKO6A3ddm3RN Sb+HcM/mbfr74EP1BL83JNAlVmTerjMm9v3CT/0qIC7vgF6zv/qvz+c/w9IWTYtD INZRJeduBmFVBDdsnZDqMP3RkMECTMU3k/nGfYt5ki0l6tCava2ID0+2r6nxalPY kxU2rt2v5Tx19ki3ZmePGE+BKo+caJGap8KuI8LA5Q== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddruddugedgledvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvve fukfhfgggtuggjsehttdertddttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhn hhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrhhnpedtteduffektd ffledvgefhuedvkeeiueehiedugfejudffgeehffelleekteejtdenucffohhmrghinhep ohgsjhgpohhffhhsvghtshdrihhsnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrg hmpehmrghilhhfrhhomhepphhssehpkhhsrdhimhdpnhgspghrtghpthhtohepvddpmhho uggvpehsmhhtphhouhhtpdhrtghpthhtohepphgvfhhfsehpvghffhdrnhgvthdprhgtph htthhopehgihhtsehvghgvrhdrkhgvrhhnvghlrdhorhhg X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 19 Aug 2024 11:39:55 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id a837db9e (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 19 Aug 2024 15:39:26 +0000 (UTC) Date: Mon, 19 Aug 2024 17:39:54 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Jeff King Subject: [PATCH 04/10] reftable/reader: inline `init_reader()` Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Most users use an allocated version of the `reftable_reader`, except for some tests. We are about to convert the reader to become refcounted though, and providing the ability to keep a reader on the stack makes this conversion harder than necessary. Update the tests to use `reftable_reader_new()` instead to prepare for this change. Signed-off-by: Patrick Steinhardt --- reftable/reader.c | 122 +++++++++++++++++++------------------- reftable/reader.h | 2 - reftable/readwrite_test.c | 73 ++++++++++++----------- 3 files changed, 98 insertions(+), 99 deletions(-) diff --git a/reftable/reader.c b/reftable/reader.c index ea4fdea90b..9239679ad9 100644 --- a/reftable/reader.c +++ b/reftable/reader.c @@ -162,58 +162,6 @@ static int parse_footer(struct reftable_reader *r, uint8_t *footer, return err; } -int init_reader(struct reftable_reader *r, struct reftable_block_source *source, - const char *name) -{ - struct reftable_block footer = { NULL }; - struct reftable_block header = { NULL }; - int err = 0; - uint64_t file_size = block_source_size(source); - - /* Need +1 to read type of first block. */ - uint32_t read_size = header_size(2) + 1; /* read v2 because it's larger. */ - memset(r, 0, sizeof(struct reftable_reader)); - - if (read_size > file_size) { - err = REFTABLE_FORMAT_ERROR; - goto done; - } - - err = block_source_read_block(source, &header, 0, read_size); - if (err != read_size) { - err = REFTABLE_IO_ERROR; - goto done; - } - - if (memcmp(header.data, "REFT", 4)) { - err = REFTABLE_FORMAT_ERROR; - goto done; - } - r->version = header.data[4]; - if (r->version != 1 && r->version != 2) { - err = REFTABLE_FORMAT_ERROR; - goto done; - } - - r->size = file_size - footer_size(r->version); - r->source = *source; - r->name = xstrdup(name); - r->hash_id = 0; - - err = block_source_read_block(source, &footer, r->size, - footer_size(r->version)); - if (err != footer_size(r->version)) { - err = REFTABLE_IO_ERROR; - goto done; - } - - err = parse_footer(r, footer.data, header.data); -done: - reftable_block_done(&footer); - reftable_block_done(&header); - return err; -} - struct table_iter { struct reftable_reader *r; uint8_t typ; @@ -637,16 +585,68 @@ void reader_close(struct reftable_reader *r) FREE_AND_NULL(r->name); } -int reftable_reader_new(struct reftable_reader **p, - struct reftable_block_source *src, char const *name) +int reftable_reader_new(struct reftable_reader **out, + struct reftable_block_source *source, char const *name) { - struct reftable_reader *rd = reftable_calloc(1, sizeof(*rd)); - int err = init_reader(rd, src, name); - if (err == 0) { - *p = rd; - } else { - block_source_close(src); - reftable_free(rd); + struct reftable_block footer = { 0 }; + struct reftable_block header = { 0 }; + struct reftable_reader *r; + uint64_t file_size = block_source_size(source); + uint32_t read_size; + int err; + + REFTABLE_CALLOC_ARRAY(r, 1); + + /* + * We need one extra byte to read the type of first block. We also + * pretend to always be reading v2 of the format because it is larger. + */ + read_size = header_size(2) + 1; + if (read_size > file_size) { + err = REFTABLE_FORMAT_ERROR; + goto done; + } + + err = block_source_read_block(source, &header, 0, read_size); + if (err != read_size) { + err = REFTABLE_IO_ERROR; + goto done; + } + + if (memcmp(header.data, "REFT", 4)) { + err = REFTABLE_FORMAT_ERROR; + goto done; + } + r->version = header.data[4]; + if (r->version != 1 && r->version != 2) { + err = REFTABLE_FORMAT_ERROR; + goto done; + } + + r->size = file_size - footer_size(r->version); + r->source = *source; + r->name = xstrdup(name); + r->hash_id = 0; + + err = block_source_read_block(source, &footer, r->size, + footer_size(r->version)); + if (err != footer_size(r->version)) { + err = REFTABLE_IO_ERROR; + goto done; + } + + err = parse_footer(r, footer.data, header.data); + if (err) + goto done; + + *out = r; + +done: + reftable_block_done(&footer); + reftable_block_done(&header); + if (err) { + reftable_free(r); + block_source_close(source); } return err; } diff --git a/reftable/reader.h b/reftable/reader.h index d8cbfd6404..762cd6de66 100644 --- a/reftable/reader.h +++ b/reftable/reader.h @@ -52,8 +52,6 @@ struct reftable_reader { struct reftable_reader_offsets log_offsets; }; -int reader_init(struct reftable_reader *r, struct reftable_block_source *source, - const char *name); void reader_close(struct reftable_reader *r); const char *reader_name(struct reftable_reader *r); diff --git a/reftable/readwrite_test.c b/reftable/readwrite_test.c index 101cdb5cd6..2f2ff787b2 100644 --- a/reftable/readwrite_test.c +++ b/reftable/readwrite_test.c @@ -195,7 +195,7 @@ static void test_log_write_read(void) struct reftable_log_record log = { NULL }; int n; struct reftable_iterator it = { NULL }; - struct reftable_reader rd = { NULL }; + struct reftable_reader *reader; struct reftable_block_source source = { NULL }; struct strbuf buf = STRBUF_INIT; struct reftable_writer *w = @@ -236,10 +236,10 @@ static void test_log_write_read(void) block_source_from_strbuf(&source, &buf); - err = init_reader(&rd, &source, "file.log"); + err = reftable_reader_new(&reader, &source, "file.log"); EXPECT_ERR(err); - reftable_reader_init_ref_iterator(&rd, &it); + reftable_reader_init_ref_iterator(reader, &it); err = reftable_iterator_seek_ref(&it, names[N - 1]); EXPECT_ERR(err); @@ -254,7 +254,7 @@ static void test_log_write_read(void) reftable_iterator_destroy(&it); reftable_ref_record_release(&ref); - reftable_reader_init_log_iterator(&rd, &it); + reftable_reader_init_log_iterator(reader, &it); err = reftable_iterator_seek_log(&it, ""); EXPECT_ERR(err); @@ -279,7 +279,7 @@ static void test_log_write_read(void) /* cleanup. */ strbuf_release(&buf); free_names(names); - reader_close(&rd); + reftable_reader_free(reader); } static void test_log_zlib_corruption(void) @@ -288,7 +288,7 @@ static void test_log_zlib_corruption(void) .block_size = 256, }; struct reftable_iterator it = { 0 }; - struct reftable_reader rd = { 0 }; + struct reftable_reader *reader; struct reftable_block_source source = { 0 }; struct strbuf buf = STRBUF_INIT; struct reftable_writer *w = @@ -331,18 +331,18 @@ static void test_log_zlib_corruption(void) block_source_from_strbuf(&source, &buf); - err = init_reader(&rd, &source, "file.log"); + err = reftable_reader_new(&reader, &source, "file.log"); EXPECT_ERR(err); - reftable_reader_init_log_iterator(&rd, &it); + reftable_reader_init_log_iterator(reader, &it); err = reftable_iterator_seek_log(&it, "refname"); EXPECT(err == REFTABLE_ZLIB_ERROR); reftable_iterator_destroy(&it); /* cleanup. */ + reftable_reader_free(reader); strbuf_release(&buf); - reader_close(&rd); } static void test_table_read_write_sequential(void) @@ -352,7 +352,7 @@ static void test_table_read_write_sequential(void) int N = 50; struct reftable_iterator it = { NULL }; struct reftable_block_source source = { NULL }; - struct reftable_reader rd = { NULL }; + struct reftable_reader *reader; int err = 0; int j = 0; @@ -360,10 +360,10 @@ static void test_table_read_write_sequential(void) block_source_from_strbuf(&source, &buf); - err = init_reader(&rd, &source, "file.ref"); + err = reftable_reader_new(&reader, &source, "file.ref"); EXPECT_ERR(err); - reftable_reader_init_ref_iterator(&rd, &it); + reftable_reader_init_ref_iterator(reader, &it); err = reftable_iterator_seek_ref(&it, ""); EXPECT_ERR(err); @@ -381,11 +381,11 @@ static void test_table_read_write_sequential(void) reftable_ref_record_release(&ref); } EXPECT(j == N); + reftable_iterator_destroy(&it); + reftable_reader_free(reader); strbuf_release(&buf); free_names(names); - - reader_close(&rd); } static void test_table_write_small_table(void) @@ -404,7 +404,7 @@ static void test_table_read_api(void) char **names; struct strbuf buf = STRBUF_INIT; int N = 50; - struct reftable_reader rd = { NULL }; + struct reftable_reader *reader; struct reftable_block_source source = { NULL }; int err; int i; @@ -415,10 +415,10 @@ static void test_table_read_api(void) block_source_from_strbuf(&source, &buf); - err = init_reader(&rd, &source, "file.ref"); + err = reftable_reader_new(&reader, &source, "file.ref"); EXPECT_ERR(err); - reftable_reader_init_ref_iterator(&rd, &it); + reftable_reader_init_ref_iterator(reader, &it); err = reftable_iterator_seek_ref(&it, names[0]); EXPECT_ERR(err); @@ -431,7 +431,7 @@ static void test_table_read_api(void) } reftable_iterator_destroy(&it); reftable_free(names); - reader_close(&rd); + reftable_reader_free(reader); strbuf_release(&buf); } @@ -440,7 +440,7 @@ static void test_table_read_write_seek(int index, int hash_id) char **names; struct strbuf buf = STRBUF_INIT; int N = 50; - struct reftable_reader rd = { NULL }; + struct reftable_reader *reader; struct reftable_block_source source = { NULL }; int err; int i = 0; @@ -453,18 +453,18 @@ static void test_table_read_write_seek(int index, int hash_id) block_source_from_strbuf(&source, &buf); - err = init_reader(&rd, &source, "file.ref"); + err = reftable_reader_new(&reader, &source, "file.ref"); EXPECT_ERR(err); - EXPECT(hash_id == reftable_reader_hash_id(&rd)); + EXPECT(hash_id == reftable_reader_hash_id(reader)); if (!index) { - rd.ref_offsets.index_offset = 0; + reader->ref_offsets.index_offset = 0; } else { - EXPECT(rd.ref_offsets.index_offset > 0); + EXPECT(reader->ref_offsets.index_offset > 0); } for (i = 1; i < N; i++) { - reftable_reader_init_ref_iterator(&rd, &it); + reftable_reader_init_ref_iterator(reader, &it); err = reftable_iterator_seek_ref(&it, names[i]); EXPECT_ERR(err); err = reftable_iterator_next_ref(&it, &ref); @@ -480,7 +480,7 @@ static void test_table_read_write_seek(int index, int hash_id) strbuf_addstr(&pastLast, names[N - 1]); strbuf_addstr(&pastLast, "/"); - reftable_reader_init_ref_iterator(&rd, &it); + reftable_reader_init_ref_iterator(reader, &it); err = reftable_iterator_seek_ref(&it, pastLast.buf); if (err == 0) { struct reftable_ref_record ref = { NULL }; @@ -498,7 +498,7 @@ static void test_table_read_write_seek(int index, int hash_id) reftable_free(names[i]); } reftable_free(names); - reader_close(&rd); + reftable_reader_free(reader); } static void test_table_read_write_seek_linear(void) @@ -530,7 +530,7 @@ static void test_table_refs_for(int indexed) int i = 0; int n; int err; - struct reftable_reader rd; + struct reftable_reader *reader; struct reftable_block_source source = { NULL }; struct strbuf buf = STRBUF_INIT; @@ -579,18 +579,18 @@ static void test_table_refs_for(int indexed) block_source_from_strbuf(&source, &buf); - err = init_reader(&rd, &source, "file.ref"); + err = reftable_reader_new(&reader, &source, "file.ref"); EXPECT_ERR(err); if (!indexed) { - rd.obj_offsets.is_present = 0; + reader->obj_offsets.is_present = 0; } - reftable_reader_init_ref_iterator(&rd, &it); + reftable_reader_init_ref_iterator(reader, &it); err = reftable_iterator_seek_ref(&it, ""); EXPECT_ERR(err); reftable_iterator_destroy(&it); - err = reftable_reader_refs_for(&rd, &it, want_hash); + err = reftable_reader_refs_for(reader, &it, want_hash); EXPECT_ERR(err); j = 0; @@ -611,7 +611,7 @@ static void test_table_refs_for(int indexed) strbuf_release(&buf); free_names(want_names); reftable_iterator_destroy(&it); - reader_close(&rd); + reftable_reader_free(reader); } static void test_table_refs_for_no_index(void) @@ -928,11 +928,11 @@ static void test_corrupt_table_empty(void) { struct strbuf buf = STRBUF_INIT; struct reftable_block_source source = { NULL }; - struct reftable_reader rd = { NULL }; + struct reftable_reader *reader; int err; block_source_from_strbuf(&source, &buf); - err = init_reader(&rd, &source, "file.log"); + err = reftable_reader_new(&reader, &source, "file.log"); EXPECT(err == REFTABLE_FORMAT_ERROR); } @@ -941,13 +941,14 @@ static void test_corrupt_table(void) uint8_t zeros[1024] = { 0 }; struct strbuf buf = STRBUF_INIT; struct reftable_block_source source = { NULL }; - struct reftable_reader rd = { NULL }; + struct reftable_reader *reader; int err; strbuf_add(&buf, zeros, sizeof(zeros)); block_source_from_strbuf(&source, &buf); - err = init_reader(&rd, &source, "file.log"); + err = reftable_reader_new(&reader, &source, "file.log"); EXPECT(err == REFTABLE_FORMAT_ERROR); + strbuf_release(&buf); } From patchwork Mon Aug 19 15:39:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13768510 Received: from fhigh2-smtp.messagingengine.com (fhigh2-smtp.messagingengine.com [103.168.172.153]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A2C2413BAC6 for ; Mon, 19 Aug 2024 15:40:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.153 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724082002; cv=none; b=WMVLXwdMPuRIA98G3QozaUcwl8jPSzUDfOJQkW1zYLzljxd0XfJaQMuFEDMHWyImihOcke70TBO3vSdUUUQPq7jhGz3UbVQmQedec9yQy26GAeXhx349kzOXwxFzU6ZONcTp1DOe9vTnIYzQT/JY8v1hR93NYCIJSDc/yz907Ug= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724082002; c=relaxed/simple; bh=c7ri+KL6g60uB4uCdWTqUduOOezqvYG8mAVWC0Eu+EQ=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=BE6om4bwi6wGZMjMKrSWe4k+KZWDXxo91aBVcV8XAweZUNNoJS1SN+3QwpIU+56XZPeZusXaoLoVNIoz3TMhA0WPfrg4P2gqWHQKtB2j/x88ebLN5Fz9AJbifOPrrOTpm4dawqll/QLeGegjQWI1icTeDKPzsApO6w4y4v/XlN0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=USjaqHUB; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=IikLgTFn; arc=none smtp.client-ip=103.168.172.153 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="USjaqHUB"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="IikLgTFn" Received: from phl-compute-04.internal (phl-compute-04.nyi.internal [10.202.2.44]) by mailfhigh.nyi.internal (Postfix) with ESMTP id BD68E1151A97; Mon, 19 Aug 2024 11:39:59 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-04.internal (MEProxy); Mon, 19 Aug 2024 11:39:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm1; t=1724081999; x=1724168399; bh=YXxSjlMxy5 /jGhfWX5XSGF3IrlQRUzt/l4rWWfGGeH0=; b=USjaqHUBNjhts1CXgy7O90rj9p Wd9E83iJMMQyBF4SUIOroP+gCuuCXb0qtxTu4qIo+bA5wC/DjAaoHGZ22rGRgN98 oMrKEBzTH2jmyf6ze5KbxhwKjrmsZwFqR5ztIlChu5+S5d91IBKos128N+yvA3hH Yao/nE2d67wkjlgMH0Z00h0bFwyauCCDcBSH3NqSWWdrqwSKRcxdQ7z0n6T4QIUZ /y2Np072KrnLRzHj8SCmJUY5uzrjEDt+f1FNAIf9JA4y6cSZpF4IJbJFgPGFsSb2 Hg+EJlBJnkXOuH2uaLc2LM4lF5xFv8qmMGPOBK5vUOszdZP05eGvTg/kiBFA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; t=1724081999; x=1724168399; bh=YXxSjlMxy5/jGhfWX5XSGF3IrlQR Uzt/l4rWWfGGeH0=; b=IikLgTFn+kgC7WoBduRwnmdy4RK8EguiW8wSKgFb3wp7 McOJQpdWP4kZyFvnqJ1rop/QuPhHyMC2J0uV19OmBSkB8ckA8khRin0Z1U4fP8jc C55vrIfo28bS/JUeX5UaR5Yp5L5KHfwa6EyFTuKzTOvJn7Te1no1qEkRF+h/ckey a2NXpsOZna1HoPoU9BOX9V2E4TgW+XU5SEaK/ihaboh7USlr4uKycc/Y2Ej/utG5 gyQhcZsWlpcj9VFaf1yJLmymO/5qCGpSFcBaIAxCX9Vlj/HWDShxxSaIeU8TYCLo kQ2+yyr4N71mpkO+ReBYe/PyXihlbAKrKDhz0l7xag== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddruddugedgledvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvve fukfhfgggtuggjsehttdertddttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhn hhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrhhnpeevkeekfffhie dtleduiefgjedttedvledvudehgfeugedugffhueekhfejvdektdenucevlhhushhtvghr ufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhmpdhnsg gprhgtphhtthhopedvpdhmohguvgepshhmthhpohhuthdprhgtphhtthhopehpvghffhes phgvfhhfrdhnvghtpdhrtghpthhtohepghhithesvhhgvghrrdhkvghrnhgvlhdrohhrgh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 19 Aug 2024 11:39:58 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id c8db3ba2 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 19 Aug 2024 15:39:29 +0000 (UTC) Date: Mon, 19 Aug 2024 17:39:57 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Jeff King Subject: [PATCH 05/10] reftable/reader: inline `reader_close()` Message-ID: <4a9fe15042795f5b2bd4d4e17cf771d86e4eea47.1724080006.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Same as with the preceding commit, we also provide a `reader_close()` function that allows the caller to close a reader without freeing it. This is unnecessary now that all users will have an allocated version of the reader. Inline it into `reftable_reader_free()`. Signed-off-by: Patrick Steinhardt --- reftable/reader.c | 9 ++------- reftable/reader.h | 1 - reftable/stack.c | 5 +---- 3 files changed, 3 insertions(+), 12 deletions(-) diff --git a/reftable/reader.c b/reftable/reader.c index 9239679ad9..037417fcf6 100644 --- a/reftable/reader.c +++ b/reftable/reader.c @@ -579,12 +579,6 @@ void reftable_reader_init_log_iterator(struct reftable_reader *r, reader_init_iter(r, it, BLOCK_TYPE_LOG); } -void reader_close(struct reftable_reader *r) -{ - block_source_close(&r->source); - FREE_AND_NULL(r->name); -} - int reftable_reader_new(struct reftable_reader **out, struct reftable_block_source *source, char const *name) { @@ -655,7 +649,8 @@ void reftable_reader_free(struct reftable_reader *r) { if (!r) return; - reader_close(r); + block_source_close(&r->source); + FREE_AND_NULL(r->name); reftable_free(r); } diff --git a/reftable/reader.h b/reftable/reader.h index 762cd6de66..88b4f3b421 100644 --- a/reftable/reader.h +++ b/reftable/reader.h @@ -52,7 +52,6 @@ struct reftable_reader { struct reftable_reader_offsets log_offsets; }; -void reader_close(struct reftable_reader *r); const char *reader_name(struct reftable_reader *r); void reader_init_iter(struct reftable_reader *r, diff --git a/reftable/stack.c b/reftable/stack.c index c72435b059..0ac9cdf8de 100644 --- a/reftable/stack.c +++ b/reftable/stack.c @@ -290,7 +290,6 @@ static int reftable_stack_reload_once(struct reftable_stack *st, const char *name = reader_name(cur[i]); stack_filename(&table_path, st, name); - reader_close(cur[i]); reftable_reader_free(cur[i]); /* On Windows, can only unlink after closing. */ @@ -299,10 +298,8 @@ static int reftable_stack_reload_once(struct reftable_stack *st, } done: - for (i = 0; i < new_readers_len; i++) { - reader_close(new_readers[i]); + for (i = 0; i < new_readers_len; i++) reftable_reader_free(new_readers[i]); - } reftable_free(new_readers); reftable_free(cur); strbuf_release(&table_path); From patchwork Mon Aug 19 15:40:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13768511 Received: from fout7-smtp.messagingengine.com (fout7-smtp.messagingengine.com [103.168.172.150]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CC830161320 for ; Mon, 19 Aug 2024 15:40:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.150 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724082005; cv=none; b=hBbMFiywXvqbSqB5g8TiOc0uVtuux2zqRM2ggEk6FFwNcnWpO+SFLMtxEoUIljfwVtQPCtMPr1ASpa5SpEtXMcPNWLmNMM/RFv4/juhNR/BlQgg6qBf+BuI7MSZHIzP8PXCthkj4K5OY8x+kJw1YGfNPDvVB4aoBa91HAx9pYRQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724082005; c=relaxed/simple; bh=FEr8ardXQayQ2TphZ73DWd5/HmhupTNwRaduRG4G6dY=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=j+DI6DzbS+a9YW//WpJ+t/ysfF1wwOUNT7nsaxUCQ7z6/BrtVof1xm9mI7wA+67CX36prcqIvDsKKeGhJ+elxh2lVwR6cu9PPyhGzlbRTLkWK7stAIAl+lS6Izjrknsvd0vpb23SkccIQPKqVMGsmao5laOgHNYsboXmxle+eOk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=QOi/aqaS; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=MiEKK922; arc=none smtp.client-ip=103.168.172.150 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="QOi/aqaS"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="MiEKK922" Received: from phl-compute-08.internal (phl-compute-08.nyi.internal [10.202.2.48]) by mailfout.nyi.internal (Postfix) with ESMTP id D86F8138FC3F; Mon, 19 Aug 2024 11:40:02 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-08.internal (MEProxy); Mon, 19 Aug 2024 11:40:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm1; t=1724082002; x=1724168402; bh=iTJegvQEzg opvHU7GaYWJs0yY5xgDY2Vv5QJY921bUk=; b=QOi/aqaSDIiFPZXj31le+a/wEi fpQGYt8cJ27PcgMOcD0uxyruvWmr1C5qByjdm/M0j576eCWJHS6TJ0S1wLcUz0sp qKxDmyeh4fshsXLlQPtWqRvB5ti0pdJ2emXIVdw0D4BcgAPh/PejxsPUHBnqbrsy uXRY2UQsLWxwlXtmUYMYyseFkTbh9i+t8rGbsJTOkz/EWMTy2mnczS9NEa1fqqcZ lATf5xCJHDZ4bywMge7rn9LO3fUgby5l6ww/MIms+zxqvO1UwDCqvn6X6SCDQzQx ny0QpLS5JLl/kG2DzsoaVktvNemozzHB2ADtepLJxYpM5YEDrXyleGt5P3Sw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; t=1724082002; x=1724168402; bh=iTJegvQEzgopvHU7GaYWJs0yY5xg DY2Vv5QJY921bUk=; b=MiEKK922FV1QvhFymyAeN0c5L+KuWjaS/TjCQa0gE5Cd pxuG5ca8txw8Ox3HYRPHBKgSBe7OgTlbCr//zqi+W/uEp/vewdrHEbzp3wQ39900 6OnGZwK9IlOPRG0kwrOvckOeY2PJRinHJgOFyXEDMO8OatBkn30mlTdV8T81Lddh BQvH3OPLYxmVW4QIyeHlif3ZNS1kR+TJWuSR1NPMIYW0XgzKoHg4f3S1kOpAiw8u uLO59elBt8Trb/5WMsgFW7sA7fL7dlkqc4ncYAEdiR7/4JlNTM8sQhi9NUWIoM8m X7BJB+8Xz7Vrm1talWwMWEIW3ipuuhC9wM05Eg4HrA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddruddugedgledvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvve fukfhfgggtuggjsehttdertddttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhn hhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrhhnpeevkeekfffhie dtleduiefgjedttedvledvudehgfeugedugffhueekhfejvdektdenucevlhhushhtvghr ufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhmpdhnsg gprhgtphhtthhopedvpdhmohguvgepshhmthhpohhuthdprhgtphhtthhopehgihhtsehv ghgvrhdrkhgvrhhnvghlrdhorhhgpdhrtghpthhtohepphgvfhhfsehpvghffhdrnhgvth X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 19 Aug 2024 11:40:01 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 1c538200 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 19 Aug 2024 15:39:32 +0000 (UTC) Date: Mon, 19 Aug 2024 17:40:00 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Jeff King Subject: [PATCH 06/10] reftable/stack: fix broken refnames in `write_n_ref_tables()` Message-ID: <4965402e7bf5a09da551a8cdc8c12dd87515e2ba.1724080006.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The `write_n_ref_tables()` helper function writes N references in separate tables. We never reset the computed name of those references though, leading us to end up with unexpected names. Fix this by resetting the buffer. Signed-off-by: Patrick Steinhardt --- reftable/stack_test.c | 1 + 1 file changed, 1 insertion(+) diff --git a/reftable/stack_test.c b/reftable/stack_test.c index 42044ed8a3..de0669b7b8 100644 --- a/reftable/stack_test.c +++ b/reftable/stack_test.c @@ -125,6 +125,7 @@ static void write_n_ref_tables(struct reftable_stack *st, .value_type = REFTABLE_REF_VAL1, }; + strbuf_reset(&buf); strbuf_addf(&buf, "refs/heads/branch-%04u", (unsigned) i); ref.refname = buf.buf; set_test_hash(ref.value.val1, i); From patchwork Mon Aug 19 15:40:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13768512 Received: from fout7-smtp.messagingengine.com (fout7-smtp.messagingengine.com [103.168.172.150]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C5F78161320 for ; Mon, 19 Aug 2024 15:40:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.150 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724082009; cv=none; b=r0ITI+ldcHyAAcIEhIPl+FVyDVRi9vjfW6hPL4QTIaEWt/ade+OxAUWExitd/oR/tyy2SGDP+YS4Yt+zLlN5thES6VZac1Hev8vbIxgb/stbKpTCZg2l6XAJAefif6lM+jygv6nDiCtYui+bYVhluqePY/yw9rctXGyoeVu8blk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724082009; c=relaxed/simple; bh=EfZxkguDU0v+bWLzcI608qN3qqOraERCl9qE3rZhHLM=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=RGRpvq+b0Fq1XSnjU8QUWeKaJdACgsBUF38knyeDI9jAYboRJ95m76ePAwUT/LIsw3bv0FUuc+aTBWikea7BtlUdzffpbIP+pFIHhYTSEt5n5GOA2YfOMHQZfkxT3qrIxG4Z/YLXUWRRz3Cl3U1wSbmjc8UOCSyjXyTkOc4CSDY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=ngtkItGl; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=dK3fqsDV; arc=none smtp.client-ip=103.168.172.150 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="ngtkItGl"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="dK3fqsDV" Received: from phl-compute-06.internal (phl-compute-06.nyi.internal [10.202.2.46]) by mailfout.nyi.internal (Postfix) with ESMTP id E2F82138FF3A; Mon, 19 Aug 2024 11:40:06 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-06.internal (MEProxy); Mon, 19 Aug 2024 11:40:06 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm1; t=1724082006; x=1724168406; bh=Wu8sfZVfcb bcORKjzT2qa59A83bW2tKFIu2PvDPSazE=; b=ngtkItGlw8E6yH9TFarEcsOba7 PZA5iCcZL9aw4Ft+bDR7Kq0P3lbzx8z5FbTCG20jGGTLBzy3N8kTAZyYW0Dpxo3d dSG2gTjLXlKLV0TrOciLJU2zVeDRTNJMb3EBc+tltjDsT3yt+F0W1cuMXdm1DlHN Jrnj+n3s529Hj+5zrl9NxfKvYEaii2e+WhgT+JYcY11APShGIw6rKHeTTkd/mBgU +/IBfXp/p7NdXWo42ZEIzE9G+3DvMiQA6V0iCU3TNdjabhC7q+Wk0rcaYD6h+SMu HnPgswGvY8/rSbnF06sTYSVMaMa57sJtzHdOhi+cL0RqM+o+GTn48DmG1Wig== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; t=1724082006; x=1724168406; bh=Wu8sfZVfcbbcORKjzT2qa59A83bW 2tKFIu2PvDPSazE=; b=dK3fqsDVMzZZb4GEtGCcp+1ifcmfQmOSyGxnMiOyzQLS dbi20BJPTD3C1WYQq+SzGzeFSIvaU4xCAZLkjdgXt/MWVSW38IhSLRtupi50ok+r 16NYgr5TKGnCMZcnURdXxpPbn20wD8aCSnsn6TSx35EWVqCcdeLupXmbRrMqx5YG 6xNdSbUI53sKlaGEUfcnZeSZFPUn90VUq7EHJRtKyb2OFWgFgbcD6200qMBTmoBA OcQMcTDWW2WimUnevWsHNuVVF/bdYbQRO+oXPtIqZoRDxx8+OFWKq7KokznCHwo/ oYNwRsXEV8pv4QrJsxvx4eRCkGArobNPELnCPGdSEA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddruddugedgledvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvve fukfhfgggtuggjsehttdertddttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhn hhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrhhnpeevkeekfffhie dtleduiefgjedttedvledvudehgfeugedugffhueekhfejvdektdenucevlhhushhtvghr ufhiiigvpedunecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhmpdhnsg gprhgtphhtthhopedvpdhmohguvgepshhmthhpohhuthdprhgtphhtthhopehgihhtsehv ghgvrhdrkhgvrhhnvghlrdhorhhgpdhrtghpthhtohepphgvfhhfsehpvghffhdrnhgvth X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 19 Aug 2024 11:40:06 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 0ebab0c6 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 19 Aug 2024 15:39:36 +0000 (UTC) Date: Mon, 19 Aug 2024 17:40:03 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Jeff King Subject: [PATCH 07/10] reftable/reader: introduce refcounting Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: It was recently reported that concurrent reads and writes may cause the reftable backend to segfault. The root cause of this is that we do not properly keep track of reftable readers across reloads. Suppose that you have a reftable iterator and then decide to reload the stack while iterating through the iterator. When the stack has been rewritten since we have created the iterator, then we would end up discarding a subset of readers that may still be in use by the iterator. The consequence is that we now try to reference deallocated memory, which of course segfaults. One way to trigger this is in t5616, where some background maintenance jobs have been leaking from one test into another. This leads to stack traces like the following one: + git -c protocol.version=0 -C pc1 fetch --filter=blob:limit=29999 --refetch origin AddressSanitizer:DEADLYSIGNAL ================================================================= ==657994==ERROR: AddressSanitizer: SEGV on unknown address 0x7fa0f0ec6089 (pc 0x55f23e52ddf9 bp 0x7ffe7bfa1700 sp 0x7ffe7bfa1700 T0) ==657994==The signal is caused by a READ memory access. #0 0x55f23e52ddf9 in get_var_int reftable/record.c:29 #1 0x55f23e53295e in reftable_decode_keylen reftable/record.c:170 #2 0x55f23e532cc0 in reftable_decode_key reftable/record.c:194 #3 0x55f23e54e72e in block_iter_next reftable/block.c:398 #4 0x55f23e5573dc in table_iter_next_in_block reftable/reader.c:240 #5 0x55f23e5573dc in table_iter_next reftable/reader.c:355 #6 0x55f23e5573dc in table_iter_next reftable/reader.c:339 #7 0x55f23e551283 in merged_iter_advance_subiter reftable/merged.c:69 #8 0x55f23e55169e in merged_iter_next_entry reftable/merged.c:123 #9 0x55f23e55169e in merged_iter_next_void reftable/merged.c:172 #10 0x55f23e537625 in reftable_iterator_next_ref reftable/generic.c:175 #11 0x55f23e2cf9c6 in reftable_ref_iterator_advance refs/reftable-backend.c:464 #12 0x55f23e2d996e in ref_iterator_advance refs/iterator.c:13 #13 0x55f23e2d996e in do_for_each_ref_iterator refs/iterator.c:452 #14 0x55f23dca6767 in get_ref_map builtin/fetch.c:623 #15 0x55f23dca6767 in do_fetch builtin/fetch.c:1659 #16 0x55f23dca6767 in fetch_one builtin/fetch.c:2133 #17 0x55f23dca6767 in cmd_fetch builtin/fetch.c:2432 #18 0x55f23dba7764 in run_builtin git.c:484 #19 0x55f23dba7764 in handle_builtin git.c:741 #20 0x55f23dbab61e in run_argv git.c:805 #21 0x55f23dbab61e in cmd_main git.c:1000 #22 0x55f23dba4781 in main common-main.c:64 #23 0x7fa0f063fc89 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 #24 0x7fa0f063fd44 in __libc_start_main_impl ../csu/libc-start.c:360 #25 0x55f23dba6ad0 in _start (git+0xadfad0) (BuildId: 803b2b7f59beb03d7849fb8294a8e2145dd4aa27) While it is somewhat awkward that the maintenance processes survive tests in the first place, it is totally expected that reftables should work alright with concurrent writers. Seemingly they don't. The only underlying resource that we need to care about in this context is the reftable reader, which is responsible for reading a single table from disk. These readers get discarded immediately (unless reused) when calling `reftable_stack_reload()`, which is wrong. We can only close them once we know that there are no iterators using them anymore. Prepare for a fix by converting the reftable readers to be refcounted. Reported-by: Jeff King Signed-off-by: Patrick Steinhardt --- reftable/reader.c | 16 ++++++++++++++-- reftable/reader.h | 2 ++ reftable/readwrite_test.c | 18 +++++++++--------- reftable/reftable-reader.h | 15 ++++++++++++--- reftable/stack.c | 8 ++++---- reftable/stack_test.c | 6 ++---- t/helper/test-reftable.c | 2 +- t/unit-tests/t-reftable-merged.c | 4 ++-- 8 files changed, 46 insertions(+), 25 deletions(-) diff --git a/reftable/reader.c b/reftable/reader.c index 037417fcf6..64a0953e68 100644 --- a/reftable/reader.c +++ b/reftable/reader.c @@ -621,6 +621,7 @@ int reftable_reader_new(struct reftable_reader **out, r->source = *source; r->name = xstrdup(name); r->hash_id = 0; + r->refcount = 1; err = block_source_read_block(source, &footer, r->size, footer_size(r->version)); @@ -645,10 +646,21 @@ int reftable_reader_new(struct reftable_reader **out, return err; } -void reftable_reader_free(struct reftable_reader *r) +void reftable_reader_incref(struct reftable_reader *r) +{ + if (!r->refcount) + BUG("cannot increment ref counter of dead reader"); + r->refcount++; +} + +void reftable_reader_decref(struct reftable_reader *r) { if (!r) return; + if (!r->refcount) + BUG("cannot decrement ref counter of dead reader"); + if (--r->refcount) + return; block_source_close(&r->source); FREE_AND_NULL(r->name); reftable_free(r); @@ -812,7 +824,7 @@ int reftable_reader_print_blocks(const char *tablename) } done: - reftable_reader_free(r); + reftable_reader_decref(r); table_iter_close(&ti); return err; } diff --git a/reftable/reader.h b/reftable/reader.h index 88b4f3b421..3710ee09b4 100644 --- a/reftable/reader.h +++ b/reftable/reader.h @@ -50,6 +50,8 @@ struct reftable_reader { struct reftable_reader_offsets ref_offsets; struct reftable_reader_offsets obj_offsets; struct reftable_reader_offsets log_offsets; + + uint64_t refcount; }; const char *reader_name(struct reftable_reader *r); diff --git a/reftable/readwrite_test.c b/reftable/readwrite_test.c index 2f2ff787b2..0494e7955a 100644 --- a/reftable/readwrite_test.c +++ b/reftable/readwrite_test.c @@ -279,7 +279,7 @@ static void test_log_write_read(void) /* cleanup. */ strbuf_release(&buf); free_names(names); - reftable_reader_free(reader); + reftable_reader_decref(reader); } static void test_log_zlib_corruption(void) @@ -341,7 +341,7 @@ static void test_log_zlib_corruption(void) reftable_iterator_destroy(&it); /* cleanup. */ - reftable_reader_free(reader); + reftable_reader_decref(reader); strbuf_release(&buf); } @@ -383,7 +383,7 @@ static void test_table_read_write_sequential(void) EXPECT(j == N); reftable_iterator_destroy(&it); - reftable_reader_free(reader); + reftable_reader_decref(reader); strbuf_release(&buf); free_names(names); } @@ -431,7 +431,7 @@ static void test_table_read_api(void) } reftable_iterator_destroy(&it); reftable_free(names); - reftable_reader_free(reader); + reftable_reader_decref(reader); strbuf_release(&buf); } @@ -498,7 +498,7 @@ static void test_table_read_write_seek(int index, int hash_id) reftable_free(names[i]); } reftable_free(names); - reftable_reader_free(reader); + reftable_reader_decref(reader); } static void test_table_read_write_seek_linear(void) @@ -611,7 +611,7 @@ static void test_table_refs_for(int indexed) strbuf_release(&buf); free_names(want_names); reftable_iterator_destroy(&it); - reftable_reader_free(reader); + reftable_reader_decref(reader); } static void test_table_refs_for_no_index(void) @@ -657,7 +657,7 @@ static void test_write_empty_table(void) EXPECT(err > 0); reftable_iterator_destroy(&it); - reftable_reader_free(rd); + reftable_reader_decref(rd); strbuf_release(&buf); } @@ -863,7 +863,7 @@ static void test_write_multiple_indices(void) reftable_iterator_destroy(&it); reftable_writer_free(writer); - reftable_reader_free(reader); + reftable_reader_decref(reader); strbuf_release(&writer_buf); strbuf_release(&buf); } @@ -919,7 +919,7 @@ static void test_write_multi_level_index(void) reftable_iterator_destroy(&it); reftable_writer_free(writer); - reftable_reader_free(reader); + reftable_reader_decref(reader); strbuf_release(&writer_buf); strbuf_release(&buf); } diff --git a/reftable/reftable-reader.h b/reftable/reftable-reader.h index 8a05c84685..a600452b56 100644 --- a/reftable/reftable-reader.h +++ b/reftable/reftable-reader.h @@ -33,6 +33,18 @@ struct reftable_reader; int reftable_reader_new(struct reftable_reader **pp, struct reftable_block_source *src, const char *name); +/* + * Manage the reference count of the reftable reader. A newly initialized + * reader starts with a refcount of 1 and will be deleted once the refcount has + * reached 0. + * + * This is required because readers may have longer lifetimes than the stack + * they belong to. The stack may for example be reloaded while the old tables + * are still being accessed by an iterator. + */ +void reftable_reader_incref(struct reftable_reader *reader); +void reftable_reader_decref(struct reftable_reader *reader); + /* Initialize a reftable iterator for reading refs. */ void reftable_reader_init_ref_iterator(struct reftable_reader *r, struct reftable_iterator *it); @@ -44,9 +56,6 @@ void reftable_reader_init_log_iterator(struct reftable_reader *r, /* returns the hash ID used in this table. */ uint32_t reftable_reader_hash_id(struct reftable_reader *r); -/* closes and deallocates a reader. */ -void reftable_reader_free(struct reftable_reader *); - /* return an iterator for the refs pointing to `oid`. */ int reftable_reader_refs_for(struct reftable_reader *r, struct reftable_iterator *it, uint8_t *oid); diff --git a/reftable/stack.c b/reftable/stack.c index 0ac9cdf8de..8e85f8b4d9 100644 --- a/reftable/stack.c +++ b/reftable/stack.c @@ -186,7 +186,7 @@ void reftable_stack_destroy(struct reftable_stack *st) if (names && !has_name(names, name)) { stack_filename(&filename, st, name); } - reftable_reader_free(st->readers[i]); + reftable_reader_decref(st->readers[i]); if (filename.len) { /* On Windows, can only unlink after closing. */ @@ -290,7 +290,7 @@ static int reftable_stack_reload_once(struct reftable_stack *st, const char *name = reader_name(cur[i]); stack_filename(&table_path, st, name); - reftable_reader_free(cur[i]); + reftable_reader_decref(cur[i]); /* On Windows, can only unlink after closing. */ unlink(table_path.buf); @@ -299,7 +299,7 @@ static int reftable_stack_reload_once(struct reftable_stack *st, done: for (i = 0; i < new_readers_len; i++) - reftable_reader_free(new_readers[i]); + reftable_reader_decref(new_readers[i]); reftable_free(new_readers); reftable_free(cur); strbuf_release(&table_path); @@ -1534,7 +1534,7 @@ static void remove_maybe_stale_table(struct reftable_stack *st, uint64_t max, goto done; update_idx = reftable_reader_max_update_index(rd); - reftable_reader_free(rd); + reftable_reader_decref(rd); if (update_idx <= max) { unlink(table_path.buf); diff --git a/reftable/stack_test.c b/reftable/stack_test.c index de0669b7b8..bc3bf77274 100644 --- a/reftable/stack_test.c +++ b/reftable/stack_test.c @@ -1036,10 +1036,8 @@ static void test_reftable_stack_compaction_concurrent(void) static void unclean_stack_close(struct reftable_stack *st) { /* break abstraction boundary to simulate unclean shutdown. */ - int i = 0; - for (; i < st->readers_len; i++) { - reftable_reader_free(st->readers[i]); - } + for (size_t i = 0; i < st->readers_len; i++) + reftable_reader_decref(st->readers[i]); st->readers_len = 0; FREE_AND_NULL(st->readers); } diff --git a/t/helper/test-reftable.c b/t/helper/test-reftable.c index 87c2f42aae..f6d855a9d7 100644 --- a/t/helper/test-reftable.c +++ b/t/helper/test-reftable.c @@ -152,7 +152,7 @@ static int dump_reftable(const char *tablename) done: reftable_merged_table_free(mt); - reftable_reader_free(r); + reftable_reader_decref(r); return err; } diff --git a/t/unit-tests/t-reftable-merged.c b/t/unit-tests/t-reftable-merged.c index 8f51aca1b6..081d3c8b69 100644 --- a/t/unit-tests/t-reftable-merged.c +++ b/t/unit-tests/t-reftable-merged.c @@ -115,7 +115,7 @@ merged_table_from_records(struct reftable_ref_record **refs, static void readers_destroy(struct reftable_reader **readers, const size_t n) { for (size_t i = 0; i < n; i++) - reftable_reader_free(readers[i]); + reftable_reader_decref(readers[i]); reftable_free(readers); } @@ -437,7 +437,7 @@ static void t_default_write_opts(void) err = reftable_merged_table_new(&merged, &rd, 1, GIT_SHA1_FORMAT_ID); check(!err); - reftable_reader_free(rd); + reftable_reader_decref(rd); reftable_merged_table_free(merged); strbuf_release(&buf); } From patchwork Mon Aug 19 15:40:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13768513 Received: from fout7-smtp.messagingengine.com (fout7-smtp.messagingengine.com [103.168.172.150]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CD86612EBD6 for ; Mon, 19 Aug 2024 15:40:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.150 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724082012; cv=none; b=to9aMkVfDfDgVaWwiUTROM/rowp+mHs2jIsrLyuAGIPbrKzsu3FX4qf5JwnVR3RkxJEZBUiuI4a/9NLJnCJpVs3f/YYF6klu42QfRbLupJp8D5iu5xZx/JStpgeI0AooHfNH8mzGBUch2XGtGrVtBna6AAI6dExLXIH4W10n4Eo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724082012; c=relaxed/simple; bh=sTp6cTGyLPzok3ZHNT73VgVsFAYASJpIUVLd8ZaXJYQ=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=nfHg8MpvsRnnw5FDRKJQ//12ZIerVGoEkqw2Y+9rGLIz6Uk8KTG+F/TUPGWSqHXWFilOBoqJUf2eclyZFV6Z7ELJWDi2/WSlE4T5OUUADsPh18XJsF27GqaesKJKD91dcUiLAnvRhcSlGOMyDljk5mPMTFqYAzdXMunY0ge9mRM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=DSrkifLn; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=M4xn3YqO; arc=none smtp.client-ip=103.168.172.150 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="DSrkifLn"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="M4xn3YqO" Received: from phl-compute-03.internal (phl-compute-03.nyi.internal [10.202.2.43]) by mailfout.nyi.internal (Postfix) with ESMTP id D91C6138FC65; Mon, 19 Aug 2024 11:40:09 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-03.internal (MEProxy); Mon, 19 Aug 2024 11:40:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm1; t=1724082009; x=1724168409; bh=ZEEsth5dCw BA5u8Df3iVvEetdfgzg770syaXj96zqM8=; b=DSrkifLn7Pf25RJt2pug5v7fy5 T0vHphQHdZA6LT7am5+KCmPwWUTX4l0OAvzojPxuFBWdiQoIm1/UNTpqg7XNn7oo oV9J0zWaFAPd6QTR/Ccb3omPBYGHqaTupJyVSOSRiA35BtbReAKc+YsC4lL4nrhK 2kIGtxqDtCFnniqwjsfPZY7cIHLpqK1EJTgyf+R3TXKUEoFirIJkvZ59qO4kCgLL 753AxhWpj3zIfvBZDlp/EeHpJu3KNMuIVhz1DvcpjdGPXn5EvXpYUYMQI2hxc+a6 gktk1am8Kf2KAMgVWdAxfh3lBSUq2GHzgsTXRhkb0p1eKQJDgTJJBA5rgbVA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; t=1724082009; x=1724168409; bh=ZEEsth5dCwBA5u8Df3iVvEetdfgz g770syaXj96zqM8=; b=M4xn3YqOdrGh0raD/XQ9vauf14KlySJV0N/TgA7jWZDg u+PLa6SqOT1MBnsZ5N5J3/b8nMl/bn0vrZ8oNqTdipXOJf0Whx96FxGREUzE+HEO DRJ9z4sj4/FeHtxodD9WA1qIH1lW9f8kQQq+3/kb99D7pM5W6Lzt7rwVkbiwEY/V Hmr8a07GRLOzHe9Gj/YolCjwRLLJRkkKt4SdnKflsd5998tQlF83jUM2rwm6WpdI Zi2z9gSOhWDgBZnIqJHcXAa2ShC6/B5Sc36IPpQhhy6ScowVk9ZTYhBWQ31eefsp 4g6h+q6u7U0D9ihaBediGv+es5nGaPXoaI995win9Q== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddruddugedgledvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvve fukfhfgggtuggjsehttdertddttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhn hhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrhhnpeevkeekfffhie dtleduiefgjedttedvledvudehgfeugedugffhueekhfejvdektdenucevlhhushhtvghr ufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhmpdhnsg gprhgtphhtthhopedvpdhmohguvgepshhmthhpohhuthdprhgtphhtthhopehpvghffhes phgvfhhfrdhnvghtpdhrtghpthhtohepghhithesvhhgvghrrdhkvghrnhgvlhdrohhrgh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 19 Aug 2024 11:40:09 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id a4a70f46 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 19 Aug 2024 15:39:39 +0000 (UTC) Date: Mon, 19 Aug 2024 17:40:07 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Jeff King Subject: [PATCH 08/10] reftable/reader: keep readers alive during iteration Message-ID: <026820562882afb31d7224c90722e09bef835340.1724080006.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The lifetime of a table iterator may surive the lifetime of a reader when the stack gets reloaded. Keep the reader from being released by increasing its refcount while the iterator is still being used. Signed-off-by: Patrick Steinhardt --- reftable/reader.c | 2 ++ reftable/stack_test.c | 49 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 51 insertions(+) diff --git a/reftable/reader.c b/reftable/reader.c index 64a0953e68..f877099087 100644 --- a/reftable/reader.c +++ b/reftable/reader.c @@ -175,6 +175,7 @@ static int table_iter_init(struct table_iter *ti, struct reftable_reader *r) { struct block_iter bi = BLOCK_ITER_INIT; memset(ti, 0, sizeof(*ti)); + reftable_reader_incref(r); ti->r = r; ti->bi = bi; return 0; @@ -262,6 +263,7 @@ static void table_iter_close(struct table_iter *ti) { table_iter_block_done(ti); block_iter_close(&ti->bi); + reftable_reader_decref(ti->r); } static int table_iter_next_block(struct table_iter *ti) diff --git a/reftable/stack_test.c b/reftable/stack_test.c index bc3bf77274..91e716dc0a 100644 --- a/reftable/stack_test.c +++ b/reftable/stack_test.c @@ -1076,6 +1076,54 @@ static void test_reftable_stack_compaction_concurrent_clean(void) clear_dir(dir); } +static void test_reftable_stack_read_across_reload(void) +{ + struct reftable_write_options opts = { 0 }; + struct reftable_stack *st1 = NULL, *st2 = NULL; + struct reftable_ref_record rec = { 0 }; + struct reftable_iterator it = { 0 }; + char *dir = get_tmp_dir(__LINE__); + int err; + + /* Create a first stack and set up an iterator for it. */ + err = reftable_new_stack(&st1, dir, &opts); + EXPECT_ERR(err); + write_n_ref_tables(st1, 2); + EXPECT(st1->merged->readers_len == 2); + reftable_stack_init_ref_iterator(st1, &it); + err = reftable_iterator_seek_ref(&it, ""); + EXPECT_ERR(err); + + /* Set up a second stack for the same directory and compact it. */ + err = reftable_new_stack(&st2, dir, &opts); + EXPECT_ERR(err); + EXPECT(st2->merged->readers_len == 2); + err = reftable_stack_compact_all(st2, NULL); + EXPECT_ERR(err); + + /* + * Verify that we can continue to use the old iterator even after we + * have reloaded its stack. + */ + err = reftable_stack_reload(st1); + EXPECT_ERR(err); + EXPECT(st2->merged->readers_len == 1); + err = reftable_iterator_next_ref(&it, &rec); + EXPECT_ERR(err); + EXPECT(!strcmp(rec.refname, "refs/heads/branch-0000")); + err = reftable_iterator_next_ref(&it, &rec); + EXPECT_ERR(err); + EXPECT(!strcmp(rec.refname, "refs/heads/branch-0001")); + err = reftable_iterator_next_ref(&it, &rec); + EXPECT(err > 0); + + reftable_ref_record_release(&rec); + reftable_iterator_destroy(&it); + reftable_stack_destroy(st1); + reftable_stack_destroy(st2); + clear_dir(dir); +} + int stack_test_main(int argc, const char *argv[]) { RUN_TEST(test_empty_add); @@ -1098,6 +1146,7 @@ int stack_test_main(int argc, const char *argv[]) RUN_TEST(test_reftable_stack_auto_compaction_fails_gracefully); RUN_TEST(test_reftable_stack_update_index_check); RUN_TEST(test_reftable_stack_uptodate); + RUN_TEST(test_reftable_stack_read_across_reload); RUN_TEST(test_suggest_compaction_segment); RUN_TEST(test_suggest_compaction_segment_nothing); return 0; From patchwork Mon Aug 19 15:40:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13768514 Received: from fhigh2-smtp.messagingengine.com (fhigh2-smtp.messagingengine.com [103.168.172.153]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E5A25136663 for ; Mon, 19 Aug 2024 15:40:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.153 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724082015; cv=none; b=ZJsQHUfmGsNYH05oyveG46oVTZSuRt8jPq9nL10PFOCalC487nrJP5kJk9tqwS20Km17UmXcR91LlOoQcSUvgHyaCuFMCKhigHkAdubhVw3DRnvIY7maTaYTQZ7Ikzu0whLXvK5XNRZS3CeiFN13511Jv0WvVJ396eEiciWS5P0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724082015; c=relaxed/simple; bh=C0osimSr/2u8oyDAF3s0d+73XIFHSMXlilEYb44AXco=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=VMEACzRpxYO8183G0MKqd/dFF4JLSjWaz8N9WTdxEWDunNaWxeaydnNjemQcXV9H+kLlJZrENhh8nJmCcRSq5QFE5F1r47Ltb8opix+eFbzYZ/eSSz5xBRGoOvn85hEA2VucWrK+FwJ/L49Qvkq7XntSefGb5C6Zs9k3N8Dgfzs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=nMsLOcYM; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=VbKcDTqH; arc=none smtp.client-ip=103.168.172.153 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="nMsLOcYM"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="VbKcDTqH" Received: from phl-compute-08.internal (phl-compute-08.nyi.internal [10.202.2.48]) by mailfhigh.nyi.internal (Postfix) with ESMTP id 082051151A97; Mon, 19 Aug 2024 11:40:13 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-08.internal (MEProxy); Mon, 19 Aug 2024 11:40:13 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm1; t=1724082013; x=1724168413; bh=xrKgS1KWJv KfEF0tpQAiFTm+IwM7edflzQ8GpaeTiRc=; b=nMsLOcYMTOz5bCgJe8MpxOmm1q nfyTQw8lBirbWQC1VP6NfDgBEayNZYznZpEO2yCA6huZGa5Ac6H0s1zhbF5bMFle qxQnRmw09OUxdKKRwfLbNGggi8c2hhxZTCrbtp/fFQnnl19CLafDCxwucwSUUF+3 VgRd27qSoHqQEqbWVqHiHb4FJgivlvLHIRcx+9ATDB3vcKylJCo3kgN5IgjEno98 7e8H2IOhrbOIClaRAuNpFNCUplUFO6X/BcKrVlLcIbkCPyv3VXg5t7uCFSYJd4RN qYamRXJRTMIqrcw73K2rd40mlKb7gD0Tas178YAOFDmCg2tR7YhYyyKJgJcA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; t=1724082013; x=1724168413; bh=xrKgS1KWJvKfEF0tpQAiFTm+IwM7 edflzQ8GpaeTiRc=; b=VbKcDTqHPpHNQWdRmoxe+8NlytKjzvg8QsXBGfFMwkeI Nl4gua/RQ4vDi2nw0uJQXyger4Uok+NLlXKJksIqEw5OnyyiOqT/Vqi3Bv7AiH5x R/hFG8X35Dyg752t0KdZW+W+eUl/n1n7EHMDHyTrKaXE1LSThjH2RfkX2Rn23stH IXZ6Q2+Dd5HEl8z/SM9r4gDSbmsMIgcMP46dfCnqMKQaQPoGNkI4GyjlTewV1kY0 /cZ0xJTDVfCRkqadYmgOfE5JCZ/7vTojLTKtd8Z6yzW4QqNToKlpa7Gx25RD1p2t MdnMuXCGgyu75YxJELY0jwhsxZl1wl+WhGgaY59sKw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddruddugedgledvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvve fukfhfgggtuggjsehttdertddttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhn hhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrhhnpeevkeekfffhie dtleduiefgjedttedvledvudehgfeugedugffhueekhfejvdektdenucevlhhushhtvghr ufhiiigvpedunecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhmpdhnsg gprhgtphhtthhopedvpdhmohguvgepshhmthhpohhuthdprhgtphhtthhopehgihhtsehv ghgvrhdrkhgvrhhnvghlrdhorhhgpdhrtghpthhtohepphgvfhhfsehpvghffhdrnhgvth X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 19 Aug 2024 11:40:12 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id e2df5c57 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 19 Aug 2024 15:39:42 +0000 (UTC) Date: Mon, 19 Aug 2024 17:40:10 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Jeff King Subject: [PATCH 09/10] reftable/stack: reorder swapping in the reloaded stack contents Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The code flow of how we swap in the reloaded stack contents is somewhat convoluted because we switch back and forth between swapping in different parts of the stack. Reorder the code to simplify it. We now first close and unlink the old tables which do not get reused before we update the stack to point to the new stack. Signed-off-by: Patrick Steinhardt --- reftable/stack.c | 34 +++++++++++++++++++--------------- 1 file changed, 19 insertions(+), 15 deletions(-) diff --git a/reftable/stack.c b/reftable/stack.c index 8e85f8b4d9..0247222258 100644 --- a/reftable/stack.c +++ b/reftable/stack.c @@ -273,30 +273,34 @@ static int reftable_stack_reload_once(struct reftable_stack *st, if (err < 0) goto done; - st->readers_len = new_readers_len; - if (st->merged) - reftable_merged_table_free(st->merged); - if (st->readers) { - reftable_free(st->readers); - } - st->readers = new_readers; - new_readers = NULL; - new_readers_len = 0; - - new_merged->suppress_deletions = 1; - st->merged = new_merged; + /* + * Close the old, non-reused readers and proactively try to unlink + * them. This is done for systems like Windows, where the underlying + * file of such an open reader wouldn't have been possible to be + * unlinked by the compacting process. + */ for (i = 0; i < cur_len; i++) { if (cur[i]) { const char *name = reader_name(cur[i]); stack_filename(&table_path, st, name); - reftable_reader_decref(cur[i]); - - /* On Windows, can only unlink after closing. */ unlink(table_path.buf); } } + /* Update the stack to point to the new tables. */ + if (st->merged) + reftable_merged_table_free(st->merged); + new_merged->suppress_deletions = 1; + st->merged = new_merged; + + if (st->readers) + reftable_free(st->readers); + st->readers = new_readers; + st->readers_len = new_readers_len; + new_readers = NULL; + new_readers_len = 0; + done: for (i = 0; i < new_readers_len; i++) reftable_reader_decref(new_readers[i]); From patchwork Mon Aug 19 15:40:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13768515 Received: from fhigh2-smtp.messagingengine.com (fhigh2-smtp.messagingengine.com [103.168.172.153]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ABB3A16C841 for ; Mon, 19 Aug 2024 15:40:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.153 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724082018; cv=none; b=Zi6PYDga295gE408oz6O85CUzHqa09p0tm7nmpKBGKX3wrLH+Lc5QGJJKJ1xQe//rtdnLpVlxJKRN1r4/0FdrxEttB5RdUV6/8uHL3LOunS2PDEq+y38gjJtHJlf3d6xgRXvDP+SWC7/PBzBGDbQOoGcybNPUICXHbVEfh760PI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1724082018; c=relaxed/simple; bh=uTuNQN2aeI5JQ7bygsG0vSAFo+okIgdf8vWrkSSYYtk=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=gzxv0s5UWTlYdoSA2y6BH5UB8pvEBIxf1kBOMvr4EfKtvG8SgCuIWRIVQ9QwP/TMTW8Wl5CSb+1AI/LwPWfMKXrCit2onOvQM9RtL/4Dol5xPntQkwz7kD7wNw8iTh3zIsqO/q6jUQdKW7LaF/mT37oBKMKG9it1I3vL8cdO/3w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=oitTj2Pc; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=mBGKiyhf; arc=none smtp.client-ip=103.168.172.153 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="oitTj2Pc"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="mBGKiyhf" Received: from phl-compute-01.internal (phl-compute-01.nyi.internal [10.202.2.41]) by mailfhigh.nyi.internal (Postfix) with ESMTP id 10B3B1151777; Mon, 19 Aug 2024 11:40:16 -0400 (EDT) Received: from phl-mailfrontend-01 ([10.202.2.162]) by phl-compute-01.internal (MEProxy); Mon, 19 Aug 2024 11:40:16 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm1; t=1724082016; x=1724168416; bh=3dhawzIhf3 g+dK1xM3sD3m391psSw0rHv4B9Ng53oaQ=; b=oitTj2PceGgGNxB9iAS5YjLbpw KBU8BLWOK9GZqws0sGuok/U0nI3jYAVdRfJnOzjMp3dCgpzpApDHGom/Lk9bGh8h zUkDnpEHPklNPzPBCx824j7q0qTMncgc3ewfimTP4a/AoGhN9Yj+/ypmskbdkfXb LvEa9fkU3t/E1ZTsRWLBO2wCedKxkGsjZ4On3EGQX5VBlvgrH7pklWV0fwRVgsbc 1HrBF3XqOQTP9NzmScRDFgiCfewkoFYw2b1nTDFmx/SskmTbDRhne16AzmV+BYZs icXXyW099IULebMExvGFALPMBVZNw86squP5McyAZtEXQkltmShtyjgJaFKw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm1; t=1724082016; x=1724168416; bh=3dhawzIhf3g+dK1xM3sD3m391psS w0rHv4B9Ng53oaQ=; b=mBGKiyhfjPvd5YZxaPFhS7kcVLRv0jHcHXopybqp1z+Q W8ppz1ggdLNYG2mu1k7Hj/XicZPx2IGjoUGUOLu8jvlCV/w5wbHPwrBKReqtjF9v mtl6L4NIt60x0AGCm4TWLTs8bjQusn90XmDR8cKYuegGRsbob9VnaRVRsIa2upwY L8VicWIbRPFJ0OhhnISx1ZoCpPjpnv1CU+iwgDrPDcqUAULU3NsjMm49a88Ora9y 2OPNKvKIUy7nzSX5FMYSAUp+kSTvNheL+L5TGHL8DMckuDwj35tC85+a7ijDqQo7 o5Wv1dkDPC2aJOEJWyM9mbLUz7v6T0Ezif9fsz48Rw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddruddugedgledvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvve fukfhfgggtuggjsehttdertddttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhn hhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrhhnpeevkeekfffhie dtleduiefgjedttedvledvudehgfeugedugffhueekhfejvdektdenucevlhhushhtvghr ufhiiigvpedunecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhmpdhnsg gprhgtphhtthhopedvpdhmohguvgepshhmthhpohhuthdprhgtphhtthhopehgihhtsehv ghgvrhdrkhgvrhhnvghlrdhorhhgpdhrtghpthhtohepphgvfhhfsehpvghffhdrnhgvth X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 19 Aug 2024 11:40:15 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id b8a29f6a (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 19 Aug 2024 15:39:45 +0000 (UTC) Date: Mon, 19 Aug 2024 17:40:14 +0200 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Jeff King Subject: [PATCH 10/10] reftable/stack: fix segfault when reload with reused readers fails Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: It is expected that reloading the stack fails with concurrent writers, e.g. because a table that we just wanted to read just got compacted. In case we decided to reuse readers this will cause a segfault though because we unconditionally release all new readers, including the reused ones. As those are still referenced by the current stack, the result is that we will eventually try to dereference those already-freed readers. Fix this bug by incrementing the refcount of reused readers temporarily. Signed-off-by: Patrick Steinhardt --- reftable/stack.c | 23 +++++++++++++++++ reftable/stack_test.c | 59 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 82 insertions(+) diff --git a/reftable/stack.c b/reftable/stack.c index 0247222258..ce0a35216b 100644 --- a/reftable/stack.c +++ b/reftable/stack.c @@ -226,6 +226,8 @@ static int reftable_stack_reload_once(struct reftable_stack *st, { size_t cur_len = !st->merged ? 0 : st->merged->readers_len; struct reftable_reader **cur = stack_copy_readers(st, cur_len); + struct reftable_reader **reused = NULL; + size_t reused_len = 0, reused_alloc = 0; size_t names_len = names_length(names); struct reftable_reader **new_readers = reftable_calloc(names_len, sizeof(*new_readers)); @@ -245,6 +247,18 @@ static int reftable_stack_reload_once(struct reftable_stack *st, if (cur[i] && 0 == strcmp(cur[i]->name, name)) { rd = cur[i]; cur[i] = NULL; + + /* + * When reloading the stack fails, we end up + * releasing all new readers. This also + * includes the reused readers, even though + * they are still in used by the old stack. We + * thus need to keep them alive here, which we + * do by bumping their refcount. + */ + REFTABLE_ALLOC_GROW(reused, reused_len + 1, reused_alloc); + reused[reused_len++] = rd; + reftable_reader_incref(rd); break; } } @@ -301,10 +315,19 @@ static int reftable_stack_reload_once(struct reftable_stack *st, new_readers = NULL; new_readers_len = 0; + /* + * Decrement the refcount of reused readers again. This only needs to + * happen on the successful case, because on the unsuccessful one we + * decrement their refcount via `new_readers`. + */ + for (i = 0; i < reused_len; i++) + reftable_reader_decref(reused[i]); + done: for (i = 0; i < new_readers_len; i++) reftable_reader_decref(new_readers[i]); reftable_free(new_readers); + reftable_free(reused); reftable_free(cur); strbuf_release(&table_path); return err; diff --git a/reftable/stack_test.c b/reftable/stack_test.c index 91e716dc0a..30cdb1739c 100644 --- a/reftable/stack_test.c +++ b/reftable/stack_test.c @@ -10,6 +10,7 @@ license that can be found in the LICENSE file or at #include "system.h" +#include "copy.h" #include "reftable-reader.h" #include "merged.h" #include "basics.h" @@ -1124,6 +1125,63 @@ static void test_reftable_stack_read_across_reload(void) clear_dir(dir); } +static void test_reftable_stack_reload_with_missing_table(void) +{ + struct reftable_write_options opts = { 0 }; + struct reftable_stack *st = NULL; + struct reftable_ref_record rec = { 0 }; + struct reftable_iterator it = { 0 }; + struct strbuf table_path = STRBUF_INIT, content = STRBUF_INIT; + char *dir = get_tmp_dir(__LINE__); + int err; + + /* Create a first stack and set up an iterator for it. */ + err = reftable_new_stack(&st, dir, &opts); + EXPECT_ERR(err); + write_n_ref_tables(st, 2); + EXPECT(st->merged->readers_len == 2); + reftable_stack_init_ref_iterator(st, &it); + err = reftable_iterator_seek_ref(&it, ""); + EXPECT_ERR(err); + + /* + * Update the tables.list file with some garbage data, while reusing + * our old readers. This should trigger a partial reload of the stack, + * where we try to reuse our old readers. + */ + strbuf_addf(&content, "%s\n", st->readers[0]->name); + strbuf_addf(&content, "%s\n", st->readers[1]->name); + strbuf_addstr(&content, "garbage\n"); + strbuf_addf(&table_path, "%s.lock", st->list_file); + write_file_buf(table_path.buf, content.buf, content.len); + err = rename(table_path.buf, st->list_file); + EXPECT_ERR(err); + + err = reftable_stack_reload(st); + EXPECT(err == -4); + EXPECT(st->merged->readers_len == 2); + + /* + * Even though the reload has failed, we should be able to continue + * using the iterator. + */ + err = reftable_iterator_next_ref(&it, &rec); + EXPECT_ERR(err); + EXPECT(!strcmp(rec.refname, "refs/heads/branch-0000")); + err = reftable_iterator_next_ref(&it, &rec); + EXPECT_ERR(err); + EXPECT(!strcmp(rec.refname, "refs/heads/branch-0001")); + err = reftable_iterator_next_ref(&it, &rec); + EXPECT(err > 0); + + reftable_ref_record_release(&rec); + reftable_iterator_destroy(&it); + reftable_stack_destroy(st); + strbuf_release(&table_path); + strbuf_release(&content); + clear_dir(dir); +} + int stack_test_main(int argc, const char *argv[]) { RUN_TEST(test_empty_add); @@ -1147,6 +1205,7 @@ int stack_test_main(int argc, const char *argv[]) RUN_TEST(test_reftable_stack_update_index_check); RUN_TEST(test_reftable_stack_uptodate); RUN_TEST(test_reftable_stack_read_across_reload); + RUN_TEST(test_reftable_stack_reload_with_missing_table); RUN_TEST(test_suggest_compaction_segment); RUN_TEST(test_suggest_compaction_segment_nothing); return 0;