From patchwork Fri Jan 26 10:31:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13532375 Received: from out2-smtp.messagingengine.com (out2-smtp.messagingengine.com [66.111.4.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5D8541B582 for ; Fri, 26 Jan 2024 10:31:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=66.111.4.26 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706265107; cv=none; b=IddKNYu7Ne/dXQNsdVIKLNBBM7F++bceUPunqW7uCISb3LW4f61L+aH4yIH4vWC0Xk8M392Z6gxaFRvhJrGhgrQD/nk21Qp+Sx94ioGs51SA/pNg5Tkmt7hdpNBt0PCOiCGenQukOScouUpkb1weNnjGx78ihRIZ8ax4/hfERZI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706265107; c=relaxed/simple; bh=fdpXZTvbUVaFFfjZEDf5T9PpB6428W+0FUWxh8i75ls=; h=Date:From:To:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=dlzV1e+Nq4d5B+rdldfizZMxVfz7VnH9afOm+egYqAWCQyp1HSk3Ob939bq9kOvVrAtRSTYQ3pfMCoRFTTfrigUsS4UbZ0L5eBZ7wido0fq7XrIs34HhOjMke4HjYgHgNXmZN3k87WcBgmul+9257CfvPbq3QEnq2cHayf3vfo4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=ESRStHCF; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=nHjK5QpR; arc=none smtp.client-ip=66.111.4.26 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="ESRStHCF"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="nHjK5QpR" Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id 655B65C0085 for ; Fri, 26 Jan 2024 05:31:45 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute3.internal (MEProxy); Fri, 26 Jan 2024 05:31:45 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1706265105; x=1706351505; bh=HrjVUcdA1c PPfAVRzg5Y04T7GcBWAY1weCdYg/YyVbw=; b=ESRStHCFBwO1rijSP29AinYcCO eV6KXEZVY9aJZ4HFzllmqGIc7SNnjEiEwjvnAIOgeABxSDJhVAnyp86W3f8NwNAY BjXG6RINVn+xLhapplTNfnHFSZfQCpLkHB2SaSKKYknkKzPexUQvbdKuxxPj2zMb 9090iIj7ja2r4g8qzzXwGQkTeO5wl0WOnEI+3l8jTgXzeqdFhBl1E9nBvs+JYV6K 2B9zs3fqQL2DR/RKXt105cAD35n1UhcCkyTaEmRWRR0bwFQPMD3pcJOXO+GvRswQ mwLOMMRLPi5vz+o1k+aqMcoA6ZyXXvFaivvap31EV/4O664Vl1lHQ85cwzsg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1706265105; x=1706351505; bh=HrjVUcdA1cPPfAVRzg5Y04T7GcBW AY1weCdYg/YyVbw=; b=nHjK5QpR/Px0XP0PxdMwMrkUTEb64edBLqMJQC1W+vlG 6H5TohvIzG1qgL2g4hSNMRQAxzkuEvdd5w6NIs40udNbnKynLGVb1BLjNt24I1uC KHDoSog6XKz96ZO0yPIfIjfk9+jMuDqxyridHHbg+5r9yw8m0tCR5DgIE1M6Bm/8 bWniTqq7Zg5qo1LmNow05vDTiTOlabWdRFoRa6c5HZtL0HqQ4vY4B94OeG9y712k eqWq4LI1WHktTmRV7uyLzDfmy+xhJzad+pyRyg5Qbp6WBOZMdnQI8mm8rIRdH1SH t7+76dwMvwxiELXqA0Uu1N0LU1E59i3KecGhcbSyaA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrvdeljedgudehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuffkfhggtggujgesghdtre ertddtvdenucfhrhhomheprfgrthhrihgtkhcuufhtvghinhhhrghrughtuceophhssehp khhsrdhimheqnecuggftrfgrthhtvghrnhepheeghfdtfeeuffehkefgffduleffjedthf dvjeektdfhhedvlefgtefgvdettdfhnecuvehluhhsthgvrhfuihiivgeptdenucfrrghr rghmpehmrghilhhfrhhomhepphhssehpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Fri, 26 Jan 2024 05:31:44 -0500 (EST) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 2acece6b (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Fri, 26 Jan 2024 10:28:26 +0000 (UTC) Date: Fri, 26 Jan 2024 11:31:34 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Subject: [PATCH 1/5] reftable/reader: be more careful about errors in indexed seeks Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: When doing an indexed seek we first need to do a linear seek in order to find the index block for our wanted key. We do not check the returned error of the linear seek though. This is likely not an issue because the next call to `table_iter_next()` would return error, too. But it very much is a code smell when an error variable is being assigned to without actually checking it. Safeguard the code by checking for errors. Signed-off-by: Patrick Steinhardt --- reftable/reader.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/reftable/reader.c b/reftable/reader.c index 64dc366fb1..278f727a3d 100644 --- a/reftable/reader.c +++ b/reftable/reader.c @@ -509,6 +509,9 @@ static int reader_seek_indexed(struct reftable_reader *r, goto done; err = reader_seek_linear(&index_iter, &want_index); + if (err < 0) + goto done; + while (1) { err = table_iter_next(&index_iter, &index_result); table_iter_block_done(&index_iter); From patchwork Fri Jan 26 10:31:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13532376 Received: from out2-smtp.messagingengine.com (out2-smtp.messagingengine.com [66.111.4.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 49BE11B596 for ; Fri, 26 Jan 2024 10:31:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=66.111.4.26 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706265108; cv=none; b=XBtiWgD2pj8J072eS4E6Ex9GnFPfE4Yh4OeDnQ7lGELf/5Iwh9rhPGcXB9ujL9nxGDRTdKINIcLw96glV9sw0uWxrar3tTf8l6avylLR+TJ5PiKi6TqxbRi+Y5tv0Z217Oc5Yc/kFMQcoD3IAu7pT3u13wLqWBvHcAbhhd5qbBw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706265108; c=relaxed/simple; bh=Jxh3yw53FbP8JzUVqUtN4dKRTyLJlLq/yEdbq9QwG9g=; h=Date:From:To:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=YDi5XKU8jP5qduDcPEqPuTZ6kTJ5CYdCTEsdh4dAWTi/2D399InO0LJf+e7prKflHWYlCkMowSmH4Uy/HE/FqrjEtTgYhVIvkQRJ1UbUckYhGnuyPgVsmembDPY85NnIrsjQ1yI5P5ZVMQ+QlBc9AZo1YBbFaAKN10Tqrpf4ZXk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=fImXYY5y; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=tzUqTjte; arc=none smtp.client-ip=66.111.4.26 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="fImXYY5y"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="tzUqTjte" Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 53F0C5C010A for ; Fri, 26 Jan 2024 05:31:46 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Fri, 26 Jan 2024 05:31:46 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1706265106; x=1706351506; bh=u9CDuxME6v zFFZ3cuWGCDqimiQADWAD+aee/K6sRHgk=; b=fImXYY5yjdfa1GX7jQwcJMBrFl 3Kr8gEoruNC4Lm9lazcUlEHFFdlJ2xBSRhrV+fynhOIouI256p6+RmJd4AF0oVOG yTRSrUVfTJNXhFRzpTcELE+WpbN4a3rClTPbBkQyoExM22ieAjlBQtMXsEQEiJJE lU4NBll2BEGk/jcdtLNZU5F/SkvNjg0T68yIhxaYaWaI+QzGbdMvyXBkOq0g5F21 yVMq/Egc3jjl9kL4zv+Z1sy7nNdKl+l1E4s1NiNOgm2KE32W+LnTqaW1itXV3Ppq 7b2S9gX3P3pyQ5dEPbFzscqr93CYhDXSCPjuw5SVDEKncop4WFMf/x2VLb+w== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1706265106; x=1706351506; bh=u9CDuxME6vzFFZ3cuWGCDqimiQAD WAD+aee/K6sRHgk=; b=tzUqTjte6JGWf9w+5AhWDAI5IXd3AMD0qdUm7KPv7uZ6 /ivfq+4hWlXDh1fuV/qfECZMYiy3XmvdzzWIS6FRDrsaDAMzXvtIBJ/CjklXajyH oelqhBy7Tv3aPuyhYvLVSHrrR4uGah+e2IQvzfe4vzahNhVvFquqykn1POKLCPnN RolyvftLDlP6xq0xWQ9hrLvouFumq/GDKJYH2LIAbRInvApMVIQquGABx+AMa9o9 FaGZqT2RnZZyLi50XBZdWqEf35+lkmSpRkcoNB4TxL7QN9ulosPEwDIDE7OWrB0K ppc30sjh1mgPVoldfecbLNImffkwmrFws6Hmn2E0eg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrvdeljedgudehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuffkfhggtggujgesghdtre ertddtvdenucfhrhhomheprfgrthhrihgtkhcuufhtvghinhhhrghrughtuceophhssehp khhsrdhimheqnecuggftrfgrthhtvghrnhepheeghfdtfeeuffehkefgffduleffjedthf dvjeektdfhhedvlefgtefgvdettdfhnecuvehluhhsthgvrhfuihiivgeptdenucfrrghr rghmpehmrghilhhfrhhomhepphhssehpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Fri, 26 Jan 2024 05:31:45 -0500 (EST) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 7477d9d2 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Fri, 26 Jan 2024 10:28:30 +0000 (UTC) Date: Fri, 26 Jan 2024 11:31:39 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Subject: [PATCH 2/5] reftable/writer: use correct type to iterate through index entries Message-ID: <88541d03bef7c6ff425d330b153f3e3d94ca7567.1706263918.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The reftable writer is tracking the number of blocks it has to index via the `index_len` variable. But while this variable is of type `size_t`, some sites use an `int` to loop through the index entries. Convert the code to consistently use `size_t`. Signed-off-by: Patrick Steinhardt --- reftable/writer.c | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/reftable/writer.c b/reftable/writer.c index 92935baa70..5a0b87b406 100644 --- a/reftable/writer.c +++ b/reftable/writer.c @@ -379,20 +379,21 @@ int reftable_writer_add_logs(struct reftable_writer *w, static int writer_finish_section(struct reftable_writer *w) { + struct reftable_block_stats *bstats = NULL; uint8_t typ = block_writer_type(w->block_writer); uint64_t index_start = 0; int max_level = 0; - int threshold = w->opts.unpadded ? 1 : 3; + size_t threshold = w->opts.unpadded ? 1 : 3; int before_blocks = w->stats.idx_stats.blocks; - int err = writer_flush_block(w); - int i = 0; - struct reftable_block_stats *bstats = NULL; + int err; + + err = writer_flush_block(w); if (err < 0) return err; while (w->index_len > threshold) { struct reftable_index_record *idx = NULL; - int idx_len = 0; + size_t i, idx_len; max_level++; index_start = w->next; @@ -630,11 +631,8 @@ int reftable_writer_close(struct reftable_writer *w) static void writer_clear_index(struct reftable_writer *w) { - int i = 0; - for (i = 0; i < w->index_len; i++) { + for (size_t i = 0; i < w->index_len; i++) strbuf_release(&w->index[i].last_key); - } - FREE_AND_NULL(w->index); w->index_len = 0; w->index_cap = 0; From patchwork Fri Jan 26 10:31:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13532377 Received: from out2-smtp.messagingengine.com (out2-smtp.messagingengine.com [66.111.4.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EB31A1B598 for ; Fri, 26 Jan 2024 10:31:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=66.111.4.26 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706265109; cv=none; b=LDn50PVpUxjvXCvtVmxTY+x2WKHLtgTnCtjJK47FguNaFKTTzfLwu83pSsbr38qOt21vXMGrKspMXpFlAR+50erTYa9NKK1Gj9ZgVJzbJGb/XOKKU3/zc1snJSJds98q05z4eWik/kC04bWu0JnjrkTmMxUvbLhtKA2ypkv3kzc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706265109; c=relaxed/simple; bh=OE6gNBxWsueAZ04PutfdW6Tz5UwRgWpNC+Zi5vjCL9A=; h=Date:From:To:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=AJ3V5zT98Ha18b+wyEVPqFK/bOMvZHjSWDfb95vVs0K2gelZKhK+ba0LEmr3SrYm6ld6+nDhqQRtrxDPDKWV+Em3+jj/6DZuinkLQgbr4PzuAYxLqTToCFOfwuGxszwk9QIWXFAtgUw6giBHUe21xN62LGUS+cKv83s74r/jD8E= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=MJVZiHNs; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=nD0Z8f0t; arc=none smtp.client-ip=66.111.4.26 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="MJVZiHNs"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="nD0Z8f0t" Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 4323C5C0121 for ; Fri, 26 Jan 2024 05:31:47 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Fri, 26 Jan 2024 05:31:47 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1706265107; x=1706351507; bh=dI1RiSYF/6 3y7pDlTvatrs6i/PlQ4cfE/1tQq3ZzT5c=; b=MJVZiHNsaAcAspAL50s+2wCr06 irvDEl7uzdSUVhC2sYssLYQUFKzla1cO7bH2sl7W5vrhIIrcwfg8w39LUmY6FZRh dPIU5YESElOQ60jzlGkvj4B1dWLZXiEUdPoBd8RcXp2I5GCBQC7FwbYrvWRfpXF+ 80UmPWXMEgy2bUgwzy8I1CpsH2nO70h3AUCRgceLSeXuSKCI7MarVkvTyriJzx3j dtdh+FoEqSjkeY+VpqpiVVqTpbM3WHH4S9oeAkYL4Jp1VtLOSQH4uuapCDz2UHQC UbZEISuaDmCZ+dSWTI89DGQTC0Ahml6Jt+C6g1oCk1K8Sxh2oXcyIflwd7ww== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1706265107; x=1706351507; bh=dI1RiSYF/63y7pDlTvatrs6i/PlQ 4cfE/1tQq3ZzT5c=; b=nD0Z8f0tAYoEEE8l6F1bH6rrGrUsHB/MmssSq6Ti3FAx 1PWKh74nQvJjAYBevWElta9MbfJHrEnvyApwn/5JaJC0QA8x6O8s3NxR0+rp3uAH XagM8vZEUGrX7QUQdZwbZ04dfTRjCmIZtA4y4T/S0WGLaPp+R7aXiHLTALIYPVyK u8a2iJbqybNkxmrIj8wTePCpTnRAMbA3zqoCPRczSGSj6lZyzm9WpCdqYFJ+DeyT +0P+0aYCKu9eMBtd640LgX/zAM1Y2kpOZiVrunjIL3EZr/feOoeMhtRgHktP36BB 4AYkTDJJYiBY8LZxpimcrW4XHhquDtcfVEceQH7VQg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrvdeljedgudehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuffkfhggtggujgesghdtre ertddtvdenucfhrhhomheprfgrthhrihgtkhcuufhtvghinhhhrghrughtuceophhssehp khhsrdhimheqnecuggftrfgrthhtvghrnhepheeghfdtfeeuffehkefgffduleffjedthf dvjeektdfhhedvlefgtefgvdettdfhnecuvehluhhsthgvrhfuihiivgeptdenucfrrghr rghmpehmrghilhhfrhhomhepphhssehpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Fri, 26 Jan 2024 05:31:46 -0500 (EST) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 85852169 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Fri, 26 Jan 2024 10:28:34 +0000 (UTC) Date: Fri, 26 Jan 2024 11:31:43 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Subject: [PATCH 3/5] reftable/writer: simplify writing index records Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: When finishing the current section we may end up writing index records for the section to the table. The logic to do so essentially copies what we already have in `writer_add_record()`, making this more complicated than it really has to be. Simplify the code by using `writer_add_record()` instead. Signed-off-by: Patrick Steinhardt --- reftable/writer.c | 19 ++++--------------- 1 file changed, 4 insertions(+), 15 deletions(-) diff --git a/reftable/writer.c b/reftable/writer.c index 5a0b87b406..2525f236b9 100644 --- a/reftable/writer.c +++ b/reftable/writer.c @@ -405,6 +405,7 @@ static int writer_finish_section(struct reftable_writer *w) w->index = NULL; w->index_len = 0; w->index_cap = 0; + for (i = 0; i < idx_len; i++) { struct reftable_record rec = { .type = BLOCK_TYPE_INDEX, @@ -412,26 +413,14 @@ static int writer_finish_section(struct reftable_writer *w) .idx = idx[i], }, }; - if (block_writer_add(w->block_writer, &rec) == 0) { - continue; - } - err = writer_flush_block(w); + err = writer_add_record(w, &rec); if (err < 0) return err; - - writer_reinit_block_writer(w, BLOCK_TYPE_INDEX); - - err = block_writer_add(w->block_writer, &rec); - if (err != 0) { - /* write into fresh block should always succeed - */ - abort(); - } } - for (i = 0; i < idx_len; i++) { + + for (i = 0; i < idx_len; i++) strbuf_release(&idx[i].last_key); - } reftable_free(idx); } From patchwork Fri Jan 26 10:31:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13532378 Received: from out2-smtp.messagingengine.com (out2-smtp.messagingengine.com [66.111.4.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1F0C81B80E for ; Fri, 26 Jan 2024 10:31:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=66.111.4.26 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706265112; cv=none; b=gbZnyprs0dpAjd0h4TI3KRJgsVf1K2REgGjgYdFqTHUZqVgaFnhPWwI/JkjIIjWWLKBMRYRqGoCB7FszKwbThIUBssc5e2dj822JXcg4SICnRsewnx/ZHkq+alyiahN0tBlHycUT7uQ8NT4apMJi3psYiAEyhKOZSoCnK/XMFLA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706265112; c=relaxed/simple; bh=9agBEGZ14TW/0p4Jn3XtRYZeYDyf/ycS72CW7kW/oTQ=; h=Date:From:To:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=mrK2DHq+dJYgp/YFQ0O5OiJFbwU0GrUslRTmaSGddKKempO6BQzbthRTA1Srmmsdv+WIXbGVkahDrksPyxwRduItZ4jogMhf5elJcVutDRDSBrnkdBtFyjogOG7DaarzpFgbxRfKXOpf2sfN87JDgLTd6wB+PwtRcWLwKUtCxXA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=RH7O70b+; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=FvY0l62h; arc=none smtp.client-ip=66.111.4.26 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="RH7O70b+"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="FvY0l62h" Received: from compute7.internal (compute7.nyi.internal [10.202.2.48]) by mailout.nyi.internal (Postfix) with ESMTP id 2AB795C010A for ; Fri, 26 Jan 2024 05:31:50 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute7.internal (MEProxy); Fri, 26 Jan 2024 05:31:50 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1706265110; x=1706351510; bh=jvloKCGrbR 9bszjIT6AOagvyaB1YMCFqsOlZidWApXM=; b=RH7O70b+0UkUxR/EUhWVG9cJ89 luHlYL4sUMFKDpy85m9wEoznZHgYreDOD4nZmO9eGbITmd4zMPvrwb+sTmrZj8fP 0FyBAXxdp+Nup6lf//LaNHSCl4nuYpbj4EnpZtBQRdxOeZDN8MQN4Mom08tWNwov Y1OMQXmktbHBwsF6PTk2lMymuSGtdmKgbCz+3RyHTIi8vjXj8reXlbkzP7Eq0frt bbPpVJRgS3QvluznA3pkQQVQ7u4enQMiX9D3t4Q7FZfLZ4CIPaGYfFcVQ3fZEXh8 lgfwdhWaH3juB291iOl6pQ8/FDTLNTqGR9m0YXDVuMXfAnsboVdKJbc6uR5A== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1706265110; x=1706351510; bh=jvloKCGrbR9bszjIT6AOagvyaB1Y MCFqsOlZidWApXM=; b=FvY0l62ho7Xg9GhS1+sVoyieucixD4hN57WdecsFOU4/ qM6Nzi+IGfo9g1PfiaG4JJbAXFQkn0TEkmCJt8YZKST9kooepZxt5MFKp/gz8W/8 wFfYYx+AbXfmTgs8M5sjUv+uuGN8KEse6B2PCr3Ut0p72CMQQ+GdHa/RhiyP16kJ HXI9GDQSgQ4HhdXiQYX0ysB9k0ja4zYyLavb7bClqu0XNA7xRn1x5kGKfciPwZlo lc4NyecTT2W0+Fw6cb2PeFqhvMzchc0+3+fFjhTJckA0isJAPwGZONdt82X4vJU+ D2pqaERMatmYqeyqEMoBsD3azuFPyk1yfWfhUvcf+Q== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrvdeljedgudehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuffkfhggtggujgesghdtre ertddtvdenucfhrhhomheprfgrthhrihgtkhcuufhtvghinhhhrghrughtuceophhssehp khhsrdhimheqnecuggftrfgrthhtvghrnhepheeghfdtfeeuffehkefgffduleffjedthf dvjeektdfhhedvlefgtefgvdettdfhnecuvehluhhsthgvrhfuihiivgeptdenucfrrghr rghmpehmrghilhhfrhhomhepphhssehpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Fri, 26 Jan 2024 05:31:49 -0500 (EST) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 4d5a041e (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Fri, 26 Jan 2024 10:28:39 +0000 (UTC) Date: Fri, 26 Jan 2024 11:31:47 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Subject: [PATCH 4/5] reftable/writer: fix writing multi-level indices Message-ID: <9c6622c4095a00f91885a3f96863f81a7b342f84.1706263918.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: When finishing a section we will potentially write an index that makes it more efficient to look up relevant blocks. The index records written will encode, for each block of the indexed section, what the offset of that block is as well as the last key of that block. Thus, the reader would iterate through the index records to find the first key larger or equal to the wanted key and then use the encoded offset to look up the desired block. When there are a lot of blocks to index though we may end up writing multiple index blocks, too. To not require a linear search across all index blocks we instead end up writing a multi-level index. Instead of referring to the block we are after, an index record may point to another index block. The reader will then access the highest-level index and follow down the chain of index blocks until it hits the sought-after block. It has been observed though that it is impossible to seek ref records of the last ref block when using a multi-level index. While the multi-level index exists and looks fine for most of the part, the highest-level index was missing an index record pointing to the last block of the next index. Thus, every additional level made more refs become unseekable at the end of the ref section. The root cause is that we are not flushing the last block of the current level once done writing the level. Consequently, it wasn't recorded in the blocks that need to be indexed by the next-higher level and thus we forgot about it. Fix this bug by flushing blocks after we have written all index records. Signed-off-by: Patrick Steinhardt --- reftable/readwrite_test.c | 56 +++++++++++++++++++++++++++++++++++++++ reftable/writer.c | 8 +++--- 2 files changed, 60 insertions(+), 4 deletions(-) diff --git a/reftable/readwrite_test.c b/reftable/readwrite_test.c index 6b99daeaf2..853923397e 100644 --- a/reftable/readwrite_test.c +++ b/reftable/readwrite_test.c @@ -866,6 +866,61 @@ static void test_write_multiple_indices(void) strbuf_release(&buf); } +static void test_write_multi_level_index(void) +{ + struct reftable_write_options opts = { + .block_size = 100, + }; + struct strbuf writer_buf = STRBUF_INIT, buf = STRBUF_INIT; + struct reftable_block_source source = { 0 }; + struct reftable_iterator it = { 0 }; + const struct reftable_stats *stats; + struct reftable_writer *writer; + struct reftable_reader *reader; + int err; + + writer = reftable_new_writer(&strbuf_add_void, &noop_flush, &writer_buf, &opts); + reftable_writer_set_limits(writer, 1, 1); + for (size_t i = 0; i < 200; i++) { + struct reftable_ref_record ref = { + .update_index = 1, + .value_type = REFTABLE_REF_VAL1, + .value.val1 = {i}, + }; + + strbuf_reset(&buf); + strbuf_addf(&buf, "refs/heads/%03" PRIuMAX, (uintmax_t)i); + ref.refname = buf.buf, + + err = reftable_writer_add_ref(writer, &ref); + EXPECT_ERR(err); + } + reftable_writer_close(writer); + + /* + * The written refs should be sufficiently large to result in a + * multi-level index. + */ + stats = reftable_writer_stats(writer); + EXPECT(stats->ref_stats.max_index_level == 2); + + block_source_from_strbuf(&source, &writer_buf); + err = reftable_new_reader(&reader, &source, "filename"); + EXPECT_ERR(err); + + /* + * Seeking the last ref should work as expected. + */ + err = reftable_reader_seek_ref(reader, &it, "refs/heads/199"); + EXPECT_ERR(err); + + reftable_iterator_destroy(&it); + reftable_writer_free(writer); + reftable_reader_free(reader); + strbuf_release(&writer_buf); + strbuf_release(&buf); +} + static void test_corrupt_table_empty(void) { struct strbuf buf = STRBUF_INIT; @@ -916,5 +971,6 @@ int readwrite_test_main(int argc, const char *argv[]) RUN_TEST(test_write_object_id_length); RUN_TEST(test_write_object_id_min_length); RUN_TEST(test_write_multiple_indices); + RUN_TEST(test_write_multi_level_index); return 0; } diff --git a/reftable/writer.c b/reftable/writer.c index 2525f236b9..24fce5630a 100644 --- a/reftable/writer.c +++ b/reftable/writer.c @@ -419,15 +419,15 @@ static int writer_finish_section(struct reftable_writer *w) return err; } + err = writer_flush_block(w); + if (err < 0) + return err; + for (i = 0; i < idx_len; i++) strbuf_release(&idx[i].last_key); reftable_free(idx); } - err = writer_flush_block(w); - if (err < 0) - return err; - writer_clear_index(w); bstats = writer_reftable_block_stats(w, typ); From patchwork Fri Jan 26 10:31:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13532379 Received: from out2-smtp.messagingengine.com (out2-smtp.messagingengine.com [66.111.4.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E8BF71B80E for ; Fri, 26 Jan 2024 10:31:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=66.111.4.26 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706265116; cv=none; b=hS5TB0OrxW+gLIIU6UI9f4XLgiZcPqoBviDRow925ZuHfg1t0ENWg0r6JipqAjPVkwH8ZPg4llauzXS1wK91dWuZ4QIzIEq7N9Bk7DCPik0pV0aL5kHQ2I0uAcSAqHJsIzd+fTM4vUl6z0mceerf1t8qhzW0agSAIZ1bTY1umAA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706265116; c=relaxed/simple; bh=1v/nHfdwS8kPmN8on6pGM3FwVdrO0Rhv36OUdjW85zE=; h=Date:From:To:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=SSNyXaUCJdxMUssN9ZVJmTL7ohU4vUWaATXdtS4hsbq7YtD/jJgXZ58+ZiV3lw6dvbwKLEb3jrEJV75xgoUjZa0X8rb2UD6Wqq4jmiPiTExX5QNNkXFjQvzMSEp3siHxXnBivkPkVLriHdaPegfiJkmwvmrjK+fWoMW+QpYf/Ro= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=qij7Nkmk; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=eqFcp4kP; arc=none smtp.client-ip=66.111.4.26 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="qij7Nkmk"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="eqFcp4kP" Received: from compute7.internal (compute7.nyi.internal [10.202.2.48]) by mailout.nyi.internal (Postfix) with ESMTP id 1311C5C010A for ; Fri, 26 Jan 2024 05:31:54 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute7.internal (MEProxy); Fri, 26 Jan 2024 05:31:54 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1706265114; x=1706351514; bh=nQUMYDwSYb COlEfm4Lao2i8Wcz7akk874ms2hxBLpsU=; b=qij7NkmkQbfq060zrN+4WCI8my oUi5CyxTly8joFXPBxbBZ1CpjV6pZLXxSOZtrBxBuOey4RjHM+71+mRMCvqkbP50 AGEjM4BerPctgrW5U+fv6ac9SGUY6qtFTdYr2Mo+x3O9bviO4EHE4xuiVDw6HxNx ujJyxUuxUj8OZ/WNW6Vh0Lzy2fPpEZhrIsDTRvgTaFUFMwkkohPN0PUCNJl+ktpo mQgFjCVFwUfFAkJgAWK1hwzyjlUXjWW0Pk7ws8AK06Uc4jrYd6TTwx95JZ8W6FyR gIvbiWJmXN6JNk/0iYER6CxstZIjylNqurNJ3I3wda3HtiH8J2IYdrNEVxEg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1706265114; x=1706351514; bh=nQUMYDwSYbCOlEfm4Lao2i8Wcz7a kk874ms2hxBLpsU=; b=eqFcp4kPw/I0DAtvPWdcj/snfux0VmHUYcx4ZhOBD+F7 Eaw3z994VmWtKicuZANszwbVMlvi6cSnuzU+vrVDPisM76qpKd62biAC1rGfn1/e EEjeQ5WfiB3xpcmctsfBwMH5sfiluKmlD1WTRjcdi09bae9j+ybBgiaH9ECXUsSW ywx+uYY8h24LrjMZHT/nqdmXy2/J2drMC9laWO6wqk1X+ZVPORTe/YO8DTKj5qTh +d0lfxGbeZHrgAvyTb3xFIksWB8bF9GZi2Khck79wJuzMwQM/fFynUiQJy821X58 imXGdRqqjADOZKCU1a0AO0FzaxvcW/PY6lAvCbkX0Q== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvkedrvdeljedgudehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuffkfhggtggujgesghdtre ertddtvdenucfhrhhomheprfgrthhrihgtkhcuufhtvghinhhhrghrughtuceophhssehp khhsrdhimheqnecuggftrfgrthhtvghrnhepheeghfdtfeeuffehkefgffduleffjedthf dvjeektdfhhedvlefgtefgvdettdfhnecuvehluhhsthgvrhfuihiivgepudenucfrrghr rghmpehmrghilhhfrhhomhepphhssehpkhhsrdhimh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Fri, 26 Jan 2024 05:31:53 -0500 (EST) Received: by vm-mail (OpenSMTPD) with ESMTPSA id d641479f (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO) for ; Fri, 26 Jan 2024 10:28:43 +0000 (UTC) Date: Fri, 26 Jan 2024 11:31:51 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Subject: [PATCH 5/5] reftable: document reading and writing indices Message-ID: <7850e65878e42c6ae5064554a3681a3ea1a7ae6e.1706263918.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The way the index gets written and read is not trivial at all and requires the reader to piece together a bunch of parts to figure out how it works. Add some documentation to hopefully make this easier to understand for the next reader. Signed-off-by: Patrick Steinhardt --- reftable/reader.c | 27 +++++++++++++++++++++++++++ reftable/writer.c | 23 +++++++++++++++++++++++ 2 files changed, 50 insertions(+) diff --git a/reftable/reader.c b/reftable/reader.c index 278f727a3d..6011d0aa04 100644 --- a/reftable/reader.c +++ b/reftable/reader.c @@ -508,11 +508,38 @@ static int reader_seek_indexed(struct reftable_reader *r, if (err < 0) goto done; + /* + * The index may consist of multiple levels, where each level may have + * multiple index blocks. We start by doing a linear search in the + * highest layer that identifies the relevant index block as well as + * the record inside that block that corresponds to our wanted key. + */ err = reader_seek_linear(&index_iter, &want_index); if (err < 0) goto done; + /* + * Traverse down the levels until we find a non-index entry. + */ while (1) { + /* + * In case we seek a record that does not exist the index iter + * will tell us that the iterator is over. This works because + * the last index entry of the current level will contain the + * last key it knows about. So in case our seeked key is larger + * than the last indexed key we know that it won't exist. + * + * There is one subtlety in the layout of the index section + * that makes this work as expected: the highest-level index is + * at end of the section and will point backwards and thus we + * start reading from the end of the index section, not the + * beginning. + * + * If that wasn't the case and the order was reversed then the + * linear seek would seek into the lower levels and traverse + * all levels of the index only to find out that the key does + * not exist. + */ err = table_iter_next(&index_iter, &index_result); table_iter_block_done(&index_iter); if (err != 0) diff --git a/reftable/writer.c b/reftable/writer.c index 24fce5630a..6028789dee 100644 --- a/reftable/writer.c +++ b/reftable/writer.c @@ -391,6 +391,24 @@ static int writer_finish_section(struct reftable_writer *w) if (err < 0) return err; + /* + * When the section we are about to index has a lot of blocks then the + * index itself may span across multiple blocks, as well. This would + * require a linear scan over index blocks only to find the desired + * indexed block, which is inefficient. Instead, we write a multi-level + * index where index records of level N+1 will refer to index blocks of + * level N. This isn't constant time, either, but at least logarithmic. + * + * This loop handles writing this multi-level index. Note that we write + * the lowest-level index pointing to the indexed blocks first. We then + * continue writing additional index levels until the current level has + * less blocks than the threshold so that the highest level will be at + * the end of the index section. + * + * Readers are thus required to start reading the index section from + * its end, which is why we set `index_start` to the beginning of the + * last index section. + */ while (w->index_len > threshold) { struct reftable_index_record *idx = NULL; size_t i, idx_len; @@ -428,6 +446,11 @@ static int writer_finish_section(struct reftable_writer *w) reftable_free(idx); } + /* + * The index may still contain a number of index blocks lower than the + * threshold. Clear it so that these entries don't leak into the next + * index section. + */ writer_clear_index(w); bstats = writer_reftable_block_stats(w, typ);