From patchwork Tue Nov 5 09:11:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13862684 Received: from fhigh-b6-smtp.messagingengine.com (fhigh-b6-smtp.messagingengine.com [202.12.124.157]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0220F1D0178 for ; Tue, 5 Nov 2024 09:12:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=202.12.124.157 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730797930; cv=none; b=UcninI+L9G2YhI3Ag588vl28Lze6zkTsewEmVVebNxWYnwTOfcyRnaJbH+50LDy0+wYCLdpHn7tZLVqNrA9qx5aR3bRO7uShRqZCZZ2CRK8uXWBgcYzvQP2RMrgoi4X6Sn5LfDF1CSCEk+SS9wpEYqKctaEndthbyu022Mv46Ac= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730797930; c=relaxed/simple; bh=Tm1oNdX9U6in/xWokSHxBUDwSGLPW+NLibKjijzcEEo=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=pK9l0HBUuk/IXplPVtks5HNlWcmRtJtBC3kODYHO37vDhq9WJFCr/eQpwV4MHo5Mk6avcTPifxvij9yim1XkLu/BHeX87aptu5dwd8v6UW1rJLljc+Kj+dJbqxGkYXefhRqWbcyQhpry28JtrKWdLeam115oBlNWoxM+C6yxeus= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=JGiDuFXA; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=iqCN7wmW; arc=none smtp.client-ip=202.12.124.157 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="JGiDuFXA"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="iqCN7wmW" Received: from phl-compute-02.internal (phl-compute-02.phl.internal [10.202.2.42]) by mailfhigh.stl.internal (Postfix) with ESMTP id 04884254012F; Tue, 5 Nov 2024 04:12:06 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-02.internal (MEProxy); Tue, 05 Nov 2024 04:12:07 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1730797926; x=1730884326; bh=wYHPK8fOUi o7+oxZJpDpyLQkA6xgtH6hjFcp9Fy2bMU=; b=JGiDuFXA6LHpM5Qan+16RmaYhp nSGpyYlhVKKtV+y99wbxTdeoZI235ILgP7Gqyl4aQHHW5pjFdx4pm6KcHjCkQqpD Up2377C+Wb6lE88znnhSGNR4ze8Y6J5nZZdju1OdXkYzq5zVq/N2M+tVzwicNS+b RcNbbs1QcfqR1zhuIU25Qh52zEMo8RdglI8zgJJblpiUL2g2+4xmuBbOlqzwqRxO TvyIspMGF6S0DwsjtXPfktkw6Zi4OZog/wxWz3tYcWq7ih+1rpSHEvj7UcCGwMhG wJ9S4vFUTmrvOUnZyGzbpGjpdtVUNekDduTzJ2z/BcDlQH7dCb3OjXmMaLxA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t= 1730797926; x=1730884326; bh=wYHPK8fOUio7+oxZJpDpyLQkA6xgtH6hjFc p9Fy2bMU=; b=iqCN7wmW7owe9R60m41yD5agJ5VgTgrwet1ueToglooGIBrUqnf FBSfhF3tOB61meSwqkzUTJdgWM5nXU8V6XbPG1aww3FCesi18xzBakZXwZ35ObsJ tlGm1RvHy3eRJPXPvNbO+odx+YxljmyGIhY7Fh0DV2nXkDdDYEYUMgCqvEyQNP/R lOzkTLU6B9EEpHJIZJhQyL6y9nd9XKIoJz2pwHFJxwf4xLoDP09Ualu2ZNwRieHI aPNk9i3DdDysMEzvvPZ94fK2Grb06rX81t5SApptuBzj3ZOScGbVRUG43BDz+11a XJ4kbYsiNIclyHX8zMixrjPkm3KEE0Ar+IQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddrvdelkedgtdduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvve fukfhfgggtuggjsehttdertddttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhn hhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrhhnpeevkeekfffhie dtleduiefgjedttedvledvudehgfeugedugffhueekhfejvdektdenucevlhhushhtvghr ufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhmpdhnsg gprhgtphhtthhopedvpdhmohguvgepshhmthhpohhuthdprhgtphhtthhopehgihhtshht vghrsehpohgsohigrdgtohhmpdhrtghpthhtohepghhithesvhhgvghrrdhkvghrnhgvlh drohhrgh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 5 Nov 2024 04:12:06 -0500 (EST) Received: by vm-mail (OpenSMTPD) with ESMTPSA id dfd7a2c3 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Tue, 5 Nov 2024 09:11:44 +0000 (UTC) Date: Tue, 5 Nov 2024 10:11:59 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Junio C Hamano Subject: [PATCH v2 1/8] refs/reftable: encapsulate reftable stack Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The reftable ref store needs to keep track of multiple stacks, one for the main worktree and an arbitrary number of stacks for worktrees. This is done by storing pointers to `struct reftable_stack`, which we then access directly. Wrap the stack in a new `struct reftable_backend`. This will allow us to attach more data to each respective stack in subsequent commits. Signed-off-by: Patrick Steinhardt --- refs/reftable-backend.c | 129 +++++++++++++++++++++++----------------- 1 file changed, 73 insertions(+), 56 deletions(-) diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c index f560bc2b67..116cc5ec23 100644 --- a/refs/reftable-backend.c +++ b/refs/reftable-backend.c @@ -34,6 +34,23 @@ */ #define REF_UPDATE_VIA_HEAD (1 << 8) +struct reftable_backend { + struct reftable_stack *stack; +}; + +static int reftable_backend_init(struct reftable_backend *be, + const char *path, + const struct reftable_write_options *opts) +{ + return reftable_new_stack(&be->stack, path, opts); +} + +static void reftable_backend_release(struct reftable_backend *be) +{ + reftable_stack_destroy(be->stack); + be->stack = NULL; +} + struct reftable_ref_store { struct ref_store base; @@ -41,17 +58,17 @@ struct reftable_ref_store { * The main stack refers to the common dir and thus contains common * refs as well as refs of the main repository. */ - struct reftable_stack *main_stack; + struct reftable_backend main_backend; /* * The worktree stack refers to the gitdir in case the refdb is opened * via a worktree. It thus contains the per-worktree refs. */ - struct reftable_stack *worktree_stack; + struct reftable_backend worktree_backend; /* * Map of worktree stacks by their respective worktree names. The map * is populated lazily when we try to resolve `worktrees/$worktree` refs. */ - struct strmap worktree_stacks; + struct strmap worktree_backends; struct reftable_write_options write_options; unsigned int store_flags; @@ -97,21 +114,21 @@ static struct reftable_ref_store *reftable_be_downcast(struct ref_store *ref_sto * like `worktrees/$worktree/refs/heads/foo` as worktree stacks will store * those references in their normalized form. */ -static struct reftable_stack *stack_for(struct reftable_ref_store *store, - const char *refname, - const char **rewritten_ref) +static struct reftable_backend *backend_for(struct reftable_ref_store *store, + const char *refname, + const char **rewritten_ref) { const char *wtname; int wtname_len; if (!refname) - return store->main_stack; + return &store->main_backend; switch (parse_worktree_ref(refname, &wtname, &wtname_len, rewritten_ref)) { case REF_WORKTREE_OTHER: { static struct strbuf wtname_buf = STRBUF_INIT; struct strbuf wt_dir = STRBUF_INIT; - struct reftable_stack *stack; + struct reftable_backend *be; /* * We're using a static buffer here so that we don't need to @@ -125,37 +142,39 @@ static struct reftable_stack *stack_for(struct reftable_ref_store *store, /* * There is an edge case here: when the worktree references the * current worktree, then we set up the stack once via - * `worktree_stacks` and once via `worktree_stack`. This is + * `worktree_backends` and once via `worktree_backend`. This is * wasteful, but in the reading case it shouldn't matter. And * in the writing case we would notice that the stack is locked * already and error out when trying to write a reference via * both stacks. */ - stack = strmap_get(&store->worktree_stacks, wtname_buf.buf); - if (!stack) { + be = strmap_get(&store->worktree_backends, wtname_buf.buf); + if (!be) { strbuf_addf(&wt_dir, "%s/worktrees/%s/reftable", store->base.repo->commondir, wtname_buf.buf); - store->err = reftable_new_stack(&stack, wt_dir.buf, - &store->write_options); + CALLOC_ARRAY(be, 1); + store->err = reftable_backend_init(be, wt_dir.buf, + &store->write_options); assert(store->err != REFTABLE_API_ERROR); - strmap_put(&store->worktree_stacks, wtname_buf.buf, stack); + + strmap_put(&store->worktree_backends, wtname_buf.buf, be); } strbuf_release(&wt_dir); - return stack; + return be; } case REF_WORKTREE_CURRENT: /* * If there is no worktree stack then we're currently in the * main worktree. We thus return the main stack in that case. */ - if (!store->worktree_stack) - return store->main_stack; - return store->worktree_stack; + if (!store->worktree_backend.stack) + return &store->main_backend; + return &store->worktree_backend; case REF_WORKTREE_MAIN: case REF_WORKTREE_SHARED: - return store->main_stack; + return &store->main_backend; default: BUG("unhandled worktree reference type"); } @@ -292,7 +311,7 @@ static struct ref_store *reftable_be_init(struct repository *repo, umask(mask); base_ref_store_init(&refs->base, repo, gitdir, &refs_be_reftable); - strmap_init(&refs->worktree_stacks); + strmap_init(&refs->worktree_backends); refs->store_flags = store_flags; refs->log_all_ref_updates = repo_settings_get_log_all_ref_updates(repo); @@ -337,8 +356,8 @@ static struct ref_store *reftable_be_init(struct repository *repo, strbuf_realpath(&path, gitdir, 0); } strbuf_addstr(&path, "/reftable"); - refs->err = reftable_new_stack(&refs->main_stack, path.buf, - &refs->write_options); + refs->err = reftable_backend_init(&refs->main_backend, path.buf, + &refs->write_options); if (refs->err) goto done; @@ -354,8 +373,8 @@ static struct ref_store *reftable_be_init(struct repository *repo, strbuf_reset(&path); strbuf_addf(&path, "%s/reftable", gitdir); - refs->err = reftable_new_stack(&refs->worktree_stack, path.buf, - &refs->write_options); + refs->err = reftable_backend_init(&refs->worktree_backend, path.buf, + &refs->write_options); if (refs->err) goto done; } @@ -374,19 +393,17 @@ static void reftable_be_release(struct ref_store *ref_store) struct strmap_entry *entry; struct hashmap_iter iter; - if (refs->main_stack) { - reftable_stack_destroy(refs->main_stack); - refs->main_stack = NULL; - } + if (refs->main_backend.stack) + reftable_backend_release(&refs->main_backend); + if (refs->worktree_backend.stack) + reftable_backend_release(&refs->worktree_backend); - if (refs->worktree_stack) { - reftable_stack_destroy(refs->worktree_stack); - refs->worktree_stack = NULL; + strmap_for_each_entry(&refs->worktree_backends, &iter, entry) { + struct reftable_backend *be = entry->value; + reftable_backend_release(be); + free(be); } - - strmap_for_each_entry(&refs->worktree_stacks, &iter, entry) - reftable_stack_destroy(entry->value); - strmap_clear(&refs->worktree_stacks, 0); + strmap_clear(&refs->worktree_backends, 0); } static int reftable_be_create_on_disk(struct ref_store *ref_store, @@ -781,7 +798,7 @@ static struct ref_iterator *reftable_be_iterator_begin(struct ref_store *ref_sto required_flags |= REF_STORE_ODB; refs = reftable_be_downcast(ref_store, required_flags, "ref_iterator_begin"); - main_iter = ref_iterator_for_stack(refs, refs->main_stack, prefix, + main_iter = ref_iterator_for_stack(refs, refs->main_backend.stack, prefix, exclude_patterns, flags); /* @@ -789,14 +806,14 @@ static struct ref_iterator *reftable_be_iterator_begin(struct ref_store *ref_sto * right now. If we aren't, then we return the common reftable * iterator, only. */ - if (!refs->worktree_stack) + if (!refs->worktree_backend.stack) return &main_iter->base; /* * Otherwise we merge both the common and the per-worktree refs into a * single iterator. */ - worktree_iter = ref_iterator_for_stack(refs, refs->worktree_stack, prefix, + worktree_iter = ref_iterator_for_stack(refs, refs->worktree_backend.stack, prefix, exclude_patterns, flags); return merge_ref_iterator_begin(&worktree_iter->base, &main_iter->base, ref_iterator_select, NULL); @@ -811,7 +828,7 @@ static int reftable_be_read_raw_ref(struct ref_store *ref_store, { struct reftable_ref_store *refs = reftable_be_downcast(ref_store, REF_STORE_READ, "read_raw_ref"); - struct reftable_stack *stack = stack_for(refs, refname, &refname); + struct reftable_stack *stack = backend_for(refs, refname, &refname)->stack; int ret; if (refs->err < 0) @@ -838,7 +855,7 @@ static int reftable_be_read_symbolic_ref(struct ref_store *ref_store, { struct reftable_ref_store *refs = reftable_be_downcast(ref_store, REF_STORE_READ, "read_symbolic_ref"); - struct reftable_stack *stack = stack_for(refs, refname, &refname); + struct reftable_stack *stack = backend_for(refs, refname, &refname)->stack; struct reftable_ref_record ref = {0}; int ret; @@ -898,7 +915,7 @@ static int prepare_transaction_update(struct write_transaction_table_arg **out, struct ref_update *update, struct strbuf *err) { - struct reftable_stack *stack = stack_for(refs, update->refname, NULL); + struct reftable_stack *stack = backend_for(refs, update->refname, NULL)->stack; struct write_transaction_table_arg *arg = NULL; size_t i; int ret; @@ -1031,7 +1048,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store, goto done; } - ret = read_ref_without_reload(refs, stack_for(refs, "HEAD", NULL), "HEAD", + ret = read_ref_without_reload(refs, backend_for(refs, "HEAD", NULL)->stack, "HEAD", &head_oid, &head_referent, &head_type); if (ret < 0) goto done; @@ -1043,7 +1060,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store, struct reftable_stack *stack; const char *rewritten_ref; - stack = stack_for(refs, u->refname, &rewritten_ref); + stack = backend_for(refs, u->refname, &rewritten_ref)->stack; /* Verify that the new object ID is valid. */ if ((u->flags & REF_HAVE_NEW) && !is_null_oid(&u->new_oid) && @@ -1525,9 +1542,9 @@ static int reftable_be_pack_refs(struct ref_store *ref_store, if (refs->err) return refs->err; - stack = refs->worktree_stack; + stack = refs->worktree_backend.stack; if (!stack) - stack = refs->main_stack; + stack = refs->main_backend.stack; if (opts->flags & PACK_REFS_AUTO) ret = reftable_stack_auto_compact(stack); @@ -1782,7 +1799,7 @@ static int reftable_be_rename_ref(struct ref_store *ref_store, { struct reftable_ref_store *refs = reftable_be_downcast(ref_store, REF_STORE_WRITE, "rename_ref"); - struct reftable_stack *stack = stack_for(refs, newrefname, &newrefname); + struct reftable_stack *stack = backend_for(refs, newrefname, &newrefname)->stack; struct write_copy_arg arg = { .refs = refs, .stack = stack, @@ -1814,7 +1831,7 @@ static int reftable_be_copy_ref(struct ref_store *ref_store, { struct reftable_ref_store *refs = reftable_be_downcast(ref_store, REF_STORE_WRITE, "copy_ref"); - struct reftable_stack *stack = stack_for(refs, newrefname, &newrefname); + struct reftable_stack *stack = backend_for(refs, newrefname, &newrefname)->stack; struct write_copy_arg arg = { .refs = refs, .stack = stack, @@ -1952,11 +1969,11 @@ static struct ref_iterator *reftable_be_reflog_iterator_begin(struct ref_store * reftable_be_downcast(ref_store, REF_STORE_READ, "reflog_iterator_begin"); struct reftable_reflog_iterator *main_iter, *worktree_iter; - main_iter = reflog_iterator_for_stack(refs, refs->main_stack); - if (!refs->worktree_stack) + main_iter = reflog_iterator_for_stack(refs, refs->main_backend.stack); + if (!refs->worktree_backend.stack) return &main_iter->base; - worktree_iter = reflog_iterator_for_stack(refs, refs->worktree_stack); + worktree_iter = reflog_iterator_for_stack(refs, refs->worktree_backend.stack); return merge_ref_iterator_begin(&worktree_iter->base, &main_iter->base, ref_iterator_select, NULL); @@ -1995,7 +2012,7 @@ static int reftable_be_for_each_reflog_ent_reverse(struct ref_store *ref_store, { struct reftable_ref_store *refs = reftable_be_downcast(ref_store, REF_STORE_READ, "for_each_reflog_ent_reverse"); - struct reftable_stack *stack = stack_for(refs, refname, &refname); + struct reftable_stack *stack = backend_for(refs, refname, &refname)->stack; struct reftable_log_record log = {0}; struct reftable_iterator it = {0}; int ret; @@ -2035,7 +2052,7 @@ static int reftable_be_for_each_reflog_ent(struct ref_store *ref_store, { struct reftable_ref_store *refs = reftable_be_downcast(ref_store, REF_STORE_READ, "for_each_reflog_ent"); - struct reftable_stack *stack = stack_for(refs, refname, &refname); + struct reftable_stack *stack = backend_for(refs, refname, &refname)->stack; struct reftable_log_record *logs = NULL; struct reftable_iterator it = {0}; size_t logs_alloc = 0, logs_nr = 0, i; @@ -2084,7 +2101,7 @@ static int reftable_be_reflog_exists(struct ref_store *ref_store, { struct reftable_ref_store *refs = reftable_be_downcast(ref_store, REF_STORE_READ, "reflog_exists"); - struct reftable_stack *stack = stack_for(refs, refname, &refname); + struct reftable_stack *stack = backend_for(refs, refname, &refname)->stack; struct reftable_log_record log = {0}; struct reftable_iterator it = {0}; int ret; @@ -2169,7 +2186,7 @@ static int reftable_be_create_reflog(struct ref_store *ref_store, { struct reftable_ref_store *refs = reftable_be_downcast(ref_store, REF_STORE_WRITE, "create_reflog"); - struct reftable_stack *stack = stack_for(refs, refname, &refname); + struct reftable_stack *stack = backend_for(refs, refname, &refname)->stack; struct write_reflog_existence_arg arg = { .refs = refs, .stack = stack, @@ -2243,7 +2260,7 @@ static int reftable_be_delete_reflog(struct ref_store *ref_store, { struct reftable_ref_store *refs = reftable_be_downcast(ref_store, REF_STORE_WRITE, "delete_reflog"); - struct reftable_stack *stack = stack_for(refs, refname, &refname); + struct reftable_stack *stack = backend_for(refs, refname, &refname)->stack; struct write_reflog_delete_arg arg = { .stack = stack, .refname = refname, @@ -2352,7 +2369,7 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store, */ struct reftable_ref_store *refs = reftable_be_downcast(ref_store, REF_STORE_WRITE, "reflog_expire"); - struct reftable_stack *stack = stack_for(refs, refname, &refname); + struct reftable_stack *stack = backend_for(refs, refname, &refname)->stack; struct reftable_log_record *logs = NULL; struct reftable_log_record *rewritten = NULL; struct reftable_ref_record ref_record = {0}; From patchwork Tue Nov 5 09:12:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13862686 Received: from fout-b3-smtp.messagingengine.com (fout-b3-smtp.messagingengine.com [202.12.124.146]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 42A3C1D07B0 for ; Tue, 5 Nov 2024 09:12:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=202.12.124.146 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730797933; cv=none; b=qAXGTY1v10RfY8zuEQGIHcVjSWRVIZzwLzBl9icHK4rX5oPVSn2TTwSzDluKl+LjQrJAyYmuIpOtP9v7t6nd8kGSO8qsqovjVr2++rU1TR3Fl1GquhLSxVPCbVO5KbMd3qBfau6hjFugUmhPP4MZtuKVJeSb07sbbMVYNfDsfRQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730797933; c=relaxed/simple; bh=Yn4p7Q8WIH37ChdXXj3XObX4pgHhw1o42zRfLr0za94=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=iM32dpzujm5gQBj90i8HUN214Soe8o5O1wSwnlCBxYGHwHGI3+EefD2C1lDTZikW8sWRSLOBAyvJTzy+UlfYunlCxqABZyF52LeW7Tp+01u6JMAU7mYJ2JFnJQZzTDoEvqWjv2DfPOxkjTT4j+qLwKM8Y6aMcXL6ghmmPzp3rFo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=208pCgK6; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=EqroT7pU; arc=none smtp.client-ip=202.12.124.146 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="208pCgK6"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="EqroT7pU" Received: from phl-compute-08.internal (phl-compute-08.phl.internal [10.202.2.48]) by mailfout.stl.internal (Postfix) with ESMTP id 5D8AB1140171; Tue, 5 Nov 2024 04:12:10 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-08.internal (MEProxy); Tue, 05 Nov 2024 04:12:10 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1730797930; x=1730884330; bh=+DyBhWQ2HW W4E9RXpPWiQTGWgVF/ag8wbGrw8gTrYEA=; b=208pCgK6yjhQLKLILFUcjJU5vm tU7pJVoNdic1B6718jxvzA5o9e4KfMTa93+8f+JMQ29se93sasRfpsUs64Zz16eB 0gtO7vlEWeER/d5sKzB4oRvvIkEf9dOCJYPNrvLHbUnkekCMYVQGen2awPoWZ178 eonD2gijd069gJFrjZ2/qc8ZS5peQlc2MZu2LwcR8vA/TEp/nDaVzkG+eqUUD246 acBzA0bdWjkNpPZZtXEuPZLkT0iEZfuav69lNgrxHBuVm52zsvCzPa8Wc7jjegZF FgZg+ZGYLto3m9UNzIDhIDfTczs4/JqnOUjE6y0AfBNHjwazlDMFZLXnUL4w== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t= 1730797930; x=1730884330; bh=+DyBhWQ2HWW4E9RXpPWiQTGWgVF/ag8wbGr w8gTrYEA=; b=EqroT7pULTZscmEY1Mkd/gYdU3kFlQ+9FL96s000LVRLuHZYY65 RWo9vhwz9aXgprhDAgcdC4hr9w1I9DYYoZDyfUT0IMpvjjxfw3TDAmwS+h4v0u5X uvaguHtfBxvmBPipnzeKrrT2NmcwVe225RxJUVt1ia2h7RUgLDdCTbocY9jLTY78 ZoSkIVXJCCCAws/0X3StAHAXFE/uSPKhIdUDJIZxWndOxHQ/BFd87IamVGTUHb0T u3kMmREwwSMD9E+GrU+xxZMmPN8ODpVQFtqEPKqJxzIhhihMuWNNe308IV7irkbv sCm2Fq0asNVvUsdUPtwyXOSQ6nGhjJY5uGw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddrvdelkedgtdduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvve fukfhfgggtuggjsehttdertddttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhn hhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrhhnpeevkeekfffhie dtleduiefgjedttedvledvudehgfeugedugffhueekhfejvdektdenucevlhhushhtvghr ufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhmpdhnsg gprhgtphhtthhopedvpdhmohguvgepshhmthhpohhuthdprhgtphhtthhopehgihhtsehv ghgvrhdrkhgvrhhnvghlrdhorhhgpdhrtghpthhtohepghhithhsthgvrhesphhosghogi drtghomh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 5 Nov 2024 04:12:09 -0500 (EST) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 0efefcb6 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Tue, 5 Nov 2024 09:11:47 +0000 (UTC) Date: Tue, 5 Nov 2024 10:12:02 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Junio C Hamano Subject: [PATCH v2 2/8] refs/reftable: handle reloading stacks in the reftable backend Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: When accessing a stack we almost always have to reload the stack before reading data from it. This is mostly because Git does not have a notification mechanism for when underlying data has been changed, and thus we are forced to opportunistically reload the stack every single time to account for any changes that may have happened concurrently. Handle the reload internally in `backend_for()`. For one this forces callsites to think about whether or not they need to reload the stack. But second this makes the logic to access stacks more self-contained by letting the `struct reftable_backend` manage themselves. Signed-off-by: Patrick Steinhardt --- refs/reftable-backend.c | 150 +++++++++++++++++++++++++--------------- 1 file changed, 93 insertions(+), 57 deletions(-) diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c index 116cc5ec23..4a28dc8a9d 100644 --- a/refs/reftable-backend.c +++ b/refs/reftable-backend.c @@ -114,21 +114,25 @@ static struct reftable_ref_store *reftable_be_downcast(struct ref_store *ref_sto * like `worktrees/$worktree/refs/heads/foo` as worktree stacks will store * those references in their normalized form. */ -static struct reftable_backend *backend_for(struct reftable_ref_store *store, - const char *refname, - const char **rewritten_ref) +static int backend_for(struct reftable_backend **out, + struct reftable_ref_store *store, + const char *refname, + const char **rewritten_ref, + int reload) { + struct reftable_backend *be; const char *wtname; int wtname_len; - if (!refname) - return &store->main_backend; + if (!refname) { + be = &store->main_backend; + goto out; + } switch (parse_worktree_ref(refname, &wtname, &wtname_len, rewritten_ref)) { case REF_WORKTREE_OTHER: { static struct strbuf wtname_buf = STRBUF_INIT; struct strbuf wt_dir = STRBUF_INIT; - struct reftable_backend *be; /* * We're using a static buffer here so that we don't need to @@ -162,7 +166,7 @@ static struct reftable_backend *backend_for(struct reftable_ref_store *store, } strbuf_release(&wt_dir); - return be; + goto out; } case REF_WORKTREE_CURRENT: /* @@ -170,14 +174,27 @@ static struct reftable_backend *backend_for(struct reftable_ref_store *store, * main worktree. We thus return the main stack in that case. */ if (!store->worktree_backend.stack) - return &store->main_backend; - return &store->worktree_backend; + be = &store->main_backend; + else + be = &store->worktree_backend; + goto out; case REF_WORKTREE_MAIN: case REF_WORKTREE_SHARED: - return &store->main_backend; + be = &store->main_backend; + goto out; default: BUG("unhandled worktree reference type"); } + +out: + if (reload) { + int ret = reftable_stack_reload(be->stack); + if (ret) + return ret; + } + *out = be; + + return 0; } static int should_write_log(struct reftable_ref_store *refs, const char *refname) @@ -828,17 +845,17 @@ static int reftable_be_read_raw_ref(struct ref_store *ref_store, { struct reftable_ref_store *refs = reftable_be_downcast(ref_store, REF_STORE_READ, "read_raw_ref"); - struct reftable_stack *stack = backend_for(refs, refname, &refname)->stack; + struct reftable_backend *be; int ret; if (refs->err < 0) return refs->err; - ret = reftable_stack_reload(stack); + ret = backend_for(&be, refs, refname, &refname, 1); if (ret) return ret; - ret = read_ref_without_reload(refs, stack, refname, oid, referent, type); + ret = read_ref_without_reload(refs, be->stack, refname, oid, referent, type); if (ret < 0) return ret; if (ret > 0) { @@ -855,15 +872,15 @@ static int reftable_be_read_symbolic_ref(struct ref_store *ref_store, { struct reftable_ref_store *refs = reftable_be_downcast(ref_store, REF_STORE_READ, "read_symbolic_ref"); - struct reftable_stack *stack = backend_for(refs, refname, &refname)->stack; struct reftable_ref_record ref = {0}; + struct reftable_backend *be; int ret; - ret = reftable_stack_reload(stack); + ret = backend_for(&be, refs, refname, &refname, 1); if (ret) return ret; - ret = reftable_stack_read_ref(stack, refname, &ref); + ret = reftable_stack_read_ref(be->stack, refname, &ref); if (ret == 0 && ref.value_type == REFTABLE_REF_SYMREF) strbuf_addstr(referent, ref.value.symref); else @@ -880,7 +897,7 @@ struct reftable_transaction_update { struct write_transaction_table_arg { struct reftable_ref_store *refs; - struct reftable_stack *stack; + struct reftable_backend *be; struct reftable_addition *addition; struct reftable_transaction_update *updates; size_t updates_nr; @@ -915,27 +932,31 @@ static int prepare_transaction_update(struct write_transaction_table_arg **out, struct ref_update *update, struct strbuf *err) { - struct reftable_stack *stack = backend_for(refs, update->refname, NULL)->stack; struct write_transaction_table_arg *arg = NULL; + struct reftable_backend *be; size_t i; int ret; + ret = backend_for(&be, refs, update->refname, NULL, 0); + if (ret) + return ret; + /* * Search for a preexisting stack update. If there is one then we add * the update to it, otherwise we set up a new stack update. */ for (i = 0; !arg && i < tx_data->args_nr; i++) - if (tx_data->args[i].stack == stack) + if (tx_data->args[i].be == be) arg = &tx_data->args[i]; if (!arg) { struct reftable_addition *addition; - ret = reftable_stack_reload(stack); + ret = backend_for(&be, refs, update->refname, NULL, 1); if (ret) return ret; - ret = reftable_stack_new_addition(&addition, stack, + ret = reftable_stack_new_addition(&addition, be->stack, REFTABLE_STACK_NEW_ADDITION_RELOAD); if (ret) { if (ret == REFTABLE_LOCK_ERROR) @@ -947,7 +968,7 @@ static int prepare_transaction_update(struct write_transaction_table_arg **out, tx_data->args_alloc); arg = &tx_data->args[tx_data->args_nr++]; arg->refs = refs; - arg->stack = stack; + arg->be = be; arg->addition = addition; arg->updates = NULL; arg->updates_nr = 0; @@ -1002,6 +1023,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store, struct strbuf referent = STRBUF_INIT, head_referent = STRBUF_INIT; struct string_list affected_refnames = STRING_LIST_INIT_NODUP; struct reftable_transaction_data *tx_data = NULL; + struct reftable_backend *be; struct object_id head_oid; unsigned int head_type = 0; size_t i; @@ -1048,7 +1070,11 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store, goto done; } - ret = read_ref_without_reload(refs, backend_for(refs, "HEAD", NULL)->stack, "HEAD", + ret = backend_for(&be, refs, "HEAD", NULL, 0); + if (ret) + goto done; + + ret = read_ref_without_reload(refs, be->stack, "HEAD", &head_oid, &head_referent, &head_type); if (ret < 0) goto done; @@ -1057,10 +1083,11 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store, for (i = 0; i < transaction->nr; i++) { struct ref_update *u = transaction->updates[i]; struct object_id current_oid = {0}; - struct reftable_stack *stack; const char *rewritten_ref; - stack = backend_for(refs, u->refname, &rewritten_ref)->stack; + ret = backend_for(&be, refs, u->refname, &rewritten_ref, 0); + if (ret) + goto done; /* Verify that the new object ID is valid. */ if ((u->flags & REF_HAVE_NEW) && !is_null_oid(&u->new_oid) && @@ -1116,7 +1143,7 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store, string_list_insert(&affected_refnames, new_update->refname); } - ret = read_ref_without_reload(refs, stack, rewritten_ref, + ret = read_ref_without_reload(refs, be->stack, rewritten_ref, ¤t_oid, &referent, &u->type); if (ret < 0) goto done; @@ -1318,7 +1345,7 @@ static int transaction_update_cmp(const void *a, const void *b) static int write_transaction_table(struct reftable_writer *writer, void *cb_data) { struct write_transaction_table_arg *arg = cb_data; - uint64_t ts = reftable_stack_next_update_index(arg->stack); + uint64_t ts = reftable_stack_next_update_index(arg->be->stack); struct reftable_log_record *logs = NULL; struct ident_split committer_ident = {0}; size_t logs_nr = 0, logs_alloc = 0, i; @@ -1354,7 +1381,7 @@ static int write_transaction_table(struct reftable_writer *writer, void *cb_data struct reftable_log_record log = {0}; struct reftable_iterator it = {0}; - ret = reftable_stack_init_log_iterator(arg->stack, &it); + ret = reftable_stack_init_log_iterator(arg->be->stack, &it); if (ret < 0) goto done; @@ -1799,10 +1826,9 @@ static int reftable_be_rename_ref(struct ref_store *ref_store, { struct reftable_ref_store *refs = reftable_be_downcast(ref_store, REF_STORE_WRITE, "rename_ref"); - struct reftable_stack *stack = backend_for(refs, newrefname, &newrefname)->stack; + struct reftable_backend *be; struct write_copy_arg arg = { .refs = refs, - .stack = stack, .oldname = oldrefname, .newname = newrefname, .logmsg = logmsg, @@ -1814,10 +1840,11 @@ static int reftable_be_rename_ref(struct ref_store *ref_store, if (ret < 0) goto done; - ret = reftable_stack_reload(stack); + ret = backend_for(&be, refs, newrefname, &newrefname, 1); if (ret) goto done; - ret = reftable_stack_add(stack, &write_copy_table, &arg); + arg.stack = be->stack; + ret = reftable_stack_add(be->stack, &write_copy_table, &arg); done: assert(ret != REFTABLE_API_ERROR); @@ -1831,10 +1858,9 @@ static int reftable_be_copy_ref(struct ref_store *ref_store, { struct reftable_ref_store *refs = reftable_be_downcast(ref_store, REF_STORE_WRITE, "copy_ref"); - struct reftable_stack *stack = backend_for(refs, newrefname, &newrefname)->stack; + struct reftable_backend *be; struct write_copy_arg arg = { .refs = refs, - .stack = stack, .oldname = oldrefname, .newname = newrefname, .logmsg = logmsg, @@ -1845,10 +1871,11 @@ static int reftable_be_copy_ref(struct ref_store *ref_store, if (ret < 0) goto done; - ret = reftable_stack_reload(stack); + ret = backend_for(&be, refs, newrefname, &newrefname, 1); if (ret) goto done; - ret = reftable_stack_add(stack, &write_copy_table, &arg); + arg.stack = be->stack; + ret = reftable_stack_add(be->stack, &write_copy_table, &arg); done: assert(ret != REFTABLE_API_ERROR); @@ -2012,15 +2039,19 @@ static int reftable_be_for_each_reflog_ent_reverse(struct ref_store *ref_store, { struct reftable_ref_store *refs = reftable_be_downcast(ref_store, REF_STORE_READ, "for_each_reflog_ent_reverse"); - struct reftable_stack *stack = backend_for(refs, refname, &refname)->stack; struct reftable_log_record log = {0}; struct reftable_iterator it = {0}; + struct reftable_backend *be; int ret; if (refs->err < 0) return refs->err; - ret = reftable_stack_init_log_iterator(stack, &it); + ret = backend_for(&be, refs, refname, &refname, 0); + if (ret) + goto done; + + ret = reftable_stack_init_log_iterator(be->stack, &it); if (ret < 0) goto done; @@ -2052,16 +2083,20 @@ static int reftable_be_for_each_reflog_ent(struct ref_store *ref_store, { struct reftable_ref_store *refs = reftable_be_downcast(ref_store, REF_STORE_READ, "for_each_reflog_ent"); - struct reftable_stack *stack = backend_for(refs, refname, &refname)->stack; struct reftable_log_record *logs = NULL; struct reftable_iterator it = {0}; + struct reftable_backend *be; size_t logs_alloc = 0, logs_nr = 0, i; int ret; if (refs->err < 0) return refs->err; - ret = reftable_stack_init_log_iterator(stack, &it); + ret = backend_for(&be, refs, refname, &refname, 0); + if (ret) + goto done; + + ret = reftable_stack_init_log_iterator(be->stack, &it); if (ret < 0) goto done; @@ -2101,20 +2136,20 @@ static int reftable_be_reflog_exists(struct ref_store *ref_store, { struct reftable_ref_store *refs = reftable_be_downcast(ref_store, REF_STORE_READ, "reflog_exists"); - struct reftable_stack *stack = backend_for(refs, refname, &refname)->stack; struct reftable_log_record log = {0}; struct reftable_iterator it = {0}; + struct reftable_backend *be; int ret; ret = refs->err; if (ret < 0) goto done; - ret = reftable_stack_reload(stack); + ret = backend_for(&be, refs, refname, &refname, 1); if (ret < 0) goto done; - ret = reftable_stack_init_log_iterator(stack, &it); + ret = reftable_stack_init_log_iterator(be->stack, &it); if (ret < 0) goto done; @@ -2186,10 +2221,9 @@ static int reftable_be_create_reflog(struct ref_store *ref_store, { struct reftable_ref_store *refs = reftable_be_downcast(ref_store, REF_STORE_WRITE, "create_reflog"); - struct reftable_stack *stack = backend_for(refs, refname, &refname)->stack; + struct reftable_backend *be; struct write_reflog_existence_arg arg = { .refs = refs, - .stack = stack, .refname = refname, }; int ret; @@ -2198,11 +2232,12 @@ static int reftable_be_create_reflog(struct ref_store *ref_store, if (ret < 0) goto done; - ret = reftable_stack_reload(stack); + ret = backend_for(&be, refs, refname, &refname, 1); if (ret) goto done; + arg.stack = be->stack; - ret = reftable_stack_add(stack, &write_reflog_existence_table, &arg); + ret = reftable_stack_add(be->stack, &write_reflog_existence_table, &arg); done: return ret; @@ -2260,17 +2295,18 @@ static int reftable_be_delete_reflog(struct ref_store *ref_store, { struct reftable_ref_store *refs = reftable_be_downcast(ref_store, REF_STORE_WRITE, "delete_reflog"); - struct reftable_stack *stack = backend_for(refs, refname, &refname)->stack; + struct reftable_backend *be; struct write_reflog_delete_arg arg = { - .stack = stack, .refname = refname, }; int ret; - ret = reftable_stack_reload(stack); + ret = backend_for(&be, refs, refname, &refname, 1); if (ret) return ret; - ret = reftable_stack_add(stack, &write_reflog_delete_table, &arg); + arg.stack = be->stack; + + ret = reftable_stack_add(be->stack, &write_reflog_delete_table, &arg); assert(ret != REFTABLE_API_ERROR); return ret; @@ -2369,13 +2405,13 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store, */ struct reftable_ref_store *refs = reftable_be_downcast(ref_store, REF_STORE_WRITE, "reflog_expire"); - struct reftable_stack *stack = backend_for(refs, refname, &refname)->stack; struct reftable_log_record *logs = NULL; struct reftable_log_record *rewritten = NULL; struct reftable_ref_record ref_record = {0}; struct reftable_iterator it = {0}; struct reftable_addition *add = NULL; struct reflog_expiry_arg arg = {0}; + struct reftable_backend *be; struct object_id oid = {0}; uint8_t *last_hash = NULL; size_t logs_nr = 0, logs_alloc = 0, i; @@ -2384,11 +2420,11 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store, if (refs->err < 0) return refs->err; - ret = reftable_stack_reload(stack); + ret = backend_for(&be, refs, refname, &refname, 1); if (ret < 0) goto done; - ret = reftable_stack_init_log_iterator(stack, &it); + ret = reftable_stack_init_log_iterator(be->stack, &it); if (ret < 0) goto done; @@ -2396,11 +2432,11 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store, if (ret < 0) goto done; - ret = reftable_stack_new_addition(&add, stack, 0); + ret = reftable_stack_new_addition(&add, be->stack, 0); if (ret < 0) goto done; - ret = reftable_stack_read_ref(stack, refname, &ref_record); + ret = reftable_stack_read_ref(be->stack, refname, &ref_record); if (ret < 0) goto done; if (reftable_ref_record_val1(&ref_record)) @@ -2479,7 +2515,7 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store, arg.refs = refs; arg.records = rewritten; arg.len = logs_nr; - arg.stack = stack, + arg.stack = be->stack, arg.refname = refname, ret = reftable_addition_add(add, &write_reflog_expiry_table, &arg); From patchwork Tue Nov 5 09:12:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13862687 Received: from fout-b3-smtp.messagingengine.com (fout-b3-smtp.messagingengine.com [202.12.124.146]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 665C51CFED8 for ; Tue, 5 Nov 2024 09:12:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=202.12.124.146 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730797938; cv=none; b=dDXojQhAOM+CPool4qjd4GhTHxq31mwbr8BJELAf1VzNzyQeTrI6Ioqjlud6UmyN/eJ5qsCZFVDq/GS070Fl2C8nC4i3mgZV7XEoMHN99ZOuyPTmE6YxeXH2kaX8/vbQWtUtDp5s83I0TmePNJJd3MQI9zWPAwIAW2rhnf19oR4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730797938; c=relaxed/simple; bh=LT5GHRbUqtwVWQrz2Y6GDIMwDH2WlA1n0ob8qLwzDBQ=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=bdCQLSYZqA0iUCPFABDBxHL0nNsT+GWcKms1ONKJ+CGh7TFOr+KsnScZiia+2tHyMkiu2U0VlHchCv8ppEYjkeMVTQPWsT/xWv8Em1WShAYBf6dP5DIg+oER9BdbKK9qa6xQeBimooTuiFshu5OkqBCdDGfmtV/8tKQIIWQqZPA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=GXooCnJk; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=ChadOjt0; arc=none smtp.client-ip=202.12.124.146 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="GXooCnJk"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="ChadOjt0" Received: from phl-compute-08.internal (phl-compute-08.phl.internal [10.202.2.48]) by mailfout.stl.internal (Postfix) with ESMTP id 8E40F11401C6; Tue, 5 Nov 2024 04:12:15 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-08.internal (MEProxy); Tue, 05 Nov 2024 04:12:15 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1730797935; x=1730884335; bh=XRJfM7IAqt AsDDRGdHQZuM8Y5hl/dAO8FO3S+/wSZYE=; b=GXooCnJkz1UXmbwbxDQE1EohLv JRnvAjA0jCOPgip9o59WTAohrX0ckIVuSLViV/lmvU2kRsgLHZdx1zHB1pKZiROs iF0LNlFjMEXNrRU1EZChuXv9BsZMttXnXQHnt4GIgUwrdT0z9IAcIvIocFgDB/2x RNO4Xk6m5d9cNqUyGTLAdKR4eQk//wTwbGOxQCHZowG0TDOF1PxTDGWT/KCUGwjU nxBR6dDIV/l76G1FKL64etuUhPhRiZvWaSXPBQUqQlyy7QF7EN/SO2QRu7chPuHP knhiT8EsR7n1pUnRjmNwQO7UUqNZM43PrXleSsTGbjisBszeHn1U8VNGPhGA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t= 1730797935; x=1730884335; bh=XRJfM7IAqtAsDDRGdHQZuM8Y5hl/dAO8FO3 S+/wSZYE=; b=ChadOjt0aw3M5yjj59HBt5X+x2h5ZwZzSeyHrdQuymNL3I+vWfc MvP6/7OE/hNUOvJrZE5Gb8WkmlL1jUQna4cJkSemKAcfEjubi0eo4QjvEBY+MvJP o3VJSh+shGPXt44raJdFDrHdI60n06gPwWGahFTLYdCMxxH+f6kG36GOU1hZOxzE IBSyM6rze/dsCOsqJuZz85TpzLu+X1klC7JO7EWIRQCAj1KkPfcfuxYlQiqQqBDg 2u2ZJkmC+ls2RBj5XtOjMxUH9AVHklYU/JUbTPUqtc0hT7M8VwOauC+dfAUnIoc+ C97wv6D6vcrfV70btCH2+r0/e+P7Xj2LZww== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddrvdelkedgtdduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvve fukfhfgggtuggjsehttdertddttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhn hhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrhhnpedvvdehjeeuue ekhfeiieefteeiiefhfffgfffhkeeuhefgleduheffieeguefgfeenucffohhmrghinhep uhhpuggrthgvrdhtiienucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrih hlfhhrohhmpehpshesphhkshdrihhmpdhnsggprhgtphhtthhopedvpdhmohguvgepshhm thhpohhuthdprhgtphhtthhopehgihhtshhtvghrsehpohgsohigrdgtohhmpdhrtghpth htohepghhithesvhhgvghrrdhkvghrnhgvlhdrohhrgh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 5 Nov 2024 04:12:14 -0500 (EST) Received: by vm-mail (OpenSMTPD) with ESMTPSA id e3b168cd (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Tue, 5 Nov 2024 09:11:52 +0000 (UTC) Date: Tue, 5 Nov 2024 10:12:04 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Junio C Hamano Subject: [PATCH v2 3/8] refs/reftable: read references via `struct reftable_backend` Message-ID: <1b50655202f311c6a6ded61d4d50b1f287761d84.1730792627.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Refactor `read_ref_without_reload()` to accept a `struct reftable_stack` as input instead of accepting a `struct reftable_stack`. This allows us to implement an additional caching layer when reading refs where we can reuse reftable iterators. Signed-off-by: Patrick Steinhardt --- refs/reftable-backend.c | 110 ++++++++++++++++++++------------------ reftable/reftable-stack.h | 3 ++ reftable/stack.c | 5 ++ 3 files changed, 67 insertions(+), 51 deletions(-) diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c index 4a28dc8a9d..230adb690d 100644 --- a/refs/reftable-backend.c +++ b/refs/reftable-backend.c @@ -51,6 +51,50 @@ static void reftable_backend_release(struct reftable_backend *be) be->stack = NULL; } +static int reftable_backend_read_ref(struct reftable_backend *be, + const char *refname, + struct object_id *oid, + struct strbuf *referent, + unsigned int *type) +{ + struct reftable_ref_record ref = {0}; + int ret; + + ret = reftable_stack_read_ref(be->stack, refname, &ref); + if (ret) + goto done; + + if (ref.value_type == REFTABLE_REF_SYMREF) { + strbuf_reset(referent); + strbuf_addstr(referent, ref.value.symref); + *type |= REF_ISSYMREF; + } else if (reftable_ref_record_val1(&ref)) { + unsigned int hash_id; + + switch (reftable_stack_hash_id(be->stack)) { + case REFTABLE_HASH_SHA1: + hash_id = GIT_HASH_SHA1; + break; + case REFTABLE_HASH_SHA256: + hash_id = GIT_HASH_SHA256; + break; + default: + BUG("unhandled hash ID %d", reftable_stack_hash_id(be->stack)); + } + + oidread(oid, reftable_ref_record_val1(&ref), + &hash_algos[hash_id]); + } else { + /* We got a tombstone, which should not happen. */ + BUG("unhandled reference value type %d", ref.value_type); + } + +done: + assert(ret != REFTABLE_API_ERROR); + reftable_ref_record_release(&ref); + return ret; +} + struct reftable_ref_store { struct ref_store base; @@ -243,38 +287,6 @@ static void fill_reftable_log_record(struct reftable_log_record *log, const stru log->value.update.tz_offset = sign * atoi(tz_begin); } -static int read_ref_without_reload(struct reftable_ref_store *refs, - struct reftable_stack *stack, - const char *refname, - struct object_id *oid, - struct strbuf *referent, - unsigned int *type) -{ - struct reftable_ref_record ref = {0}; - int ret; - - ret = reftable_stack_read_ref(stack, refname, &ref); - if (ret) - goto done; - - if (ref.value_type == REFTABLE_REF_SYMREF) { - strbuf_reset(referent); - strbuf_addstr(referent, ref.value.symref); - *type |= REF_ISSYMREF; - } else if (reftable_ref_record_val1(&ref)) { - oidread(oid, reftable_ref_record_val1(&ref), - refs->base.repo->hash_algo); - } else { - /* We got a tombstone, which should not happen. */ - BUG("unhandled reference value type %d", ref.value_type); - } - -done: - assert(ret != REFTABLE_API_ERROR); - reftable_ref_record_release(&ref); - return ret; -} - static int reftable_be_config(const char *var, const char *value, const struct config_context *ctx, void *_opts) @@ -855,7 +867,7 @@ static int reftable_be_read_raw_ref(struct ref_store *ref_store, if (ret) return ret; - ret = read_ref_without_reload(refs, be->stack, refname, oid, referent, type); + ret = reftable_backend_read_ref(be, refname, oid, referent, type); if (ret < 0) return ret; if (ret > 0) { @@ -1074,8 +1086,8 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store, if (ret) goto done; - ret = read_ref_without_reload(refs, be->stack, "HEAD", - &head_oid, &head_referent, &head_type); + ret = reftable_backend_read_ref(be, "HEAD", &head_oid, + &head_referent, &head_type); if (ret < 0) goto done; ret = 0; @@ -1143,8 +1155,8 @@ static int reftable_be_transaction_prepare(struct ref_store *ref_store, string_list_insert(&affected_refnames, new_update->refname); } - ret = read_ref_without_reload(refs, be->stack, rewritten_ref, - ¤t_oid, &referent, &u->type); + ret = reftable_backend_read_ref(be, rewritten_ref, + ¤t_oid, &referent, &u->type); if (ret < 0) goto done; if (ret > 0 && !ref_update_expects_existing_old_ref(u)) { @@ -1602,7 +1614,7 @@ struct write_create_symref_arg { struct write_copy_arg { struct reftable_ref_store *refs; - struct reftable_stack *stack; + struct reftable_backend *be; const char *oldname; const char *newname; const char *logmsg; @@ -1627,7 +1639,7 @@ static int write_copy_table(struct reftable_writer *writer, void *cb_data) if (split_ident_line(&committer_ident, committer_info, strlen(committer_info))) BUG("failed splitting committer info"); - if (reftable_stack_read_ref(arg->stack, arg->oldname, &old_ref)) { + if (reftable_stack_read_ref(arg->be->stack, arg->oldname, &old_ref)) { ret = error(_("refname %s not found"), arg->oldname); goto done; } @@ -1666,7 +1678,7 @@ static int write_copy_table(struct reftable_writer *writer, void *cb_data) * the old branch and the creation of the new branch, and we cannot do * two changes to a reflog in a single update. */ - deletion_ts = creation_ts = reftable_stack_next_update_index(arg->stack); + deletion_ts = creation_ts = reftable_stack_next_update_index(arg->be->stack); if (arg->delete_old) creation_ts++; reftable_writer_set_limits(writer, deletion_ts, creation_ts); @@ -1709,8 +1721,8 @@ static int write_copy_table(struct reftable_writer *writer, void *cb_data) memcpy(logs[logs_nr].value.update.old_hash, old_ref.value.val1, GIT_MAX_RAWSZ); logs_nr++; - ret = read_ref_without_reload(arg->refs, arg->stack, "HEAD", &head_oid, - &head_referent, &head_type); + ret = reftable_backend_read_ref(arg->be, "HEAD", &head_oid, + &head_referent, &head_type); if (ret < 0) goto done; append_head_reflog = (head_type & REF_ISSYMREF) && !strcmp(head_referent.buf, arg->oldname); @@ -1753,7 +1765,7 @@ static int write_copy_table(struct reftable_writer *writer, void *cb_data) * copy over all log entries from the old reflog. Last but not least, * when renaming we also have to delete all the old reflog entries. */ - ret = reftable_stack_init_log_iterator(arg->stack, &it); + ret = reftable_stack_init_log_iterator(arg->be->stack, &it); if (ret < 0) goto done; @@ -1826,7 +1838,6 @@ static int reftable_be_rename_ref(struct ref_store *ref_store, { struct reftable_ref_store *refs = reftable_be_downcast(ref_store, REF_STORE_WRITE, "rename_ref"); - struct reftable_backend *be; struct write_copy_arg arg = { .refs = refs, .oldname = oldrefname, @@ -1840,11 +1851,10 @@ static int reftable_be_rename_ref(struct ref_store *ref_store, if (ret < 0) goto done; - ret = backend_for(&be, refs, newrefname, &newrefname, 1); + ret = backend_for(&arg.be, refs, newrefname, &newrefname, 1); if (ret) goto done; - arg.stack = be->stack; - ret = reftable_stack_add(be->stack, &write_copy_table, &arg); + ret = reftable_stack_add(arg.be->stack, &write_copy_table, &arg); done: assert(ret != REFTABLE_API_ERROR); @@ -1858,7 +1868,6 @@ static int reftable_be_copy_ref(struct ref_store *ref_store, { struct reftable_ref_store *refs = reftable_be_downcast(ref_store, REF_STORE_WRITE, "copy_ref"); - struct reftable_backend *be; struct write_copy_arg arg = { .refs = refs, .oldname = oldrefname, @@ -1871,11 +1880,10 @@ static int reftable_be_copy_ref(struct ref_store *ref_store, if (ret < 0) goto done; - ret = backend_for(&be, refs, newrefname, &newrefname, 1); + ret = backend_for(&arg.be, refs, newrefname, &newrefname, 1); if (ret) goto done; - arg.stack = be->stack; - ret = reftable_stack_add(be->stack, &write_copy_table, &arg); + ret = reftable_stack_add(arg.be->stack, &write_copy_table, &arg); done: assert(ret != REFTABLE_API_ERROR); diff --git a/reftable/reftable-stack.h b/reftable/reftable-stack.h index 54787f2ef5..ae14270ea7 100644 --- a/reftable/reftable-stack.h +++ b/reftable/reftable-stack.h @@ -149,4 +149,7 @@ struct reftable_compaction_stats { struct reftable_compaction_stats * reftable_stack_compaction_stats(struct reftable_stack *st); +/* Return the hash of the stack. */ +enum reftable_hash reftable_stack_hash_id(struct reftable_stack *st); + #endif diff --git a/reftable/stack.c b/reftable/stack.c index 1fffd75630..d97b64a40d 100644 --- a/reftable/stack.c +++ b/reftable/stack.c @@ -1791,3 +1791,8 @@ int reftable_stack_clean(struct reftable_stack *st) reftable_addition_destroy(add); return err; } + +enum reftable_hash reftable_stack_hash_id(struct reftable_stack *st) +{ + return reftable_merged_table_hash_id(st->merged); +} From patchwork Tue Nov 5 09:12:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13862688 Received: from fout-b3-smtp.messagingengine.com (fout-b3-smtp.messagingengine.com [202.12.124.146]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A008B1D130F for ; Tue, 5 Nov 2024 09:12:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=202.12.124.146 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730797941; cv=none; b=YNIUi9FKFkDLVrPUy+WWxuUPyZMOFs+SEhr5gAOEz6E+cKJYXBm7QjT211X2uXcgvkO6PMrMc9r9/7HRndlZz7nB6D9KuBVqmX4rcdyGT2pEzI7znAc3CisZgZHSsmv+fHhzxj0SSLdsCgU8Uq5h33jTBtFVQrJCbX+tSzOstG4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730797941; c=relaxed/simple; bh=UhxvmbkeaJ/ZVJJ6WS0iueKcy8Zd9eElc1vblnpiKzM=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=FPuAmQ8CgqMVo+sA4Y5Zhax+rD6ePNVq/yg8kRxrogsz1kkLFt1jMGqY2uxN3UVyF9Zc8SPU/rpStKjXfiYA+dyG127LFIz996fyJHk8qEf4703W2HhazbI1Vbm5XQwQroIhL1K8v1/GICA5D2Npm7d0ovrfJ8PlN2iLK8dEMLE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=ob/7oVhx; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=n1Abbju2; arc=none smtp.client-ip=202.12.124.146 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="ob/7oVhx"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="n1Abbju2" Received: from phl-compute-12.internal (phl-compute-12.phl.internal [10.202.2.52]) by mailfout.stl.internal (Postfix) with ESMTP id BB6541140176; Tue, 5 Nov 2024 04:12:18 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-12.internal (MEProxy); Tue, 05 Nov 2024 04:12:18 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1730797938; x=1730884338; bh=+PA3PSRR7x dmt6qh+LPksKQ5BdlRMZH9t7oN0HpQKTM=; b=ob/7oVhxmPwUe/vjipFdBDhffJ wnkcMOwbS+xoJB2hA95IDHcF3jLDYcBggUicRo+Lf11SQEz/X5dR5R5R1QibCC19 nL/eSJ9uJL3WVI2IKXF9dQvPnY14C6f8NzfSC6zNq3hcaisRjrFX5oh3E8QIZDyW tmb52hDeMLrFxyYzVGDjMyLDmJzqdt4uEAIRkO1grPQh6Xk9gklv+LVPyFxQVZHN B3sPGwhOEbZ3vgFTDA7bqrs+SRk7QMeMVrpMneLH5Vk252dpD3u+oH6XVAl3EFO/ pztYr1WvNWOH1hJ9ce2pYhPcf6Ee/uBGVz7noLznZ5ay6OhjHf7XFlbZJ0gg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t= 1730797938; x=1730884338; bh=+PA3PSRR7xdmt6qh+LPksKQ5BdlRMZH9t7o N0HpQKTM=; b=n1Abbju22xP7DgSbsW8QqMKQuo9AiVYrFW6an+3oB9eFu95Ay6b 0bI+ipMgojyjgLvoFvwSjlnoB6d0Us1bK79NHk5aD7FAFeNKE00rbuHFB9Fz1nYY qpJ5f67qcpxgmOUEKNbokzlELquNxUsNjUI4/tLZQcyfdWq8q+rl5+e/eInWrCH9 7RfxzEYHKPkUYQRMxRsLaqwDcqGsTuj8ToMjGCcsZgKABQzk8UAbgbjk+WbB5h04 Zn89eW0yU4PVrkZ7ZLCqbehFjy2dvw0UeKRgKDQsqs6mJxVg/dwbYbpJ+OF4C15N DmsOf4heUkY7rM54snB88egHdMQP6/P7YqA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddrvdelkedgtdduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvve fukfhfgggtuggjsehttdertddttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhn hhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrhhnpeevkeekfffhie dtleduiefgjedttedvledvudehgfeugedugffhueekhfejvdektdenucevlhhushhtvghr ufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhmpdhnsg gprhgtphhtthhopedvpdhmohguvgepshhmthhpohhuthdprhgtphhtthhopehgihhtsehv ghgvrhdrkhgvrhhnvghlrdhorhhgpdhrtghpthhtohepghhithhsthgvrhesphhosghogi drtghomh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 5 Nov 2024 04:12:17 -0500 (EST) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 0128a8a1 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Tue, 5 Nov 2024 09:11:55 +0000 (UTC) Date: Tue, 5 Nov 2024 10:12:10 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Junio C Hamano Subject: [PATCH v2 4/8] refs/reftable: refactor reading symbolic refs to use reftable backend Message-ID: <0906b04fc6da205cf67cca4d0dedac6f0d0adbc0.1730792627.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Refactor the callback function that reads symbolic references in the reftable backend to use `reftable_backend_read_ref()` instead of accessing the reftable stack directly. This ensures that the function will benefit from the new caching layer that we're about to introduce. Signed-off-by: Patrick Steinhardt --- refs/reftable-backend.c | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c index 230adb690d..dfe94ff969 100644 --- a/refs/reftable-backend.c +++ b/refs/reftable-backend.c @@ -884,21 +884,18 @@ static int reftable_be_read_symbolic_ref(struct ref_store *ref_store, { struct reftable_ref_store *refs = reftable_be_downcast(ref_store, REF_STORE_READ, "read_symbolic_ref"); - struct reftable_ref_record ref = {0}; struct reftable_backend *be; + struct object_id oid; + unsigned int type = 0; int ret; ret = backend_for(&be, refs, refname, &refname, 1); if (ret) return ret; - ret = reftable_stack_read_ref(be->stack, refname, &ref); - if (ret == 0 && ref.value_type == REFTABLE_REF_SYMREF) - strbuf_addstr(referent, ref.value.symref); - else + ret = reftable_backend_read_ref(be, refname, &oid, referent, &type); + if (type != REF_ISSYMREF) ret = -1; - - reftable_ref_record_release(&ref); return ret; } From patchwork Tue Nov 5 09:12:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13862689 Received: from fhigh-b6-smtp.messagingengine.com (fhigh-b6-smtp.messagingengine.com [202.12.124.157]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E2CDE1D14E3 for ; Tue, 5 Nov 2024 09:12:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=202.12.124.157 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730797943; cv=none; b=ND90DzsMITmUaeivrGV+T3zIQ4Oq/Fr0qt+Mk07LHpcjDkrS/EXDNvM/gmqrDOr0IZXDmLOEUTnL6UmFyZbWtx2gAwE8MZ6OwrNgsSvSJoZgLkRljzxrdlffzb6uhsrV/aWvrv/67ndXJpSdFBC7GEE9sTajMcdEPNfQvX7meUw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730797943; c=relaxed/simple; bh=OuzipJSYzBDeK/uEFyiYB8IV5RZH8NQnfdRifYDFKrI=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=ifRO9EMyNW0n3lbRnwJYnv5TkzRT5hFLiHB+JadDPCrVVq+P5Wpnt9hpPwNTxnl6YNgFyGHnXjkWOw1B8D/Wt6u0e//O4ZWTpXZM1NRbcdntLyfibzBPapvCjK0cBOvJ4qMeJIPctvJqj5+EJGSSh5Z2WudGBXkGUfR4OECyYEs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=Z1dcPyT5; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=m1ryVq+q; arc=none smtp.client-ip=202.12.124.157 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="Z1dcPyT5"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="m1ryVq+q" Received: from phl-compute-10.internal (phl-compute-10.phl.internal [10.202.2.50]) by mailfhigh.stl.internal (Postfix) with ESMTP id 0762D2540126; Tue, 5 Nov 2024 04:12:21 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-10.internal (MEProxy); Tue, 05 Nov 2024 04:12:21 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1730797940; x=1730884340; bh=vNzVlfVWW+ xICVC8kdsWLYsgyi3pcmHdpM96mbKAv1E=; b=Z1dcPyT58RNJHZrWGRnJSsbrhp qP4PC3QUN+5sBDew7Oy0mpPXkHq5umlBHPpJgxvMj9/sl5Wv6QPunBuGK6iswXet hrp2mKrBRCzJ5cjuo/8lAxG6yFUIbzODvpAAg7G5BZBJzkfmJdmgCAQrZBc6NBBN wgor+9ylUlmBzPur8cwk4+Xgb9WBDtg1O1uyRHFr1/2FRBVtDQU2iA10TVsZQ6Rl h8D93Hohfj0l5+hoQ2MT2hqYs7A8ld/0h7FbqIGDua6YgvxPphRtXhObIno09mAj UOEqfrtGbiKCN714/9MJq2DYzIX4ayp4xwbthCaRHP2jNIHPzlWZb+34qQLw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t= 1730797940; x=1730884340; bh=vNzVlfVWW+xICVC8kdsWLYsgyi3pcmHdpM9 6mbKAv1E=; b=m1ryVq+qspWFIO9ONN2BK8ATCqek7lCydw5rBhPdycEjtejbqFA x5poDl1hSCPLsKU5xT2dVGn+u7WMv7bkn88tL3WQRlEp4WQhq+LctjB/tDRGWi82 d+v3oeNAd9LnAALCaPXNO7QxGUjfQBK+P+1q7e10zwwGRoOPLTrchFjTzvJzElsy kqUwrzw1icE4SaH3mat0eqhxU/gUl8AW5+xGha960ADxv6dNyRWZkhKRNJL8rTP/ Iu94Mj7BYv41V7s1TTat1WdKQZT687UpCSf1irm46Y4dzMvUUbe9Qy/Bpza+dTwj VYK6Hmk0NiUzAIDyd1t8gcUXZ0xIqIyXEfw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddrvdelkedgtdduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvve fukfhfgggtuggjsehttdertddttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhn hhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrhhnpeevkeekfffhie dtleduiefgjedttedvledvudehgfeugedugffhueekhfejvdektdenucevlhhushhtvghr ufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhmpdhnsg gprhgtphhtthhopedvpdhmohguvgepshhmthhpohhuthdprhgtphhtthhopehgihhtshht vghrsehpohgsohigrdgtohhmpdhrtghpthhtohepghhithesvhhgvghrrdhkvghrnhgvlh drohhrgh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 5 Nov 2024 04:12:20 -0500 (EST) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 6ef1a316 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Tue, 5 Nov 2024 09:11:58 +0000 (UTC) Date: Tue, 5 Nov 2024 10:12:13 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Junio C Hamano Subject: [PATCH v2 5/8] refs/reftable: refactor reflog expiry to use reftable backend Message-ID: <355557ec951372a16b70558706981867a5206c87.1730792627.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Refactor the callback function that expires reflog entries in the reftable backend to use `reftable_backend_read_ref()` instead of accessing the reftable stack directly. This ensures that the function will benefit from the new caching layer that we're about to introduce. Signed-off-by: Patrick Steinhardt --- refs/reftable-backend.c | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c index dfe94ff969..9c6e9c8374 100644 --- a/refs/reftable-backend.c +++ b/refs/reftable-backend.c @@ -2412,14 +2412,15 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store, reftable_be_downcast(ref_store, REF_STORE_WRITE, "reflog_expire"); struct reftable_log_record *logs = NULL; struct reftable_log_record *rewritten = NULL; - struct reftable_ref_record ref_record = {0}; struct reftable_iterator it = {0}; struct reftable_addition *add = NULL; struct reflog_expiry_arg arg = {0}; struct reftable_backend *be; struct object_id oid = {0}; + struct strbuf referent = STRBUF_INIT; uint8_t *last_hash = NULL; size_t logs_nr = 0, logs_alloc = 0, i; + unsigned int type = 0; int ret; if (refs->err < 0) @@ -2441,12 +2442,9 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store, if (ret < 0) goto done; - ret = reftable_stack_read_ref(be->stack, refname, &ref_record); + ret = reftable_backend_read_ref(be, refname, &oid, &referent, &type); if (ret < 0) goto done; - if (reftable_ref_record_val1(&ref_record)) - oidread(&oid, reftable_ref_record_val1(&ref_record), - ref_store->repo->hash_algo); prepare_fn(refname, &oid, policy_cb_data); while (1) { @@ -2513,8 +2511,7 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store, } } - if (flags & EXPIRE_REFLOGS_UPDATE_REF && last_hash && - reftable_ref_record_val1(&ref_record)) + if (flags & EXPIRE_REFLOGS_UPDATE_REF && last_hash && !is_null_oid(&oid)) oidread(&arg.update_oid, last_hash, ref_store->repo->hash_algo); arg.refs = refs; @@ -2539,11 +2536,11 @@ static int reftable_be_reflog_expire(struct ref_store *ref_store, cleanup_fn(policy_cb_data); assert(ret != REFTABLE_API_ERROR); - reftable_ref_record_release(&ref_record); reftable_iterator_destroy(&it); reftable_addition_destroy(add); for (i = 0; i < logs_nr; i++) reftable_log_record_release(&logs[i]); + strbuf_release(&referent); free(logs); free(rewritten); return ret; From patchwork Tue Nov 5 09:12:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13862690 Received: from fhigh-b6-smtp.messagingengine.com (fhigh-b6-smtp.messagingengine.com [202.12.124.157]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2B5E71D130F for ; Tue, 5 Nov 2024 09:12:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=202.12.124.157 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730797945; cv=none; b=dpcoFrova+UA7tCXAvNBnwdLwCkKo85SmqsauMx/JhjslA7nxh1NBr4+4LObYfeQHR/crLir3mmZNFmNTQySjWHCgRd/ujcyX43T/Q2gT9owKfuL6Kv3mfSLT1mUUmBHnQ5XPx7LpetY7FT4emvP5xEcvetWcmDbrXqfbFxxN7A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730797945; c=relaxed/simple; bh=MwuG9Y++PqKsmpjR180M01ALjXhKt5oEu3IuTRxhoSQ=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=AOfLkqjh8WyPXa9gH+KcZDZkQjpJeazJbH0WUI/AwOTA734eA5Bw4LuN7xlT7SumU/3eZ52NvTH+w8Ix/2gfUm8cBDyVh4kZDqgoy8UvsXqqs4f2YniYxHcTebqUU8qpLC4qiKCZfZB+DwOXDfs2mx5jd9ekFrtqxbbjfNn0Tnw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=q6s3ooOk; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=OicCy2NG; arc=none smtp.client-ip=202.12.124.157 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="q6s3ooOk"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="OicCy2NG" Received: from phl-compute-03.internal (phl-compute-03.phl.internal [10.202.2.43]) by mailfhigh.stl.internal (Postfix) with ESMTP id 4EC81254012F; Tue, 5 Nov 2024 04:12:23 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-03.internal (MEProxy); Tue, 05 Nov 2024 04:12:23 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1730797943; x=1730884343; bh=17eAGAH/p4 ssliP6ERB4/x+Rr8CXigRxrcdDbXEB390=; b=q6s3ooOkORFkH0yQmwPsCtrKVw v1v5rPOiKv9Ea2FW175l+jCE3lFzcKm91GFUMOQ4sbZIsLvrVB9xifOAMIy38cEI oRISN9JCR/PSY0n77JzI7RzhJlAoPX2+VfViuBb/pb/1Ogz/ieHWgo2uVwWZ6Ks1 qh4yMhNQ7bT7yltTgG+NSiceUpZtvR2K1Fc2W9XwwkA1a4O7wNFuQcRN8pvfhgeK SXKMR1MMpybckTmHmgdaxA5UwySgC36IObE/wQXyLoE3vnBOyN5J7fUEE4W9NcHt KkgCAPLECJnOQ7R9Ims0HpR7d0jO5LS50NepiAmx6kgQEYf7sFSRFv2dZuMg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t= 1730797943; x=1730884343; bh=17eAGAH/p4ssliP6ERB4/x+Rr8CXigRxrcd DbXEB390=; b=OicCy2NG/zvMRgH9dYbmvlWfor0SF7NznGQNcxli4GhGSLVzBwG TXdKQgAooV7bjPosmQxesBC40krxvzH33MmIdj7GmOkgq5CA0pj8yQBprdFbX/XN JeJbgpj51ZCDHH3fLeNLlgFv+DxyHuQURYcWy7bI8oArPYuvPw5nGknOMC/CzJbm Q7tvyZiAsWfpjUgMhhdRL5Zr9FFIu0RLJe22XNewWDF42gFSi0vRRf+exo+6lthG nVQ4d8V0NAbNNBbVf3cKRwERH+pwEfSmsS8KyNHiKFiErPkizeTk9y+NDRyxEVCg pyR36pEofqrMwp7DkH5MBvp6THPn2z9vXTQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddrvdelkedgtdduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvve fukfhfgggtuggjsehttdertddttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhn hhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrhhnpeevkeekfffhie dtleduiefgjedttedvledvudehgfeugedugffhueekhfejvdektdenucevlhhushhtvghr ufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhmpdhnsg gprhgtphhtthhopedvpdhmohguvgepshhmthhpohhuthdprhgtphhtthhopehgihhtshht vghrsehpohgsohigrdgtohhmpdhrtghpthhtohepghhithesvhhgvghrrdhkvghrnhgvlh drohhrgh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 5 Nov 2024 04:12:22 -0500 (EST) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 1ca2b9d9 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Tue, 5 Nov 2024 09:12:00 +0000 (UTC) Date: Tue, 5 Nov 2024 10:12:15 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Junio C Hamano Subject: [PATCH v2 6/8] reftable/stack: add mechanism to notify callers on reload Message-ID: <71ad6c80b0218cb61394fcaea8618ce888c505ab.1730792627.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Reftable stacks are reloaded in two cases: - When calling `reftable_stack_reload()`, if the stat-cache tells us that the stack has been modified. - When committing a reftable addition. While callers can figure out the second case, they do not have a mechanism to figure out whether `reftable_stack_reload()` led to an actual reload of the on-disk data. All they can do is thus to assume that data is always being reloaded in that case. Improve the situation by introducing a new `on_reload()` callback to the reftable options. If provided, the function will be invoked every time the stack has indeed been reloaded. This allows callers to invalidate data that depends on the current stack data. Signed-off-by: Patrick Steinhardt --- reftable/reftable-writer.h | 9 +++++++++ reftable/stack.c | 4 ++++ 2 files changed, 13 insertions(+) diff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h index c85ef5a5bd..5f9afa620b 100644 --- a/reftable/reftable-writer.h +++ b/reftable/reftable-writer.h @@ -68,6 +68,15 @@ struct reftable_write_options { * fsync(3P) when unset. */ int (*fsync)(int fd); + + /* + * Callback function to execute whenever the stack is being reloaded. + * This can be used e.g. to discard cached information that relies on + * the old stack's data. The payload data will be passed as argument to + * the callback. + */ + void (*on_reload)(void *payload); + void *on_reload_payload; }; /* reftable_block_stats holds statistics for a single block type */ diff --git a/reftable/stack.c b/reftable/stack.c index d97b64a40d..5384ca9de0 100644 --- a/reftable/stack.c +++ b/reftable/stack.c @@ -548,6 +548,10 @@ static int reftable_stack_reload_maybe_reuse(struct reftable_stack *st, close(fd); free_names(names); free_names(names_after); + + if (st->opts.on_reload) + st->opts.on_reload(st->opts.on_reload_payload); + return err; } From patchwork Tue Nov 5 09:12:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13862691 Received: from fout-b3-smtp.messagingengine.com (fout-b3-smtp.messagingengine.com [202.12.124.146]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7B4B71D130F for ; Tue, 5 Nov 2024 09:12:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=202.12.124.146 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730797950; cv=none; b=rk+pj8wb7Pmt8PmFeid35mZJGJdDCLJJKMTHZiVEhNPT0p2Pm3+jucYA536iUMf2HdLZqm1YNQvHdC+LeEq8LPS0v1KQEuSiyU2NPq+vPDQk8P2h2JspQbrm97g2EXTpzm9o5JaG0PaSwlbwb64cnMzxG4nNx7LJ+v8xPOJlUmA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730797950; c=relaxed/simple; bh=ZSAY1M7Tc5AsE2DpD1nrqp9IAdUCqDO8+nq85VfKkqQ=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=NLPyKxi1jGwUKnyDUJvd0D3cJtNTN8Wx/RLhfb5l/sY9ZZxDIFuPEIiiwznC+nHhwHbrYBnbsEoA3EdHrmwNRQgmeW+auv8ilF3+bIQ9bKlKKBq/OgtKHsJvcBpC1pIs+pJH/E1Qn59cEmpQC8zs8F/Mg+KzwrrwfLUYtJvPDsA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=YLNc/0qD; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=Vgl8GDN6; arc=none smtp.client-ip=202.12.124.146 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="YLNc/0qD"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="Vgl8GDN6" Received: from phl-compute-08.internal (phl-compute-08.phl.internal [10.202.2.48]) by mailfout.stl.internal (Postfix) with ESMTP id 89EA711401B4; Tue, 5 Nov 2024 04:12:26 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-08.internal (MEProxy); Tue, 05 Nov 2024 04:12:26 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm3; t=1730797946; x=1730884346; bh=WzSAh9sdK3 puaQLlbTWS19fTLxfLpKrrVmPz4GTd0w8=; b=YLNc/0qD7ZrihpvlkyAQa2FXBz /b5uh4Du1HzVqN9piy7BqlRFiNvj8oJVwx/fFJI/ROGjaKHcRqrGjdeaFNYXX4HJ syfCcrAv3VuQFT9FpWZCBRqSmNmh1AjTupN/ECeJqNZ7HUtnyot/pe50zmukcjWS DM6utPR3mDtvcEXZGN9Bh5/e0KXH2jvDzs5M5JylzWOTwLoeUcCK4opH7hqHQkO1 9xbHq95ngZ+IyyaS0GLDAvnL5+8WngB7Rs+UtVuj9qQUXwxVQKfucIoLVZlYzYmW zJv7BBSx9rwvhLmCFZ9OV5NMqRUa2ajS3vgWDY+LiroTkBeTSyeNmDnWgwrg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t= 1730797946; x=1730884346; bh=WzSAh9sdK3puaQLlbTWS19fTLxfLpKrrVmP z4GTd0w8=; b=Vgl8GDN6UTzU658DmfthrRjFvjKp90rj1NdiB4s8uYMSUj8C02C 8ckj6rcZSi9U9+Q2eXZTvaJ50tMAV0cvD5r+hHF4CmzLM6TUMYIoz2tVE0OSrRP9 D+Jq641a1IyWrAjnMHNPm+KRWSM9XwJhYygjb0/bw+2CaIEIuIyojLwx6yrzzTBR d6kQXvLOuObQQQk43JECQHgB9ZwezupFbth6Hu0NY4k3EUiYo85oK5x7hKy3uAZa byoXTz1VsovRHifr3wTHI8HvtJadKJVXhDVqMRiS0/+xt4zauboiwWEvzkY5P/0C 6m5xxXYngTOsZB1DWkZFD0VjMwD8ro5gVNw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddrvdelkedgtdduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvve fukfhfgggtuggjsehttdertddttddvnecuhfhrohhmpefrrghtrhhitghkucfuthgvihhn hhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrhhnpeevkeekfffhie dtleduiefgjedttedvledvudehgfeugedugffhueekhfejvdektdenucevlhhushhtvghr ufhiiigvpedunecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesphhkshdrihhmpdhnsg gprhgtphhtthhopedvpdhmohguvgepshhmthhpohhuthdprhgtphhtthhopehgihhtsehv ghgvrhdrkhgvrhhnvghlrdhorhhgpdhrtghpthhtohepghhithhsthgvrhesphhosghogi drtghomh X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 5 Nov 2024 04:12:25 -0500 (EST) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 38c3f40b (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Tue, 5 Nov 2024 09:12:03 +0000 (UTC) Date: Tue, 5 Nov 2024 10:12:18 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Junio C Hamano Subject: [PATCH v2 7/8] reftable/merged: drain priority queue on reseek Message-ID: <93efd118864c96b51eb84e46d1eff358eb007a3a.1730792627.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: In 5bf96e0c39 (reftable/generic: move seeking of records into the iterator, 2024-05-13) we have refactored the reftable codebase such that iterators can be initialized once and then re-seeked multiple times. This feature is used by 1869525066 (refs/reftable: wire up support for exclude patterns, 2024-09-16) in order to skip records based on exclude patterns provided by the caller. The logic to re-seek the merged iterator is insufficient though because we don't drain the priority queue on a re-seek. This means that the queue may contain stale entries and thus reading the next record in the queue will return the wrong entry. While this is an obvious bug, it is harmless in the context of above exclude patterns: - If the queue contained stale entries that match the pattern then the caller would already know to filter out such refs. This is because our codebase is prepared to handle backends that don't have a way to efficiently implement exclude patterns. - If the queue contained stale entries that don't match the pattern we'd eventually filter out any duplicates. This is because the reftable code discards items with the same ref name and sorts any remaining entries properly. So things happen to work in this context regardless of the bug, and there is no other use case yet where we re-seek iterators. We're about to introduce a caching mechanism though where iterators are reused by the reftable backend, and that will expose the bug. Fix the issue by draining the priority queue when seeking and add a testcase that surfaces the issue. Signed-off-by: Patrick Steinhardt --- reftable/merged.c | 2 + t/unit-tests/t-reftable-merged.c | 73 ++++++++++++++++++++++++++++++++ 2 files changed, 75 insertions(+) diff --git a/reftable/merged.c b/reftable/merged.c index 5b93e20f42..bb0836e344 100644 --- a/reftable/merged.c +++ b/reftable/merged.c @@ -66,6 +66,8 @@ static int merged_iter_seek(struct merged_iter *mi, struct reftable_record *want int err; mi->advance_index = -1; + while (!merged_iter_pqueue_is_empty(mi->pq)) + merged_iter_pqueue_remove(&mi->pq); for (size_t i = 0; i < mi->subiters_len; i++) { err = iterator_seek(&mi->subiters[i].iter, want); diff --git a/t/unit-tests/t-reftable-merged.c b/t/unit-tests/t-reftable-merged.c index 2591b5e597..a12bd0e1a3 100644 --- a/t/unit-tests/t-reftable-merged.c +++ b/t/unit-tests/t-reftable-merged.c @@ -273,6 +273,78 @@ static void t_merged_seek_multiple_times(void) reftable_free(sources); } +static void t_merged_seek_multiple_times_without_draining(void) +{ + struct reftable_ref_record r1[] = { + { + .refname = (char *) "a", + .update_index = 1, + .value_type = REFTABLE_REF_VAL1, + .value.val1 = { 1 }, + }, + { + .refname = (char *) "c", + .update_index = 1, + .value_type = REFTABLE_REF_VAL1, + .value.val1 = { 2 }, + } + }; + struct reftable_ref_record r2[] = { + { + .refname = (char *) "b", + .update_index = 2, + .value_type = REFTABLE_REF_VAL1, + .value.val1 = { 3 }, + }, + { + .refname = (char *) "d", + .update_index = 2, + .value_type = REFTABLE_REF_VAL1, + .value.val1 = { 4 }, + }, + }; + struct reftable_ref_record *refs[] = { + r1, r2, + }; + size_t sizes[] = { + ARRAY_SIZE(r1), ARRAY_SIZE(r2), + }; + struct reftable_buf bufs[] = { + REFTABLE_BUF_INIT, REFTABLE_BUF_INIT, + }; + struct reftable_block_source *sources = NULL; + struct reftable_reader **readers = NULL; + struct reftable_ref_record rec = { 0 }; + struct reftable_iterator it = { 0 }; + struct reftable_merged_table *mt; + int err; + + mt = merged_table_from_records(refs, &sources, &readers, sizes, bufs, 2); + merged_table_init_iter(mt, &it, BLOCK_TYPE_REF); + + err = reftable_iterator_seek_ref(&it, "b"); + check(!err); + err = reftable_iterator_next_ref(&it, &rec); + check(!err); + err = reftable_ref_record_equal(&rec, &r2[0], REFTABLE_HASH_SIZE_SHA1); + check(err == 1); + + err = reftable_iterator_seek_ref(&it, "a"); + check(!err); + err = reftable_iterator_next_ref(&it, &rec); + check(!err); + err = reftable_ref_record_equal(&rec, &r1[0], REFTABLE_HASH_SIZE_SHA1); + check(err == 1); + + for (size_t i = 0; i < ARRAY_SIZE(bufs); i++) + reftable_buf_release(&bufs[i]); + readers_destroy(readers, ARRAY_SIZE(refs)); + reftable_ref_record_release(&rec); + reftable_iterator_destroy(&it); + reftable_merged_table_free(mt); + reftable_free(sources); +} + static struct reftable_merged_table * merged_table_from_log_records(struct reftable_log_record **logs, struct reftable_block_source **source, @@ -467,6 +539,7 @@ int cmd_main(int argc UNUSED, const char *argv[] UNUSED) TEST(t_merged_logs(), "merged table with multiple log updates for same ref"); TEST(t_merged_refs(), "merged table with multiple updates to same ref"); TEST(t_merged_seek_multiple_times(), "merged table can seek multiple times"); + TEST(t_merged_seek_multiple_times_without_draining(), "merged table can seek multiple times without draining"); TEST(t_merged_single_record(), "ref occurring in only one record can be fetched"); return test_done(); From patchwork Tue Nov 5 09:12:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13862692 Received: from fout-b3-smtp.messagingengine.com (fout-b3-smtp.messagingengine.com [202.12.124.146]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B655B1D04A9 for ; Tue, 5 Nov 2024 09:12:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=202.12.124.146 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730797951; cv=none; b=thmTCgvkagaN0E+Q4pkcKwIPZqMP9ku8H88isDm7hjYL8pfcOra5sT0HEFj0nynQuO4JWuqB4RxW4nCX4RTm2A1ojpypSBaxCIrmeDPLeP9mivNIqvVJRK84kW1UU3Q3yYxpervZNF2NbEk02q2gCb2BnWRbMg0ZRYG0dZxl6m0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730797951; c=relaxed/simple; bh=/PuXU3Z/DKlGCXSMcRCMREyi0uzuubH0dVd9ZiZGMAg=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=r2CuCzu1NSEZFfXkUQQVA1VlMNfPKvCtbSiBY5ljETlE2Y6yIIckjofTTucxwDHpQxM93wGeFnlOlbO2IxTrRlTni9UVaB07EVH6QHUO/06m1e6gbMFsiTCwtFc655AK8Biu3LgyRV0iyNhP6UPVHfYpAsG/uwOD77XH101LbSQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=xCUQewvO; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=K8awN/oY; arc=none smtp.client-ip=202.12.124.146 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="xCUQewvO"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="K8awN/oY" Received: from phl-compute-08.internal (phl-compute-08.phl.internal [10.202.2.48]) by mailfout.stl.internal (Postfix) with ESMTP id BB9CD1140171; Tue, 5 Nov 2024 04:12:28 -0500 (EST) Received: from phl-mailfrontend-02 ([10.202.2.163]) by phl-compute-08.internal (MEProxy); Tue, 05 Nov 2024 04:12:28 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-transfer-encoding:content-type:content-type:date:date :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to; s=fm3; t=1730797948; x=1730884348; bh=r3USq8gIBRthfs8tV8A6Jiwa7hsE2GVK7eSPH18idj4=; b= xCUQewvOggD/GkESD69Q91M+gogqrB8wigK6Pey33832NTkHDoN/Jj0zPX6Wi2E1 YPF8UrIyhFGIkgj2Hwru9DSXSRyxnTO4DDT8YDQz9/QAUWMr2+EJuA+3h/q5s/Ia gyJpb0OVTVJZDg5MytwhnktdFm23ClWzeqTFTgkof86wSW0YsMgty0ozSu2AOZmw MKd1JYyd/b9kJ5WPntncOt4sbWs3w0tBAo6Dd2Q7GNW1dRHTISBJks8C10SSXJ/B ZJeJkrB8BIdS/wKS8Q5UHypzmxto0YgvbsK0YqTtNisXBZxhJP/2T/sCvIiaujpy ON7EjoQAs56MEH3AmalSVQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:in-reply-to:message-id:mime-version :references:reply-to:subject:subject:to:to:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=1730797948; x= 1730884348; bh=r3USq8gIBRthfs8tV8A6Jiwa7hsE2GVK7eSPH18idj4=; b=K 8awN/oYJDk5NrnUXi1Z6K35OjepiCdtkqfstKBd1+RLjXgPQuzJy3SjPWkquOe4m JrzsriGoHyDAWUJ5VwL7+RUuUnkovioakiCBgM28g/iNIBh5ea4Jm/A90D3OdXee mcvRmx9IfVknTRrt6J3uSYeXfPRl6qB2K1zcvh/zJcnlPcbfWiXkqtV6Q3YvCi5L v+JvSYxdkbuxzPKG3E39JzJf51+3f9QGSsFx0XYCoqyBR8X9lWac/lyElyB3J3r/ AkKQ87VgOCspnB1RHj5OMGKjPWd4uhulR+TrXDLRSeg1xiA1zY/aIZnG6JF/cZwF ptocb3Nmc+mnzeWZhJWHg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddrvdelkedgtdduucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdggtfgfnhhsuhgsshgtrhhisggvpdfu rfetoffkrfgpnffqhgenuceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvve fukfhfgggtugfgjgesthekredttddtjeenucfhrhhomheprfgrthhrihgtkhcuufhtvghi nhhhrghrughtuceophhssehpkhhsrdhimheqnecuggftrfgrthhtvghrnhepvdefjeeitd etleehieetkeevfedtfedvheekvdevteffvdevveejjeelgeetvdfgnecuvehluhhsthgv rhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomhepphhssehpkhhsrdhimhdpnh gspghrtghpthhtohepvddpmhhouggvpehsmhhtphhouhhtpdhrtghpthhtohepghhithhs thgvrhesphhosghogidrtghomhdprhgtphhtthhopehgihhtsehvghgvrhdrkhgvrhhnvg hlrdhorhhg X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 5 Nov 2024 04:12:27 -0500 (EST) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 19b682d1 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Tue, 5 Nov 2024 09:12:06 +0000 (UTC) Date: Tue, 5 Nov 2024 10:12:20 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Junio C Hamano Subject: [PATCH v2 8/8] refs/reftable: reuse iterators when reading refs Message-ID: <276c27e770ace7030bdd83106c05f4fe5b07dbe5.1730792627.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: When reading references the reftable backend has to: 1. Create a new ref iterator. 2. Seek the iterator to the record we're searching for. 3. Read the record. We cannot really avoid the last two steps, but re-creating the iterator every single time we want to read a reference is kind of expensive and a waste of resources. We couldn't help it in the past though because it was not possible to reuse iterators. But starting with 5bf96e0c39 (reftable/generic: move seeking of records into the iterator, 2024-05-13) we have split up the iterator lifecycle such that creating the iterator and seeking are two different concerns. Refactor the code such that we cache iterators in the reftable backend. This cache is invalidated whenever the respective stack is reloaded such that we know to recreate the iterator in that case. This leads to a sizeable speedup when creating many refs, which requires a lot of random reference reads: Benchmark 1: update-ref: create many refs (refcount = 100000, revision = master) Time (mean ± σ): 1.793 s ± 0.010 s [User: 0.954 s, System: 0.835 s] Range (min … max): 1.781 s … 1.811 s 10 runs Benchmark 2: update-ref: create many refs (refcount = 100000, revision = HEAD) Time (mean ± σ): 1.680 s ± 0.013 s [User: 0.846 s, System: 0.831 s] Range (min … max): 1.664 s … 1.702 s 10 runs Summary update-ref: create many refs (refcount = 100000, revision = HEAD) ran 1.07 ± 0.01 times faster than update-ref: create many refs (refcount = 100000, revision = master) While 7% is not a huge win, you have to consider that the benchmark is _writing_ data, so _reading_ references is only one part of what we do. Flame graphs show that we spend around 40% of our time reading refs, so the speedup when reading refs is approximately ~2.5x that. I could not find better benchmarks where we perform a lot of random ref reads. You can also see a sizeable impact on memory usage when creating 100k references. Before this change: HEAP SUMMARY: in use at exit: 19,112,538 bytes in 200,170 blocks total heap usage: 8,400,426 allocs, 8,200,256 frees, 454,367,048 bytes allocated After this change: HEAP SUMMARY: in use at exit: 674,416 bytes in 169 blocks total heap usage: 7,929,872 allocs, 7,929,703 frees, 281,509,985 bytes allocated As an additional factor, this refactoring opens up the possibility for more performance optimizations in how we re-seek iterators. Any change that allows us to optimize re-seeking by e.g. reusing data structures would thus also directly speed up random reads. Signed-off-by: Patrick Steinhardt --- refs/reftable-backend.c | 32 +++++++++++++++++++++++++++++--- 1 file changed, 29 insertions(+), 3 deletions(-) diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c index 9c6e9c8374..4942363712 100644 --- a/refs/reftable-backend.c +++ b/refs/reftable-backend.c @@ -36,19 +36,30 @@ struct reftable_backend { struct reftable_stack *stack; + struct reftable_iterator it; }; +static void reftable_backend_on_reload(void *payload) +{ + struct reftable_backend *be = payload; + reftable_iterator_destroy(&be->it); +} + static int reftable_backend_init(struct reftable_backend *be, const char *path, - const struct reftable_write_options *opts) + const struct reftable_write_options *_opts) { - return reftable_new_stack(&be->stack, path, opts); + struct reftable_write_options opts = *_opts; + opts.on_reload = reftable_backend_on_reload; + opts.on_reload_payload = be; + return reftable_new_stack(&be->stack, path, &opts); } static void reftable_backend_release(struct reftable_backend *be) { reftable_stack_destroy(be->stack); be->stack = NULL; + reftable_iterator_destroy(&be->it); } static int reftable_backend_read_ref(struct reftable_backend *be, @@ -60,10 +71,25 @@ static int reftable_backend_read_ref(struct reftable_backend *be, struct reftable_ref_record ref = {0}; int ret; - ret = reftable_stack_read_ref(be->stack, refname, &ref); + if (!be->it.ops) { + ret = reftable_stack_init_ref_iterator(be->stack, &be->it); + if (ret) + goto done; + } + + ret = reftable_iterator_seek_ref(&be->it, refname); if (ret) goto done; + ret = reftable_iterator_next_ref(&be->it, &ref); + if (ret) + goto done; + + if (strcmp(ref.refname, refname)) { + ret = 1; + goto done; + } + if (ref.value_type == REFTABLE_REF_SYMREF) { strbuf_reset(referent); strbuf_addstr(referent, ref.value.symref);