From patchwork Mon Mar 18 10:52:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13595212 Received: from fout2-smtp.messagingengine.com (fout2-smtp.messagingengine.com [103.168.172.145]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6DCAB360AE for ; Mon, 18 Mar 2024 10:52:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.145 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710759153; cv=none; b=nHLV+2BOnRadeF3507hzRbxDfZYrusjvcLqXpvgHm8arQCiBC9HzKJ8ceTXBPCH02DJP9dYgk5bdQUwIrUed4fAj7Vd6bijaUNHA4LO2nakm52Trv5U51LEOx18qjraAcambICmu9ZQTNxS+tDzf3DBerLj2rGODYQ3x7OMUXb0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710759153; c=relaxed/simple; bh=nEv6E6JCN+i19Xb6dIpJK+9DVVSt24ACbSO2NnjRyJU=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=tmrYR4KCOgLcjKwIkKoQ5sYOcBSY7h1pS7uVzO7KQu60cQ6NSVqmS7jpPtfuKaL0ebb1gLmE5eszzUxbv0PGhjJIiJ5d4uKMgOXZk0yVVdAxkvChA2J6dzWI8BCQqHnlNC+KynP53pFzTf40ud4NpmEpbOnA2ncf9onTv8tzJg4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=L4LP9ziw; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=FUZ/GbEO; arc=none smtp.client-ip=103.168.172.145 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="L4LP9ziw"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="FUZ/GbEO" Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailfout.nyi.internal (Postfix) with ESMTP id 5E8ED13800C8; Mon, 18 Mar 2024 06:52:30 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Mon, 18 Mar 2024 06:52:30 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1710759150; x=1710845550; bh=6x11qkHIEG hset+qHakcZUV6ERKSu63h68YwnEwwm8E=; b=L4LP9ziwKKpqQNisrTm3CZFv4J X3nV/JgJzTbMfBOnHS5PleD78jCkKigExI1CAP2LaVf0NxQ7f7DHDNvra5PsM7oH TmgHiKszgTSl5kBNQM8M0ckmNxQ9DvYvisYKYE/AKa/vqHdZvF7T+FH1yfm368l7 CPZb6nZQOksJjT3nWfpyw2/4LxT9xOC/OmRy/GfZxPtrACIx0Xtzfz8bRfdO+HAf F84S7U3vUNWRzSqS/vUdTwgV5VbH52ZIt33l6zLAw3z+ul1PYdtdFl3MDsqEK3hQ U+DCdlUd64wwEeenBJQbrvM3prBJ9lDyxw4IvrqKQnoxy97USXTUKoKDbsNA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1710759150; x=1710845550; bh=6x11qkHIEGhset+qHakcZUV6ERKS u63h68YwnEwwm8E=; b=FUZ/GbEOBMYAEMR0ku2/1dBcnTiBLKCvLkFPucnVMmSQ j8RU5GNi3lyTFg2pwh1NqrQk6fo2r1mYqG4yqx0LBVwIQcNAwEF6QAdaVkSUFP3B M30MY+r9MvHa6l151c+HsDBwPfJVtPtWnSTecFjLcT6y94Ofnavpu8USco0QU99V u1+Zv7KoH0BGAXzZPjTzGQe7UD0eMEcX97KcwUcUdtJi8UUwRwb1l8t+svKX8z8C Q3n1JJJED/xS/k+YhKtU7p3J7+Qw9vVzLQBQsKKJGRQ7FEzGODiEY0vtupaoaIMA Q2+e1lV1OvsumoGMyM6WDEJG3roIlR5G79XHECp2ww== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrkeejgddvudcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpeffhffvvefukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeeukedtvedtffevleejtefgheehieegkeeluddvfeefgeehgfeltddtheejleffteen ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 18 Mar 2024 06:52:29 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id b932a93c (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 18 Mar 2024 10:47:38 +0000 (UTC) Date: Mon, 18 Mar 2024 11:52:27 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Derrick Stolee Subject: [PATCH 01/15] reftable/stack: fix error handling in `reftable_stack_init_addition()` Message-ID: <1e39d93a45db502280ecff383d53e0294f969719.1710706118.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: In `reftable_stack_init_addition()` we call `stack_uptodate()` after having created the lockfile to check whether the stack was modified concurrently, which is indicated by a positive return code from the latter function. If so, we return a `REFTABLE_LOCK_ERROR` to the caller and abort the addition. The error handling has an off-by-one though because we check whether the error code is `> 1` instead of `> 0`. Thus, instead of returning the locking error, we would return a positive value. One of the callers of `reftable_stack_init_addition()` works around this bug by repeating the error code check without the off-by-one. But other callers are subtly broken by this bug. Fix this by checking for `err > 0` instead. Signed-off-by: Patrick Steinhardt --- reftable/stack.c | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/reftable/stack.c b/reftable/stack.c index 1ecf1b9751..92d9a7facb 100644 --- a/reftable/stack.c +++ b/reftable/stack.c @@ -590,8 +590,7 @@ static int reftable_stack_init_addition(struct reftable_addition *add, err = stack_uptodate(st); if (err < 0) goto done; - - if (err > 1) { + if (err > 0) { err = REFTABLE_LOCK_ERROR; goto done; } @@ -713,10 +712,6 @@ static int stack_try_add(struct reftable_stack *st, int err = reftable_stack_init_addition(&add, st); if (err < 0) goto done; - if (err > 0) { - err = REFTABLE_LOCK_ERROR; - goto done; - } err = reftable_addition_add(&add, write_table, arg); if (err < 0) From patchwork Mon Mar 18 10:52:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13595213 Received: from fout2-smtp.messagingengine.com (fout2-smtp.messagingengine.com [103.168.172.145]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3DD4C364B4 for ; Mon, 18 Mar 2024 10:52:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.145 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710759156; cv=none; b=KsSCgtZCMlZQp3rK0fFqO31KfXvs8K/A31wNyfoIHW78h8QGNQ+ZCyj4m8mSwaZflGclSo3+JdsLfPVcj7Ru3LOyyXlqC6DnnE5Dsm71RMOxLQkleckYWe8gW8ElBq6mDmiIfRX8xRhmFDa1k+oYTaqd/wCz/NxzLE/zemDJFcc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710759156; c=relaxed/simple; bh=vIzB8nBpZdgXFDDHwimpH5wfNZmSsXKsQTtLu2t6D3A=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=OPya7Un8yOViQah56Xwac0iAj6VPucsp4Ea+en027Bz4u+bdDDLhxGYm83mcWTfc7zVIQ1YNn++yIHZGLFyJAaM47evSg8BkDdVfMqvG7u21+ief0IZQ4yWAXUPcZSqg8irdU+VuuMvXM4XBWRMRxHbNrw+UFP2k8bmM0otkB18= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=OP4OrpT+; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=rTNW/833; arc=none smtp.client-ip=103.168.172.145 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="OP4OrpT+"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="rTNW/833" Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailfout.nyi.internal (Postfix) with ESMTP id 6937A13800C8; Mon, 18 Mar 2024 06:52:34 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Mon, 18 Mar 2024 06:52:34 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1710759154; x=1710845554; bh=MtnxE3Mru5 u8JNu8QY8OPDHc3GxnOXSJ8MA8LrgteNA=; b=OP4OrpT+pyytvFwsPDDSFGehNm 4fIP5Jt3te79Yn1VADwY8r45YWEvZcU3h0Et42Bwdcq/Ny0ESilKG/poYEne54QS VIHBPyIloTrlJRjyRBnjX0UCNr2Szl8rEovaXBOFtUp7zDBUzV0S85XG/97KaHLt Rw11wMfKtTtWUzO5JnAii19qqZ6szJRobloG0PGXKf/kskyyY9MZNY+PfW1SjtIH 0Z0dEG40GT/CboZjsjXMZslIUbHcOm9KhjoHeLJfv6RfTBtob6sXLABkuHCIUNAA 2O5hxioTK6IBnT536LdlxyVwb6EVIjOQDoVwFK19LCgKGoe9jdEsKsmLVylQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1710759154; x=1710845554; bh=MtnxE3Mru5u8JNu8QY8OPDHc3Gxn OXSJ8MA8LrgteNA=; b=rTNW/833bqXMxuyXmCJOjm064gblw8DSKo0DMr6ju5PP yGeOE87hNMwbA2uja2j9bF0pw37LIuaQCdrJcm1RxMSghIxMOUrE5sn0oE+srOOd kd2KQgTmu8Tytb/okZBTFD6HnihGGU4jJImJH4bcfs7Lxn3yXroppGVWDz+3LyeM 83MIREZexPlx/JFoCdHdJBMsWnhikZSlOWmJKw+WDdcqGWzpAqBSP9XGkH24mpJf eeMwyfxl6yYhPY1U1K9qlcnFDOxHDua2PMzFQTabQxSUsVS2UrvtiBkqa5wiZUUM oL2/SjMSwxgZWLjwYFHYJJUj/fvUYcY92OfZe/FH7g== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrkeejgddvudcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpeffhffvvefukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeeukedtvedtffevleejtefgheehieegkeeluddvfeefgeehgfeltddtheejleffteen ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 18 Mar 2024 06:52:33 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 0ba850c7 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 18 Mar 2024 10:47:42 +0000 (UTC) Date: Mon, 18 Mar 2024 11:52:31 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Derrick Stolee Subject: [PATCH 02/15] reftable/error: discern locked/outdated errors Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: We currently throw two different errors into a similar-but-different error code: - Errors when trying to lock the reftable stack. - Errors when trying to write to the reftable stack which has been modified concurrently. This results in unclear error handling and user-visible error messages. Create a new `REFTABLE_OUTDATED_ERROR` so that those error conditions can be clearly told apart from each other. Adjust users of the old `REFTABLE_LOCK_ERROR` to use the new error code as required. Signed-off-by: Patrick Steinhardt --- reftable/error.c | 4 +++- reftable/reftable-error.h | 5 ++++- reftable/stack.c | 6 +++--- reftable/stack_test.c | 2 +- 4 files changed, 11 insertions(+), 6 deletions(-) diff --git a/reftable/error.c b/reftable/error.c index 0d1766735e..cfb7a0fda4 100644 --- a/reftable/error.c +++ b/reftable/error.c @@ -22,7 +22,7 @@ const char *reftable_error_str(int err) case REFTABLE_NOT_EXIST_ERROR: return "file does not exist"; case REFTABLE_LOCK_ERROR: - return "data is outdated"; + return "data is locked"; case REFTABLE_API_ERROR: return "misuse of the reftable API"; case REFTABLE_ZLIB_ERROR: @@ -35,6 +35,8 @@ const char *reftable_error_str(int err) return "invalid refname"; case REFTABLE_ENTRY_TOO_BIG_ERROR: return "entry too large"; + case REFTABLE_OUTDATED_ERROR: + return "data concurrently modified"; case -1: return "general error"; default: diff --git a/reftable/reftable-error.h b/reftable/reftable-error.h index 4c457aaaf8..e9b07c9f36 100644 --- a/reftable/reftable-error.h +++ b/reftable/reftable-error.h @@ -25,7 +25,7 @@ enum reftable_error { */ REFTABLE_NOT_EXIST_ERROR = -4, - /* Trying to write out-of-date data. */ + /* Trying to access locked data. */ REFTABLE_LOCK_ERROR = -5, /* Misuse of the API: @@ -57,6 +57,9 @@ enum reftable_error { /* Entry does not fit. This can happen when writing outsize reflog messages. */ REFTABLE_ENTRY_TOO_BIG_ERROR = -11, + + /* Trying to write out-of-date data. */ + REFTABLE_OUTDATED_ERROR = -12, }; /* convert the numeric error code to a string. The string should not be diff --git a/reftable/stack.c b/reftable/stack.c index 92d9a7facb..eaa8bb9c99 100644 --- a/reftable/stack.c +++ b/reftable/stack.c @@ -529,9 +529,9 @@ int reftable_stack_add(struct reftable_stack *st, { int err = stack_try_add(st, write, arg); if (err < 0) { - if (err == REFTABLE_LOCK_ERROR) { + if (err == REFTABLE_OUTDATED_ERROR) { /* Ignore error return, we want to propagate - REFTABLE_LOCK_ERROR. + REFTABLE_OUTDATED_ERROR. */ reftable_stack_reload(st); } @@ -591,7 +591,7 @@ static int reftable_stack_init_addition(struct reftable_addition *add, if (err < 0) goto done; if (err > 0) { - err = REFTABLE_LOCK_ERROR; + err = REFTABLE_OUTDATED_ERROR; goto done; } diff --git a/reftable/stack_test.c b/reftable/stack_test.c index 509f486623..b0c7041a4f 100644 --- a/reftable/stack_test.c +++ b/reftable/stack_test.c @@ -232,7 +232,7 @@ static void test_reftable_stack_uptodate(void) EXPECT_ERR(err); err = reftable_stack_add(st2, &write_test_ref, &ref2); - EXPECT(err == REFTABLE_LOCK_ERROR); + EXPECT(err == REFTABLE_OUTDATED_ERROR); err = reftable_stack_reload(st2); EXPECT_ERR(err); From patchwork Mon Mar 18 10:52:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13595214 Received: from fout2-smtp.messagingengine.com (fout2-smtp.messagingengine.com [103.168.172.145]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6CABD2E85B for ; Mon, 18 Mar 2024 10:52:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.145 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710759162; cv=none; b=QcZKN7PMP9f3mzphyN8vCK7mhELPmTpaPW/N6b76dvXQSRoZCHeVmaat9nXWXnWD2N2HOObkJhwl04HuhbPM8MznF1emRujwWxG6whI6OAImUlLLrSBYElbR7W0osv8Je5CeoX7onQggG8+KkjeYEUQARM6rYMi39PmvKxh4s2c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710759162; c=relaxed/simple; bh=1hDFHlMhhZjVt3nrmFvWCE08g8TZldnJyqZ6KnvDg4s=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=U3f8sP3j06utbvx8QLV/vOrAjePiPGdY4t3rL4xjbvW+b9eNnmFu+UNt0LLbgzBCF3bma6vw2ZUFco6dFqYuKhuclT/qC2gENmHHKZaLEYZ15jcxXdpyzedURjpZb8gu3HQEctkLcHTQIGXUYImMHUjWIGILrhzRLdgWWfqqi9k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=HBAx6MZD; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=KrqByIsv; arc=none smtp.client-ip=103.168.172.145 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="HBAx6MZD"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="KrqByIsv" Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailfout.nyi.internal (Postfix) with ESMTP id 81F3A13800D8; Mon, 18 Mar 2024 06:52:39 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Mon, 18 Mar 2024 06:52:39 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1710759159; x=1710845559; bh=Ny+fl7mAjf kfN5lr5eERPi/OCJ04ncOXXmkONsTr6KA=; b=HBAx6MZD88j0FXLykF45AsGBwy vqVKxCaAlRMlB8PPHY33cNsrf/pixnQRms0oMbwqli5f8Fko6PdLmoHk71Yl+/g1 4VR3mTW1NDpnIcY4l5E0fwdrERd9+gjPtYj4D+GkVUJnD5UX58j0ChtCXStqABKN P/HU/W0SGb+8SO15ft+ux7Q302ahaEzj1IcYzUFn9J71b0qjWvQi0WlULERAiop3 XcDB6P7ulCh6GKHy3GH6c6fMWEhD8HnHhuFl3VA/B7dHhFySrAdGMiqmnIdg15M2 lDqe/SUi7fJliN7Oo6QChsj84JY8fI/FQerY6Ka6qGtCwh5Zxa3tNS4z43gA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1710759159; x=1710845559; bh=Ny+fl7mAjfkfN5lr5eERPi/OCJ04 ncOXXmkONsTr6KA=; b=KrqByIsv7kkK7UzRZO7nyYHMYJrPIMXzWbR8SBp183SL 9mATJwx8uRLyVAfH68Ph/KfvKF176v8pHVf6U/DNdQyeXz0PCwXx7npFIXl2Y+Rv zkk1Z6hQHO/TXLNIvMZVimStfTpp+GiyWirPIh/BDptrwAaVIkleDpvCIWli5UNw OH3SBmnShCrOdkXY/2vwZuy/4GdVC4aL7rGS0cx19ZDcIqczDs/fJtzUk5pLlwU1 hJnsuKd6NYcg0o/nBKGI85JUEFCCvCU+xeteg6edHA2bWMTHEmbw6aLs/U5QphZW VyM5y2BW/DlnVvu+DcSwG6o0c/BxbLbt3in7yamktw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrkeejgddvudcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpeffhffvvefukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeeukedtvedtffevleejtefgheehieegkeeluddvfeefgeehgfeltddtheejleffteen ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 18 Mar 2024 06:52:38 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 540720a5 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 18 Mar 2024 10:47:47 +0000 (UTC) Date: Mon, 18 Mar 2024 11:52:36 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Derrick Stolee Subject: [PATCH 03/15] reftable/stack: use error codes when locking fails during compaction Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Compaction of a reftable stack may fail gracefully when there is a concurrent process that writes to the reftable stack and which has thus locked either the "tables.list" file or one of the tables. This is expected and can be handled gracefully by some of the callers which invoke compaction. Thus, to indicate this situation to our callers, we return a positive return code from `stack_compact_range()` and bubble it up to the caller. This kind of error handling is somewhat awkward though as many callers in the call chain never even think of handling positive return values. Thus, the result is either that such errors are swallowed by accident, or that we abort operations with an unhelpful error message. Make the code more robust by always using negative error codes when compaction fails, with `REFTABLE_LOCK_ERROR` for the described benign error case. Note that only a single callsite knew to handle positive error codes gracefully in the first place. Subsequent commits will touch up some of the other sites to handle those errors better. Signed-off-by: Patrick Steinhardt --- reftable/stack.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/reftable/stack.c b/reftable/stack.c index eaa8bb9c99..ba15c48ddd 100644 --- a/reftable/stack.c +++ b/reftable/stack.c @@ -1003,7 +1003,7 @@ static int stack_compact_range(struct reftable_stack *st, LOCK_NO_DEREF); if (err < 0) { if (errno == EEXIST) - err = 1; + err = REFTABLE_LOCK_ERROR; else err = REFTABLE_IO_ERROR; goto done; @@ -1025,7 +1025,7 @@ static int stack_compact_range(struct reftable_stack *st, table_name.buf, LOCK_NO_DEREF); if (err < 0) { if (errno == EEXIST) - err = 1; + err = REFTABLE_LOCK_ERROR; else err = REFTABLE_IO_ERROR; goto done; @@ -1075,7 +1075,7 @@ static int stack_compact_range(struct reftable_stack *st, LOCK_NO_DEREF); if (err < 0) { if (errno == EEXIST) - err = 1; + err = REFTABLE_LOCK_ERROR; else err = REFTABLE_IO_ERROR; goto done; @@ -1187,7 +1187,7 @@ static int stack_compact_range_stats(struct reftable_stack *st, struct reftable_log_expiry_config *config) { int err = stack_compact_range(st, first, last, config); - if (err > 0) + if (err == REFTABLE_LOCK_ERROR) st->stats.failures++; return err; } From patchwork Mon Mar 18 10:52:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13595215 Received: from fhigh6-smtp.messagingengine.com (fhigh6-smtp.messagingengine.com [103.168.172.157]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8B2AA2E85B for ; Mon, 18 Mar 2024 10:52:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.157 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710759166; cv=none; b=mNANFmbC5VFnBDYo6szxzGAcINV8Z9mLlx5Q5otE3dsRGjteP7Wr/KFs9YEjTUIQBkUEVQLvS9BbOQTmn9pfRpD/JEqaT/zfhxMisb32GieaTljcnfvmR3ai5TPn8Zc59BOyho+vM3Th7gzr3CXHi4zpenA+8AxHwPbj1uWSCfg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710759166; c=relaxed/simple; bh=/zANBXOsq1/YbLMV4okvUK9/rDoFGNQifq5BK0uRDWk=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=Q2yVglDltysNmTHVSk6lWWDEn3HR9Z8ERRIkqNawjjSIe/eb4sGu6dnbIjij1WXATNUvNAv48H5gzxf2xcl9bxlDgkczbNIunzUpbdmebOhoXFpiiWi0K/wrNyewWbFWvkujjHukqo6H5oztx0o1q7Vhtg7mQyC55ssztbn5Xbg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=GPneP7t6; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=QkwiEbjC; arc=none smtp.client-ip=103.168.172.157 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="GPneP7t6"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="QkwiEbjC" Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailfhigh.nyi.internal (Postfix) with ESMTP id 8A3AB114006D; Mon, 18 Mar 2024 06:52:43 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Mon, 18 Mar 2024 06:52:43 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1710759163; x=1710845563; bh=WZxBzlPduq nIBmJ6tAanerF0snzTazF0bsa5FuWVCJE=; b=GPneP7t6lUshkyTNPs0XwzbhLt DUCmkcQ2ypYSW6XXQBd5iFc74fnqnQPlEoTd7Co+RgG0Y168zJKpWO3K9XojaA6z 5S6Pa/ctKhTkL7dG8vFnULB4sEfvwLoiZR0iX3pGkj1MSVIpEj2v7QcFPqZ4l17K AnpsZRq0ZoVF/vAXspbig5tdzjONRsMPyit6vOlDkDm2fouqQFrqW0RoZxwnt1oe ptvEW9pfB3Xys7a2eC4ads9ow4diWVxRii4oIzBPl375dXFEiEZ+L2quQqhTpkw/ c20zU0bBw4yx8QCrj0yr2Lp5CjS1/KJy2vbUkZVclQIQ7AsXTLRKwci0l0xg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1710759163; x=1710845563; bh=WZxBzlPduqnIBmJ6tAanerF0snzT azF0bsa5FuWVCJE=; b=QkwiEbjCmG5A9SNB+oUU7ZUJwIvX1QD3DlTZsmzDOupV o8fB5/hGN5di3lj4H/JNsE0C/fa3DRiwrTlFVxXaq9osFsi9md50OFRv7JRio6Gp GW/JPwymOOytcUBxbwlAH/oNu/Cyz836m3CM1szYJwXVQ4OeDbEfFrs1ZGpjuBTY 0r9v6adaYqozfz/649D6F9vX+ZFwZn6uqJZ/yulFYeo2X6DS67bDqy918RI4kH2C 6Ln+DGDZYmjOUEAlhNikMDwMPphvDGjQJR4D7C9xFIid6VC3+k2wzZe3UchrM7GL X3qIhnJb2zgXEcGGSmkNRZdCRdHCHdLw+pXOgsU4Zg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrkeejgddvudcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpeffhffvvefukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeeukedtvedtffevleejtefgheehieegkeeluddvfeefgeehgfeltddtheejleffteen ucevlhhushhtvghrufhiiigvpedunecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 18 Mar 2024 06:52:42 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 57c14de8 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 18 Mar 2024 10:47:51 +0000 (UTC) Date: Mon, 18 Mar 2024 11:52:40 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Derrick Stolee Subject: [PATCH 04/15] reftable/stack: gracefully handle failed auto-compaction due to locks Message-ID: <37a18b91ca998ca5ad27d17e86d286040b6e6ee1.1710706118.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Whenever we commit a new table to the reftable stack we will end up invoking auto-compaction of the stack to keep the total number of tables at bay. This auto-compaction may fail though in case at least one of the tables which we are about to compact is locked. This is indicated by the compaction function returning a positive value. We do not handle this case though, and thus bubble that return value up the calling chain, which will ultimately cause a failure. Fix this bug by handling positive return values. Signed-off-by: Patrick Steinhardt --- reftable/stack.c | 13 ++++++++++++- t/t0610-reftable-basics.sh | 20 ++++++++++++++++++++ 2 files changed, 32 insertions(+), 1 deletion(-) diff --git a/reftable/stack.c b/reftable/stack.c index ba15c48ddd..dd5160d751 100644 --- a/reftable/stack.c +++ b/reftable/stack.c @@ -680,8 +680,19 @@ int reftable_addition_commit(struct reftable_addition *add) if (err) goto done; - if (!add->stack->disable_auto_compact) + if (!add->stack->disable_auto_compact) { + /* + * Auto-compact the stack to keep the number of tables in + * control. Note that we explicitly ignore locking issues which + * may indicate that a concurrent process is already trying to + * compact tables. This is fine, so we simply ignore that error + * condition. + */ err = reftable_stack_auto_compact(add->stack); + if (err < 0 && err != REFTABLE_LOCK_ERROR) + goto done; + err = 0; + } done: reftable_addition_close(add); diff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh index 686781192e..5f2f9baa9b 100755 --- a/t/t0610-reftable-basics.sh +++ b/t/t0610-reftable-basics.sh @@ -340,6 +340,26 @@ test_expect_success 'ref transaction: empty transaction in empty repo' ' EOF ' +test_expect_success 'ref transaction: fails gracefully when auto compaction fails' ' + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + + test_commit A && + for i in $(test_seq 10) + do + git branch branch-$i && + for table in .git/reftable/*.ref + do + touch "$table.lock" || exit 1 + done || + exit 1 + done && + test_line_count = 13 .git/reftable/tables.list + ) +' + test_expect_success 'pack-refs: compacts tables' ' test_when_finished "rm -rf repo" && git init repo && From patchwork Mon Mar 18 10:52:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13595216 Received: from out4-smtp.messagingengine.com (out4-smtp.messagingengine.com [66.111.4.28]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8EAB23611F for ; Mon, 18 Mar 2024 10:52:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=66.111.4.28 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710759171; cv=none; b=FQCyJWvH+T8ldrSGQRMluWyIeadrn31DSKjC31yzXTOJLLt7d1IDthOqaALGA2du0/UfxgOI2tjnpaM7UoBa8itpJbgsPY8xun5ZlsM1ZIcY82zH/NWAEe482t/VJYfyJbW7WFmbN0v7lDCNXzKqQ9ZFAgm8HB8BC8FMpO9kQ40= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710759171; c=relaxed/simple; bh=nGBIsuU+kS3bKWVXomXq7EbrW9Q6OtWBGA+oOOvp2y8=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=BZb87YyBQiRMCg/NojYIqR0fqGTAt0Q+4Pr/x9YzbdFcatGVMgGzVACPSPrzEGTarywhS5Lrm0PjXdredF0HpwZrcxwcy8JxDcVH62trtgo5rjekw2l0BdVJoVeyJv2CyM/vZ67k/gtCImjzAymgrIkCPFYLRgAO2aV5JlkeT4c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=nUxnKFRF; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=cJ1Zqilw; arc=none smtp.client-ip=66.111.4.28 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="nUxnKFRF"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="cJ1Zqilw" Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailout.nyi.internal (Postfix) with ESMTP id 8D37B5C005B; Mon, 18 Mar 2024 06:52:48 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Mon, 18 Mar 2024 06:52:48 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1710759168; x=1710845568; bh=ijCvTQSBf1 nsUGKqqguA/kXc1KwGVni7lM1GgKHLOWQ=; b=nUxnKFRFqZDH8r5xy66Fcx18l5 CT7dxjlShcIH8H936mxfKiQ/03LAKsAM8H0BhA6ub3W7rTxfNzKTEKXS0sJ0Vwpt 5iUFRJkOWqDLimwJabyKPm0vMtP+P8CBMdLOEcfqPUFkUZt6EkpZxLquaVEjZj2A BnhKWhnYd4Tu3Zb0BwrQsvC8xzg6/aZPx1yW7tLDjyA+lKb4ZfZpr16PLaoNpNBf vtsmVmfGSrs2kfQq4nkdKum70rBgMm/EZjficqlgSEZ4EXQQZcyidW50ILbnemlh sHInGlBsA2SNCUVviT44mSAPxG2WiwxuLjim7CC/ewU5o6BfNEbUK5cWjK/A== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1710759168; x=1710845568; bh=ijCvTQSBf1nsUGKqqguA/kXc1KwG Vni7lM1GgKHLOWQ=; b=cJ1Zqilwe2dbwjPZcQlqhgcm8oj5Tio6ixdefiDhmikH nHSmsCLEl8yb7WzWMCAVsN2agS/fohoAxr9Je7gMoWdWJ8kzaR88gmkDa6yByS9R g7v3b2eG1CBahmBYFRJbttxsjICB7vvg9QeswK/DWpuA4PIcqfiUFW3DaCcLizZe yl371rdY+eidxYEbpsNDrM/sBGzPQWukk9/sj1nktBSfY3EHELPEEfLtiptEpDk3 NOukS8XafxZADOrCdvvLrqZt2/cCfWr3smM3lHkubzp2nNJpXUnuXZN8fWUjFG3b LRA8MNyEZ9PQSFNLTM40YDozocjeu+Phr9WjF3BA9g== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrkeejgddvudcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpeffhffvvefukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeeukedtvedtffevleejtefgheehieegkeeluddvfeefgeehgfeltddtheejleffteen ucevlhhushhtvghrufhiiigvpedvnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 18 Mar 2024 06:52:47 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id f3ca0b51 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 18 Mar 2024 10:47:56 +0000 (UTC) Date: Mon, 18 Mar 2024 11:52:45 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Derrick Stolee Subject: [PATCH 05/15] refs/reftable: print errors on compaction failure Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: When git-pack-refs(1) fails in the reftable backend we end up printing no error message at all, leaving the caller puzzled as to why compaction has failed. Fix this. Signed-off-by: Patrick Steinhardt --- refs/reftable-backend.c | 6 +++++- t/t0610-reftable-basics.sh | 12 ++++++++++++ 2 files changed, 17 insertions(+), 1 deletion(-) diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c index 74dab18eda..66cdbbdb24 100644 --- a/refs/reftable-backend.c +++ b/refs/reftable-backend.c @@ -1221,8 +1221,12 @@ static int reftable_be_pack_refs(struct ref_store *ref_store, stack = refs->main_stack; ret = reftable_stack_compact_all(stack, NULL); - if (ret) + if (ret < 0) { + ret = error(_("unable to compact stack: %s"), + reftable_error_str(ret)); goto out; + } + ret = reftable_stack_clean(stack); if (ret) goto out; diff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh index 5f2f9baa9b..a53d1dc493 100755 --- a/t/t0610-reftable-basics.sh +++ b/t/t0610-reftable-basics.sh @@ -375,6 +375,18 @@ test_expect_success 'pack-refs: compacts tables' ' test_line_count = 1 repo/.git/reftable/tables.list ' +test_expect_success 'pack-refs: compaction raises locking errors' ' + test_when_finished "rm -rf repo" && + git init repo && + test_commit -C repo A && + touch repo/.git/reftable/tables.list.lock && + cat >expect <<-EOF && + error: unable to compact stack: data is locked + EOF + test_must_fail git -C repo pack-refs 2>err && + test_cmp expect err +' + test_expect_success 'pack-refs: prunes stale tables' ' test_when_finished "rm -rf repo" && git init repo && From patchwork Mon Mar 18 10:52:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13595217 Received: from fhigh6-smtp.messagingengine.com (fhigh6-smtp.messagingengine.com [103.168.172.157]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 90D7F364AA for ; Mon, 18 Mar 2024 10:52:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.157 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710759176; cv=none; b=OeO06sK2QqzJuuHNafGyeHVxTgc3R6ux2ZnovdIHBprC/CeML0xazc3VIKUi50qhe0yXsET1Rw/Ks7XBIJwJvPiS7J13jLM9xWldW0Y6k4bbJREGq4wpptRc+rzb+no55o/TcxgqLoS6Q9XyxjFrQ2sVKJRaSDQJzmx5eBjjEuM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710759176; c=relaxed/simple; bh=oxD9+zAkkIZ48ALDQz1+gdr6QJkrGrsA9j0cYKSn31U=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=fuxFhJMvFq6NcwlIE//HQbsgHsCxlwLuA19xtgIhApzhEeMzpSqjdz8dEy5NNdu1WoSHyOCvVOVkN+F8llB9XqAACln+AqWTIJmgEck3XD0SlMrw1u8PrwycVJE3zDafWIDxTnZj7ZnxGDwYYbA99+WnBlzuWFRqdkNWQ9XkYo0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=XrH819LF; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=TvKaVX/8; arc=none smtp.client-ip=103.168.172.157 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="XrH819LF"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="TvKaVX/8" Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailfhigh.nyi.internal (Postfix) with ESMTP id 98458114009B; Mon, 18 Mar 2024 06:52:53 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Mon, 18 Mar 2024 06:52:53 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1710759173; x=1710845573; bh=EgMyM0noVE Nyv7Vhv77Dn0gBi+Jbe78RZ0eKqzXtEZ8=; b=XrH819LFMxQBz+d7SK7+HflMTM ff6mQJ8X2gfmmWjQniwxeoM9L2Sa9jLCohMb5JUpu7gS5fdbFuQdwF13qRZwDNQq ivfUv1LbwABu93BLvG36NN2FEWi89OH7T9qr+PLCE1Tyhepv6qOQmmbWyr3CwxQR LylAoIXNykkBO8nf3VCe1AohMgoiqP5Ff3jcivuTarjYdVG1Zg5V2T2ElaEraVZW rKqS/y0mIdbXudoknWTn/4/hRoMumYxTWppuwlNtNrmd5/OQV/cErhPCU4T/AvRB zsbDljlio/f5EEsMrSUSMgH3QGx5u2fqEdP+cOntPim7FrMjtg44kUEzCSvQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1710759173; x=1710845573; bh=EgMyM0noVENyv7Vhv77Dn0gBi+Jb e78RZ0eKqzXtEZ8=; b=TvKaVX/85PrtD+CNU/84mIr7y8/kTmGhJkalB0Fbztax ZfK1mESD+TDYY7zad5Ve/EdyE1SLZOajwymH8rp1yJvnCVaGytrSa/9BhhcRw89R fy6+wQLUARivxAjCP6Zxr+zz4+HLiF7176xZI1P6lGYDjZY2aFoFRxuoJI/S8yb5 Y9QwkS9J5BJEro614OSFSrmSrdrKAhEToa9YztndePReL7NHEKRlDe4d90GKs+s2 DbcuIERfnfzGuwCiVQlVJof7UX3JhrK2JF0Z/Mk936kbFYMdPUuF8hhHCzmpcolB FSqG6mLI0ZLxX5EV776G8nwM1XBq9aKXWBLrWq3m0A== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrkeejgddvudcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpeffhffvvefukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeeukedtvedtffevleejtefgheehieegkeeluddvfeefgeehgfeltddtheejleffteen ucevlhhushhtvghrufhiiigvpedunecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 18 Mar 2024 06:52:52 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id c5acd902 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 18 Mar 2024 10:48:01 +0000 (UTC) Date: Mon, 18 Mar 2024 11:52:50 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Derrick Stolee Subject: [PATCH 06/15] t/helper: drop pack-refs wrapper Message-ID: <999d0c2bb8fe11766a20d102dcef0f4f38a5fbd4.1710706118.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The test helper provides a "ref-store pack-refs" wrapper that more or less directly invokes `refs_pack_refs()`. This helper is only used in a single test with the "PACK_REFS_PRUNE" and "PACK_REFS_ALL" flags. Both of these flags can directly be accessed via git-pack-refs(1) though via the `--all` and `--prune` flags, which makes the helper superfluous. Refactor the test to use git-pack-refs(1) instead of the test helper. Drop the now-unused test helper command. Signed-off-by: Patrick Steinhardt --- t/helper/test-ref-store.c | 20 -------------------- t/t0601-reffiles-pack-refs.sh | 13 +++++++++---- 2 files changed, 9 insertions(+), 24 deletions(-) diff --git a/t/helper/test-ref-store.c b/t/helper/test-ref-store.c index 7a0f6cac53..82bbf6e2e6 100644 --- a/t/helper/test-ref-store.c +++ b/t/helper/test-ref-store.c @@ -112,25 +112,6 @@ static const char **get_store(const char **argv, struct ref_store **refs) return argv + 1; } -static struct flag_definition pack_flags[] = { FLAG_DEF(PACK_REFS_PRUNE), - FLAG_DEF(PACK_REFS_ALL), - { NULL, 0 } }; - -static int cmd_pack_refs(struct ref_store *refs, const char **argv) -{ - unsigned int flags = arg_flags(*argv++, "flags", pack_flags); - static struct ref_exclusions exclusions = REF_EXCLUSIONS_INIT; - static struct string_list included_refs = STRING_LIST_INIT_NODUP; - struct pack_refs_opts pack_opts = { .flags = flags, - .exclusions = &exclusions, - .includes = &included_refs }; - - if (pack_opts.flags & PACK_REFS_ALL) - string_list_append(pack_opts.includes, "*"); - - return refs_pack_refs(refs, &pack_opts); -} - static int cmd_create_symref(struct ref_store *refs, const char **argv) { const char *refname = notnull(*argv++, "refname"); @@ -326,7 +307,6 @@ struct command { }; static struct command commands[] = { - { "pack-refs", cmd_pack_refs }, { "create-symref", cmd_create_symref }, { "delete-refs", cmd_delete_refs }, { "rename-ref", cmd_rename_ref }, diff --git a/t/t0601-reffiles-pack-refs.sh b/t/t0601-reffiles-pack-refs.sh index c309d2bae8..b1cf587347 100755 --- a/t/t0601-reffiles-pack-refs.sh +++ b/t/t0601-reffiles-pack-refs.sh @@ -32,11 +32,16 @@ test_expect_success 'prepare a trivial repository' ' HEAD=$(git rev-parse --verify HEAD) ' -test_expect_success 'pack_refs(PACK_REFS_ALL | PACK_REFS_PRUNE)' ' - N=`find .git/refs -type f | wc -l` && +test_expect_success 'pack-refs --prune --all' ' + test_path_is_missing .git/packed-refs && + git pack-refs --no-prune --all && + test_path_is_file .git/packed-refs && + N=$(find .git/refs -type f | wc -l) && test "$N" != 0 && - test-tool ref-store main pack-refs PACK_REFS_PRUNE,PACK_REFS_ALL && - N=`find .git/refs -type f` && + + git pack-refs --prune --all && + test_path_is_file .git/packed-refs && + N=$(find .git/refs -type f) && test -z "$N" ' From patchwork Mon Mar 18 10:52:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13595218 Received: from fhigh6-smtp.messagingengine.com (fhigh6-smtp.messagingengine.com [103.168.172.157]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6EC86364AA for ; Mon, 18 Mar 2024 10:52:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.157 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710759179; cv=none; b=cze5hlM9yA4INoHtdq9CBFGKgnVUQMl6qt80KTH7Y2wu23I8YADMP4dLRv23bL6fHIMVb0qETLTmLYIOjtsQvc2SUgpDd/8Ej0kLVa06L0tFRLALYtQpNUiq2PUHa55s1SmuvpCqH+hcIPqhhDS/zny4hhq2FZHuwlnv8yIvySA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710759179; c=relaxed/simple; bh=oQT8on1+l/dRIS2KaXkEBK9n5NIxHigJDuiCoWYB6wc=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=YEkTFn+SxmkNTVyJvuuZXeOt5k86Q1NvtnL4bHNdjLbjvYGH/LXBtQz+UuBv8e/cbYd5JvunaAVN/jWzA6sVRh9F0PVVa+91CsGxNLhnzvC3M8Fb7ynNk8yRSfE8NqbT+DPZ3CidzJI9NX4A3op0mTNEWOc42gqwxLstief2YxE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=F+HnD75f; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=YNitTk9N; arc=none smtp.client-ip=103.168.172.157 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="F+HnD75f"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="YNitTk9N" Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailfhigh.nyi.internal (Postfix) with ESMTP id ADA55114006D; Mon, 18 Mar 2024 06:52:57 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Mon, 18 Mar 2024 06:52:57 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1710759177; x=1710845577; bh=OzsO38GPky iKvDrw5DKmd21FLlUKuv4Rbd4fG+KDkiY=; b=F+HnD75fUa1gu+Xc5jlAA3M8wH FGHniDzEOOLZZ3VUEpv8uWcU9IKVHFKHxpb3TgfnSAd+mWePtEdLvYv2d3VmstE4 P6Lp2Ql8Ho19oRyyMGUKgNqcIPzUk6mF6vZZ14G7tAn3hqsEFuMpz264AWdBitU6 3G5cUi2zefEkmKZU/vADat1eFcdvYDQ6mmmZeSi/ByHNg3v+FwrnFaqpM9sAMZRG thvwQwzVqgmw1bh6nKKfZvP19m2P3fC4zZqymH8l2ruKv66kacSl6RFhKhjCRKfH +nbvXoxKOsCg3L4kUC9ghgcQUAadzZ+J+iOzBR1Yh8hl/DWERiyYIrcrBurQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1710759177; x=1710845577; bh=OzsO38GPkyiKvDrw5DKmd21FLlUK uv4Rbd4fG+KDkiY=; b=YNitTk9NiU5nnlKWpy4IQnpn2ayhC4ApgnAWaXj/hEM9 eX7NHEg560GWlIebhpIZjwGXozsgV9O9zCjvJXwymnNZXrzsC3NhN9P/PybkvVtI 0Be+9+ME8v4S2IOjsO0I5Ctbk04IhJ43DjnbCs++nxqGL9AxaOD2pEUMWNW/oJEO 6TI3zQU25ipeESpbwUu+4lWYutIC5Ep0IOnwBwgtzZaoOp8BOO7fUjp7oZPAl5iQ 168p38eNzIlU+Ejh7ZpkS/cbJQPTJW3VshpZDO36O9bZUJFoONYaLfDCg3AUbqGQ K5kkC4xbLn9VkMkV9MSk8Aaq6IuDx8bbVDqnEb+7GQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrkeejgddvudcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpeffhffvvefukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeeukedtvedtffevleejtefgheehieegkeeluddvfeefgeehgfeltddtheejleffteen ucevlhhushhtvghrufhiiigvpedvnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 18 Mar 2024 06:52:56 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id cef2137c (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 18 Mar 2024 10:48:05 +0000 (UTC) Date: Mon, 18 Mar 2024 11:52:54 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Derrick Stolee Subject: [PATCH 07/15] refs: move `struct pack_refs_opts` to where it's used Message-ID: <072627d82cc71a56a4e0884a15380a9df5d1d03d.1710706118.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The declaration of `struct pack_refs_opts` is in a seemingly random place. Move it so that it's located right next to its flags and functions that use it. Signed-off-by: Patrick Steinhardt --- refs.h | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/refs.h b/refs.h index 298caf6c61..139ce7113b 100644 --- a/refs.h +++ b/refs.h @@ -66,12 +66,6 @@ const char *ref_storage_format_to_name(unsigned int ref_storage_format); #define RESOLVE_REF_NO_RECURSE 0x02 #define RESOLVE_REF_ALLOW_BAD_NAME 0x04 -struct pack_refs_opts { - unsigned int flags; - struct ref_exclusions *exclusions; - struct string_list *includes; -}; - const char *refs_resolve_ref_unsafe(struct ref_store *refs, const char *refname, int resolve_flags, @@ -433,6 +427,12 @@ void warn_dangling_symrefs(FILE *fp, const char *msg_fmt, #define PACK_REFS_PRUNE 0x0001 #define PACK_REFS_ALL 0x0002 +struct pack_refs_opts { + unsigned int flags; + struct ref_exclusions *exclusions; + struct string_list *includes; +}; + /* * Write a packed-refs file for the current repository. * flags: Combination of the above PACK_REFS_* flags. From patchwork Mon Mar 18 10:52:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13595219 Received: from fhigh6-smtp.messagingengine.com (fhigh6-smtp.messagingengine.com [103.168.172.157]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B503336AFE for ; Mon, 18 Mar 2024 10:53:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.157 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710759185; cv=none; b=stDoVH47izlfjwtMhKw5hvUS5+EiUrXWaA9D5wHGzVk08clPRFqHfhX0GrjuXrABpQU35cLCsfaH48ZvTbCyumK56n3vDgRbxCpYS0rt7D0RF6YP01uz55BBxUfHO6TrSkydz6pNfvmTIbXEfkfWfFJPwECsiQTOgXMtF7rEMoE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710759185; c=relaxed/simple; bh=9SVKaQET/O+7k/zJCC7OARR5frvuYW2635WXGhc7nIQ=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=ErnKGopBcWIsAx/tgSTDMNibIWYBTTXhhfjRF8UPJdvI/4i5gCoKab7KpY/WxB1B0ZOqLEpm7OQRGRLCDMYEptUHmwAy9vujqBOKbE7S48EImg49yYQc9hnSa0Iib7XEBF3p3tqh69q1HSEE7iCqGFj2PbwPMmNlcNodCEkm9Ac= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=PQzudulQ; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=rFZIdCxs; arc=none smtp.client-ip=103.168.172.157 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="PQzudulQ"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="rFZIdCxs" Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailfhigh.nyi.internal (Postfix) with ESMTP id BA7B9114009B; Mon, 18 Mar 2024 06:53:02 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Mon, 18 Mar 2024 06:53:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1710759182; x=1710845582; bh=m9vT/HQUff 7fvzAb6+bpQXMsus2+tEtV3WmVW+iQ7XU=; b=PQzudulQxE0dVMzSrHkT5qr4iU DomQPGfgNhgrHVUAOJMxFylSfbAyLgUFKNsgSNIvdPzoYK3C5G/2xhQBgF8YZDZI gJQyYdGJotM9Nyy57+tyULPqrYTNUr1yRxaz71gpF0rRSyDPvc1ziLoG6jIovoYq 6+uijUJ8lbBVb+Wq3jMsjj73/46fLPUT83HVRrpmKG+9y1ctqkJSbB9ZSveCNyCq eJWT3PtAT9QaK2EJQxeuC55v0gTWZaeGUgsUr6LQ9aUSRI/lIJk6Ssn8JxvFjtox D6FkD36cfDqhwWYBuG+afNa7ZprG5KSp9QH8HILYQktC4yA9ee0ZHUmH1Svw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1710759182; x=1710845582; bh=m9vT/HQUff7fvzAb6+bpQXMsus2+ tEtV3WmVW+iQ7XU=; b=rFZIdCxs/rmVLKoYZGJ8Z31MnJvrBCa1vfTzJWcxSYnI 3lXG+MpXe+SaMufU0v5v06IGhnQ3MBaWyEeMz6f6MzqbYcpbKHbi5dvjilsEya2c EsBNPyyTFlO4D2MtjRHPjOxarYTEaBiNBCFcpPb2vYLf168y7UesBiZbXmZ8AgVB v2i86v/vM0p7C0tgIQGpcZnRvorBLYl1iWCQAR830e5eJakO4Og1k7v4I+tVMJCO c0BgGMoiAKyGUCkLDbvFsxfxDycaQtrahiNS5CtxtsSM/d6PNpMs/vDlZaXitYfY pF5T5CIKeWugyii9KcSmCOxD3Ir2JYdJWPQ0+XDXHg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrkeejgddvudcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpeffhffvvefukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeeukedtvedtffevleejtefgheehieegkeeluddvfeefgeehgfeltddtheejleffteen ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 18 Mar 2024 06:53:01 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 573be819 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 18 Mar 2024 10:48:10 +0000 (UTC) Date: Mon, 18 Mar 2024 11:52:59 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Derrick Stolee Subject: [PATCH 08/15] refs: remove `PACK_REFS_ALL` flag Message-ID: <919abe8eb993dbfab36136c7d9a4fd5987270519.1710706118.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The intent of the `PACK_REFS_ALL` flag is to ask the backend to compact all refs instead of only a subset of them. Thus, this flag gets passed down to `refs_pack_refs()` via `struct pack_refs_opts::flags`. But starting with 4fe42f326e (pack-refs: teach pack-refs --include option, 2023-05-12), the flag's semantics have changed. Instead of being handled by the respective backends, this flag is now getting handled by the callers of `refs_pack_refs()` which will add a single glob ("*") to the list of refs-to-be-packed. Thus, the flag serves no purpose to the ref backends anymore. Remove the flag and replace it with a local variable. Signed-off-by: Patrick Steinhardt --- builtin/pack-refs.c | 5 +++-- refs.h | 2 -- 2 files changed, 3 insertions(+), 4 deletions(-) diff --git a/builtin/pack-refs.c b/builtin/pack-refs.c index bcf383cac9..97921beef2 100644 --- a/builtin/pack-refs.c +++ b/builtin/pack-refs.c @@ -21,9 +21,10 @@ int cmd_pack_refs(int argc, const char **argv, const char *prefix) .flags = flags }; static struct string_list option_excluded_refs = STRING_LIST_INIT_NODUP; struct string_list_item *item; + int pack_all = 0; struct option opts[] = { - OPT_BIT(0, "all", &pack_refs_opts.flags, N_("pack everything"), PACK_REFS_ALL), + OPT_BOOL(0, "all", &pack_all, N_("pack everything")), OPT_BIT(0, "prune", &pack_refs_opts.flags, N_("prune loose refs (default)"), PACK_REFS_PRUNE), OPT_STRING_LIST(0, "include", pack_refs_opts.includes, N_("pattern"), N_("references to include")), @@ -38,7 +39,7 @@ int cmd_pack_refs(int argc, const char **argv, const char *prefix) for_each_string_list_item(item, &option_excluded_refs) add_ref_exclusion(pack_refs_opts.exclusions, item->string); - if (pack_refs_opts.flags & PACK_REFS_ALL) + if (pack_all) string_list_append(pack_refs_opts.includes, "*"); if (!pack_refs_opts.includes->nr) diff --git a/refs.h b/refs.h index 139ce7113b..8c8994cb29 100644 --- a/refs.h +++ b/refs.h @@ -422,10 +422,8 @@ void warn_dangling_symrefs(FILE *fp, const char *msg_fmt, /* * Flags for controlling behaviour of pack_refs() * PACK_REFS_PRUNE: Prune loose refs after packing - * PACK_REFS_ALL: Pack _all_ refs, not just tags and already packed refs */ #define PACK_REFS_PRUNE 0x0001 -#define PACK_REFS_ALL 0x0002 struct pack_refs_opts { unsigned int flags; From patchwork Mon Mar 18 10:53:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13595220 Received: from fout2-smtp.messagingengine.com (fout2-smtp.messagingengine.com [103.168.172.145]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A351F37162 for ; Mon, 18 Mar 2024 10:53:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.145 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710759190; cv=none; b=Hokb9COSeTaVetClWGN6pa7nsmgJfFALUdNeDLWrTwOiiGW94fi3JnkhM3CSBOisMnnFoX83c2P0JdvgkOvsc9ZC0lTJZpVvuppu5XBGaLwYnCKJoc1EuX2J4infpAcJKmoPSx45vCMTjwW8m8twk2vPzrAIlFfDrFY2irdZ7Yw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710759190; c=relaxed/simple; bh=IMM1/viW/LzwGwufRw/2442IL6Z7zSr4o8r8LQe6vAI=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=J6ZiyeFyS70i5XjoRl9RVzHiqfGVICMsthuCpNxr5KeErharnUxzYR7dV++4mqqebJQiWy594T6Bljwb4ja85EriU7QA9DJRUXnfZkrAjtJTxWoBq7rNToMfPWP87oKNgQ7qb579Tv1L9NXsgWDR8nV1Uz6Ex8uWX7lCn19/G24= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=l5+mqjge; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=T6EVSsio; arc=none smtp.client-ip=103.168.172.145 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="l5+mqjge"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="T6EVSsio" Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailfout.nyi.internal (Postfix) with ESMTP id C9F8C13800B8; Mon, 18 Mar 2024 06:53:06 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Mon, 18 Mar 2024 06:53:06 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1710759186; x=1710845586; bh=XKBZZRJ7hB 6UxlSt/Fjqr2jFVS1LRjVEGrsdzfMpgpA=; b=l5+mqjgeMol+Ug0Di4gaBb7MPx 8/SrVFSMH/MbXWVE5TqsvLRciKrazV8FTOpoUCuAw2WueqW523x7K80gn3PT33xM bQRh8ZCS6EOsQ7jmBo312JdGDKGeTch7k+g7kmgOg26U0xYMpJ3m8vx0NOPbinGE ehEPQ/fQ/9Gc8KLc545evtWZL5R3Me+kkN5t2gPPQSEhMlBkN8U8eKyY2I5RJ6Cp cfUoc/Y5/uV4VTrIT8Cd7mm1BYq6k6AAwiSyeXlZRFvqu6e4eh8gPuXZcgA3UxVx ej/5lKvFI0D3jpA0h76FJYcM/8x9RDqEb0oZm41BM3/McfMFvSMojwNrnCFg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1710759186; x=1710845586; bh=XKBZZRJ7hB6UxlSt/Fjqr2jFVS1L RjVEGrsdzfMpgpA=; b=T6EVSsioRjv9occoE2R06tVFC+bxoOlFG6k3vS7Z4hyL bwC9h4MHzKbg41AMwBb0L/Vcw/dU2FfQ16qJXkpZk14p9c9671RYoVTy3od52CpU 7M+RDRxyGxja4MHDSpsy6gT+ZvnGrWBY7Y0UaBHzxOq86sCUeAbsfocFC+BLJOfa h829E9UWJamev4FQRkMB8OvpQRtVakqMItibqoll0K8LhIevrfg8MzRcItqQeOQB 81D0Ztyvcm6fEuioQ2Lwu8y0XTso7gus5VR6kHhGNMeUoq+hLlkLsO0NTZ0iKXa7 9F4Dv3d5Nm1xMp6I/L1cRW6jYb2CIG4lUTHzVxcADA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrkeejgddvudcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpeffhffvvefukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeeukedtvedtffevleejtefgheehieegkeeluddvfeefgeehgfeltddtheejleffteen ucevlhhushhtvghrufhiiigvpeefnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 18 Mar 2024 06:53:05 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 34be92b6 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 18 Mar 2024 10:48:15 +0000 (UTC) Date: Mon, 18 Mar 2024 11:53:04 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Derrick Stolee Subject: [PATCH 09/15] refs/reftable: expose auto compaction via new flag Message-ID: <61ebcb2d52e829cf8d8a5b24f16a79b8ff4d0f3b.1710706118.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Under normal circumstances, the "reftable" backend will automatically perform compaction after appending to the stack. It is thus not necessary and may even be considered wasteful to run git-pack-refs(1) in "reftable"-backed repositories as it will cause the backend to compact all tables into a single one. We do exactly that though when running `git maintenance run --auto` or `git gc --auto`, which gets spawned by Git after running some specific commands. The `--auto` mode is typically only executing optimizations as needed. To do so, we already use several heuristics for the various different data structures in Git to determine whether to optimize them or not. We do not use any heuristics for refs though and instead always optimize them. Introduce a new `PACK_REFS_AUTO` flag that can be passed to the backend. When not handled by the backend we will continue to behave the exact same as we do right now, that is we optimize refs unconditionally. This is done for the "files" backend for now to retain current behaviour, even though we may eventually also want to introduce heuristics here. For the "reftable" backend though we already do have auto-compaction, so we can easily reuse that logic to implement the new auto-packing flag. Note that under normal circumstances, this should always end up being a no-op. After all, we already invoke the code for every single addition to the stack. But there are special cases where it can still be helpful to execute the auto-compaction code explicitly: - Concurrent writers may cause compaction to not run due to locks. - Callers may decide to disable compaction altogether and then pack refs at a later point due to various reasons. - Other implementations of the reftable format may do compaction differently or even not at all. Signed-off-by: Patrick Steinhardt --- refs.h | 6 +++++- refs/reftable-backend.c | 5 ++++- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/refs.h b/refs.h index 8c8994cb29..d278775e08 100644 --- a/refs.h +++ b/refs.h @@ -422,8 +422,12 @@ void warn_dangling_symrefs(FILE *fp, const char *msg_fmt, /* * Flags for controlling behaviour of pack_refs() * PACK_REFS_PRUNE: Prune loose refs after packing + * PACK_REFS_AUTO: Pack refs on a best effort basis. The heuristics and end + * result are decided by the ref backend. Backends may ignore + * this flag and fall back to a normal repack. */ -#define PACK_REFS_PRUNE 0x0001 +#define PACK_REFS_PRUNE (1 << 0) +#define PACK_REFS_AUTO (1 << 1) struct pack_refs_opts { unsigned int flags; diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c index 66cdbbdb24..135bd4e268 100644 --- a/refs/reftable-backend.c +++ b/refs/reftable-backend.c @@ -1220,7 +1220,10 @@ static int reftable_be_pack_refs(struct ref_store *ref_store, if (!stack) stack = refs->main_stack; - ret = reftable_stack_compact_all(stack, NULL); + if (opts->flags & PACK_REFS_AUTO) + ret = reftable_stack_auto_compact(stack); + else + ret = reftable_stack_compact_all(stack, NULL); if (ret < 0) { ret = error(_("unable to compact stack: %s"), reftable_error_str(ret)); From patchwork Mon Mar 18 10:53:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13595221 Received: from fout2-smtp.messagingengine.com (fout2-smtp.messagingengine.com [103.168.172.145]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 893D7374EA for ; Mon, 18 Mar 2024 10:53:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.145 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710759194; cv=none; b=BOs3nI2Ou/HWceXBMnJTdpvgslmIyFfKj15n3aTGgzHXCLd1KDTzIyojgAuTV94t+0o7X6TJSnhjK5U+ozLM69RxSxVZZ7elgbKGIYFIWUbeFaVlyv+6YgP+ChlsaqqpcmNVPwjb5/8XHjLLi5jjIIEXEuLPy6NcYt9DGTsCxI4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710759194; c=relaxed/simple; bh=wuAk0t3F5g6z6Z1pJHc4XbHgyoIs7lThnQxBamyTc2M=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=YbDuT2z5fH48qYGD2cd6GZoNObQsWYmqs3Z8486u/Pdi4ly2PE+tkHN6+HuxFAErYcHxyc+Sb3Lz7DkTqdES6bCMgjPdsbFrUzIgWW4WnPO1WuMnxpbk3+xNK+rUYnHLsBS6rlbpwv0yN27GdahtcvELe5fwLlDqyMhJ/xMVYRA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=IHku+0gd; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=Zg65xsRX; arc=none smtp.client-ip=103.168.172.145 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="IHku+0gd"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="Zg65xsRX" Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailfout.nyi.internal (Postfix) with ESMTP id D60E213800B8; Mon, 18 Mar 2024 06:53:11 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Mon, 18 Mar 2024 06:53:11 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1710759191; x=1710845591; bh=yAtfXVjQ50 LC4onQe5lyb8sFfUc0dHFM2vUMy5ASAUw=; b=IHku+0gdR9zyDP+EuSGLn52X+R fa6s2huq+HhbTAsr3bWSX4GQ0tPpKQRczULb9W0A9/0KRgtP2DRCHnprlinKtnj+ hdAbv96uxQIxDClvMXkyCoExaeeGSjBWIEnmDMjnqN0ZRAUWEH2cls0zjh4B8c9a qHzPjbk7w9STeNRKnoQrISTReZ0zxaYzU9DmhUKh1eVfa+JBMmLbjsj6nI1cWPQr 4whxQfAlhP3W9NHBIjt3OItBxXpz6Z0ZtTlpVr4oa2sMv17tvOWsGxtAiu/foRNu rLsNDSvz8UMShshXtnKvMG8HmSooj2XkFQYZ1HLphINWZajNKwaJc5aOCUuw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1710759191; x=1710845591; bh=yAtfXVjQ50LC4onQe5lyb8sFfUc0 dHFM2vUMy5ASAUw=; b=Zg65xsRXq7XVumbzZNP47CiPwY+XcFzg7XY9RyyidHCT avwiLSv2tplO52q3Fzh0e5IIhTd3OYvMKAn/MRVj1APX2/AS+5X5W6eDad3oYzRG qpgEkKRN5NYVznh43ExUh8Wh5AxKzCzk2eNc5YHeXIwLO/Plry191faFVSxtD8sD nSSjvcen9v91fm8LdTdPifKmcCGlOA25PsOC0Mf8rCnnkpCBj8O8rXRLZt2Za6Le sRXY3mAA1YrWXDyplzPzLJp1byoTjTfNim2QyP2T0V5LY2LoVPeGhEZYdwquFdP+ +MLNtn7LsIMspWFZb7QkUaFK2zS9GtaQqs1Skch+jw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrkeejgddvudcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpeffhffvvefukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeeukedtvedtffevleejtefgheehieegkeeluddvfeefgeehgfeltddtheejleffteen ucevlhhushhtvghrufhiiigvpedunecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 18 Mar 2024 06:53:10 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 1573318a (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 18 Mar 2024 10:48:19 +0000 (UTC) Date: Mon, 18 Mar 2024 11:53:08 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Derrick Stolee Subject: [PATCH 10/15] builtin/pack-refs: release allocated memory Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Some of the command line options in `cmd_pack_refs()` require us to allocate memory. This memory is never released and thus leaking, but we paper over this leak by declaring the respective variables as `static` function-level variables, which is somewhat awkward. Refactor the code to release the allocated memory and drop the `static` declaration. While at it, remove the useless `flags` variable. Signed-off-by: Patrick Steinhardt --- builtin/pack-refs.c | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/builtin/pack-refs.c b/builtin/pack-refs.c index 97921beef2..ea2baeec76 100644 --- a/builtin/pack-refs.c +++ b/builtin/pack-refs.c @@ -13,15 +13,17 @@ static char const * const pack_refs_usage[] = { int cmd_pack_refs(int argc, const char **argv, const char *prefix) { - unsigned int flags = PACK_REFS_PRUNE; - static struct ref_exclusions excludes = REF_EXCLUSIONS_INIT; - static struct string_list included_refs = STRING_LIST_INIT_NODUP; - struct pack_refs_opts pack_refs_opts = { .exclusions = &excludes, - .includes = &included_refs, - .flags = flags }; - static struct string_list option_excluded_refs = STRING_LIST_INIT_NODUP; + struct ref_exclusions excludes = REF_EXCLUSIONS_INIT; + struct string_list included_refs = STRING_LIST_INIT_NODUP; + struct pack_refs_opts pack_refs_opts = { + .exclusions = &excludes, + .includes = &included_refs, + .flags = PACK_REFS_PRUNE, + }; + struct string_list option_excluded_refs = STRING_LIST_INIT_NODUP; struct string_list_item *item; int pack_all = 0; + int ret; struct option opts[] = { OPT_BOOL(0, "all", &pack_all, N_("pack everything")), @@ -45,5 +47,10 @@ int cmd_pack_refs(int argc, const char **argv, const char *prefix) if (!pack_refs_opts.includes->nr) string_list_append(pack_refs_opts.includes, "refs/tags/*"); - return refs_pack_refs(get_main_ref_store(the_repository), &pack_refs_opts); + ret = refs_pack_refs(get_main_ref_store(the_repository), &pack_refs_opts); + + clear_ref_exclusions(&excludes); + string_list_clear(&included_refs, 0); + string_list_clear(&option_excluded_refs, 0); + return ret; } From patchwork Mon Mar 18 10:53:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13595222 Received: from fout2-smtp.messagingengine.com (fout2-smtp.messagingengine.com [103.168.172.145]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id ABC68376EC for ; Mon, 18 Mar 2024 10:53:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.145 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710759198; cv=none; b=JjWiPmbOitfnX9/GYMFakVHeRu9vCVBtWpLRQSbS5pURK2sjlDq6wOcGafIl9h5MuiztQbLv2IC5LNj+uizGdtKhOlbZEoon/iyP1xfNZSSKbPsu4wRH7UKbicAQoBdwCKMiQmw7tjGuCJ2dufigMSeUqxE08bM6zfE92T8EJBs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710759198; c=relaxed/simple; bh=xim6e9zk5yD8yB7HbU9Hg8g7oidPF3zorQnNwdjg6Js=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=t2wtPTRSh3XSfYT1WBp4MKaRV7a3z8iLT/oCnmaMJzbGu7EuchyfdDaSyLMs7ef47WT9ey5UI1d4efJuKPfNHM4xJGbRt9NCJ72F9E5mq2V1adBLvmuEPE5MkYRIppFwmNcmHRJeF+YX4ANyIIUwAgis5lmZubV2RVyG3jAd9DQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=N5VMLHey; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=U2xiKl27; arc=none smtp.client-ip=103.168.172.145 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="N5VMLHey"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="U2xiKl27" Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailfout.nyi.internal (Postfix) with ESMTP id D946313800B8; Mon, 18 Mar 2024 06:53:15 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute5.internal (MEProxy); Mon, 18 Mar 2024 06:53:15 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1710759195; x=1710845595; bh=KFrt4V0XpV jNJ5VgIdrBlUv53iRXlnUuNtyTcu3Z3nA=; b=N5VMLHeyvKuixCSbYufqsE8YNe 0DLNgQzlo4rS+QbWcR+vWk4eSqy6omUtIxCCz8Gn7iCn4HTBKFUIa7l0udZQIW8o KBgejwmFiNyY8EOLmJfMDVXqJi/SKCWOz2wuWalYUP4Bx8itIB6lZkJExBFbnd8r Lb+Lhmkm/Y2QiVlp5w7fnARZ8yOd2TI/0Mbz98FJqd32dH3VFl1DV9EyNLl1eC3M it0pAIYy9X+7s/tuYwrfbjVrEorqZtyUWDIzNUTxxjB/7UuT2Fqg5ZNSTuwSzNIO MehYdmINriY1r7CyDGysikSN5bbc59KAezA9OgZbDUV2IqZVInUWtzxaeUTQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1710759195; x=1710845595; bh=KFrt4V0XpVjNJ5VgIdrBlUv53iRX lnUuNtyTcu3Z3nA=; b=U2xiKl27ZA5aDoW7+C2ObkxkkXdqiyRsE1N5YenqpY45 t2HzG5eQ1Q1vBcCDs3V0gJ/97wcaR43fakldTgk0EU+l4MZFqCxYkqwr+kAzlxoa 4KRWf2ootb5oGf5xIw2X4IVAp2XZaPnSRPdzWBAPcnUPygijTfedR0i08nzR/Rs1 W7+He0/kQh2hV938TMAfG31DNFOsEBCC5xkokzoXem89Lw0etSh2jV9oCg8YcbQk mABhaBT5GvaB35rPUrk+j0OlhRr+k1mBZ7Nshc7jkrTAilRrmWsorQV9jZdIxvpt iJ5RaLqwoaOj5OuIHtj7Aj3xr3Ajbl4yslu69TYW1w== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrkeejgddvudcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpeffhffvvefukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeeukedtvedtffevleejtefgheehieegkeeluddvfeefgeehgfeltddtheejleffteen ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 18 Mar 2024 06:53:15 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 294c562e (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 18 Mar 2024 10:48:24 +0000 (UTC) Date: Mon, 18 Mar 2024 11:53:13 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Derrick Stolee Subject: [PATCH 11/15] builtin/pack-refs: introduce new "--auto" flag Message-ID: <8727f08bab8459ca608915a6320c817711742a87.1710706118.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Calling git-pack-refs(1) will unconditionally cause it to pack all requested refs regardless of the current state of the ref database. For example: - With the "files" backend we will end up rewriting the complete "packed-refs" file even if only a single ref would require compaction. - With the "reftable" backend we will end up always compacting all tables into a single table. This behaviour can be completely unnecessary depending on the backend and is thus wasteful. With the introduction of the `PACK_REFS_AUTO` flag in the preceding commit we can improve this and let the backends decide for themselves whether to pack refs in the first place. Expose this functionality via a new "--auto" flag in git-pack-refs(1), which mirrors the same flag in both git-gc(1) and git-maintenance(1). Signed-off-by: Patrick Steinhardt --- Documentation/git-pack-refs.txt | 15 ++++++++++++++- builtin/pack-refs.c | 3 ++- t/t0601-reffiles-pack-refs.sh | 7 +++++++ t/t0610-reftable-basics.sh | 34 +++++++++++++++++++++++++++++++++ 4 files changed, 57 insertions(+), 2 deletions(-) diff --git a/Documentation/git-pack-refs.txt b/Documentation/git-pack-refs.txt index 284956acb3..2dcabaf74c 100644 --- a/Documentation/git-pack-refs.txt +++ b/Documentation/git-pack-refs.txt @@ -8,7 +8,7 @@ git-pack-refs - Pack heads and tags for efficient repository access SYNOPSIS -------- [verse] -'git pack-refs' [--all] [--no-prune] [--include ] [--exclude ] +'git pack-refs' [--all] [--no-prune] [--auto] [--include ] [--exclude ] DESCRIPTION ----------- @@ -60,6 +60,19 @@ with many branches of historical interests. The command usually removes loose refs under `$GIT_DIR/refs` hierarchy after packing them. This option tells it not to. +--auto:: + +Pack refs as needed depending on the current state of the ref database. The +behavior depends on the ref format used by the repository and may change in the +future. ++ + - "files": No special handling for `--auto` has been implemented. ++ + - "reftable": Tables are compacted such that they form a geometric + sequence. For two tables N and N+1, where N+1 is newer, this + maintains the property that N is at least twice as big as N+1. Only + tables that violate this property are compacted. + --include :: Pack refs based on a `glob(7)` pattern. Repetitions of this option diff --git a/builtin/pack-refs.c b/builtin/pack-refs.c index ea2baeec76..db40825666 100644 --- a/builtin/pack-refs.c +++ b/builtin/pack-refs.c @@ -7,7 +7,7 @@ #include "revision.h" static char const * const pack_refs_usage[] = { - N_("git pack-refs [--all] [--no-prune] [--include ] [--exclude ]"), + N_("git pack-refs [--all] [--no-prune] [--auto] [--include ] [--exclude ]"), NULL }; @@ -28,6 +28,7 @@ int cmd_pack_refs(int argc, const char **argv, const char *prefix) struct option opts[] = { OPT_BOOL(0, "all", &pack_all, N_("pack everything")), OPT_BIT(0, "prune", &pack_refs_opts.flags, N_("prune loose refs (default)"), PACK_REFS_PRUNE), + OPT_BIT(0, "auto", &pack_refs_opts.flags, N_("auto-pack refs as needed"), PACK_REFS_AUTO), OPT_STRING_LIST(0, "include", pack_refs_opts.includes, N_("pattern"), N_("references to include")), OPT_STRING_LIST(0, "exclude", &option_excluded_refs, N_("pattern"), diff --git a/t/t0601-reffiles-pack-refs.sh b/t/t0601-reffiles-pack-refs.sh index b1cf587347..219a495451 100755 --- a/t/t0601-reffiles-pack-refs.sh +++ b/t/t0601-reffiles-pack-refs.sh @@ -164,6 +164,13 @@ test_expect_success 'test --exclude takes precedence over --include' ' git pack-refs --include "refs/heads/pack*" --exclude "refs/heads/pack*" && test -f .git/refs/heads/dont_pack5' +test_expect_success '--auto packs and prunes refs as usual' ' + git branch auto && + test_path_is_file .git/refs/heads/auto && + git pack-refs --auto --all && + test_path_is_missing .git/refs/heads/auto +' + test_expect_success 'see if up-to-date packed refs are preserved' ' git branch q && git pack-refs --all --prune && diff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh index a53d1dc493..6de7529575 100755 --- a/t/t0610-reftable-basics.sh +++ b/t/t0610-reftable-basics.sh @@ -387,6 +387,40 @@ test_expect_success 'pack-refs: compaction raises locking errors' ' test_cmp expect err ' +test_expect_success 'pack-refs: auto compaction' ' + test_when_finished "rm -rf repo" && + git init repo && + ( + cd repo && + + test_commit A && + + # The tables should have been auto-compacted, and thus auto + # compaction should not have to do anything. + ls -1 .git/reftable >tables-expect && + test_line_count = 4 tables-expect && + git pack-refs --auto && + ls -1 .git/reftable >tables-actual && + test_cmp tables-expect tables-actual && + + # Lock all tables write some refs. Auto-compaction will be + # unable to compact tables and thus fails gracefully, leaving + # the stack in a sub-optimal state. + ls .git/reftable/*.ref | + while read table + do + touch "$table.lock" || exit 1 + done && + git branch B && + git branch C && + rm .git/reftable/*.lock && + test_line_count = 5 .git/reftable/tables.list && + + git pack-refs --auto && + test_line_count = 1 .git/reftable/tables.list + ) +' + test_expect_success 'pack-refs: prunes stale tables' ' test_when_finished "rm -rf repo" && git init repo && From patchwork Mon Mar 18 10:53:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13595223 Received: from fhigh6-smtp.messagingengine.com (fhigh6-smtp.messagingengine.com [103.168.172.157]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9709C381A4 for ; Mon, 18 Mar 2024 10:53:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.157 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710759203; cv=none; b=uL0pkvPW/ib9qLuBqb3pCEYLs/ETDSPVZ5VCwO5xHCp1kXLVcw3lUYIjgp+Gb0NsP5RH+I8IW+3FXa3eoLs4nII0VU18qeMsNbZnzhkXOISY4nFRJONU4Cezta30XrUYBJMwvJoQK1E6u00X5RKl9wdi2cBfbs50YPaZruioQvw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710759203; c=relaxed/simple; bh=uA9ohyiCQuT2CVly4E0YxGX1OwRJCr7ZIHMd5AItVss=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=PYcm5uEJDfJ09edSts1rtfxDj6RqraW2hwdBCBzKyggXNnYDUxzzSR0tP847KNVpu559ErxeNIp8eZ14dL8m4s+utbUSvfdFzamGUcc04Bfj4uCJmKa5naprejb7mSsIJynmMtXkLOO9MLYtw1fm8FYRc0RLy7PHrkKYTVie840= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=eyq9/tXL; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=SYu6ZiBH; arc=none smtp.client-ip=103.168.172.157 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="eyq9/tXL"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="SYu6ZiBH" Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailfhigh.nyi.internal (Postfix) with ESMTP id E20A3114009B; Mon, 18 Mar 2024 06:53:20 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Mon, 18 Mar 2024 06:53:20 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1710759200; x=1710845600; bh=bSlEPbwMHZ Ywu+ZRrfl3qNrPWEu8vPt88gJuHuIkbus=; b=eyq9/tXL/bq7EoH6ukkyZt7oDD ZXwltBC+Vk+Q+4k3Vc+BIZokVec6ZVqJC+NzYF04CneOp/3O3j/DJSbkHDMm1N2I a7QRU7jF3zLGyO/d/N2Hb2aB/+TATuvZcvUNNXs7QC/6Yq55SmFqFfSGb2Vl6Yjh 6oUUGSV+GH+nnofP+AFyYA5yavRfFw9xdAz2Z3eX+n0Q1j6rD/qu7+WuzqX8z2Es J+bCAF2hzVixButm1YhSaPeMpeh2uz+plEntkg4hRQDcHwEDuPgeDdCG6zSxCasn ZnKzeJpwBiTyMilChTMDb44d0vdOrorT7xAqrSi+SARzFyquL9e7YPMv1mYA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1710759200; x=1710845600; bh=bSlEPbwMHZYwu+ZRrfl3qNrPWEu8 vPt88gJuHuIkbus=; b=SYu6ZiBHOYHy6YX3d0RtP4sskvKKkkLQzRvmRC3DCVlu x54MWWNmf5K90XOne9BClME6xiOoDwiajcJ9cEt8KGiUeXzhwpEjpEZK2hYndNSK aAxkSYZ1cMKzZsOeauLygHm5qeK+xVJPGy824zk2hi7WuNXkNn7Ais1k7YdpVztl 87ocaN0iubYr9CZpkQt1xRH5C8/I7RNYHNWEf+LPUceTtybyeMFZevLos6eg42Nj UBC1Mf2EOXJPDB7Cgo1rRxlxYJjmEf14VcZndq4ZDO7ZvHB4SQbLsdRoDqH7r1We luAQW2qP9lqlZwEbG9YNc/VVmWv2IGIh6v7lPQjsIw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrkeejgddvvdcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpeffhffvvefukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeeukedtvedtffevleejtefgheehieegkeeluddvfeefgeehgfeltddtheejleffteen ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 18 Mar 2024 06:53:20 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id c39799fc (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 18 Mar 2024 10:48:29 +0000 (UTC) Date: Mon, 18 Mar 2024 11:53:18 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Derrick Stolee Subject: [PATCH 12/15] builtin/gc: move `struct maintenance_run_opts` Message-ID: <65c9ff3ee51c982203e8b76378536958836db7d5.1710706118.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: We're about to start using `struct maintenance_run_opts` in `maintenance_task_pack_refs()`. Move its definition up to prepare for this. Signed-off-by: Patrick Steinhardt --- builtin/gc.c | 53 ++++++++++++++++++++++++++-------------------------- 1 file changed, 26 insertions(+), 27 deletions(-) diff --git a/builtin/gc.c b/builtin/gc.c index cb80ced6cb..e0029c88f9 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -180,7 +180,32 @@ static void gc_config(void) git_config(git_default_config, NULL); } -struct maintenance_run_opts; +enum schedule_priority { + SCHEDULE_NONE = 0, + SCHEDULE_WEEKLY = 1, + SCHEDULE_DAILY = 2, + SCHEDULE_HOURLY = 3, +}; + +static enum schedule_priority parse_schedule(const char *value) +{ + if (!value) + return SCHEDULE_NONE; + if (!strcasecmp(value, "hourly")) + return SCHEDULE_HOURLY; + if (!strcasecmp(value, "daily")) + return SCHEDULE_DAILY; + if (!strcasecmp(value, "weekly")) + return SCHEDULE_WEEKLY; + return SCHEDULE_NONE; +} + +struct maintenance_run_opts { + int auto_flag; + int quiet; + enum schedule_priority schedule; +}; + static int maintenance_task_pack_refs(MAYBE_UNUSED struct maintenance_run_opts *opts) { struct child_process cmd = CHILD_PROCESS_INIT; @@ -773,26 +798,6 @@ static const char *const builtin_maintenance_run_usage[] = { NULL }; -enum schedule_priority { - SCHEDULE_NONE = 0, - SCHEDULE_WEEKLY = 1, - SCHEDULE_DAILY = 2, - SCHEDULE_HOURLY = 3, -}; - -static enum schedule_priority parse_schedule(const char *value) -{ - if (!value) - return SCHEDULE_NONE; - if (!strcasecmp(value, "hourly")) - return SCHEDULE_HOURLY; - if (!strcasecmp(value, "daily")) - return SCHEDULE_DAILY; - if (!strcasecmp(value, "weekly")) - return SCHEDULE_WEEKLY; - return SCHEDULE_NONE; -} - static int maintenance_opt_schedule(const struct option *opt, const char *arg, int unset) { @@ -809,12 +814,6 @@ static int maintenance_opt_schedule(const struct option *opt, const char *arg, return 0; } -struct maintenance_run_opts { - int auto_flag; - int quiet; - enum schedule_priority schedule; -}; - /* Remember to update object flag allocation in object.h */ #define SEEN (1u<<0) From patchwork Mon Mar 18 10:53:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13595224 Received: from fout2-smtp.messagingengine.com (fout2-smtp.messagingengine.com [103.168.172.145]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B93A1381A4 for ; Mon, 18 Mar 2024 10:53:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.145 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710759208; cv=none; b=iG8kKegQYflBwNYqku3NTxAVBiIDuH1k1L+MJc9hT/59f47XLlFoOhOVJi1HLd8X8ledEsiSpow/werhJ+8sRol4nAXUzYnkA0JhUezCGX0FSrzdQptSm9bXcHqPVmztOF0B+b+OT1cpoA+ozVLIhQNC+aXivWb4XXR3rJLzf+A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710759208; c=relaxed/simple; bh=KyPvVSJIncliUNEsdSMnhy1RivlojMLRXZ8lgmC/CgU=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=iserblQIywGD9P7x1zNUYqsbZjzM6TT30OP3/rwlI3D0ufI40LbpOHo9NWDkmA9u/wL4/hKYSrcbyqCfKPrYbsq5avM3R0HW4kmeGK0vAFCH5PRH3fi5OOSZ8aDHpuOpvbvJK9d1obqRFyuQ0po0S7pQ+C2wCiNpRn7PzMZIXXo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=RvzQ12o/; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=WQ70OVyQ; arc=none smtp.client-ip=103.168.172.145 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="RvzQ12o/"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="WQ70OVyQ" Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailfout.nyi.internal (Postfix) with ESMTP id E444A13800B8; Mon, 18 Mar 2024 06:53:25 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Mon, 18 Mar 2024 06:53:25 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1710759205; x=1710845605; bh=PQOo5aifmg NDdsxHz7CMh7vVnHPe/r07M0VgP5TwfjQ=; b=RvzQ12o/BE6hiJFacytYtD2BwD FjGb1K02jDpbPu2q70chQVcOgfxT3RR4zJjV8RHdF5R8JV0W7rQr2GnjtlsmYuXS XtDPNUlfjayKB0WUBB9nVxKZiL6itpt3c5gB+2yQQ/yc77V1K5DyTBPTwx6V6gpv Eu32JGlAG6W4yK76vtYzJjk2vlVxTOmdpA9oFKzquOIQePqA7OkyIM47lZscP9Ra VhPcCJghXkUJbQwhTdeQgQToPvNrl1+XJBc/8fsOGHVUuGA1FUslxLDPtIhuSVJL Mmm4NY7JIZs/FqAQ2LqONFcV/5zxUb4yx2JufmoBkgv5reU6PQZnIusOfblg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1710759205; x=1710845605; bh=PQOo5aifmgNDdsxHz7CMh7vVnHPe /r07M0VgP5TwfjQ=; b=WQ70OVyQv+mPO/BRKcs7cRMzjn3YJ0r5UxXnYJ7K0ZkX x+reNOFJe7CX1HZwbG4H3D4m1Mswrh91osGlso6Z/ua/LJV8+CSORUI/k9HsOaN6 xFdEaEddj3qZzI03b37PxMH4OKFPz498fNv/21HwqtEi6+zqMUAL38kmwvsnXSk2 GIkUh08TN/t8qqNgKaGw+wEHqwkfk7ZLTMYXit9wjoXwsoxHtKK21JRKFqdcrkKm Ekn/heHHtQBKBp1bLmmDvytmGVZR8d3sX1rn4+6DMwaOGESGZ4xp0DnTN/2aP0fd d585xR3+RldptRJAFPx7dYHhFk73fR1hf8xs6GqCQA== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrkeejgddvudcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpeffhffvvefukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeeukedtvedtffevleejtefgheehieegkeeluddvfeefgeehgfeltddtheejleffteen ucevlhhushhtvghrufhiiigvpedunecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 18 Mar 2024 06:53:25 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id e4755ce4 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 18 Mar 2024 10:48:34 +0000 (UTC) Date: Mon, 18 Mar 2024 11:53:22 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Derrick Stolee Subject: [PATCH 13/15] t6500: extract objects with "17" prefix Message-ID: <817de1a88f2fc5edd8c113da2c49636a12ce0725.1710706119.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The ".git/obects/17/" shard is somewhat special because it is used by git-gc(1) to estimate how many objects there are by extrapolating the number of objects in that shard, only. In t6500 we thus have a hard coded set of data that, when written to the object database, result in blobs starting with that prefix. We are about to need such "17"-prefixed objects in another test suite. Extract them into "t/oid-info/hash-info" so that they can be reused by other tests. Signed-off-by: Patrick Steinhardt --- t/oid-info/hash-info | 12 ++++++++++++ t/t6500-gc.sh | 30 +++++++----------------------- 2 files changed, 19 insertions(+), 23 deletions(-) diff --git a/t/oid-info/hash-info b/t/oid-info/hash-info index d0736dd1a0..b8a5bcb187 100644 --- a/t/oid-info/hash-info +++ b/t/oid-info/hash-info @@ -15,3 +15,15 @@ empty_blob sha256:473a0f4c3be8a93681a267e3b1e9a7dcda1185436fe141f7749120a3037218 empty_tree sha1:4b825dc642cb6eb9a060e54bf8d69288fbee4904 empty_tree sha256:6ef19b41225c5369f1c104d45d8d85efa9b057b53b14b4b9b939dd74decc5321 + +blob17_1 sha1:263 +blob17_1 sha256:34 + +blob17_2 sha1:410 +blob17_2 sha256:174 + +blob17_3 sha1:523 +blob17_3 sha256:313 + +blob17_4 sha1:790 +blob17_4 sha256:481 diff --git a/t/t6500-gc.sh b/t/t6500-gc.sh index 18fe1c25e6..43d40175f8 100755 --- a/t/t6500-gc.sh +++ b/t/t6500-gc.sh @@ -11,23 +11,7 @@ test_expect_success 'setup' ' # behavior, make sure we always pack everything to one pack by # default git config gc.bigPackThreshold 2g && - - # These are simply values which, when hashed as a blob with a newline, - # produce a hash where the first byte is 0x17 in their respective - # algorithms. - test_oid_cache <<-EOF - obj1 sha1:263 - obj1 sha256:34 - - obj2 sha1:410 - obj2 sha256:174 - - obj3 sha1:523 - obj3 sha256:313 - - obj4 sha1:790 - obj4 sha256:481 - EOF + test_oid_init ' test_expect_success 'gc empty repository' ' @@ -114,8 +98,8 @@ test_expect_success 'pre-auto-gc hook can stop auto gc' ' # We need to create two object whose sha1s start with 17 # since this is what git gc counts. As it happens, these # two blobs will do so. - test_commit "$(test_oid obj1)" && - test_commit "$(test_oid obj2)" && + test_commit "$(test_oid blob17_1)" && + test_commit "$(test_oid blob17_2)" && git gc --auto >../out.actual 2>../err.actual ) && @@ -146,13 +130,13 @@ test_expect_success 'auto gc with too many loose objects does not attempt to cre # We need to create two object whose sha1s start with 17 # since this is what git gc counts. As it happens, these # two blobs will do so. - test_commit "$(test_oid obj1)" && - test_commit "$(test_oid obj2)" && + test_commit "$(test_oid blob17_1)" && + test_commit "$(test_oid blob17_2)" && # Our first gc will create a pack; our second will create a second pack git gc --auto && ls .git/objects/pack/pack-*.pack | sort >existing_packs && - test_commit "$(test_oid obj3)" && - test_commit "$(test_oid obj4)" && + test_commit "$(test_oid blob17_3)" && + test_commit "$(test_oid blob17_4)" && git gc --auto 2>err && test_grep ! "^warning:" err && From patchwork Mon Mar 18 10:53:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13595225 Received: from fhigh6-smtp.messagingengine.com (fhigh6-smtp.messagingengine.com [103.168.172.157]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DAAE7381A4 for ; Mon, 18 Mar 2024 10:53:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.157 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710759213; cv=none; b=HlyDazWkPo7QYyPtm1KsuiN94r3SRwVjZJ+UbSXjV0haD/o7nHkaBXkxsMbxMpEiKjRzLw8HPuQFzJMAKA/7y2Z2nkkB/EqlIhM8RqivNy52Ix96Gy33s2SItWENrf1RXc++8o5ExsoOHX9h9Vnb0ekIaiw0QWTG3A+GjLeAAzE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710759213; c=relaxed/simple; bh=RfD3CuZo9m8GZ5sv3Qs6bLWRNBROJTzgBan3ptNnD4o=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=FqSkkDEOgQdWk0+dFExCd9MQmiIBgO3csGpyrO73eV7KHoNO2A8K/QRmnM73GKuwC/YM5qdnRv+aa87fgzfMdEQVvDd7MR1/jPBOTSR7+WVHcM+4LUB8ByLC6loGCcjek+pFiAJg7FoIzzES+k5Fpf426erGJaEi4geOCMxYBhk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=jTGWoykM; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=DjZvvDAp; arc=none smtp.client-ip=103.168.172.157 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="jTGWoykM"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="DjZvvDAp" Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailfhigh.nyi.internal (Postfix) with ESMTP id E63D9114010D; Mon, 18 Mar 2024 06:53:30 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Mon, 18 Mar 2024 06:53:30 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1710759210; x=1710845610; bh=mzmFEhmqr3 JhSpOLKxNt+AwzNPFvcb/vI88tQOD4Hgs=; b=jTGWoykM8uPZaQ3KOUmP0xVIK2 QgQOJ/0RptTtAc3L2QgCObMMj3CzH8fKv/dZ5RVkie8on1zokXTI/FYCDzd3LUGX 2GxlQAo1gfNbd9Ct0Odf6cm3qk9ITrONxIonq5sHXVQz1WQg5NNr5nWD6UMNly5I Uy4tPio7xpzN2eurQnDV2mxYd8MbhKPSmtIU0zgHaxo8+bB0FTo5c0cHN+9LD3zh HQZ7p0vz4EjM2QiNjOHbfm69CfMgC7i3RLavU4in9iB05ql25D3VhZMubvxnyS55 71AWQlrifglxiXJGkpS5/rQKDTQaXqMF2ZO9nVZaoQ8FqeuM5zRNehXt/qjw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1710759210; x=1710845610; bh=mzmFEhmqr3JhSpOLKxNt+AwzNPFv cb/vI88tQOD4Hgs=; b=DjZvvDApL2kDwUXQPUMez/QaokqWbPjhxwZK/FRQ2nZA hU3M0jHrxXM5zERkiBKbnkjb8jpUYrmSJ/5VSsCzL78T/+psWvN/6nhGecWBdz29 BfhZf5HdLxHkLzv8sDJMGzgj2wMKVHg8Jnp9dltlrlJRxb0wQYfZKnbT4bjS4gZk AFrZ7OaeR3OqkI/AMP7G0yU651D/tLcdXWaH0V3j4iUgmtoth8BBtl5LNii5TNh8 jDlybTdXxhNdzRIUb+4rmxiKx2FEPT1CzI0a6qFYvCDHG7Ja3J4gVjMxhMI9RN2d GTe/rT1k3m8OE/9txOWWh/wBVEt4sEM7rQktQCiErg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrkeejgddvudcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpeffhffvvefukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeeukedtvedtffevleejtefgheehieegkeeluddvfeefgeehgfeltddtheejleffteen ucevlhhushhtvghrufhiiigvpedvnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 18 Mar 2024 06:53:30 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 1b9956f1 (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 18 Mar 2024 10:48:38 +0000 (UTC) Date: Mon, 18 Mar 2024 11:53:28 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Derrick Stolee Subject: [PATCH 14/15] builtin/gc: forward git-gc(1)'s `--auto` flag when packing refs Message-ID: <474cf66b26f5e459dccd60a2dcd1a09eff86178c.1710706119.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Forward the `--auto` flag to git-pack-refs(1) when it has been invoked with this flag itself. This does not change anything for the "files" backend, which will continue to eagerly pack refs. But it does ensure that the "reftable" backend only compacts refs as required. This change does not impact git-maintenance(1) as it won't ever end up executing the pack-refs task when run with `--auto`. Signed-off-by: Patrick Steinhardt --- builtin/gc.c | 21 ++++++++++++--------- t/t0610-reftable-basics.sh | 19 ++++++++++++++++--- 2 files changed, 28 insertions(+), 12 deletions(-) diff --git a/builtin/gc.c b/builtin/gc.c index e0029c88f9..bf1f2a621a 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -212,6 +212,9 @@ static int maintenance_task_pack_refs(MAYBE_UNUSED struct maintenance_run_opts * cmd.git_cmd = 1; strvec_pushl(&cmd.args, "pack-refs", "--all", "--prune", NULL); + if (opts->auto_flag) + strvec_push(&cmd.args, "--auto"); + return run_command(&cmd); } @@ -572,7 +575,7 @@ static int report_last_gc_error(void) return ret; } -static void gc_before_repack(void) +static void gc_before_repack(struct maintenance_run_opts *opts) { /* * We may be called twice, as both the pre- and @@ -583,7 +586,7 @@ static void gc_before_repack(void) if (done++) return; - if (pack_refs && maintenance_task_pack_refs(NULL)) + if (pack_refs && maintenance_task_pack_refs(opts)) die(FAILED_RUN, "pack-refs"); if (prune_reflogs) { @@ -599,7 +602,6 @@ static void gc_before_repack(void) int cmd_gc(int argc, const char **argv, const char *prefix) { int aggressive = 0; - int auto_gc = 0; int quiet = 0; int force = 0; const char *name; @@ -608,6 +610,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix) int keep_largest_pack = -1; timestamp_t dummy; struct child_process rerere_cmd = CHILD_PROCESS_INIT; + struct maintenance_run_opts opts = {0}; struct option builtin_gc_options[] = { OPT__QUIET(&quiet, N_("suppress progress reporting")), @@ -618,7 +621,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix) OPT_MAGNITUDE(0, "max-cruft-size", &max_cruft_size, N_("with --cruft, limit the size of new cruft packs")), OPT_BOOL(0, "aggressive", &aggressive, N_("be more thorough (increased runtime)")), - OPT_BOOL_F(0, "auto", &auto_gc, N_("enable auto-gc mode"), + OPT_BOOL_F(0, "auto", &opts.auto_flag, N_("enable auto-gc mode"), PARSE_OPT_NOCOMPLETE), OPT_BOOL_F(0, "force", &force, N_("force running gc even if there may be another gc running"), @@ -663,7 +666,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix) if (quiet) strvec_push(&repack, "-q"); - if (auto_gc) { + if (opts.auto_flag) { /* * Auto-gc should be least intrusive as possible. */ @@ -688,7 +691,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix) if (lock_repo_for_gc(force, &pid)) return 0; - gc_before_repack(); /* dies on failure */ + gc_before_repack(&opts); /* dies on failure */ delete_tempfile(&pidfile); /* @@ -713,7 +716,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix) name = lock_repo_for_gc(force, &pid); if (name) { - if (auto_gc) + if (opts.auto_flag) return 0; /* be quiet on --auto */ die(_("gc is already running on machine '%s' pid %"PRIuMAX" (use --force if not)"), name, (uintmax_t)pid); @@ -728,7 +731,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix) atexit(process_log_file_at_exit); } - gc_before_repack(); + gc_before_repack(&opts); if (!repository_format_precious_objects) { struct child_process repack_cmd = CHILD_PROCESS_INIT; @@ -783,7 +786,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix) !quiet && !daemonized ? COMMIT_GRAPH_WRITE_PROGRESS : 0, NULL); - if (auto_gc && too_many_loose_objects()) + if (opts.auto_flag && too_many_loose_objects()) warning(_("There are too many unreachable loose objects; " "run 'git prune' to remove them.")); diff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh index 6de7529575..b37d8bf3b1 100755 --- a/t/t0610-reftable-basics.sh +++ b/t/t0610-reftable-basics.sh @@ -387,7 +387,9 @@ test_expect_success 'pack-refs: compaction raises locking errors' ' test_cmp expect err ' -test_expect_success 'pack-refs: auto compaction' ' +for command in pack-refs gc +do +test_expect_success "$command: auto compaction" ' test_when_finished "rm -rf repo" && git init repo && ( @@ -395,14 +397,24 @@ test_expect_success 'pack-refs: auto compaction' ' test_commit A && + # We need a bit of setup to ensure that git-gc(1) actually + # triggers, and that it does not write anything to the refdb. + git config gc.auto 1 && + git config gc.autoDetach 0 && + git config gc.reflogExpire never && + git config gc.reflogExpireUnreachable never && + test_oid blob17_1 | git hash-object -w --stdin && + # The tables should have been auto-compacted, and thus auto # compaction should not have to do anything. ls -1 .git/reftable >tables-expect && test_line_count = 4 tables-expect && - git pack-refs --auto && + git $command --auto && ls -1 .git/reftable >tables-actual && test_cmp tables-expect tables-actual && + test_oid blob17_2 | git hash-object -w --stdin && + # Lock all tables write some refs. Auto-compaction will be # unable to compact tables and thus fails gracefully, leaving # the stack in a sub-optimal state. @@ -416,10 +428,11 @@ test_expect_success 'pack-refs: auto compaction' ' rm .git/reftable/*.lock && test_line_count = 5 .git/reftable/tables.list && - git pack-refs --auto && + git $command --auto && test_line_count = 1 .git/reftable/tables.list ) ' +done test_expect_success 'pack-refs: prunes stale tables' ' test_when_finished "rm -rf repo" && From patchwork Mon Mar 18 10:53:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Patrick Steinhardt X-Patchwork-Id: 13595226 Received: from fout2-smtp.messagingengine.com (fout2-smtp.messagingengine.com [103.168.172.145]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9DFA43839E for ; Mon, 18 Mar 2024 10:53:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.145 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710759218; cv=none; b=QMveaF9Rvd9tst6BhjXcUomLGAJm1RRBYeUY42fNve6pXfMA4tHtgj/IxTYC2LX7r2F3NF8E/3mELLfemm6p85uIROZmDHqA2SM/FGgxBWbxTgnnfvWYx1mrReU23hTggXxkSp52APUxGTxKKejaBZSr0PKFKV+L6rcQK3HBFXE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1710759218; c=relaxed/simple; bh=+4e2zK3QfQQbR+QQ/OhktXmEYc5Wp842d7P8eSIMLeE=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=DSrHG9dDRSHBzmQFaaetscWL6k+7Irx6OmTMgTfCA4wwLskiQFhNLL+OKMxefhc4zoqJyPQzr8ZdzkizEqNqH4CwwwBZ6eEYclaTOwfWr2MMwfJuZ5hIOgY4ERbnk2XtJiMtIhmioS4R7JeDiQVNY/DMxiWvRcGA2TFuqQZplNE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im; spf=pass smtp.mailfrom=pks.im; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b=Gz58Ru3B; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=IZktG0yE; arc=none smtp.client-ip=103.168.172.145 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=pks.im Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=pks.im Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=pks.im header.i=@pks.im header.b="Gz58Ru3B"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="IZktG0yE" Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailfout.nyi.internal (Postfix) with ESMTP id F01C2138011E; Mon, 18 Mar 2024 06:53:35 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Mon, 18 Mar 2024 06:53:35 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pks.im; h=cc:cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1710759215; x=1710845615; bh=5nyKR3q0WE 96Qv/b7bNr3VECpAzaHiW0RoNpbFDZvVk=; b=Gz58Ru3Bb+rHV36w71nji2+nvR TRGeQzlhGrN0iWyVfbV3LwrwPKW2QKI6E2b1LJeNIQj0VNOMSOQzqu3DI6v/kzE6 B15uDDSqOXwG5nhcRftquzfRAKq626D2XHMBlCUh9n3ibko3hCe9BYAk8chLzmTO tMGl91deqIPJF320AdNAmEkCrmINokq86CDNyOAONhqDiGcVX1CCcBw1hFl8h80C cUizicO3z112hJ/GJv7jRJpkoMA+hRdKs/2VKx7qiuBqZGYmSlqKySgUmOLugBYO owxSuyQleR0L+8EQP8KIH0RdaTZPkBsifMNc4QhozLiVt0NaBg3mlkYNIMWQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; t=1710759215; x=1710845615; bh=5nyKR3q0WE96Qv/b7bNr3VECpAza HiW0RoNpbFDZvVk=; b=IZktG0yEJKlhE1QqGuWxQ6yqS6zxIGf6Ed2gx7F2uetQ MQPMs6iRr9/cvSEeisiHxCGWGoP2Pg33XnggNwPC993iI0J1PwRWa/ItKH8gsA/M afM4X71t2PM4PVGVncq9wKQWQriCIM8TkRyYCU3UOMLczYZvPy+zu+glT0Q7RpC0 BLvJwD44dfc1IlLeE6QLF+4OuaojNPr3+wV5EaBCpCwvkqRHuwn9PDzsaColIS7F VfmOrEtk4KSr5QZFlGVBax4ZXWovb7esuEYdAhm+VI2yzDw0vO8xCd6YF6JOXY8Z 5zxjEq/Uphn08Shc98y8yTeNkmYZD8TEHXFwdExbLw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrkeejgddvudcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpeffhffvvefukfhfgggtuggjsehgtderredttddvnecuhfhrohhmpefrrghtrhhi tghkucfuthgvihhnhhgrrhguthcuoehpshesphhkshdrihhmqeenucggtffrrghtthgvrh hnpeeukedtvedtffevleejtefgheehieegkeeluddvfeefgeehgfeltddtheejleffteen ucevlhhushhtvghrufhiiigvpedvnecurfgrrhgrmhepmhgrihhlfhhrohhmpehpshesph hkshdrihhm X-ME-Proxy: Feedback-ID: i197146af:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 18 Mar 2024 06:53:35 -0400 (EDT) Received: by vm-mail (OpenSMTPD) with ESMTPSA id 5b01057d (TLSv1.3:TLS_AES_256_GCM_SHA384:256:NO); Mon, 18 Mar 2024 10:48:43 +0000 (UTC) Date: Mon, 18 Mar 2024 11:53:32 +0100 From: Patrick Steinhardt To: git@vger.kernel.org Cc: Derrick Stolee Subject: [PATCH 15/15] builtin/gc: pack refs when using `git maintenance run --auto` Message-ID: <71f4800d36bcc77d8f36c5fc7b48eccbb90e6a93.1710706119.git.ps@pks.im> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: When running `git maintenance run --auto`, then the various subtasks will only run as needed. Thus, we for example end up only packing loose objects if we hit a certain threshold. Interestingly enough, the "pack-refs" task is actually _never_ executed when the auto-flag is set because it does not have a condition at all. As 41abfe15d9 (maintenance: add pack-refs task, 2021-02-09) mentions: The 'auto_condition' function pointer is left NULL for now. We could extend this in the future to have a condition check if pack-refs should be run during 'git maintenance run --auto'. It is not quite clear from that quote whether it is actually intended that the task doesn't run at all in this mode. Also, no test was added to verify this behaviour. Ultimately though, it feels quite surprising that `git maintenance run --auto --task=pack-refs` would quietly never do anything at all. In any case, now that we do have the logic in place to let ref backends decide whether or not to repack refs, it does make sense to wire it up accordingly. With the "reftable" backend we will thus now perform auto-compaction, which optimizes the refdb as needed. But for the "files" backend we now unconditionally pack refs as it does not yet know to handle the "auto" flag. Arguably, this can be seen as a bug fix given that previously the task never did anything at all. Eventually though we should amend the "files" backend to use some heuristics for auto compaction, as well. Signed-off-by: Patrick Steinhardt --- builtin/gc.c | 12 +++++++++++- t/t0601-reffiles-pack-refs.sh | 10 ++++++++++ t/t0610-reftable-basics.sh | 2 +- 3 files changed, 22 insertions(+), 2 deletions(-) diff --git a/builtin/gc.c b/builtin/gc.c index bf1f2a621a..3c874b248b 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -206,6 +206,16 @@ struct maintenance_run_opts { enum schedule_priority schedule; }; +static int pack_refs_condition(void) +{ + /* + * The auto-repacking logic for refs is handled by the ref backends and + * exposed via `git pack-refs --auto`. We thus always return truish + * here and let the backend decide for us. + */ + return 1; +} + static int maintenance_task_pack_refs(MAYBE_UNUSED struct maintenance_run_opts *opts) { struct child_process cmd = CHILD_PROCESS_INIT; @@ -1298,7 +1308,7 @@ static struct maintenance_task tasks[] = { [TASK_PACK_REFS] = { "pack-refs", maintenance_task_pack_refs, - NULL, + pack_refs_condition, }, }; diff --git a/t/t0601-reffiles-pack-refs.sh b/t/t0601-reffiles-pack-refs.sh index 219a495451..7d4ab0b91a 100755 --- a/t/t0601-reffiles-pack-refs.sh +++ b/t/t0601-reffiles-pack-refs.sh @@ -370,4 +370,14 @@ test_expect_success 'pack-refs does not drop broken refs during deletion' ' test_cmp expect actual ' +test_expect_success 'maintenance --auto unconditionally packs loose refs' ' + git update-ref refs/heads/something HEAD && + test_path_is_file .git/refs/heads/something && + git rev-parse refs/heads/something >expect && + git maintenance run --task=pack-refs --auto && + test_path_is_missing .git/refs/heads/something && + git rev-parse refs/heads/something >actual && + test_cmp expect actual +' + test_done diff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh index b37d8bf3b1..931d888bbb 100755 --- a/t/t0610-reftable-basics.sh +++ b/t/t0610-reftable-basics.sh @@ -387,7 +387,7 @@ test_expect_success 'pack-refs: compaction raises locking errors' ' test_cmp expect err ' -for command in pack-refs gc +for command in pack-refs gc "maintenance run --task=pack-refs" do test_expect_success "$command: auto compaction" ' test_when_finished "rm -rf repo" &&