From patchwork Thu Apr 4 18:29:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Justin Tobler X-Patchwork-Id: 13618123 Received: from mail-wr1-f41.google.com (mail-wr1-f41.google.com [209.85.221.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AC6B9130E30 for ; Thu, 4 Apr 2024 18:29:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.41 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712255376; cv=none; b=U9TgiRW1fYvrju9GVQGNBBEaMnKt8dcHsyWCzErih1Wt4Ljkcu3C1Jd+UEhJ+Yp7MxKIk0SxEtDLgpmRC5dD/QxezuOaZaoXJt1O3fLtOWMuZJiZuQ6wRVBUjFVly1yd+lakm766SRuEqOmJczHa8L4Z5zY2im2sm0brcWvvI0E= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712255376; c=relaxed/simple; bh=XlF8UTd4PZpcoolOPS1+NnwXn0eu3NN2nbOCySTq6yU=; h=Message-Id:In-Reply-To:References:From:Date:Subject:Content-Type: MIME-Version:To:Cc; b=fJPLT0e4Kxcf4gqt8PJQB++TnzKXFXOvd8xV+jiQOtJdaN+yIy/kJNfu9RopXJY/uVlhf+l8v4KG0ZG4qd++2w3o8IJH4vaSXcdVZvSUBZQXUWaUEjtrgOVumQ/e3xsiX+nSuTbNWl96WDPisDoAEY/WShUoxcSqoYLYpVLdePU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=VdI2cLOp; arc=none smtp.client-ip=209.85.221.41 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="VdI2cLOp" Received: by mail-wr1-f41.google.com with SMTP id ffacd0b85a97d-34388753650so618222f8f.3 for ; Thu, 04 Apr 2024 11:29:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712255372; x=1712860172; darn=vger.kernel.org; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=wELA0xpHxohhRdXRAyqLhSgDFfxFxa1EhpuYj2LhL54=; b=VdI2cLOpb7yiBbJLL/6gI48go+ETSBcSWePs2aVycn5TP7dnXfszS124tvxwfbeGJl 73xw2f2ilqoIAQYeUCmVEg4sOfyyXlVT7BEchRI4beX9JrMOuoRzPNDBG5dxxwU+3e3v oqjrdCEhnlAAr2bFSfqln8C41/RwtpX+GJ7PTDMvjcapFsBYXUlQdD1dev8dNqbB+RnN MipHiz1b+d/FO615zGsHOznbDGBruOPptChE7a3QxEknkMAXLXmw4S9/J5x500kB7iXd jORh5jNI52ew3AE/Tlz5UP+kMeGGZpRdIvKErTrF/qjdB5VFRiKTqnbC87mmNjQaYPds 9vxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712255372; x=1712860172; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wELA0xpHxohhRdXRAyqLhSgDFfxFxa1EhpuYj2LhL54=; b=w6+s5WgO4YwhBKjLg/k5wcZPLdlPZNtvZu833NgMXkz2cmd/bPnKTHyV9tXTaTTwLw QpKsTr/Wiy1Zb9ei5wTJXz3EyHMv7NZR8hBT11feULJR6fhwjgaHf3LA3xhGlOSo9hFu PAOENjMd2lgiGAz3DQlsNLB+A1RyK5qyPpuPsICPZ6zYejCfmFmN98blyytmICBociyB 1+/FGWZZUYypbxdYmmiwVmDlQ9l0H2S7Gp0NP3oMNJgt5rF/mcsdVIl8cBMUpGuOZ2f9 8LgV9irU2PINvXVSiSZUz8Zrnjee/WKgDSl1GTNxNi3QJACxDgfkyAm4agx7eijf1oFW amsA== X-Gm-Message-State: AOJu0Yxfj/bpf3HmCHab0ZZcEMOe86GFpWe0nao1ZeVjrZyer9C3DuU6 cFisiDojxEFpUbpgF8tAOlKMCUpKHlJ09wZtt1qoun0gMen5bLlybdONvgcS X-Google-Smtp-Source: AGHT+IHiwKWquuDB/o9qxUqGgBTUU7IHy7bvoJnMrW/HO4fnoN77dJwp9aVLGIkQASLFbHFFeYRNNw== X-Received: by 2002:adf:9b9a:0:b0:33e:34aa:d78a with SMTP id d26-20020adf9b9a000000b0033e34aad78amr297498wrc.8.1712255372027; Thu, 04 Apr 2024 11:29:32 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id y4-20020a056000108400b00343bccbe0c7sm24920wrw.26.2024.04.04.11.29.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 04 Apr 2024 11:29:31 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Thu, 04 Apr 2024 18:29:27 +0000 Subject: [PATCH v5 1/3] reftable/stack: allow disabling of auto-compaction Fcc: Sent Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 To: git@vger.kernel.org Cc: Patrick Steinhardt , Karthik Nayak , Han-Wen Nienhuys , Justin Tobler , Justin Tobler From: Justin Tobler From: Justin Tobler Move the `disable_auto_compact` option into `reftable_write_options` to allow a stack to be configured with auto-compaction disabled. In a subsequent commit, this is used to disable auto-compaction when a specific environment variable is set. Signed-off-by: Justin Tobler --- reftable/reftable-writer.h | 3 +++ reftable/stack.c | 2 +- reftable/stack.h | 1 - reftable/stack_test.c | 11 ++++++----- 4 files changed, 10 insertions(+), 7 deletions(-) diff --git a/reftable/reftable-writer.h b/reftable/reftable-writer.h index 7c7cae5f99b..155bf0bbe2a 100644 --- a/reftable/reftable-writer.h +++ b/reftable/reftable-writer.h @@ -46,6 +46,9 @@ struct reftable_write_options { * is a single line, and add '\n' if missing. */ unsigned exact_log_message : 1; + + /* boolean: Prevent auto-compaction of tables. */ + unsigned disable_auto_compact : 1; }; /* reftable_block_stats holds statistics for a single block type */ diff --git a/reftable/stack.c b/reftable/stack.c index dde50b61d69..1a7cdad12c9 100644 --- a/reftable/stack.c +++ b/reftable/stack.c @@ -680,7 +680,7 @@ int reftable_addition_commit(struct reftable_addition *add) if (err) goto done; - if (!add->stack->disable_auto_compact) { + if (!add->stack->config.disable_auto_compact) { /* * Auto-compact the stack to keep the number of tables in * control. It is possible that a concurrent writer is already diff --git a/reftable/stack.h b/reftable/stack.h index d919455669e..c862053025f 100644 --- a/reftable/stack.h +++ b/reftable/stack.h @@ -19,7 +19,6 @@ struct reftable_stack { int list_fd; char *reftable_dir; - int disable_auto_compact; struct reftable_write_options config; diff --git a/reftable/stack_test.c b/reftable/stack_test.c index 351e35bd86d..4fec823f14f 100644 --- a/reftable/stack_test.c +++ b/reftable/stack_test.c @@ -325,7 +325,7 @@ static void test_reftable_stack_transaction_api_performs_auto_compaction(void) * we can ensure that we indeed honor this setting and have * better control over when exactly auto compaction runs. */ - st->disable_auto_compact = i != n; + st->config.disable_auto_compact = i != n; err = reftable_stack_new_addition(&add, st); EXPECT_ERR(err); @@ -497,6 +497,7 @@ static void test_reftable_stack_add(void) struct reftable_write_options cfg = { .exact_log_message = 1, .default_permissions = 0660, + .disable_auto_compact = 1, }; struct reftable_stack *st = NULL; char *dir = get_tmp_dir(__LINE__); @@ -508,7 +509,6 @@ static void test_reftable_stack_add(void) err = reftable_new_stack(&st, dir, cfg); EXPECT_ERR(err); - st->disable_auto_compact = 1; for (i = 0; i < N; i++) { char buf[256]; @@ -935,7 +935,9 @@ static void test_empty_add(void) static void test_reftable_stack_auto_compaction(void) { - struct reftable_write_options cfg = { 0 }; + struct reftable_write_options cfg = { + .disable_auto_compact = 1, + }; struct reftable_stack *st = NULL; char *dir = get_tmp_dir(__LINE__); @@ -945,7 +947,6 @@ static void test_reftable_stack_auto_compaction(void) err = reftable_new_stack(&st, dir, cfg); EXPECT_ERR(err); - st->disable_auto_compact = 1; /* call manually below for coverage. */ for (i = 0; i < N; i++) { char name[100]; struct reftable_ref_record ref = { @@ -994,7 +995,7 @@ static void test_reftable_stack_add_performs_auto_compaction(void) * we can ensure that we indeed honor this setting and have * better control over when exactly auto compaction runs. */ - st->disable_auto_compact = i != n; + st->config.disable_auto_compact = i != n; strbuf_reset(&refname); strbuf_addf(&refname, "branch-%04d", i); From patchwork Thu Apr 4 18:29:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Justin Tobler X-Patchwork-Id: 13618124 Received: from mail-wm1-f42.google.com (mail-wm1-f42.google.com [209.85.128.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CD9F3130A58 for ; Thu, 4 Apr 2024 18:29:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712255377; cv=none; b=LOlLZYX+QdLVMqbGxFyDfquKj2AaihpZDchrhHK3UewN1Fkh18DbVvwhNUyNLJgDpX2GxSyuvrUx+NhuVzlVdtGhIgeJxNLGK0zMW/OgzkNUA0ZOn73txqUgMCyA4Wh9MVoc6wIPxybbm9hJ+a0aO2PEVCLJtVtr/NH1ZcUsIek= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712255377; c=relaxed/simple; bh=SGRoB3Ls4NSX8DFUeyjq2hUNeDsu0sOmDj8QzzpWxtQ=; h=Message-Id:In-Reply-To:References:From:Date:Subject:Content-Type: MIME-Version:To:Cc; b=tiwG7KqG6sKAmQSI1wDcvSrkZGcel8tZxf3EsVto2ahlHkRn9M8hkuT1BFoWzb8oYsOeE3fogyCWueHNs9jWklVAxaTLCW8m3j+LTVbUN/63L3O5QwLrQ11VFtDc66pEk09mDKedA5sNW5RmI9pxp8I4sfTKZt/mE3GhUEX/Cdo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=Wva7UqRZ; arc=none smtp.client-ip=209.85.128.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Wva7UqRZ" Received: by mail-wm1-f42.google.com with SMTP id 5b1f17b1804b1-415584360c0so9815155e9.1 for ; Thu, 04 Apr 2024 11:29:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712255374; x=1712860174; darn=vger.kernel.org; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=KXd5dyR56DSwC7VqXb+akUfx07s4BGr6g8LRyw/vigM=; b=Wva7UqRZCRyPjvpUm84vh/V7LsMjkMjccw8oKa+skEK77UiwmljgN7hRMhJMXGtQuy 8F6XcR1IOh1OcPUlTn/PN3HGZxtIXSh3UOeRNScdmHa9Ly3gmZsstw/6pP3jUVYxvWTP tsmcmUMcgpQeM+4Cy/jjB0oMkis5GhAwhS21B+ZrOG4JVkj3YnkeERuEh6QDfzi4xi9v xEumg/W7a6ljuurGP8Y4culw3noolMnULS15RLBfRdtcle+Ye968VAfBwasd1khVwsoR h226SnMmEEuWvppIN9frnv3kv9G09/oEoHnz9ka/TH5am0s+9LihDuYQ9WNEQgnXPCEv k2WA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712255374; x=1712860174; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KXd5dyR56DSwC7VqXb+akUfx07s4BGr6g8LRyw/vigM=; b=l2+/mDTCs/mG7kALmrItReAmZlHfk0RPnR4PdSnefQ7gZPHYm0qp4NMqsMpMs/6Eef vkseOdKDzdkT610k9bd78D1AZki9J6to0rzSBZaVOQ06RHjrnkS0ORqC1gmjus+aEizE y6gujUGZBu4/i306y7GTmztArVzv8XPwm97amoDwOV+9D8qMsx8NAW0MWRd9VMiPbjVh TXm4hCUq3NrxFWd/wFhUZBU3u9jUen9zw5XkPEvekkC/PHg9TrMqJF2wj7yl0/Slwin8 HNI5v0FEDXFVXwwzsDBPOiIleq6CphfJ5MSoKQhV5mjndym/D/m2sbqQaTjyVnoEhAK9 zaeA== X-Gm-Message-State: AOJu0YyEFdP1iJXD7weJLXpOi7gJc2X+rkUsCxX/leZADC1zfsB2nnJm 4zlgsW1vopPYfdH4xh/Ykl6hqp9blIxQ3cEC/OTUYrBVgEaGM0rHdkCvcQJg X-Google-Smtp-Source: AGHT+IGecg1owiHQSqj6Tg1OGMRP1XUF/+FanQjM4gTCjpntS5woMy+F8t93TJlhOVNdwBFc/g2iLw== X-Received: by 2002:adf:fac6:0:b0:343:68ea:c0a8 with SMTP id a6-20020adffac6000000b0034368eac0a8mr2493413wrs.34.1712255373756; Thu, 04 Apr 2024 11:29:33 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id dr20-20020a5d5f94000000b0033ea499c645sm27063wrb.4.2024.04.04.11.29.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 04 Apr 2024 11:29:32 -0700 (PDT) Message-Id: <7c4fe0e9ec597203ee37d2c2503be319e87ff5ee.1712255369.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 04 Apr 2024 18:29:28 +0000 Subject: [PATCH v5 2/3] reftable/stack: add env to disable autocompaction Fcc: Sent Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 To: git@vger.kernel.org Cc: Patrick Steinhardt , Karthik Nayak , Han-Wen Nienhuys , Justin Tobler , Justin Tobler From: Justin Tobler From: Justin Tobler In future tests it will be neccesary to create repositories with a set number of tables. To make this easier, introduce the `GIT_TEST_REFTABLE_AUTOCOMPACTION` environment variable that, when set to false, disables autocompaction of reftables. Signed-off-by: Justin Tobler --- refs/reftable-backend.c | 4 ++++ t/t0610-reftable-basics.sh | 21 +++++++++++++++++++++ 2 files changed, 25 insertions(+) diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c index 0bed6d2ab48..6b6191f89dd 100644 --- a/refs/reftable-backend.c +++ b/refs/reftable-backend.c @@ -18,6 +18,7 @@ #include "../reftable/reftable-merged.h" #include "../setup.h" #include "../strmap.h" +#include "parse.h" #include "refs-internal.h" /* @@ -248,6 +249,9 @@ static struct ref_store *reftable_be_init(struct repository *repo, refs->write_options.hash_id = repo->hash_algo->format_id; refs->write_options.default_permissions = calc_shared_perm(0666 & ~mask); + if (!git_env_bool("GIT_TEST_REFTABLE_AUTOCOMPACTION", 1)) + refs->write_options.disable_auto_compact = 1; + /* * Set up the main reftable stack that is hosted in GIT_COMMON_DIR. * This stack contains both the shared and the main worktree refs. diff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh index 931d888bbbc..c9e10b34684 100755 --- a/t/t0610-reftable-basics.sh +++ b/t/t0610-reftable-basics.sh @@ -299,6 +299,27 @@ test_expect_success 'ref transaction: writes cause auto-compaction' ' test_line_count = 1 repo/.git/reftable/tables.list ' +test_expect_success 'ref transaction: env var disables compaction' ' + test_when_finished "rm -rf repo" && + + git init repo && + test_commit -C repo A && + + start=$(wc -l X-Patchwork-Id: 13618125 Received: from mail-lj1-f181.google.com (mail-lj1-f181.google.com [209.85.208.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 96D82130AC4 for ; Thu, 4 Apr 2024 18:29:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712255379; cv=none; b=WEb3koST3J62/05af9VJYP2lw2nJehLxTLKfVoBnd85rt83AmTW4fTjmoX1nPwj6p+m+okPEqeBhNuXOufBwESta0vZUGz24h/EDJztaV3l/zvFn5KTp7xZEy5hc6e7PCdl4uVwvp736vF4XxMdUFQml5KLkekXpwxTYZ/9dsx4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1712255379; c=relaxed/simple; bh=t4bu3VO3CdFdG3XgoR7nC0RSluZbspY/09hBfLI1Wkg=; h=Message-Id:In-Reply-To:References:From:Date:Subject:MIME-Version: Content-Type:To:Cc; b=c/0Kx5mnMqAiylOXJeHlQOWkih8q8PDglEhrGMpDys6qdT6b5jVSjAQ6OiYoapwT5hG4x5eb2F6m3RxNqilNFdf8n/7QvVhCaH/UyUyU4BnPg13b3XgR2LfnQK8I+zS8rrfrB/Kjz+W/7lQVb2FNjEypy5UnPZS5aGon9L3Ev1M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=lzSHw+ld; arc=none smtp.client-ip=209.85.208.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="lzSHw+ld" Received: by mail-lj1-f181.google.com with SMTP id 38308e7fff4ca-2d717603aa5so17437201fa.0 for ; Thu, 04 Apr 2024 11:29:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712255375; x=1712860175; darn=vger.kernel.org; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=5hbh+l1VXKX0a9XBC9dEfE3rTRjybVL9js5cSqQwDC4=; b=lzSHw+ldjrxLnUBoaKWEn6UWgTJGww22aRMjlZtsjCYBWqZK+jHM4YSA3kcaISfhFI 2vpooG4ydwoo178ggCR/f1kteydg+b364gtBzdwKhjc6b54YTqZ+qFbbGJTj4KR4A7FV TjVRZ7WeUp3n8mwpdj6mqfeg5joAOj5ZujXjKJD3OUBQkV0utGu4iZzUDxcHYhkaedow LiR2DzOLqJq15BRMiVVt14Kdjn9YP8wL4ooAhXhJEKl+TxqKNLz7FnMfChXXr+dZ4VY2 9hb1MPoRVvZJlRiBwWQvyqVaxrXeKpLjEarOjwZVsRyQFDK3l/zpqFNHNr9JpSk/2Hnw W8cg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712255375; x=1712860175; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5hbh+l1VXKX0a9XBC9dEfE3rTRjybVL9js5cSqQwDC4=; b=WCuDIHWPl+MM/vlByzkyP4t9LNPIrwO9YCLG9YDh3xi8vXGa9OOhjTr4tQlx8a06JC zYYPKWRUHMfZo2Qd64iVhnp3jJtbkX0mAG5syhnfXfJ+Pa2Rb6lavvaXLslK1DamZSvI o/9BwMnRGmaQsPslDGTaZw57lsa9Vpd+WIHa9PVf+Z9HO/siH7IJKvgE3MjgG7RnkVAz OfjS/bPqi22H0OSxsT7F58XMa63+nVhwc+34i62oqa5/Cy/wIBb+khQRVe1myt2j1qdY 9k6RGMYfgZlFOVYrgFEkgXRJzOHi8C/BUnsJHkMAsG18L4U5pff3gXGDFFPVOo3Y6qD9 +uWg== X-Gm-Message-State: AOJu0YyIgb/SXxVsu+RUKoBcG5vadCr8OLXdTImpmeBprkENTSAp1mBj VdVdc1aTelusJDEujkfcYoYLGUdSO2TDBYLouM02Xj2CM6O5186AL+x6S+bn X-Google-Smtp-Source: AGHT+IH3uNPXb7F/JRWk052gv4erL+62/LMzpP2SyTbI+/JEOJdc2JMVOmazqsDLCyCKZCyrIO+Msg== X-Received: by 2002:a2e:8095:0:b0:2d8:5b5f:e6d4 with SMTP id i21-20020a2e8095000000b002d85b5fe6d4mr1629165ljg.30.1712255374845; Thu, 04 Apr 2024 11:29:34 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id t10-20020a05600c198a00b004156afd6843sm3537106wmq.18.2024.04.04.11.29.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 04 Apr 2024 11:29:34 -0700 (PDT) Message-Id: <8f124acf0f8358f98cf8872cc82dfb4bc1ef9398.1712255369.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 04 Apr 2024 18:29:29 +0000 Subject: [PATCH v5 3/3] reftable/stack: use geometric table compaction Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: Patrick Steinhardt , Karthik Nayak , Han-Wen Nienhuys , Justin Tobler , Justin Tobler From: Justin Tobler From: Justin Tobler To reduce the number of on-disk reftables, compaction is performed. Contiguous tables with the same binary log value of size are grouped into segments. The segment that has both the lowest binary log value and contains more than one table is set as the starting point when identifying the compaction segment. Since segments containing a single table are not initially considered for compaction, if the table appended to the list does not match the previous table log value, no compaction occurs for the new table. It is therefore possible for unbounded growth of the table list. This can be demonstrated by repeating the following sequence: git branch -f foo git branch -d foo Each operation results in a new table being written with no compaction occurring until a separate operation produces a table matching the previous table log value. Instead, to avoid unbounded growth of the table list, the compaction strategy is updated to ensure tables follow a geometric sequence after each operation by individually evaluating each table in reverse index order. This strategy results in a much simpler and more robust algorithm compared to the previous one while also maintaining a minimal ordered set of tables on-disk. When creating 10 thousand references, the new strategy has no performance impact: Benchmark 1: update-ref: create refs sequentially (revision = HEAD~) Time (mean ± σ): 26.516 s ± 0.047 s [User: 17.864 s, System: 8.491 s] Range (min … max): 26.447 s … 26.569 s 10 runs Benchmark 2: update-ref: create refs sequentially (revision = HEAD) Time (mean ± σ): 26.417 s ± 0.028 s [User: 17.738 s, System: 8.500 s] Range (min … max): 26.366 s … 26.444 s 10 runs Summary update-ref: create refs sequentially (revision = HEAD) ran 1.00 ± 0.00 times faster than update-ref: create refs sequentially (revision = HEAD~) Some tests in `t0610-reftable-basics.sh` assert the on-disk state of tables and are therefore updated to specify the correct new table count. Since compaction is more aggressive in ensuring tables maintain a geometric sequence, the expected table count is reduced in these tests. In `reftable/stack_test.c` tests related to `sizes_to_segments()` are removed because the function is no longer needed. Also, the `test_suggest_compaction_segment()` test is updated to better showcase and reflect the new geometric compaction behavior. Signed-off-by: Justin Tobler --- reftable/stack.c | 123 +++++++++++++++++++------------------ reftable/stack.h | 3 - reftable/stack_test.c | 66 ++++---------------- t/t0610-reftable-basics.sh | 50 ++++++++++----- 4 files changed, 111 insertions(+), 131 deletions(-) diff --git a/reftable/stack.c b/reftable/stack.c index 1a7cdad12c9..80266bcbab1 100644 --- a/reftable/stack.c +++ b/reftable/stack.c @@ -1216,75 +1216,76 @@ static int segment_size(struct segment *s) return s->end - s->start; } -int fastlog2(uint64_t sz) -{ - int l = 0; - if (sz == 0) - return 0; - for (; sz; sz /= 2) { - l++; - } - return l - 1; -} - -struct segment *sizes_to_segments(size_t *seglen, uint64_t *sizes, size_t n) -{ - struct segment *segs = reftable_calloc(n, sizeof(*segs)); - struct segment cur = { 0 }; - size_t next = 0, i; - - if (n == 0) { - *seglen = 0; - return segs; - } - for (i = 0; i < n; i++) { - int log = fastlog2(sizes[i]); - if (cur.log != log && cur.bytes > 0) { - struct segment fresh = { - .start = i, - }; - - segs[next++] = cur; - cur = fresh; - } - - cur.log = log; - cur.end = i + 1; - cur.bytes += sizes[i]; - } - segs[next++] = cur; - *seglen = next; - return segs; -} - struct segment suggest_compaction_segment(uint64_t *sizes, size_t n) { - struct segment min_seg = { - .log = 64, - }; - struct segment *segs; - size_t seglen = 0, i; - - segs = sizes_to_segments(&seglen, sizes, n); - for (i = 0; i < seglen; i++) { - if (segment_size(&segs[i]) == 1) - continue; + struct segment seg = { 0 }; + uint64_t bytes; + size_t i; - if (segs[i].log < min_seg.log) - min_seg = segs[i]; - } + /* + * If there are no tables or only a single one then we don't have to + * compact anything. The sequence is geometric by definition already. + */ + if (n <= 1) + return seg; - while (min_seg.start > 0) { - size_t prev = min_seg.start - 1; - if (fastlog2(min_seg.bytes) < fastlog2(sizes[prev])) + /* + * Find the ending table of the compaction segment needed to restore the + * geometric sequence. Note that the segment end is exclusive. + * + * To do so, we iterate backwards starting from the most recent table + * until a valid segment end is found. If the preceding table is smaller + * than the current table multiplied by the geometric factor (2), the + * compaction segment end has been identified. + * + * Tables after the ending point are not added to the byte count because + * they are already valid members of the geometric sequence. Due to the + * properties of a geometric sequence, it is not possible for the sum of + * these tables to exceed the value of the ending point table. + * + * Example table size sequence requiring no compaction: + * 64, 32, 16, 8, 4, 2, 1 + * + * Example table size sequence where compaction segment end is set to + * the last table. Since the segment end is exclusive, the last table is + * excluded during subsequent compaction and the table with size 3 is + * the final table included: + * 64, 32, 16, 8, 4, 3, 1 + */ + for (i = n - 1; i > 0; i--) { + if (sizes[i - 1] < sizes[i] * 2) { + seg.end = i + 1; + bytes = sizes[i]; break; + } + } - min_seg.start = prev; - min_seg.bytes += sizes[prev]; + /* + * Find the starting table of the compaction segment by iterating + * through the remaining tables and keeping track of the accumulated + * size of all tables seen from the segment end table. The previous + * table is compared to the accumulated size because the tables from the + * segment end are merged backwards recursively. + * + * Note that we keep iterating even after we have found the first + * starting point. This is because there may be tables in the stack + * preceding that first starting point which violate the geometric + * sequence. + * + * Example compaction segment start set to table with size 32: + * 128, 32, 16, 8, 4, 3, 1 + */ + for (; i > 0; i--) { + uint64_t curr = bytes; + bytes += sizes[i - 1]; + + if (sizes[i - 1] < curr * 2) { + seg.start = i - 1; + seg.bytes = bytes; + } } - reftable_free(segs); - return min_seg; + return seg; } static uint64_t *stack_table_sizes_for_compaction(struct reftable_stack *st) diff --git a/reftable/stack.h b/reftable/stack.h index c862053025f..d43efa47607 100644 --- a/reftable/stack.h +++ b/reftable/stack.h @@ -32,12 +32,9 @@ int read_lines(const char *filename, char ***lines); struct segment { size_t start, end; - int log; uint64_t bytes; }; -int fastlog2(uint64_t sz); -struct segment *sizes_to_segments(size_t *seglen, uint64_t *sizes, size_t n); struct segment suggest_compaction_segment(uint64_t *sizes, size_t n); #endif diff --git a/reftable/stack_test.c b/reftable/stack_test.c index 4fec823f14f..1df3ffce526 100644 --- a/reftable/stack_test.c +++ b/reftable/stack_test.c @@ -770,59 +770,13 @@ static void test_reftable_stack_hash_id(void) clear_dir(dir); } -static void test_log2(void) -{ - EXPECT(1 == fastlog2(3)); - EXPECT(2 == fastlog2(4)); - EXPECT(2 == fastlog2(5)); -} - -static void test_sizes_to_segments(void) -{ - uint64_t sizes[] = { 2, 3, 4, 5, 7, 9 }; - /* .................0 1 2 3 4 5 */ - - size_t seglen = 0; - struct segment *segs = - sizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes)); - EXPECT(segs[2].log == 3); - EXPECT(segs[2].start == 5); - EXPECT(segs[2].end == 6); - - EXPECT(segs[1].log == 2); - EXPECT(segs[1].start == 2); - EXPECT(segs[1].end == 5); - reftable_free(segs); -} - -static void test_sizes_to_segments_empty(void) -{ - size_t seglen = 0; - struct segment *segs = sizes_to_segments(&seglen, NULL, 0); - EXPECT(seglen == 0); - reftable_free(segs); -} - -static void test_sizes_to_segments_all_equal(void) -{ - uint64_t sizes[] = { 5, 5 }; - size_t seglen = 0; - struct segment *segs = - sizes_to_segments(&seglen, sizes, ARRAY_SIZE(sizes)); - EXPECT(seglen == 1); - EXPECT(segs[0].start == 0); - EXPECT(segs[0].end == 2); - reftable_free(segs); -} - static void test_suggest_compaction_segment(void) { - uint64_t sizes[] = { 128, 64, 17, 16, 9, 9, 9, 16, 16 }; - /* .................0 1 2 3 4 5 6 */ + uint64_t sizes[] = { 512, 64, 17, 16, 9, 9, 9, 16, 2, 16 }; struct segment min = suggest_compaction_segment(sizes, ARRAY_SIZE(sizes)); - EXPECT(min.start == 2); - EXPECT(min.end == 7); + EXPECT(min.start == 1); + EXPECT(min.end == 10); } static void test_suggest_compaction_segment_nothing(void) @@ -933,6 +887,16 @@ static void test_empty_add(void) reftable_stack_destroy(st2); } +static int fastlog2(uint64_t sz) +{ + int l = 0; + if (sz == 0) + return 0; + for (; sz; sz /= 2) + l++; + return l - 1; +} + static void test_reftable_stack_auto_compaction(void) { struct reftable_write_options cfg = { @@ -1122,7 +1086,6 @@ static void test_reftable_stack_compaction_concurrent_clean(void) int stack_test_main(int argc, const char *argv[]) { RUN_TEST(test_empty_add); - RUN_TEST(test_log2); RUN_TEST(test_names_equal); RUN_TEST(test_parse_names); RUN_TEST(test_read_file); @@ -1143,9 +1106,6 @@ int stack_test_main(int argc, const char *argv[]) RUN_TEST(test_reftable_stack_update_index_check); RUN_TEST(test_reftable_stack_uptodate); RUN_TEST(test_reftable_stack_validate_refname); - RUN_TEST(test_sizes_to_segments); - RUN_TEST(test_sizes_to_segments_all_equal); - RUN_TEST(test_sizes_to_segments_empty); RUN_TEST(test_suggest_compaction_segment); RUN_TEST(test_suggest_compaction_segment_nothing); return 0; diff --git a/t/t0610-reftable-basics.sh b/t/t0610-reftable-basics.sh index c9e10b34684..8eec093788d 100755 --- a/t/t0610-reftable-basics.sh +++ b/t/t0610-reftable-basics.sh @@ -293,7 +293,7 @@ test_expect_success 'ref transaction: writes cause auto-compaction' ' test_line_count = 1 repo/.git/reftable/tables.list && test_commit -C repo --no-tag A && - test_line_count = 2 repo/.git/reftable/tables.list && + test_line_count = 1 repo/.git/reftable/tables.list && test_commit -C repo --no-tag B && test_line_count = 1 repo/.git/reftable/tables.list @@ -320,6 +320,19 @@ test_expect_success 'ref transaction: env var disables compaction' ' test_line_count -lt $expected repo/.git/reftable/tables.list ' +test_expect_success 'ref transaction: alternating table sizes are compacted' ' + test_when_finished "rm -rf repo" && + + git init repo && + test_commit -C repo A && + for i in $(test_seq 5) + do + git -C repo branch -f foo && + git -C repo branch -d foo || return 1 + done && + test_line_count = 2 repo/.git/reftable/tables.list +' + check_fsync_events () { local trace="$1" && shift && @@ -345,7 +358,7 @@ test_expect_success 'ref transaction: writes are synced' ' git -C repo -c core.fsync=reference \ -c core.fsyncMethod=fsync update-ref refs/heads/branch HEAD && check_fsync_events trace2.txt <<-EOF - "name":"hardware-flush","count":2 + "name":"hardware-flush","count":4 EOF ' @@ -377,7 +390,7 @@ test_expect_success 'ref transaction: fails gracefully when auto compaction fail done || exit 1 done && - test_line_count = 13 .git/reftable/tables.list + test_line_count = 10 .git/reftable/tables.list ) ' @@ -387,8 +400,8 @@ test_expect_success 'pack-refs: compacts tables' ' test_commit -C repo A && ls -1 repo/.git/reftable >table-files && - test_line_count = 4 table-files && - test_line_count = 3 repo/.git/reftable/tables.list && + test_line_count = 3 table-files && + test_line_count = 2 repo/.git/reftable/tables.list && git -C repo pack-refs && ls -1 repo/.git/reftable >table-files && @@ -429,7 +442,7 @@ test_expect_success "$command: auto compaction" ' # The tables should have been auto-compacted, and thus auto # compaction should not have to do anything. ls -1 .git/reftable >tables-expect && - test_line_count = 4 tables-expect && + test_line_count = 3 tables-expect && git $command --auto && ls -1 .git/reftable >tables-actual && test_cmp tables-expect tables-actual && @@ -447,7 +460,7 @@ test_expect_success "$command: auto compaction" ' git branch B && git branch C && rm .git/reftable/*.lock && - test_line_count = 5 .git/reftable/tables.list && + test_line_count = 4 .git/reftable/tables.list && git $command --auto && test_line_count = 1 .git/reftable/tables.list @@ -479,7 +492,7 @@ do umask $umask && git init --shared=true repo && test_commit -C repo A && - test_line_count = 3 repo/.git/reftable/tables.list + test_line_count = 2 repo/.git/reftable/tables.list ) && git -C repo pack-refs && test_expect_perms "-rw-rw-r--" repo/.git/reftable/tables.list && @@ -847,12 +860,16 @@ test_expect_success 'worktree: pack-refs in main repo packs main refs' ' test_when_finished "rm -rf repo worktree" && git init repo && test_commit -C repo A && + + GIT_TEST_REFTABLE_AUTOCOMPACTION=false \ git -C repo worktree add ../worktree && + GIT_TEST_REFTABLE_AUTOCOMPACTION=false \ + git -C worktree update-ref refs/worktree/per-worktree HEAD && - test_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list && - test_line_count = 4 repo/.git/reftable/tables.list && + test_line_count = 4 repo/.git/worktrees/worktree/reftable/tables.list && + test_line_count = 3 repo/.git/reftable/tables.list && git -C repo pack-refs && - test_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list && + test_line_count = 4 repo/.git/worktrees/worktree/reftable/tables.list && test_line_count = 1 repo/.git/reftable/tables.list ' @@ -860,13 +877,17 @@ test_expect_success 'worktree: pack-refs in worktree packs worktree refs' ' test_when_finished "rm -rf repo worktree" && git init repo && test_commit -C repo A && + + GIT_TEST_REFTABLE_AUTOCOMPACTION=false \ git -C repo worktree add ../worktree && + GIT_TEST_REFTABLE_AUTOCOMPACTION=false \ + git -C worktree update-ref refs/worktree/per-worktree HEAD && - test_line_count = 3 repo/.git/worktrees/worktree/reftable/tables.list && - test_line_count = 4 repo/.git/reftable/tables.list && + test_line_count = 4 repo/.git/worktrees/worktree/reftable/tables.list && + test_line_count = 3 repo/.git/reftable/tables.list && git -C worktree pack-refs && test_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list && - test_line_count = 4 repo/.git/reftable/tables.list + test_line_count = 3 repo/.git/reftable/tables.list ' test_expect_success 'worktree: creating shared ref updates main stack' ' @@ -880,6 +901,7 @@ test_expect_success 'worktree: creating shared ref updates main stack' ' test_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list && test_line_count = 1 repo/.git/reftable/tables.list && + GIT_TEST_REFTABLE_AUTOCOMPACTION=false \ git -C worktree update-ref refs/heads/shared HEAD && test_line_count = 1 repo/.git/worktrees/worktree/reftable/tables.list && test_line_count = 2 repo/.git/reftable/tables.list