From patchwork Thu May 23 16:38:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13672048 Received: from mail-oa1-f46.google.com (mail-oa1-f46.google.com [209.85.160.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1A0781B7FD for ; Thu, 23 May 2024 16:38:05 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716482287; cv=none; b=UVQJUE0GSkcZFvnvnIhNWLysEDdgGvWFTugr0echIYow7hkVyOloAtrz25+z5MXEgrtaj9Kq2haab+Y+CKRVuR2ow8C6m3AWVzYMa2uRr0382ctW4sVXXSQJUBRggv6277yVIYFyk7C57dgHJqCq1Vxjk+32iwrzCTzSW9Nz7JA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716482287; c=relaxed/simple; bh=HwtvqW7EpQf3K92TDX76FgVY+/5E30HaN6cm5OHlsl0=; h=Date:From:To:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=uooouXWW+lMbA7fGxjEUESypbBI78XsiM5NxezR+FRhX/SI1UKmLhnFWDOa7s3kKf1gS7Du+4OvOe9C6jBfz9trBTfMCPbyFFfXgLbgNo6oCE4SIXd6iM2C8hNlrzOIW6/ZX0Xwf1pS9QF2S5Ip5XFTOqMoTUD43g9idm36B5cw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=Doos42h8; arc=none smtp.client-ip=209.85.160.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="Doos42h8" Received: by mail-oa1-f46.google.com with SMTP id 586e51a60fabf-24c39cedc7bso2263761fac.0 for ; Thu, 23 May 2024 09:38:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1716482285; x=1717087085; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=YGLvESH22iwQ49ZcJDUaie53EKwSWaDoLHaSGMma2Sk=; b=Doos42h8pispxT43FSEPRe/uwvRUSoJL1TljGOS39aa2oDWJtx8uDKZykO0zB/b7+R /IaBBzfyX1gkTdOMt9ko5KErlIU63cxuxkcgkyJW0o3acCXff7ydaiXArU+RmSmmSZNB SXIvfWZeZiHdCqmAz0Jy7kdrArkPa0wjlEPwmuK511Mj4ommMOnJATtgElFk381Zm68k 0/EHp0RN9IJQQNN80u/pXzg5+nXYxK5Vq8T9INpgtIBpYkYXr6XvikDW4ZcLZiyZgNkt HaxebHK3nhRkyue9RMjncmQRV2smb3DzrtDGIbNE99zEeV4+8PMndXHVStF6Qhgh5Atc 1UnA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716482285; x=1717087085; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=YGLvESH22iwQ49ZcJDUaie53EKwSWaDoLHaSGMma2Sk=; b=dZQ9mSmRrZoRkhEcz9HwKvPV9Wmd3oiw+WGdWqjadJL0eWyLLUy/2srcOwIYgym7J1 rvUkeyDedDru2fxfn3bbS5IvP+V/RtOzvyDTRlVyUwDSnRLlgDaiTFkxDB/Cbc1bEbf7 oRJ7gObWgAAgJly+bSqHzuORoQ4t+sDbVCdxVQR5wFG8gGA4gNYrUnk+uInAoW2C0iJ0 MJWM580vepkhLF3k562gtt1CEN9q0SPAY8g6oP3GMBCsqa4Wf7DFAGmMd468NEYg8qan TR9Tw88z0WzgzpS2M8tiiPWXl62agrFMg5WPkM3APP0oIf3oyZFSmgF1IFNATLMxg3Lq JxjQ== X-Gm-Message-State: AOJu0Yznl2FQa51R6vUZ0y+d1ndQqAJd18xxi541WO4Dh73b0nb9JSaF OFaIpnIG8uX5lpMHd2RZPSjBFpDvDc+724kJNWtsdN8JAnUw+WblrRL9q5jC4eN2b/e1zMU9U6n x X-Google-Smtp-Source: AGHT+IHc5ZPMY07sT4HU8lXtTsUUkBnNHMN43xNmvBRZbnYlT3q6i9StH5Rl/LvOhVBunIW6h/EayA== X-Received: by 2002:a05:6870:a507:b0:245:40dd:8292 with SMTP id 586e51a60fabf-24c68bc9036mr5949438fac.33.1716482284868; Thu, 23 May 2024 09:38:04 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-43e137a737asm149301961cf.4.2024.05.23.09.38.04 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 May 2024 09:38:04 -0700 (PDT) Date: Thu, 23 May 2024 12:38:03 -0400 From: Taylor Blau To: git@vger.kernel.org Subject: [PATCH 1/8] midx-write.c: tolerate `--preferred-pack` without bitmaps Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: When passing a preferred pack to the MIDX write machinery, we ensure that the given preferred pack is non-empty since 5d3cd09a808 (midx: reject empty `--preferred-pack`'s, 2021-08-31). However packs are only loaded (via `write_midx_internal()`, though a subsequent patch will refactor this code out to its own function) when the `MIDX_WRITE_REV_INDEX` flag is set. So if a caller runs: $ git multi-pack-index write --preferred-pack=... with both (a) an existing MIDX, and (b) specifies a pack from that MIDX as the preferred one, without passing `--bitmap`, then the check added in 5d3cd09a808 will result in a segfault. Note that packs loaded from disk which don't appear in an existing MIDX do not trigger this issue, as those packs are loaded unconditionally. We conditionally load packs from a MIDX since we tolerate MIDXs whose packs do not resolve (i.e., via the MIDX write after removing unreferenced packs via 'git multi-pack-index expire'). In practice, this isn't possible to trigger when running `git multi-pack-index write` from via `git repack`, as the latter always passes `--stdin-packs`, which prevents us from loading an existing MIDX, as it forces all packs to be read from disk. But a future commit in this series will change that behavior to unconditionally load an existing MIDX, even with `--stdin-packs`, making this behavior trigger-able from 'repack' much more easily. Prevent this from being an issue by removing the segfault altogether by calling `prepare_midx_pack()` on packs loaded from an existing MIDX when either the `MIDX_WRITE_REV_INDEX` flag is set *or* we specified a `--preferred-pack`. Signed-off-by: Taylor Blau --- midx-write.c | 8 +++++++- t/t5319-multi-pack-index.sh | 23 +++++++++++++++++++++++ 2 files changed, 30 insertions(+), 1 deletion(-) diff --git a/midx-write.c b/midx-write.c index 9d096d5a28..03e95ae821 100644 --- a/midx-write.c +++ b/midx-write.c @@ -930,11 +930,17 @@ static int write_midx_internal(const char *object_dir, for (i = 0; i < ctx.m->num_packs; i++) { ALLOC_GROW(ctx.info, ctx.nr + 1, ctx.alloc); - if (flags & MIDX_WRITE_REV_INDEX) { + if (flags & MIDX_WRITE_REV_INDEX || + preferred_pack_name) { /* * If generating a reverse index, need to have * packed_git's loaded to compare their * mtimes and object count. + * + * If a preferred pack is specified, + * need to have packed_git's loaded to + * ensure the chosen preferred pack has + * a non-zero object count. */ if (prepare_midx_pack(the_repository, ctx.m, i)) { error(_("could not load pack")); diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index dd09134db0..10d2a6bf92 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -350,6 +350,29 @@ test_expect_success 'preferred packs must be non-empty' ' ) ' +test_expect_success 'preferred pack from existing MIDX without bitmaps' ' + git init preferred-without-bitmaps && + ( + cd preferred-without-bitmaps && + + test_commit one && + pack="$(git pack-objects --all $objdir/pack/pack X-Patchwork-Id: 13672049 Received: from mail-qv1-f45.google.com (mail-qv1-f45.google.com [209.85.219.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A0A341B7FD for ; Thu, 23 May 2024 16:38:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716482291; cv=none; b=QJYcZXR3YbXW/3bM51eaSBWgCU1kIU24wFqgTOLJ9BZJFB6HXemGnraT1jxv9L/IdMU6j425ZTgG3EAH03vJYfFcuOVsJpb/phhNMcVnRUVtSHNxPhLSlxV+pnKmX9mI2yDN+sLYjzvXoR4hO7ppFG1apklNPgpnXcRY5lbU2LQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716482291; c=relaxed/simple; bh=ZyceiAae5RU+OvI+I3CaLTBJM5AnmJq38BTdXfF/N1M=; h=Date:From:To:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=RzQZzKnBSUQI/ioEc2gksDoDXoEwUelrhOFj8ImrjQsjwQoh+2J1NT0fJjIMb+B3qzUnmjyGoqfzQGltJAoFkQTIXNfGxvPoVf9cONtdfGjTSSSK3aQlfhlwSFfZyF7Ycvxvj0OrAFkwX1jAA1Cnj8BpSWzD+U7/dM4eU1+/keI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=PUZHBIvk; arc=none smtp.client-ip=209.85.219.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="PUZHBIvk" Received: by mail-qv1-f45.google.com with SMTP id 6a1803df08f44-69b782287f9so11963476d6.0 for ; Thu, 23 May 2024 09:38:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1716482288; x=1717087088; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=Vuq7/weQdgsrPaJmu+zeXqY9YtNhcV5N63jj8imtUjI=; b=PUZHBIvksSlyGtwuGP7s9ejmyXt4kxvbprJlFqrYfW00CZZgoajyPJZoMXxFKV+004 KXKG8d8FJILafhBcjJo1RFWXhJpgGpt6tZgbKjGkBf1fJUW1KOycOrdssi3oFH4LBTPL Euwk155k8xWFz6r1iUavuvf6BE402TEDewVbaFvC3nv4TNCEo+0y0awih6U0hdX+s10C qOc9SG1wH8W7WdN484I4YKaY1gnq9Rv0nGdV5vm7ZG2T6dtMb2KXxFGtUplrohAeHuQ3 4DWZ0voN2/Qm6goObeiaLGMOjiMYGnm9xbcnDHllGnFnqJcGbKJA7rdFf4kiuOngnQy8 ssJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716482288; x=1717087088; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=Vuq7/weQdgsrPaJmu+zeXqY9YtNhcV5N63jj8imtUjI=; b=eV2LOKQ+hz28qR6UqTGkZtDAWb/ogBmLt5qMH0sQ/JRkP/G7CLolrxp/ZIa+ydEq7z qZgIJJK95Smmsmvoo8oAcudn0BuKZmOo25k1wTD0CCJRnhsEL6+QwqsGSL5azQGn2Lx+ a3d+AHLfxWxpZxGdRZJ95pw5cBW5NhVZ+3kddK0tn5sdHADnsfZvxecrkpqrHfT5YtTD BojO/Juu4EWq5jEbUgO0E/oCiNlO0ExpSNw0HhV4EzHdeGcdr7bHLcDLZhF4Dc0PoJ09 qcXaEXajEXhBioBAemrKwBqxn03+opN2yGxsnK5eiQsqFfVQublM4J2q3YJP9X7EQtQL tZrw== X-Gm-Message-State: AOJu0YzfuinPtIEU3bB1hsC9BhknDC/i4dCxlhkzi5VqEoq3PCEp6i9y fdW/qr5sFKpuR0O/N/f/r+Vj0h5yWO029H7ZCsx9hp2Z0AleCCmbs7SS5mfntEtzGzEwaQgkXGZ m X-Google-Smtp-Source: AGHT+IG3KN9kRLGk/mUywVRDeRUQQfL1ujLRNWYGWhmUh3+iHwa5MkGQoSvbrRRtpu107mJ+21WcXw== X-Received: by 2002:a05:6214:5d86:b0:6ab:97af:9aa6 with SMTP id 6a1803df08f44-6ab97af9be4mr14628926d6.8.1716482288162; Thu, 23 May 2024 09:38:08 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6a15f17966fsm144331786d6.23.2024.05.23.09.38.07 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 May 2024 09:38:07 -0700 (PDT) Date: Thu, 23 May 2024 12:38:06 -0400 From: Taylor Blau To: git@vger.kernel.org Subject: [PATCH 2/8] midx-write.c: reduce argument count for `get_sorted_entries()` Message-ID: <07dad5a5812794be6e355b1e0eb3722d452f292b.1716482279.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The function `midx-write.c::get_sorted_entries()` is responsible for constructing the array of OIDs from a given list of packs which will comprise the MIDX being written. The singular call-site for this function looks something like: ctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr, ctx.preferred_pack_idx); This function has five formal arguments, all of which are members of the shared `struct write_midx_context` used to track various pieces of information about the MIDX being written. The function `get_sorted_entries()` dates back to fe1ed56f5e4 (midx: sort and deduplicate objects from packfiles, 2018-07-12), which came shortly after 396f257018a (multi-pack-index: read packfile list, 2018-07-12). The latter patch introduced the `pack_list` structure, which was a precursor to the structure we now know as `write_midx_context` (c.f. 577dc49696a (midx: rename pack_info to write_midx_context, 2021-02-18)). At the time, `get_sorted_entries()` likely could have used the pack_list structure introduced earlier in 396f257018a, but understandably did not since the structure only contained three fields (only two of which were relevant to `get_sorted_entries()`) at the time. Simplify the declaration of this function by taking a single pointer to the whole `struct write_midx_context` instead of various members within it. Signed-off-by: Taylor Blau --- midx-write.c | 41 ++++++++++++++++++----------------------- 1 file changed, 18 insertions(+), 23 deletions(-) diff --git a/midx-write.c b/midx-write.c index 03e95ae821..ad32e8953d 100644 --- a/midx-write.c +++ b/midx-write.c @@ -299,21 +299,17 @@ static void midx_fanout_add_pack_fanout(struct midx_fanout *fanout, * Copy only the de-duplicated entries (selected by most-recent modified time * of a packfile containing the object). */ -static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m, - struct pack_info *info, - uint32_t nr_packs, - size_t *nr_objects, - int preferred_pack) +static struct pack_midx_entry *get_sorted_entries(struct write_midx_context *ctx) { uint32_t cur_fanout, cur_pack, cur_object; size_t alloc_objects, total_objects = 0; struct midx_fanout fanout = { 0 }; struct pack_midx_entry *deduplicated_entries = NULL; - uint32_t start_pack = m ? m->num_packs : 0; + uint32_t start_pack = ctx->m ? ctx->m->num_packs : 0; - for (cur_pack = start_pack; cur_pack < nr_packs; cur_pack++) + for (cur_pack = start_pack; cur_pack < ctx->nr; cur_pack++) total_objects = st_add(total_objects, - info[cur_pack].p->num_objects); + ctx->info[cur_pack].p->num_objects); /* * As we de-duplicate by fanout value, we expect the fanout @@ -324,25 +320,25 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m, ALLOC_ARRAY(fanout.entries, fanout.alloc); ALLOC_ARRAY(deduplicated_entries, alloc_objects); - *nr_objects = 0; + ctx->entries_nr = 0; for (cur_fanout = 0; cur_fanout < 256; cur_fanout++) { fanout.nr = 0; - if (m) - midx_fanout_add_midx_fanout(&fanout, m, cur_fanout, - preferred_pack); + if (ctx->m) + midx_fanout_add_midx_fanout(&fanout, ctx->m, cur_fanout, + ctx->preferred_pack_idx); - for (cur_pack = start_pack; cur_pack < nr_packs; cur_pack++) { - int preferred = cur_pack == preferred_pack; + for (cur_pack = start_pack; cur_pack < ctx->nr; cur_pack++) { + int preferred = cur_pack == ctx->preferred_pack_idx; midx_fanout_add_pack_fanout(&fanout, - info, cur_pack, + ctx->info, cur_pack, preferred, cur_fanout); } - if (-1 < preferred_pack && preferred_pack < start_pack) - midx_fanout_add_pack_fanout(&fanout, info, - preferred_pack, 1, + if (-1 < ctx->preferred_pack_idx && ctx->preferred_pack_idx < start_pack) + midx_fanout_add_pack_fanout(&fanout, ctx->info, + ctx->preferred_pack_idx, 1, cur_fanout); midx_fanout_sort(&fanout); @@ -356,12 +352,12 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m, &fanout.entries[cur_object].oid)) continue; - ALLOC_GROW(deduplicated_entries, st_add(*nr_objects, 1), + ALLOC_GROW(deduplicated_entries, st_add(ctx->entries_nr, 1), alloc_objects); - memcpy(&deduplicated_entries[*nr_objects], + memcpy(&deduplicated_entries[ctx->entries_nr], &fanout.entries[cur_object], sizeof(struct pack_midx_entry)); - (*nr_objects)++; + ctx->entries_nr++; } } @@ -1055,8 +1051,7 @@ static int write_midx_internal(const char *object_dir, } } - ctx.entries = get_sorted_entries(ctx.m, ctx.info, ctx.nr, &ctx.entries_nr, - ctx.preferred_pack_idx); + ctx.entries = get_sorted_entries(&ctx); ctx.large_offsets_needed = 0; for (i = 0; i < ctx.entries_nr; i++) { From patchwork Thu May 23 16:38:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13672050 Received: from mail-qt1-f172.google.com (mail-qt1-f172.google.com [209.85.160.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C7C3B1B7FD for ; Thu, 23 May 2024 16:38:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716482294; cv=none; b=ZZ4TtP210cgAf0Gd2+YiHd7l525ig8ENe0FeennjTtJmK47pPGJRGiBXcL6Kb1Z4a92HGv8bwLoL1xBKmaL3ZRSS5HhOlcPU0vqR/q1ltVCaqQkZe1i5p2HJ3Q+PM5x8wJ2ezYwp7OmWUe0G0BbAHaKq3TQtHSUsKpg1okAeE+Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716482294; c=relaxed/simple; bh=gbR48D26zhvme0vFimQ50jGk/WDH0xdrkPg8hNzctck=; h=Date:From:To:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=i+45es4TzJEGxpQEoWssAKvAB//IO56q62UZCU8qr4Gg1hj8tneiULYUc8OZ3HfzohdvjVadu8GL4A+AiUsj++ksfjLUhCwRBk2BZJ+eWdtdWeDeK+LR4vlQj3Hz7wyj61Uunl60pgYDicd6nRv4pJqGej9Xb01O9h4fU2zFrTs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=qNZzoaRY; arc=none smtp.client-ip=209.85.160.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="qNZzoaRY" Received: by mail-qt1-f172.google.com with SMTP id d75a77b69052e-43df052337bso6996951cf.3 for ; Thu, 23 May 2024 09:38:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1716482291; x=1717087091; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=Qz1tGE7pSt1jwGAO9LCpidn3GllthPkuTtX/D1wUtY8=; b=qNZzoaRYS07yDTsXQ02i6vsZ53wwoUA/KJkWeMWnHr+AEbsD9RFs9zpWYrP+6oRjh4 oQnFBLAeiEg4JVia+RkLhQH0xz/x5YcccEta8rwFZxnHaAwubplmr4cRQ32yceMO21sC raHZ54WBS2oGc7jrUWJevKusO+nGXVtUQRBrIXsAO43m7TzJz+paDOZR5Yp9ofFZTfpx JT8CGZ5ga+JvFgtf5aQw/KFTl5Hej50C6NLkZKjxVfyGhqLAXgS0br3NGvwuMcC3NKRH 9avTelWB3DkjW+vh0wPg9Pr07/T5FKrkRNW0rIGo0lyEkRmoLUfOfPdQw2FbpSfmbAsx ET6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716482291; x=1717087091; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=Qz1tGE7pSt1jwGAO9LCpidn3GllthPkuTtX/D1wUtY8=; b=KlfjIdSuU3TW7Eojw1OQ9i+56OSBAeb/mb0dENzQvce6yXE9ygRkAU6/osKaF3TEYw nDK0yB+T1zILn9rNs/jmiSKmE8Q2IRZFsd4fqth92KqsjdkwbhwGIuiIxPK1ruD2SP51 wEvCcLuhx4TaZAG2hpo7lzvHHx/6m9rbd33PECRrm6GigPsDLpLTi3npiuyrmTiYcRvE fViUQhTlCRmzOkkg/Z/sl/v/WUSbAy32yOkZLotM0LEObfUTJgyDvnd1a4UeFAbhS45J RXhrUnp0FiidKU4Ah1mDXrAzJUSzvW4f822Yk4zdrynzsPcDiAaTDbmPFp1QpX1C7gAK 1sJw== X-Gm-Message-State: AOJu0YwQb7nSmnCmtcdlGqHK8mCSIABqmXQMbHrCTk1fxg7RGHu74fMX wibE2vwd7r+w8uWqu7yODhr7/VXUZgILsgPe+fwYxtvc/qP5lRh9HiX3xStR1bzPoagyu9DKnyI G X-Google-Smtp-Source: AGHT+IFnOq5LcL8SxElf2qe4Jc4tBlAbmrAfwpK05k7Cpi4nMiEBPJxEnoJg344gra/exQhb5LLHig== X-Received: by 2002:a05:622a:1111:b0:43a:b531:5e88 with SMTP id d75a77b69052e-43f9e172db8mr63707151cf.68.1716482291329; Thu, 23 May 2024 09:38:11 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-43e046467a4sm169848981cf.83.2024.05.23.09.38.10 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 May 2024 09:38:10 -0700 (PDT) Date: Thu, 23 May 2024 12:38:09 -0400 From: Taylor Blau To: git@vger.kernel.org Subject: [PATCH 3/8] midx-write.c: pass `start_pack` to `get_sorted_entries()` Message-ID: <7acf4557dcb2240cb43eadebfd21b5c37515ba7f.1716482279.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The function `get_sorted_entries()` is broadly responsible for building an array of the objects to be written into a MIDX based on the provided list of packs. If we have loaded an existing MIDX, however, we may not use all of its packs, despite loading them into the ctx->info array. The existing implementation simply skips past the first ctx->m->num_packs (if ctx->m is non-NULL, indicating that we loaded an existing MIDX). Future changes (outside the scope of this patch series) to the MIDX code will require us to skip *at most* that number[^1]. We could tag each pack with a bit that indicates the pack's contents should be included in the MIDX. But we can just as easily determine the number of packs to skip by passing in the number of packs we learned about after processing an existing MIDX. [^1]: Kind of. The real number will be bounded by the number of packs in a MIDX layer, and the number of packs in its base layer(s), but that concept hasn't been fully defined yet. Signed-off-by: Taylor Blau --- midx-write.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/midx-write.c b/midx-write.c index ad32e8953d..cf7e391b6e 100644 --- a/midx-write.c +++ b/midx-write.c @@ -299,13 +299,13 @@ static void midx_fanout_add_pack_fanout(struct midx_fanout *fanout, * Copy only the de-duplicated entries (selected by most-recent modified time * of a packfile containing the object). */ -static struct pack_midx_entry *get_sorted_entries(struct write_midx_context *ctx) +static struct pack_midx_entry *get_sorted_entries(struct write_midx_context *ctx, + uint32_t start_pack) { uint32_t cur_fanout, cur_pack, cur_object; size_t alloc_objects, total_objects = 0; struct midx_fanout fanout = { 0 }; struct pack_midx_entry *deduplicated_entries = NULL; - uint32_t start_pack = ctx->m ? ctx->m->num_packs : 0; for (cur_pack = start_pack; cur_pack < ctx->nr; cur_pack++) total_objects = st_add(total_objects, @@ -886,7 +886,7 @@ static int write_midx_internal(const char *object_dir, { struct strbuf midx_name = STRBUF_INIT; unsigned char midx_hash[GIT_MAX_RAWSZ]; - uint32_t i; + uint32_t i, start_pack; struct hashfile *f = NULL; struct lock_file lk; struct write_midx_context ctx = { 0 }; @@ -954,6 +954,8 @@ static int write_midx_internal(const char *object_dir, } } + start_pack = ctx.nr; + ctx.pack_paths_checked = 0; if (flags & MIDX_PROGRESS) ctx.progress = start_delayed_progress(_("Adding packfiles to multi-pack-index"), 0); @@ -1051,7 +1053,7 @@ static int write_midx_internal(const char *object_dir, } } - ctx.entries = get_sorted_entries(&ctx); + ctx.entries = get_sorted_entries(&ctx, start_pack); ctx.large_offsets_needed = 0; for (i = 0; i < ctx.entries_nr; i++) { From patchwork Thu May 23 16:38:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13672051 Received: from mail-qk1-f174.google.com (mail-qk1-f174.google.com [209.85.222.174]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D8C911B7FD for ; Thu, 23 May 2024 16:38:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.174 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716482297; cv=none; b=ULesWhl9oO12z6wWUROALWLmlqD7CDMVEUc3VooibogIRjrfTTkKCYif+Qf9j4wR+lR48pGStaEM9S/F8QgjrMvemkBxkpljpdVYC8SO04obLH+vCBrw6Yu5lrK01p50WLoWZaGlGFh1oJ96lWTtCpcv3vWXi9Wz8v97qd3ueEw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716482297; c=relaxed/simple; bh=o3id65niulCMX++GDzA3VFolhNsb7ubDDbSO568qZuk=; h=Date:From:To:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=aOlNsNBORIkzIvqZTrwwq9TCXsFdwLe5zs52tcOJWLbUtpZQJF5tnBTeQLR9qOadQYyYwQSURq7+LV+7Lb9DlOG+AdLeX3awiuLRSGdyu/tWtF33n1+ZRvla4bjRBMGHv+bDoSF1P+h078W1YooFdiSR7/4KqU/IgEbH121M6mQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=VoWJxx6F; arc=none smtp.client-ip=209.85.222.174 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="VoWJxx6F" Received: by mail-qk1-f174.google.com with SMTP id af79cd13be357-794a028b8feso96704085a.1 for ; Thu, 23 May 2024 09:38:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1716482294; x=1717087094; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=r17ecaxES+jsXGGPceE1uS0rUQtx2r9DaLWAKfBvml8=; b=VoWJxx6FlDmiKGKvRLXT++CKQP+cv4hpYnUWnMqckwXY4piQVb7qx0wwhK0jNR7kBv somgaGsJr0tNwo2epFbfpeuP8+vSXDOaEKRbWXUr9QfpgvYSBbUUMRaAtprMmy03o2E/ WgySjuPglFZut8VNDjj6MVBF5AuL5ZawfmVrbqLTY/hW6iaJhXEP0TeZIFlr+Axv/h12 6/11JQDQxQ4s9IqQeI7mG0HasEBGSoKNEq8Qk75AMmVUA+aADzpQfikePdMIvmhkSXUV n+5d338+cb4o8431/IAVpsar0N4lZ8+FTrL0FhG5Q2bAZ/5lrpHmfPnPYpkdu+AW7ETf uVEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716482294; x=1717087094; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=r17ecaxES+jsXGGPceE1uS0rUQtx2r9DaLWAKfBvml8=; b=dIS7NLscVL4ohiw5YAqQplOitAKoW/4il6ehkKqj/6315jrgPnSX2fMexQKdLY2uHW M2uHKB/2P6dL/dLupVUjoVnNpLe+rYox+KgnmajCnztl5Fdm91KgCVapgRxv4iMwbQzG jz6RCaoR65pw4am4asB9Pmv3/f4DixUvQdMV7uWqovXl3MPJgJkOkGwTRHtmGJIzr2hO +LTu0OvSBt9IkSc3lweYUHvJaHQe/xiz7ragu+OrhYwwJ2YfrRHsKGSFDOMf/LrD7jsE MMZNYN6A2+g25UnMmAbRgI64knt4eG/mm5L7Zn0C1hDdjba1F/f+OY5S17d+L/qNe8dr HrUg== X-Gm-Message-State: AOJu0YwuvP/6DfvUC6WY0j3CNpIC6z3eFNwqdpug+P3A90MKsX26kiTy CkxP8ypaDh/UWAewoOmTSgDbz0+BHRpMsgqV+NKsU4aJc6jV7q6aWaMgvPWWaQwy/2BLLIpu85k z X-Google-Smtp-Source: AGHT+IHfn8YWGI8uPXt/K1k0GAfS5AvXCiNA9gqJornexu0joz61k9uhuY9PRT43HGtL5VPNop8TkQ== X-Received: by 2002:a05:622a:18a0:b0:43e:2f13:6072 with SMTP id d75a77b69052e-43fa7455dcamr51066951cf.9.1716482294524; Thu, 23 May 2024 09:38:14 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-43e144a98a2sm147051961cf.16.2024.05.23.09.38.14 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 May 2024 09:38:14 -0700 (PDT) Date: Thu, 23 May 2024 12:38:13 -0400 From: Taylor Blau To: git@vger.kernel.org Subject: [PATCH 4/8] midx-write.c: extract `should_include_pack()` Message-ID: <3908546ea85eb36a27ce6bd681a3c2152ff005f5.1716482279.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The add_pack_to_midx() callback used via for_each_file_in_pack_dir() is used to add packs with .idx files to the MIDX being written. Within this function, we have a pair of checks that discards packs which: - appear in an existing MIDX, if we successfully read an existing MIDX from disk - or, appear in the "to_include" list, if invoking the MIDX write machinery with the `--stdin-packs` command-line argument. In a future commit will want to call a slight variant of these checks from the code that reuses all packs from an existing MIDX, as well as the current location via add_pack_to_midx(). The latter will be modified in subsequent commits to only reuse packs which appear in the to_include list, if one was given. Prepare for that step by extracting these checks as a subroutine that may be called from both places. Signed-off-by: Taylor Blau --- midx-write.c | 48 ++++++++++++++++++++++++++++-------------------- 1 file changed, 28 insertions(+), 20 deletions(-) diff --git a/midx-write.c b/midx-write.c index cf7e391b6e..f593cf1593 100644 --- a/midx-write.c +++ b/midx-write.c @@ -100,6 +100,32 @@ struct write_midx_context { struct string_list *to_include; }; +static int should_include_pack(const struct write_midx_context *ctx, + const char *file_name) +{ + /* + * Note that at most one of ctx->m and ctx->to_include are set, + * so we are testing midx_contains_pack() and + * string_list_has_string() independently (guarded by the + * appropriate NULL checks). + * + * We could support passing to_include while reusing an existing + * MIDX, but don't currently since the reuse process drags + * forward all packs from an existing MIDX (without checking + * whether or not they appear in the to_include list). + * + * If we added support for that, these next two conditional + * should be performed independently (likely checking + * to_include before the existing MIDX). + */ + if (ctx->m && midx_contains_pack(ctx->m, file_name)) + return 0; + else if (ctx->to_include && + !string_list_has_string(ctx->to_include, file_name)) + return 0; + return 1; +} + static void add_pack_to_midx(const char *full_path, size_t full_path_len, const char *file_name, void *data) { @@ -108,29 +134,11 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len, if (ends_with(file_name, ".idx")) { display_progress(ctx->progress, ++ctx->pack_paths_checked); - /* - * Note that at most one of ctx->m and ctx->to_include are set, - * so we are testing midx_contains_pack() and - * string_list_has_string() independently (guarded by the - * appropriate NULL checks). - * - * We could support passing to_include while reusing an existing - * MIDX, but don't currently since the reuse process drags - * forward all packs from an existing MIDX (without checking - * whether or not they appear in the to_include list). - * - * If we added support for that, these next two conditional - * should be performed independently (likely checking - * to_include before the existing MIDX). - */ - if (ctx->m && midx_contains_pack(ctx->m, file_name)) - return; - else if (ctx->to_include && - !string_list_has_string(ctx->to_include, file_name)) + + if (!should_include_pack(ctx, file_name)) return; ALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc); - p = add_packed_git(full_path, full_path_len, 0); if (!p) { warning(_("failed to add packfile '%s'"), From patchwork Thu May 23 16:38:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13672052 Received: from mail-oi1-f169.google.com (mail-oi1-f169.google.com [209.85.167.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4C1B11B7FD for ; Thu, 23 May 2024 16:38:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716482300; cv=none; b=dwAoInEEU+nucWnc3jpp856P/IFwN+QKZKrkfGipyL/lTx4Y8PShWObCKKD4+G/v68aHFzA8ISI5UvIxAohbZZVfRSuHlJghpGp0st1m+IzIZwiarBYq2otVCHCMgvOPDnU4leKo+MjOAfNFhWkHXqbAaukY9osxIbfJEIUm1QI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716482300; c=relaxed/simple; bh=xVCZ3Ypwgu0PBwWR1lCoNPt5uuHM1mDSHmdRA+dUUsI=; h=Date:From:To:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=WYht/oG/ATub7F/NBPW3XHF7xrg7twtbZ3jcPrVZEHv3oF3DCM2MWFqEyDw4hS4UnZVZsUJoLBIiLa9GcVD4u7V481lUph8gBqXt1fqznXduTTxCWM7LWJrOqYz35YFKLjZFeRG0ubIlWNqNv1j02vjzUKnFdPnihWKIX7lJXhA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=EogCyZ+b; arc=none smtp.client-ip=209.85.167.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="EogCyZ+b" Received: by mail-oi1-f169.google.com with SMTP id 5614622812f47-3c99501317bso3152295b6e.3 for ; Thu, 23 May 2024 09:38:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1716482298; x=1717087098; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=xUGSTQ5ZEP/VfHPFtdC4NsjLcbjdVBBQdzH3OFMbxdA=; b=EogCyZ+bLw85GfDjxBK1tCyeFbm+XqM6BGZoAOxhWkm24LNEAtlwQKbYC14+3rOzFC OLv4dBnpWsGv+K207/bQIapioxQSb8YGG5pBo8srSBFfey9tcns/m+Ot6+yqYT3hPg8b S0F5sr8I4NgqbfAQBsUl8VKgHB0Jb5ATZm9OUZvPev+ZvAOxoxMxaojlP8QQYZ11fqWE nmzsbmOp3CZ7mIbVisj6ZVF/adLJ6j2jRYa7WT6IbZq8t9rW1+jpF+G+FipQOmJsxJaJ 41EbQVvbWcJGEmrQbes92c3TY5wZs4DmNrutkxfCS+qjeNllNmcLMTPJ5kboLg7HbJNN OCoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716482298; x=1717087098; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=xUGSTQ5ZEP/VfHPFtdC4NsjLcbjdVBBQdzH3OFMbxdA=; b=hAerMhHWE8e3GeEkvrPb9Pc/i2p/nHnRJLtae3Lt98wsqpnPGqfYDOINtL9bC7VhIF ItibsapLOLWBih66OcLvKpx/TFsrcZq9BNTceXlEpYSk2cP8IieqA41QcrgogKBMSw5G oxZeWVKHs4DzT0+VjbL3KW4rgZj4Rr6e+suv7OxeQUyBG4uyIq87tuk5Po+eAp87AXoH 994g6/Z9ejZepDdRxmDYkQy7X1s4ON8eW07QT2gNI0c2VEWkHHeT5ZNI/WoI538y2WpQ WaqKRZpbZYSilVwDEPzxOAd12IC3aIlpr7uQIWclfw4qGCMMKmLmerTZdO8b6PmrVjZB Vx8A== X-Gm-Message-State: AOJu0YzaRq+Lv7JCC6vDkoDMIt/bUOoHsZFvknTGVi2AVpmtslM9rmy1 6fmCPaVHSXnL1icwWsDAIW9WBNONT9VRdkoeMBydqaPkg0CGyKzMA706iQzYEljJUl8nOtV+YQd g X-Google-Smtp-Source: AGHT+IGgBjHZh2vUe8e/sEFkr4XsnoxZqE3nBe3oSizX426e1UOSJwSz7IPpNRi5YWAln3q0p1J84Q== X-Received: by 2002:a05:6358:7305:b0:186:119d:8c16 with SMTP id e5c5f4694b2df-19791dded60mr849729655d.23.1716482297923; Thu, 23 May 2024 09:38:17 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6a15f1cd250sm145656586d6.98.2024.05.23.09.38.17 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 May 2024 09:38:17 -0700 (PDT) Date: Thu, 23 May 2024 12:38:16 -0400 From: Taylor Blau To: git@vger.kernel.org Subject: [PATCH 5/8] midx-write.c: extract `fill_packs_from_midx()` Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: When write_midx_internal() loads an existing MIDX, all packs are copied forward into the new MIDX. Improve the readability of write_midx_internal() by extracting this functionality out into a separate function. Signed-off-by: Taylor Blau --- midx-write.c | 68 +++++++++++++++++++++++++++++----------------------- 1 file changed, 38 insertions(+), 30 deletions(-) diff --git a/midx-write.c b/midx-write.c index f593cf1593..9712ac044f 100644 --- a/midx-write.c +++ b/midx-write.c @@ -885,6 +885,40 @@ static struct multi_pack_index *lookup_multi_pack_index(struct repository *r, return result; } +static int fill_packs_from_midx(struct write_midx_context *ctx, + const char *preferred_pack_name, uint32_t flags) +{ + uint32_t i; + + for (i = 0; i < ctx->m->num_packs; i++) { + ALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc); + + if (flags & MIDX_WRITE_REV_INDEX || preferred_pack_name) { + /* + * If generating a reverse index, need to have + * packed_git's loaded to compare their + * mtimes and object count. + * + * + * If a preferred pack is specified, need to + * have packed_git's loaded to ensure the chosen + * preferred pack has a non-zero object count. + */ + if (prepare_midx_pack(the_repository, ctx->m, i)) + return error(_("could not load pack")); + + if (open_pack_index(ctx->m->packs[i])) + die(_("could not open index for %s"), + ctx->m->packs[i]->pack_name); + } + + fill_pack_info(&ctx->info[ctx->nr++], ctx->m->packs[i], + ctx->m->pack_names[i], i); + } + + return 0; +} + static int write_midx_internal(const char *object_dir, struct string_list *packs_to_include, struct string_list *packs_to_drop, @@ -930,36 +964,10 @@ static int write_midx_internal(const char *object_dir, ctx.info = NULL; ALLOC_ARRAY(ctx.info, ctx.alloc); - if (ctx.m) { - for (i = 0; i < ctx.m->num_packs; i++) { - ALLOC_GROW(ctx.info, ctx.nr + 1, ctx.alloc); - - if (flags & MIDX_WRITE_REV_INDEX || - preferred_pack_name) { - /* - * If generating a reverse index, need to have - * packed_git's loaded to compare their - * mtimes and object count. - * - * If a preferred pack is specified, - * need to have packed_git's loaded to - * ensure the chosen preferred pack has - * a non-zero object count. - */ - if (prepare_midx_pack(the_repository, ctx.m, i)) { - error(_("could not load pack")); - result = 1; - goto cleanup; - } - - if (open_pack_index(ctx.m->packs[i])) - die(_("could not open index for %s"), - ctx.m->packs[i]->pack_name); - } - - fill_pack_info(&ctx.info[ctx.nr++], ctx.m->packs[i], - ctx.m->pack_names[i], i); - } + if (ctx.m && fill_packs_from_midx(&ctx, preferred_pack_name, + flags) < 0) { + result = 1; + goto cleanup; } start_pack = ctx.nr; From patchwork Thu May 23 16:38:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13672053 Received: from mail-qv1-f50.google.com (mail-qv1-f50.google.com [209.85.219.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8525018633 for ; Thu, 23 May 2024 16:38:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716482304; cv=none; b=V7aRvFaQvBmhxxsjJkVm5/d1/wzCOAuEn/qF0Embq4nKIA4lcWaz/S2fVtVPKXlWpVCNOcXALAujeKRuJLjdWMpXN335uGp0JKIvoh/NIEXuJm9ilq/ACKU4cX3pE6UR5S7f1Eh2WLSFfSyA4NCyoIZp2E+jTzRJxhCIDKpYcaE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716482304; c=relaxed/simple; bh=dE/+cG9vCp1CP7435dVyoXKk/eQxQNVe/s6AVB6/8QI=; h=Date:From:To:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=sW/khNnShapSTy1d217XcREm7Ic29Ho/mZkvM1UtBpqejD/mAADsE03fYFVmxzFfoFdR/9wnqL1s9EsUL6bWun4qENxWtYGsSgrV0MD5kaGm3LJCp1IVxwPgV21AV1E2t9ETZcHRK9yyDxM1w56Eq4vM2weeTPd1mFcvgNheCbk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=WxGqtxm7; arc=none smtp.client-ip=209.85.219.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="WxGqtxm7" Received: by mail-qv1-f50.google.com with SMTP id 6a1803df08f44-6a8691d724eso15808506d6.0 for ; Thu, 23 May 2024 09:38:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1716482301; x=1717087101; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=545QmfmwDNiPAOw6ahXSjCuxGWiceNxLOOVvFrCYxQQ=; b=WxGqtxm7ssKWnJ9J9Zz+dRQcRhdsAcX8tsQnsVRilFPXS7n7zZYY2KJFqF/Sm+UDKP aYBnv/i3lOxBpriHh25jd5LXbttE2vu4w0V7qpC/0SylPdvO233LmyLjjXPc+LNJLQmc fkUF9VYPdNGVRpMxEjNBv2YEQv51bjgWd2YMONu+2kgnpI45FrQFKBBvhWoxzRqxj+J9 kZ59IQS7oIyYGOQZ3WEZ2VAtOd4azOKn42X22bMmWf9oY6hThn2nzmAVjSU7/ZTbTl8Z O3iZDn2aMI6/h99bHZ5TV5YQEYLs+zGWqPwD2v3/C0L0InVEfLBzsKPTXEIo6ZZbnqC2 stqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716482301; x=1717087101; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=545QmfmwDNiPAOw6ahXSjCuxGWiceNxLOOVvFrCYxQQ=; b=GyaCFBqGKclJZeGpbA6miSw9aX0CNWcSQnip/GPOxNSu7O3dUKOM34BpNy4H2bvAPj rDomLNgH1GgAT1+sNpiVzbe3NAAa5M6cN5QlQRoLIjfWIHeOlDRZj4U88YFmyifg8azq 0udgiE+kST3TsSTZ6oZTMFkt+lMz6axbpb3B+OZfZOFdONjm4HZLqYWQ+JXBUT0GWVDj qfYTE9C8AxeAh1qae1xW7522XDbmaQWS75uHBrb/XjK7/f+Uzo42+6WNP6Ywrk9Rj5Ef DTX4sP2s8FIOa5rsjm1df1nO9XVNxWKBHnx45zGLe1+UPuYH65kU3SG0BcpYEyXo926P OItQ== X-Gm-Message-State: AOJu0YwBvoimjrr1P4mERkdBrx+Kcfqat1k5HXSRiN33GWawgWHsHkPk 6oLrZGYhwW4bGoaAAr3penITqpiX1WtXA0UfnjVXaTDzmMQzpzF+8H9hf+D+oFS/3LVl/26m67T A X-Google-Smtp-Source: AGHT+IGqriRKWww1T0QwRwDKBqaQlR72bYTHoodMdxcYaWUrYJpUoKPvFcL3xIX8KBiN7c0I429tVA== X-Received: by 2002:a05:6214:4309:b0:6a0:d21b:c914 with SMTP id 6a1803df08f44-6ab808f321bmr65246336d6.51.1716482301055; Thu, 23 May 2024 09:38:21 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6ab9585fc19sm5142336d6.49.2024.05.23.09.38.20 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 May 2024 09:38:20 -0700 (PDT) Date: Thu, 23 May 2024 12:38:19 -0400 From: Taylor Blau To: git@vger.kernel.org Subject: [PATCH 6/8] midx-write.c: support reading an existing MIDX with `packs_to_include` Message-ID: <61268114c6562ba882210fd94b3f336efcb5c486.1716482279.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Avoid unconditionally copying all packs from an existing MIDX into a new MIDX by checking that packs added via `fill_packs_from_midx()` don't appear in the `to_include` set, if one was provided. Do so by calling `should_include_pack()` from both `add_pack_to_midx()` and `fill_packs_from_midx()`. In order to make this work, teach `should_include_pack()` a new "exclude_from_midx" parameter, which allows skipping the first check. This is done so that the caller in `fill_packs_from_midx()` doesn't reject all of the packs it provided since they appear in an existing MIDX by definition. Signed-off-by: Taylor Blau --- midx-write.c | 42 +++++++++++------------------------------- 1 file changed, 11 insertions(+), 31 deletions(-) diff --git a/midx-write.c b/midx-write.c index 9712ac044f..36ac4ab65b 100644 --- a/midx-write.c +++ b/midx-write.c @@ -101,27 +101,13 @@ struct write_midx_context { }; static int should_include_pack(const struct write_midx_context *ctx, - const char *file_name) + const char *file_name, + int exclude_from_midx) { - /* - * Note that at most one of ctx->m and ctx->to_include are set, - * so we are testing midx_contains_pack() and - * string_list_has_string() independently (guarded by the - * appropriate NULL checks). - * - * We could support passing to_include while reusing an existing - * MIDX, but don't currently since the reuse process drags - * forward all packs from an existing MIDX (without checking - * whether or not they appear in the to_include list). - * - * If we added support for that, these next two conditional - * should be performed independently (likely checking - * to_include before the existing MIDX). - */ - if (ctx->m && midx_contains_pack(ctx->m, file_name)) + if (exclude_from_midx && ctx->m && midx_contains_pack(ctx->m, file_name)) return 0; - else if (ctx->to_include && - !string_list_has_string(ctx->to_include, file_name)) + if (ctx->to_include && !string_list_has_string(ctx->to_include, + file_name)) return 0; return 1; } @@ -135,7 +121,7 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len, if (ends_with(file_name, ".idx")) { display_progress(ctx->progress, ++ctx->pack_paths_checked); - if (!should_include_pack(ctx, file_name)) + if (!should_include_pack(ctx, file_name, 1)) return; ALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc); @@ -891,6 +877,9 @@ static int fill_packs_from_midx(struct write_midx_context *ctx, uint32_t i; for (i = 0; i < ctx->m->num_packs; i++) { + if (!should_include_pack(ctx, ctx->m->pack_names[i], 0)) + continue; + ALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc); if (flags & MIDX_WRITE_REV_INDEX || preferred_pack_name) { @@ -945,15 +934,7 @@ static int write_midx_internal(const char *object_dir, die_errno(_("unable to create leading directories of %s"), midx_name.buf); - if (!packs_to_include) { - /* - * Only reference an existing MIDX when not filtering which - * packs to include, since all packs and objects are copied - * blindly from an existing MIDX if one is present. - */ - ctx.m = lookup_multi_pack_index(the_repository, object_dir); - } - + ctx.m = lookup_multi_pack_index(the_repository, object_dir); if (ctx.m && !midx_checksum_valid(ctx.m)) { warning(_("ignoring existing multi-pack-index; checksum mismatch")); ctx.m = NULL; @@ -962,6 +943,7 @@ static int write_midx_internal(const char *object_dir, ctx.nr = 0; ctx.alloc = ctx.m ? ctx.m->num_packs : 16; ctx.info = NULL; + ctx.to_include = packs_to_include; ALLOC_ARRAY(ctx.info, ctx.alloc); if (ctx.m && fill_packs_from_midx(&ctx, preferred_pack_name, @@ -978,8 +960,6 @@ static int write_midx_internal(const char *object_dir, else ctx.progress = NULL; - ctx.to_include = packs_to_include; - for_each_file_in_pack_dir(object_dir, add_pack_to_midx, &ctx); stop_progress(&ctx.progress); From patchwork Thu May 23 16:38:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13672054 Received: from mail-qv1-f44.google.com (mail-qv1-f44.google.com [209.85.219.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D6F5421A06 for ; Thu, 23 May 2024 16:38:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716482307; cv=none; b=rWN6QXzPLjQu+qLwoQ6st6L1e4jmX4g5j3q2u4smcCjEnPzTDNfgtrpO1xhusbfo3hmMJ4l7nwbVc4DdKacZdeQ+Yj5+241zYc58sJ7ZkeHm01lNuosSrxftKOJDmqZLyELbL5bMzRlElmqDi4NuwMHQ1gf67uVt+2BJu1SiId4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716482307; c=relaxed/simple; bh=PRE9Dg0ztwG/uhXNrXgKE1kbMuG35HWy7L5vQpC5r64=; h=Date:From:To:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=A1v3ns+AxHHLMDbvK5zzs7iZhKd7BPIkimsrFwKULHxTOuhr3L6WC2xoXs5OgCigqh2txVKiis07hu5Bfb6JNaeeCjYjB5eHkx+Asz4bjs0UcfWOe/SCTlnxFwgy5lMURpBbwMwrREZfCamZEwQAqHM1j8AxaO5fGyrZa5JhHLo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=B2XErPlY; arc=none smtp.client-ip=209.85.219.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="B2XErPlY" Received: by mail-qv1-f44.google.com with SMTP id 6a1803df08f44-69b24162dd6so32916766d6.2 for ; Thu, 23 May 2024 09:38:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1716482304; x=1717087104; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=O70Of1lM6JS3QdfGAW06FGIQHCm6akX1ozhJZJPsCfM=; b=B2XErPlYAtdo5ytqb6IYYYRlQdemTalYhVzVzpDBSC2+lHqMg9lAn9m7ud1btBBUaw combNGjOulQtMAvH4Tw0ShdOVr8XGPpnfhyxH3XhOSE267tSK21SJZ6IaYbygdbx1/Nz Q84y0/pHxHJ7c9ltOjimLzQ07I6h9qSckmlmEO8VltwadeWLCzhUusgrjqlEFMp7lgUv /4A1AZQJgHWPclA5/xNLLKI0TpLKC8v5EvwZgOGb+Vy/y1R9kzSegsAjTsdQMLfGVdtS 4VSnvwhYfbtw0x+bG0Q729nAGtKz+zHH2o0liCLZW5FK+RpNege4zg0P0OsVTAMMSpDY c1gw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716482304; x=1717087104; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=O70Of1lM6JS3QdfGAW06FGIQHCm6akX1ozhJZJPsCfM=; b=tRbCCYrp/1PXh9n5xEUI9UPu8ZHSe9+q+9YbfIHSxSnTys+uxWt1JWGlCFX7Gb12om H2bFiFLiXMG3MUws6jlX/jkfMNGv0TRDrkX4avsWCW2Gs/CWdu2C36b9Ihvsn+3zbHJZ 8pqqOaASfk/0qz+WSvAjJ9aGMVzAoS1QwzbZinX73k+EKJWj+18mn3YyHU9vAoWQ2VbF Bof9W987GpmHWl6ccs3tqNyInS2mudfs7SEog4EZGdrJf1Z4+695hq0aaoMpUjMuthNO 5s1tnndFsl6ahhEQcqM7mESRmRRdtoUWdM1dzw1FiQcKuYEFn5UGSNhLEZ9KuPyydbdl cWlA== X-Gm-Message-State: AOJu0YxBVyXSramw8yzTGEi4iWezj+2g+xVrnBzNAsShqjW0kBwR1CXi 9h0gO2OE2QD6yxLvp/0yROxu1S0/saAQW8pl47LwLpsIwJd7qtYj9eG1n1ZhqXpEhEp9PpwEiKC f X-Google-Smtp-Source: AGHT+IHVjZGgOE5XktEPn+ulsfj4lFvX1LdjWi+IEX49LunQ/ciMiRXIAyjY+u8OmwM/1Vy0soQX3w== X-Received: by 2002:a05:6214:3b85:b0:6a9:879f:63fd with SMTP id 6a1803df08f44-6ab7f749fbemr64805946d6.37.1716482304235; Thu, 23 May 2024 09:38:24 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6ab8af00036sm12741566d6.63.2024.05.23.09.38.23 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 May 2024 09:38:23 -0700 (PDT) Date: Thu, 23 May 2024 12:38:22 -0400 From: Taylor Blau To: git@vger.kernel.org Subject: [PATCH 7/8] midx: replace `get_midx_rev_filename()` with a generic helper Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Commit f894081deae (pack-revindex: read multi-pack reverse indexes, 2021-03-30) introduced the `get_midx_rev_filename()` helper (later modified by commit 60980aed786 (midx.c: write MIDX filenames to strbuf, 2021-10-26)). This function returns the location of the classic ".rev" files we used to write for MIDXs (prior to 95e8383bac1 (midx.c: make changing the preferred pack safe, 2022-01-25)), which is always of the form: $GIT_DIR/objects/pack/multi-pack-index-$HASH.rev Replace this function with a generic helper that populates a strbuf with the above form, replacing the ".rev" extension with a caller-provided argument. This will allow us to remove a similarly-defined function in the pack-bitmap code (used to determine the location of a MIDX .bitmap file) by reimplementing it in terms of `get_midx_filename_ext()`. Signed-off-by: Taylor Blau --- midx.c | 14 ++++++++------ midx.h | 6 +++++- pack-revindex.c | 3 ++- 3 files changed, 15 insertions(+), 8 deletions(-) diff --git a/midx.c b/midx.c index 6f07de3688..bc4797196f 100644 --- a/midx.c +++ b/midx.c @@ -24,14 +24,16 @@ const unsigned char *get_midx_checksum(struct multi_pack_index *m) } void get_midx_filename(struct strbuf *out, const char *object_dir) +{ + get_midx_filename_ext(out, object_dir, NULL, NULL); +} + +void get_midx_filename_ext(struct strbuf *out, const char *object_dir, + const unsigned char *hash, const char *ext) { strbuf_addf(out, "%s/pack/multi-pack-index", object_dir); -} - -void get_midx_rev_filename(struct strbuf *out, struct multi_pack_index *m) -{ - get_midx_filename(out, m->object_dir); - strbuf_addf(out, "-%s.rev", hash_to_hex(get_midx_checksum(m))); + if (ext) + strbuf_addf(out, "-%s.%s", hash_to_hex(hash), ext); } static int midx_read_oid_fanout(const unsigned char *chunk_start, diff --git a/midx.h b/midx.h index dc477dff44..8554f2d616 100644 --- a/midx.h +++ b/midx.h @@ -74,9 +74,13 @@ struct multi_pack_index { #define MIDX_WRITE_BITMAP_HASH_CACHE (1 << 3) #define MIDX_WRITE_BITMAP_LOOKUP_TABLE (1 << 4) +#define MIDX_EXT_REV "rev" +#define MIDX_EXT_BITMAP "bitmap" + const unsigned char *get_midx_checksum(struct multi_pack_index *m); void get_midx_filename(struct strbuf *out, const char *object_dir); -void get_midx_rev_filename(struct strbuf *out, struct multi_pack_index *m); +void get_midx_filename_ext(struct strbuf *out, const char *object_dir, + const unsigned char *hash, const char *ext); struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local); int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id); diff --git a/pack-revindex.c b/pack-revindex.c index a7624d8be8..fc63aa76a2 100644 --- a/pack-revindex.c +++ b/pack-revindex.c @@ -381,7 +381,8 @@ int load_midx_revindex(struct multi_pack_index *m) trace2_data_string("load_midx_revindex", the_repository, "source", "rev"); - get_midx_rev_filename(&revindex_name, m); + get_midx_filename_ext(&revindex_name, m->object_dir, + get_midx_checksum(m), MIDX_EXT_REV); ret = load_revindex_from_disk(revindex_name.buf, m->num_objects, From patchwork Thu May 23 16:38:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13672055 Received: from mail-qv1-f48.google.com (mail-qv1-f48.google.com [209.85.219.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0044F18C31 for ; Thu, 23 May 2024 16:38:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716482310; cv=none; b=jccJamSaB3EZiXvrOBCQxWr00Bi4TAg2eNjADhJe/AoKkv0P8rQUSjVuWkH28dUjAiTJnzcPDCXVEh88bcWPJRVIcWmwJX8KQNxU0iTKr9rjIBiCsHniF73jDFH7V+kCoXy6PC8I048F1t0M/Y8tnHC1EKwHq8NlLO749blBsQ8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1716482310; c=relaxed/simple; bh=fRjknsG7JbWps1QQZ8hk1CSPuww6hk0RptZN2NrZqKs=; h=Date:From:To:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=qcWpMlMBVp+dh8wLV1CAjaXeE+GGYhVhocXsae3VdSnxvF3rZc0DRGPH214LGWIDiT8kpaEaoLTyZoF/wIFIpPcryRW01YyEE4dHJMqB4bzEFQJhlpC2nILP9gR6rWW52WJCoeO1R/yMGWybHyKXKvcVV6n6iiDX3zNDI6DpPlk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=a7ffCnrU; arc=none smtp.client-ip=209.85.219.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="a7ffCnrU" Received: by mail-qv1-f48.google.com with SMTP id 6a1803df08f44-6a04bae513cso18198536d6.1 for ; Thu, 23 May 2024 09:38:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1716482307; x=1717087107; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=C9H17QM3gRZ3brFw2+qTqduwXQbs1ZUdChn/r+Loe8c=; b=a7ffCnrUccPfbOyeLqW1CKNZKB2XM523kouGMb6MLBeLP8Jw2RFXyOMhfpho9WKoeW MWIgv0janAWEoFH7MMmw25LyvoP9bIZ6IJwZ/58rr1H+7bye8HQJZ5KUViT6Si9EHJRW NRS5TcG5uS/3uZSDhRmBJZ25dBYNIa0fe/99A60BI3kpDZdYxS0m1bus0sOtzFhVvZWl 8cXx16G0rrxhh3Zgi6CKFfjDNqOeHExJfjhbvlrfC18NHzF0B2TDFdovYhZTlCKEJ25Q etfoVYK1qz4wuCQCutGB6uDaDI+EHZ1wFcTcoEQ5RDEsxG6atI/vBsd3uFCqiP+vhqY1 Rbrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716482307; x=1717087107; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=C9H17QM3gRZ3brFw2+qTqduwXQbs1ZUdChn/r+Loe8c=; b=M4h6OGOUXlhxkYqUllaDe4nvpXYNvY5g3M4C1C1dQ9evsIaJkfP8q+hqoW6Ys7FVne QmnwlPKZZkmBiSDMhaKzSTRgT0x1nTEjOKDHT2zq+qs18uKzLhXPZAiuf8maQH1gmsRN KjnT5MPr2xxNCdEJvChaNzr2pWVKguFtoPKz+KgeuaQYBFnlg6hKwqYrRjZ2s9lCvayv lPIrHh2fSGuBtyUv/uKYoCdBAOcwY9+CPFszCamOfx9v9N8R+o2BioImN49O/OLKosht w6HCoLCKM5ecPzr6iUaXWji9MqxXQntBBltds1XlpStkSr2tT9Ub9XgLN8B7ZPZgVyum Q8LQ== X-Gm-Message-State: AOJu0YyknAzYrF+z5nKbKuTm5KkJfHxf1ni561fajnx1ou3n1tTsAtaw sP0yI7OWyXRHJ03MJPyTebZVFO4znkyGnDBNv1tDpvNHOc2Z3lIoCIsxgqV32rXhzNOGHXKJF2D X X-Google-Smtp-Source: AGHT+IHKm78l3r2CgFYckcKKmYlfkdKx2DNgmFerjXHeP/DWSCDvo3RXp3bypWX93xdI/kVd/hV7hg== X-Received: by 2002:a05:6214:2c02:b0:6a0:cc6b:14da with SMTP id 6a1803df08f44-6ab80900dd3mr61347316d6.47.1716482307420; Thu, 23 May 2024 09:38:27 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6ab9b4f82d3sm648776d6.48.2024.05.23.09.38.26 for (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 May 2024 09:38:27 -0700 (PDT) Date: Thu, 23 May 2024 12:38:26 -0400 From: Taylor Blau To: git@vger.kernel.org Subject: [PATCH 8/8] pack-bitmap.c: reimplement `midx_bitmap_filename()` with helper Message-ID: <79e3f7f83f4085eab03de6d0809e8b5d1bae4411.1716482279.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Now that we have the `get_midx_filename_ext()` helper, we can reimplement the `midx_bitmap_filename()` function in terms of it. Signed-off-by: Taylor Blau --- pack-bitmap.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/pack-bitmap.c b/pack-bitmap.c index 35c5ef9d3c..fe8e8a51d3 100644 --- a/pack-bitmap.c +++ b/pack-bitmap.c @@ -309,9 +309,8 @@ static int load_bitmap_entries_v1(struct bitmap_index *index) char *midx_bitmap_filename(struct multi_pack_index *midx) { struct strbuf buf = STRBUF_INIT; - - get_midx_filename(&buf, midx->object_dir); - strbuf_addf(&buf, "-%s.bitmap", hash_to_hex(get_midx_checksum(midx))); + get_midx_filename_ext(&buf, midx->object_dir, get_midx_checksum(midx), + MIDX_EXT_BITMAP); return strbuf_detach(&buf, NULL); }