From patchwork Tue Aug 6 15:36:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13755133 Received: from mail-yw1-f175.google.com (mail-yw1-f175.google.com [209.85.128.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4BE8917BD9 for ; Tue, 6 Aug 2024 15:36:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958606; cv=none; b=FXeVLwnuuZTc7WRLm9l9RgRkPDT+56w5woN1L1tvJvrv5J6MTBqI5J3t64pkkMNNJ3LdfyeR03Blbah9laDH0yHXIGYk1rr/xkR2mEGm9rqPmTX608KRiK8JvSn6tACu5rXEtznif8b+/+GJ5kSm6++UT81pytpmHYZLhVHczqI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958606; c=relaxed/simple; bh=+8q1ugDEV/tnnBF090tg6c121DuKfoi7HlXYjvqI11s=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=j1Qi+BvOfDlVaoaPXaBQHYeZn5UOxz/FpcY6GxeW1y8KeSPjIFBqwx9iDDxwRvFv8vVMiCq+P/bP1xYfGTaGPrETBkWFgzkKYD+Oeg/EZnTuIegieaSDxUvHJf5NvZCs2vtscYg5/fa+IqQsFYErGZlNOCNhL41uhIFavmKaWRs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=NxbY4wQZ; arc=none smtp.client-ip=209.85.128.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="NxbY4wQZ" Received: by mail-yw1-f175.google.com with SMTP id 00721157ae682-6510c0c8e29so8133857b3.0 for ; Tue, 06 Aug 2024 08:36:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1722958603; x=1723563403; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=+BYfeeuZWteNS4zNSvLlmmyY5kI6hua9LVyJMTOfSYg=; b=NxbY4wQZoC3WR1Qo2sUI4tE69t7m6dNWZk88rmh/PJnO9mdqk5eQ4jrAMHkxGAtpI+ b2ns5cJ9T6q8ZimOS5yGz8lVo42XGyWvLYRARuG4gvIK01xe/5ESquOhhztCgp/1ZzIC QblztpNPsurKTCAByByZss/Jike1Qor/KkGx1jbnuzZ4fLYbCwsVrBXtuvSBA+CVjg7L /2soGrFBkIMSO2JLGLTbr9qUio/B+u2UZn3ATj8Ty3KEUTBRzdzfiqJxSGqs8gJp0w8w ACZx6niEvE4eTrB+ZR0R729vfuBNpiDjH8EygZs+fQKNDHEMT8IlPQuWVtWUjMpGrr8O z9BQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722958603; x=1723563403; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=+BYfeeuZWteNS4zNSvLlmmyY5kI6hua9LVyJMTOfSYg=; b=B6+rt+57cR29MLdJaMLulpGr1VPlo8cueYkp2heB8aLYiG4g6V+LIdrIqQVLc1Z/fN g4o8bXmJUQakSnqAmwoyCqpiMKDtqMTS463jtNdwsCcM04WFukOal94oQQPyi2sUQjoA D1XYRmFA2lLpvmsWLo+COqERHESU1C8f0eD3W8mw+LBSIN1aQB+z9FfqnotPk5doJ0Tc NJgojPEN6QNc0iFNzKBFY/dPETRFx4AROUmexCfnpHbqrmYGwOS9Z200UHD/qxdMXpCD 6GPlM5ssgA+FZbLmdL6T3AD4kHhmq9i+nr3wF47b4e0isZ6jSRYCGPjvScWH9lPbklWg kXvQ== X-Gm-Message-State: AOJu0Yz4WDUmKdhmniZwyl7I750m3/YXg+U8WrcIBFMlDh+wUT/J86Fj Ru2irWbLpikynde4ci1xYUHD97N4GLpN4lW4Feh/kr8KM/2jBu1BpN6M4ZbN7DoTIH6YRrlBEr7 n X-Google-Smtp-Source: AGHT+IHut5q1Bc+j/056RMaTrZrsDSFB+nMNYnI15wO6oqN9yPluZ6YUH/vxZNokUO5hvPLO3hJLZw== X-Received: by 2002:a81:5b02:0:b0:62c:f82b:553f with SMTP id 00721157ae682-68963706da8mr160537397b3.31.1722958602972; Tue, 06 Aug 2024 08:36:42 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 00721157ae682-68a12d138e1sm15738147b3.87.2024.08.06.08.36.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Aug 2024 08:36:42 -0700 (PDT) Date: Tue, 6 Aug 2024 11:36:41 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v3 01/19] Documentation: describe incremental MIDX format Message-ID: <90b21b11ed7ba11e58a3bfda2c70672f0d7fa16d.1722958596.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Prepare to implement incremental multi-pack indexes (MIDXs) over the next several commits by first describing the relevant prerequisites (like a new chunk in the MIDX format, the directory structure for incremental MIDXs, etc.) The format is described in detail in the patch contents below, but the high-level description is as follows. Incremental MIDXs live in $GIT_DIR/objects/pack/multi-pack-index.d, and each `*.midx` within that directory has a single "parent" MIDX, which is the MIDX layer immediately before it in the MIDX chain. The chain order resides in a file 'multi-pack-index-chain' in the same directory. Signed-off-by: Taylor Blau --- Documentation/technical/multi-pack-index.txt | 103 +++++++++++++++++++ 1 file changed, 103 insertions(+) diff --git a/Documentation/technical/multi-pack-index.txt b/Documentation/technical/multi-pack-index.txt index f2221d2b44..cc063b30be 100644 --- a/Documentation/technical/multi-pack-index.txt +++ b/Documentation/technical/multi-pack-index.txt @@ -61,6 +61,109 @@ Design Details - The MIDX file format uses a chunk-based approach (similar to the commit-graph file) that allows optional data to be added. +Incremental multi-pack indexes +------------------------------ + +As repositories grow in size, it becomes more expensive to write a +multi-pack index (MIDX) that includes all packfiles. To accommodate +this, the "incremental multi-pack indexes" feature allows for combining +a "chain" of multi-pack indexes. + +Each individual component of the chain need only contain a small number +of packfiles. Appending to the chain does not invalidate earlier parts +of the chain, so repositories can control how much time is spent +updating the MIDX chain by determining the number of packs in each layer +of the MIDX chain. + +=== Design state + +At present, the incremental multi-pack indexes feature is missing two +important components: + + - The ability to rewrite earlier portions of the MIDX chain (i.e., to + "compact" some collection of adjacent MIDX layers into a single + MIDX). At present the only supported way of shrinking a MIDX chain + is to rewrite the entire chain from scratch without the `--split` + flag. ++ +There are no fundamental limitations that stand in the way of being able +to implement this feature. It is omitted from the initial implementation +in order to reduce the complexity, but will be added later. + + - Support for reachability bitmaps. The classic single MIDX + implementation does support reachability bitmaps (see the section + titled "multi-pack-index reverse indexes" in + linkgit:gitformat-pack[5] for more details). ++ +As above, there are no fundamental limitations that stand in the way of +extending the incremental MIDX format to support reachability bitmaps. +The design below specifically takes this into account, and support for +reachability bitmaps will be added in a future patch series. It is +omitted from the current implementation for the same reason as above. ++ +In brief, to support reachability bitmaps with the incremental MIDX +feature, the concept of the pseudo-pack order is extended across each +layer of the incremental MIDX chain to form a concatenated pseudo-pack +order. This concatenation takes place in the same order as the chain +itself (in other words, the concatenated pseudo-pack order for a chain +`{$H1, $H2, $H3}` would be the pseudo-pack order for `$H1`, followed by +the pseudo-pack order for `$H2`, followed by the pseudo-pack order for +`$H3`). ++ +The layout will then be extended so that each layer of the incremental +MIDX chain can write a `*.bitmap`. The objects in each layer's bitmap +are offset by the number of objects in the previous layers of the chain. + +=== File layout + +Instead of storing a single `multi-pack-index` file (with an optional +`.rev` and `.bitmap` extension) in `$GIT_DIR/objects/pack`, incremental +MIDXs are stored in the following layout: + +---- +$GIT_DIR/objects/pack/multi-pack-index.d/ +$GIT_DIR/objects/pack/multi-pack-index.d/multi-pack-index-chain +$GIT_DIR/objects/pack/multi-pack-index.d/multi-pack-index-$H1.midx +$GIT_DIR/objects/pack/multi-pack-index.d/multi-pack-index-$H2.midx +$GIT_DIR/objects/pack/multi-pack-index.d/multi-pack-index-$H3.midx +---- + +The `multi-pack-index-chain` file contains a list of the incremental +MIDX files in the chain, in order. The above example shows a chain whose +`multi-pack-index-chain` file would contain the following lines: + +---- +$H1 +$H2 +$H3 +---- + +The `multi-pack-index-$H1.midx` file contains the first layer of the +multi-pack-index chain. The `multi-pack-index-$H2.midx` file contains +the second layer of the chain, and so on. + +When both an incremental- and non-incremental MIDX are present, the +non-incremental MIDX is always read first. + +=== Object positions for incremental MIDXs + +In the original multi-pack-index design, we refer to objects via their +lexicographic position (by object IDs) within the repository's singular +multi-pack-index. In the incremental multi-pack-index design, we refer +to objects via their index into a concatenated lexicographic ordering +among each component in the MIDX chain. + +If `objects_nr()` is a function that returns the number of objects in a +given MIDX layer, then the index of an object at lexicographic position +`i` within, say, $H3 is defined as: + +---- +objects_nr($H2) + objects_nr($H1) + i +---- + +(in the C implementation, this is often computed as `i + +m->num_objects_in_base`). + Future Work ----------- From patchwork Tue Aug 6 15:36:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13755134 Received: from mail-yw1-f171.google.com (mail-yw1-f171.google.com [209.85.128.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4DAD01799B for ; Tue, 6 Aug 2024 15:36:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958608; cv=none; b=dr9VlZYGH2GJJrgn2TAVvVR7B/YlPLtyNE4K/AR1+qRQR+gIT/CslRkqZDYISyL56jpGyYupsIEM+Ta1qHMLhHLyHUA2C+U+aKPCbETJVaJg0/Rjox6HLcrfiBXyfTmeXdewWqZV8MiWhJBZ3ZTLlm583SXFCwshLBNPgZU1xhU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958608; c=relaxed/simple; bh=WYRKYLDHjvT9vT9KG2Me0U6hkzKbLoYSQYS6vpqqTIc=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=e7DYASkHlJwfRY3J7cYt/zA3lNs0ujxqcwvb5NSGtjYKzyLZuiLxsMxtNX5/CeNQlenkYtLJDEcQBuyZ94AfkEuoRTYJ02Yd6dNb/u0t56GI3FGUD6S8OqGhQeAQ/tEZmjy2Sv0NjI6rekszH2tav8t7aBEu3HWDEucIxEG8wds= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=klhojAlx; arc=none smtp.client-ip=209.85.128.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="klhojAlx" Received: by mail-yw1-f171.google.com with SMTP id 00721157ae682-66acac24443so7348107b3.1 for ; Tue, 06 Aug 2024 08:36:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1722958606; x=1723563406; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=FBvHRKHw/kNLS3YJbTE9AEvtdTYxS0fv16LncCXaCj8=; b=klhojAlx9SPodLXPX/A12S+8TDeEnJ3wpvy4r8Inru04EuFv3pb59pMxwEwBeMhVBM cIYzuDgugpgmpF0dN80Hj6eyxKOymRx8TYY0DcK+iMZvRMbtIjNR1Sktny7Knry17kBX ToQfXBr8cGgaQrtEorsgMi6XUZmvFCj0uVescv35yVguKHkApWBnlAun+5A3MgF/9tct cqyYlJcAYf+KFtG9IFermtG9hfD1PCmOqT01X2zAMKvMZrAsXmG8CenUCJnqx+Kh+x35 Plg4hyYjoRYAKOk7zfzWYcLdvcwEwQ1fAcfMf+jfqMkl8o1SFW6C0gXAgDMjHElcrIog ELBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722958606; x=1723563406; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=FBvHRKHw/kNLS3YJbTE9AEvtdTYxS0fv16LncCXaCj8=; b=K28ICpWHG/GcwirWYT3E3XPOr0/lYF/UdASkxMuHvN6RUq59/1sB/jy3gElzM6V9/+ KYbaIgZEt9WySNnTtrLpzqFnvDXY6t2OY8mLhAIfJq1SZt4jfrRN66qG2M7Q8fyo1Lnk f/0zRQL3fJ6v2nUhZoFEB8pvHWlkYZGxIqQaOf0QneuEbCY34p6uSXimCNiQIhgNsc48 p6gIuMEnQ5wyAbEg1p/ahxmsGGR3NeE4eq1+Mtcw5yzW3aJj/cfo6XBr85dKGeo0lJKu sWOrmU9kmzgU5u9jRgHjl5MRkd/d87kM2e7mpxEbPXkvH/sciBc+ZDYfs8OgUR0cV1c8 zsmA== X-Gm-Message-State: AOJu0Yxuwscx7LvETZY0rxT/W/lA0z69B59LX+DYBNGPxh3NYGONJEyh Qf893Ty1DXSFpZTI/+/BJ3dZa+s/C0SNjkrr0GJK8WXNgLQLRnE2BjX8Z/ZUnRn4rcHc9V6SK30 H X-Google-Smtp-Source: AGHT+IEtkvaFD84afgjY5aOzVXOEJuYnIRdBZYfUQGW3ur8wgx0DmukHNno11VFL5pThEtckLmuobQ== X-Received: by 2002:a81:9c07:0:b0:650:82e0:63b1 with SMTP id 00721157ae682-689641a6125mr171241277b3.41.1722958606106; Tue, 06 Aug 2024 08:36:46 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 00721157ae682-68a135ca5c0sm15956127b3.117.2024.08.06.08.36.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Aug 2024 08:36:45 -0700 (PDT) Date: Tue, 6 Aug 2024 11:36:44 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v3 02/19] midx: add new fields for incremental MIDX chains Message-ID: <0d3b19c59ff31eb17d283c61f615957e89137fb1.1722958596.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The incremental MIDX chain feature is designed around the idea of indexing into a concatenated lexicographic ordering of object IDs present in the MIDX. When given an object position, the MIDX machinery needs to be able to locate both (a) which MIDX layer contains the given object, and (b) at what position *within that MIDX layer* that object appears. To do this, three new fields are added to the `struct multi_pack_index`: - struct multi_pack_index *base_midx; - uint32_t num_objects_in_base; - uint32_t num_packs_in_base; These three fields store the pieces of information suggested by their respective field names. In turn, the `num_objects_in_base` and `num_packs_in_base` fields are used to crawl backwards along the `base_midx` pointer to locate the appropriate position for a given object within the MIDX that contains it. The following commits will update various parts of the MIDX machinery (as well as their callers from outside of midx.c and midx-write.c) to be aware and make use of these fields when performing object lookups. Signed-off-by: Taylor Blau --- midx.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/midx.h b/midx.h index 8554f2d616..020e49f77c 100644 --- a/midx.h +++ b/midx.h @@ -63,6 +63,10 @@ struct multi_pack_index { const unsigned char *chunk_revindex; size_t chunk_revindex_len; + struct multi_pack_index *base_midx; + uint32_t num_objects_in_base; + uint32_t num_packs_in_base; + const char **pack_names; struct packed_git **packs; char object_dir[FLEX_ARRAY]; From patchwork Tue Aug 6 15:37:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13755135 Received: from mail-yw1-f180.google.com (mail-yw1-f180.google.com [209.85.128.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 66CED1799B for ; Tue, 6 Aug 2024 15:37:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958642; cv=none; b=iqiiIZAdYHoihbHVZYBwxzbj9ZpTOzFQPfQ+SjyqoXP4kgTYfMOewQ2TYmCAPi6U4X1BkfR/fJqRqEXE7/DbNiYDi2pzMznG+JEQMeosa2zcHdcvshOYyYspC5dc13aiQLqPaffmJAfSS9ZzF1fMfDMOc3f4nEScOyN5tUzuQFE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958642; c=relaxed/simple; bh=T2ABrx9nYv8rEbdKuGVBmkdIaR0TnfAFUkN7OvTioq4=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=LMaphLMhEg+BRk1BxC9k01BltvNJOnqp9LgZdXi6jxeXkBHgt8BFSOPyCM2uvUdc9w2sgwXNwHf0gb91AFbz3MRpEBnCd9QbtAxDc7fSZje/rrPpq+ghqeWo8sgwLQiGAWVm75xf/OVbRBrmt2cOsYSTZeJR47Ux19VTnEaFMGI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=mpHpidRl; arc=none smtp.client-ip=209.85.128.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="mpHpidRl" Received: by mail-yw1-f180.google.com with SMTP id 00721157ae682-68aa9c894c7so7471877b3.1 for ; Tue, 06 Aug 2024 08:37:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1722958640; x=1723563440; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=MkDZ8LohH7UWQYLGoTItOAFXWqRR6j/yiQyB0dNtufU=; b=mpHpidRlXlMVcT1NkCO9djOu1FZGjI4zGn3avzv++4tKgu+laxCn8hVuATF+ffDrB6 MR6s8auCxU/q2FQPAJU8URkwk/TGfmkEyDeY/E9Ntax2azJKPSVkYlKyH4jH0gEcf74U KF1jHFK76DcKs7IJbXxKdkyQxcjGJIlcA3XnxR3EYJ8KPhMfdo97NM9a0NBtw5ZPMDAX XWjEWoXqppwe5Rtlo0GFPGHwec99HiBpScJpsQMsz9vHGToU0qPZ2m6b2AXWjXyA6UTx dU2L9Yr1LfDecXn2WVB844SRrU7zmszq/AE5jR/zCo1Jc+fBiguVjt7yH4kYVjlTNe3t ICvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722958640; x=1723563440; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=MkDZ8LohH7UWQYLGoTItOAFXWqRR6j/yiQyB0dNtufU=; b=hs6ELHIjsYvrxHmauPiQ09ii7t7BUz6N9ZygqpERu9+9TCs9tzyBHvsfP0o23x2qMw +cQ3GlFRZNFlTni7k2w6XST0SKmFkfe711eX1Ctp2RR7MVkNiXEQiy2VCP5yrGHnZihe CYXavSZP3LbwF8E8FuibvmRpWae9xzPuBSh8pb2str0Pi6ODkxpR+39dOMPrl08pOqSx xnM/THGJgoZW2WqqJaZGiVBZ5iDGXdgFDgNDKWe8bsQAQJovk+OCCIB5EmU9uSKxHCzd gt1ZKFhqXKQIoU6VbKdO5yE6qm7M+FOz8otCWyq2Z4Bh0d+jFGZWbhjOgv0ov7NA9M8M 4Dhg== X-Gm-Message-State: AOJu0YxBNF1fWGRzqLJl4FMSJvlrlQjj9wSYOt2NoZdv5Fcoyxr6y/yL WzMazsTYkQK/6B7yv15HgE9+I1gxo+4z6BphTLOHQYdxjhc9p1o6yhnBvl/kaWGsslUr6iBVNid C X-Google-Smtp-Source: AGHT+IHHeOKPtouuiqB7ks4gp8ne3z8BWUWNuclvpfH4Rc2NAsv6O9WhEaeVv3TqUkOCLwi2nyRW+w== X-Received: by 2002:a81:8901:0:b0:630:f7c9:80d6 with SMTP id 00721157ae682-6896160e208mr162328677b3.27.1722958640090; Tue, 06 Aug 2024 08:37:20 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 00721157ae682-68a136bfa21sm15902177b3.119.2024.08.06.08.37.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Aug 2024 08:37:19 -0700 (PDT) Date: Tue, 6 Aug 2024 11:37:18 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v3 03/19] midx: teach `nth_midxed_pack_int_id()` about incremental MIDXs Message-ID: <5cd742b67761913bd9693102a8b823f353b60f7b.1722958596.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The function `nth_midxed_pack_int_id()` takes in a object position in MIDX lexicographic order and returns an identifier of the pack from which that object was selected in the MIDX. Currently, the given object position is an index into the lexicographic order of objects in a single MIDX. Change this position to instead refer into the concatenated lexicographic order of all MIDXs in a MIDX chain. This has two visible effects within the implementation of `prepare_midx_pack()`: - First, the given position is now an index into the concatenated lexicographic order of all MIDXs in the order in which they appear in the MIDX chain. - Second the pack ID returned from this function is now also in the concatenated order of packs among all layers of the MIDX chain in the same order that they appear in the MIDX chain. To do this, introduce the first of two general purpose helpers, this one being `midx_for_object()`. `midx_for_object()` takes a double pointer to a `struct multi_pack_index` as well as an object `pos` in terms of the entire MIDX chain[^1]. The function chases down the '->base_midx' field until it finds the MIDX layer within the chain that contains the given object. It then: - modifies the double pointer to point to the containing MIDX, instead of the tip of the chain, and - returns the MIDX-local position[^2] at which the given object can be found. Use this function within `nth_midxed_pack_int_id()` so that the `pos` it expects is now relative to the entire MIDX chain, and that it returns the appropriate pack position for that object. [^1]: As a reminder, this means that the object is identified among the objects contained in all layers of the incremental MIDX chain, not any particular layer. For example, consider MIDX chain with two individual MIDXs, one with 4 objects and another with 3 objects. If the MIDX with 4 objects appears earlier in the chain, then asking for object 6 would return the second object in the MIDX with 3 objects. [^2]: Building on the previous example, asking for object 6 in a MIDX chain with (4, 3) objects, respectively, this would set the double pointer to point at the MIDX containing three objects, and would return an index to the second object within that MIDX. Signed-off-by: Taylor Blau --- midx.c | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/midx.c b/midx.c index 3992b05465..39d358da20 100644 --- a/midx.c +++ b/midx.c @@ -242,6 +242,23 @@ void close_midx(struct multi_pack_index *m) free(m); } +static uint32_t midx_for_object(struct multi_pack_index **_m, uint32_t pos) +{ + struct multi_pack_index *m = *_m; + while (m && pos < m->num_objects_in_base) + m = m->base_midx; + + if (!m) + BUG("NULL multi-pack-index for object position: %"PRIu32, pos); + + if (pos >= m->num_objects + m->num_objects_in_base) + die(_("invalid MIDX object position, MIDX is likely corrupt")); + + *_m = m; + + return pos - m->num_objects_in_base; +} + int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id) { struct strbuf pack_name = STRBUF_INIT; @@ -334,8 +351,10 @@ off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos) uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos) { - return get_be32(m->chunk_object_offsets + - (off_t)pos * MIDX_CHUNK_OFFSET_WIDTH); + pos = midx_for_object(&m, pos); + + return m->num_packs_in_base + get_be32(m->chunk_object_offsets + + (off_t)pos * MIDX_CHUNK_OFFSET_WIDTH); } int fill_midx_entry(struct repository *r, From patchwork Tue Aug 6 15:37:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13755136 Received: from mail-yw1-f171.google.com (mail-yw1-f171.google.com [209.85.128.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 816F21799B for ; Tue, 6 Aug 2024 15:37:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958645; cv=none; b=SLCXkqeabLp+ET9nyiPEJl+X/kzcUqly37NfUoYOux/Yomgz6BXDcvL+uk21/fS9uGN1HMCQBK9TS6cFISzgXlRo0d9xKx73Y+SiZfxDfnUIcRMuIgJ2g1y1AMHBdS7nbCUBwY+V4zfm45LplV+UUyI9gWo9KRYU/6dx7xS3yq4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958645; c=relaxed/simple; bh=5h2V6oQr4RtflglTo8kZZr5xy3UJTt9j6m7rESDDBio=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=C+Virjqq92HMWE9YX+04EdytZ/yXwdM5TfFAVdFjmsZ806hZUGAZxw2rVMcWynVUIDBAVRiaeqN93sUE2QbdAKDG4ox2TqWAVInjWlO3tXbrJljNofCTfI4RW3yqdZfxiZx5oHgiZ3OL09gR4+JVGmoUDdPf+/mQ2mbgfgtaCPI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=sKJ6KtAu; arc=none smtp.client-ip=209.85.128.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="sKJ6KtAu" Received: by mail-yw1-f171.google.com with SMTP id 00721157ae682-66108213e88so7225327b3.1 for ; Tue, 06 Aug 2024 08:37:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1722958643; x=1723563443; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=3J/MlDsCQ3mjy9gHP2REiPbO5oTcjMjayOcVr3EoWfY=; b=sKJ6KtAusAijF9fDDniEf3OqiOFxoFglBh9Uciz6MSuK/eudZQbluw8ICkvMA+Gs8+ XbeQJjlbow9Ia/dvrdWo/f35B1i1Q5eqadBb8XUJ7v1EIPUctzkiJVRoDaSXwD1JcT4S jVsfec1Y0PrIHlSmaRUyNCCSRsuCBGreu7g0BpQwlJEkpDOUBGA+WKfKOh1XfmoMHABe cfJ5rRKEzT8MHmi1wFj6MKp4HpFFOSXEL/8oG3lPISvi3p5zsdvfGrwJprRQchubZrOT XhanTpsmJ25FaMB2wNa7ql3GJnml0uvRxyhRon6gGUkwdpHRLEUQYnLC/JZ7BmW7PhOW EnmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722958643; x=1723563443; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=3J/MlDsCQ3mjy9gHP2REiPbO5oTcjMjayOcVr3EoWfY=; b=J13zaYN41zqmmexT4p7ZlqwPf5mO/lLa2pcy7nphoB9pQao+vjdC/MvetBnB003i/J CNpUJ1eqH5JmYI5GH+CAwpFVFoHB0EaS8gCwxPYL6V3XXGroIIz2Ra+QgWUAf1eM3S8R crzODi+X4kf6rLigNAWpfOQBuAU7FxAFlxesJ8HBA2FXhC7ey+72So9agMNGI2b4nzOs 1TasZKNV/Am7RlwQDRBMZCQTGoa8FnPdj6S+YQpcu2c+imGvBwlHqws40POXRGUnm23D +rqPSrtPSU/vAzQKOiab0H1r73HxitShZnavkBlUi9+a8LEVcAe2Hhqbg5QSc4o6zx/6 Tnqg== X-Gm-Message-State: AOJu0YyV3n9wK+0Yvo+wPpJcOWZ+ovCO5qa/D+A8p/iM5OT2qsKdzgf7 4PHuqQ2lBbqNufkuJUx9UnkB/Fm07ckZkckU7yYCblxweicUCj5BQ1jJ4w/wEAKaNrVwx7Ljp7v a X-Google-Smtp-Source: AGHT+IFvdHIjp/lekRl0vW2m6YA7zanP9ZvqUSxnwweAHw4mSAXLZP9wewmInE4SvkJDs5FCHn/1bg== X-Received: by 2002:a05:690c:b1a:b0:664:c24f:c248 with SMTP id 00721157ae682-696c48ea44amr30809357b3.32.1722958643141; Tue, 06 Aug 2024 08:37:23 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 00721157ae682-68d2a3e037esm11532217b3.4.2024.08.06.08.37.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Aug 2024 08:37:22 -0700 (PDT) Date: Tue, 6 Aug 2024 11:37:21 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v3 04/19] midx: teach `prepare_midx_pack()` about incremental MIDXs Message-ID: <372104c73ded1dd7035b16f5aa48827560c4e064.1722958596.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The function `prepare_midx_pack()` is part of the midx.h API and loads the pack identified by the MIDX-local 'pack_int_id'. This patch prepares that function to be aware of an incremental MIDX world. To do this, introduce the second of the two general purpose helpers mentioned in the previous commit. This commit introduces `midx_for_pack()`, which is the pack-specific analog of `midx_for_object()`, and works in the same fashion. Like `midx_for_object()`, this function chases down the '->base_midx' field until it finds the MIDX layer within the chain that contains the given pack. Use this function within `prepare_midx_pack()` so that the `pack_int_id` it expects is now relative to the entire MIDX chain, and that it prepares the given pack in the appropriate MIDX. Signed-off-by: Taylor Blau --- midx.c | 28 +++++++++++++++++++++++----- 1 file changed, 23 insertions(+), 5 deletions(-) diff --git a/midx.c b/midx.c index 39d358da20..07b3981a7a 100644 --- a/midx.c +++ b/midx.c @@ -259,14 +259,32 @@ static uint32_t midx_for_object(struct multi_pack_index **_m, uint32_t pos) return pos - m->num_objects_in_base; } -int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id) +static uint32_t midx_for_pack(struct multi_pack_index **_m, + uint32_t pack_int_id) { - struct strbuf pack_name = STRBUF_INIT; - struct packed_git *p; + struct multi_pack_index *m = *_m; + while (m && pack_int_id < m->num_packs_in_base) + m = m->base_midx; - if (pack_int_id >= m->num_packs) + if (!m) + BUG("NULL multi-pack-index for pack ID: %"PRIu32, pack_int_id); + + if (pack_int_id >= m->num_packs + m->num_packs_in_base) die(_("bad pack-int-id: %u (%u total packs)"), - pack_int_id, m->num_packs); + pack_int_id, m->num_packs + m->num_packs_in_base); + + *_m = m; + + return pack_int_id - m->num_packs_in_base; +} + +int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, + uint32_t pack_int_id) +{ + struct strbuf pack_name = STRBUF_INIT; + struct packed_git *p; + + pack_int_id = midx_for_pack(&m, pack_int_id); if (m->packs[pack_int_id]) return 0; From patchwork Tue Aug 6 15:37:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13755137 Received: from mail-yw1-f176.google.com (mail-yw1-f176.google.com [209.85.128.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 71A0F1799B for ; Tue, 6 Aug 2024 15:37:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958648; cv=none; b=FajtsysBlcSz0yKhwdVG2dCs+6H+9S1av8cBhqx9/1+b0xCEArqaL5KPNuw1ZSYa6NQREL7u5CdTspGAZsHioSViIRl1G90ESr70/L/A2p4jlcT5dr7U4PVnKd88zNXvLQ+nsk0ao83rt9H7VZjqnE1IhNBBw/Wpo3UxczC/+oU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958648; c=relaxed/simple; bh=0QTD/S3loZbpDanEny7YsLfKvDpI20eU36WXVz8jNvE=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=sMx3GubNNpSBGlPW+MIiIZMX+TkdRZcYCcljcKD2HoO6CLAAsgW5bbQ8s1cbJi5Fx+QtDj98xUU+eF+luyq/fIy9KO1Gi5qfsPp1lNa9Ga9X54Lh5ggpu5vDu/ySl+dn1e5msXTgX3NOzDafvdqJvAYRXcEh85VVjd9iQtTbvRI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=BseqLjA9; arc=none smtp.client-ip=209.85.128.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="BseqLjA9" Received: by mail-yw1-f176.google.com with SMTP id 00721157ae682-690af536546so8783307b3.3 for ; Tue, 06 Aug 2024 08:37:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1722958646; x=1723563446; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=K+jJH0I0zUZUGxrC82+YHBH2XRXd0I0zLiSIY0smqO8=; b=BseqLjA9y5I7kJoUNhQUhrMdjTT+1/BVpCPCkGUviGAxXQwUCvN3VJnQ583XRBVzpK bNRr2gajHx53vfUDzgfDNjEInyfMuxAFKZ5OT5Rg40q1DeNQ+c5NHhj5cOowl7YoI9VW KutrmVPG64gMU5zZ3SPnbLkgnAkWR3we1KUuAhMe0EzQN0CsTbpOSyrc3GsDAQO/ZAsD Rzd6qNTPGRU8OK5ssPyDh+JwXi9YtLrEVgLw0LbFCscEBAa9ZegHMJjswbk2lTgIEeJd sgYNUV0E/YGgB1s3vVngmDUAaNKH83mBibZzuL88Waczgj1WThjHXKfACwPOE73u4MhJ 2qYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722958646; x=1723563446; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=K+jJH0I0zUZUGxrC82+YHBH2XRXd0I0zLiSIY0smqO8=; b=JEkRUBATltV0ogDUtJOivkL1QknpL06eGqJW/9z/ZKEgtwFvTXgGDTL3w03TBL9CB+ 88buGLEVWvU/QUqr/6SX+miJjctoPYzYiF0tLykT3vEP/LvFALEA7jqcU3zYr9PBCquL KWbBYt3ttpNeVqwHNeIwpqn9Wfe2kZ0s+3wwUTX4al1bIKVp0rcKHLckNo3o9fR7ZQ4C L6LWos4Hn1VRFALysk1PyNFdkojQB46tb0Ozjp1dxCJSENqZDZ2XyhGn6hxhlpAknwXx 6B5g9yckM2/Zimkkr3CSUC0ERq/m+uVzm7Q3ZG0guPhG9trkkn/OOnYNN4BLh4+N0IpB 3G/Q== X-Gm-Message-State: AOJu0YzJe0lOngVGygDMaWv7W6QrxoTqednFgpyyWIqjfl9bDiQW5Q7W +SmdHQwm4YK41aVAfeZRAqzo9yhXLajZlisTetHw1UGWobOaUWCDQzkW8srTj+V++2yTH6+T2ty U X-Google-Smtp-Source: AGHT+IEZ/SHDxUEZuKtVZQNjq43orXWCvXScpcOM/tT/4bBkeighPItijZHGm5XVmf0uKutJpOfOkg== X-Received: by 2002:a0d:f585:0:b0:687:f1f3:6738 with SMTP id 00721157ae682-68960589767mr170009467b3.19.1722958646209; Tue, 06 Aug 2024 08:37:26 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 00721157ae682-68f3452478csm10809897b3.86.2024.08.06.08.37.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Aug 2024 08:37:25 -0700 (PDT) Date: Tue, 6 Aug 2024 11:37:24 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v3 05/19] midx: teach `nth_midxed_object_oid()` about incremental MIDXs Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The function `nth_midxed_object_oid()` returns the object ID for a given object position in the MIDX lexicographic order. Teach this function to instead operate over the concatenated lexicographic order defined in an earlier step so that it is able to be used with incremental MIDXs. To do this, we need to both (a) adjust the bounds check for the given 'n', as well as record the MIDX-local position after chasing the `->base_midx` pointer to find the MIDX which contains that object. Signed-off-by: Taylor Blau --- midx.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/midx.c b/midx.c index 07b3981a7a..64a051cca1 100644 --- a/midx.c +++ b/midx.c @@ -338,9 +338,11 @@ struct object_id *nth_midxed_object_oid(struct object_id *oid, struct multi_pack_index *m, uint32_t n) { - if (n >= m->num_objects) + if (n >= m->num_objects + m->num_objects_in_base) return NULL; + n = midx_for_object(&m, n); + oidread(oid, m->chunk_oid_lookup + st_mult(m->hash_len, n), the_repository->hash_algo); return oid; From patchwork Tue Aug 6 15:37:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13755138 Received: from mail-yw1-f173.google.com (mail-yw1-f173.google.com [209.85.128.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 98BA21799B for ; Tue, 6 Aug 2024 15:37:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958652; cv=none; b=kU9f0nzh/iMNrnWvi/VDHiJPYDFtEGusmAANnceEyKIz6Te5NQr6o4Mexnj+CC7OEM8TkIXLmjXMxf5vCufpnmSy0mC5TY8BtGcAZLKh5K/TIPzh+AZd0qIM70CFkTPCWKbImoLDB3OszyzA4nE8CfIMAkQ7ZZO8cYBDzXJNvBA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958652; c=relaxed/simple; bh=mc8zUPuWI14y/tvtqfFhMpcqObJ/bAciaX50ecgUFPM=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=e3dYd8/VcBnadpgQ22M8ESLyaTSAU7zZZ9+FplnPR2GQtRN3oh2KH7aqh6jMidGYyFkm1yVfVTrWgP01owqv7behmaa2cpPszqAmz5pw/RlZ+YoVtdaD/aUiVuAmX3RXwF1esga3OzS9PTt7vrabgxyeCX32b6zRsEg16GhtnWE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=N7aQegov; arc=none smtp.client-ip=209.85.128.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="N7aQegov" Received: by mail-yw1-f173.google.com with SMTP id 00721157ae682-68d30057ae9so6677467b3.1 for ; Tue, 06 Aug 2024 08:37:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1722958649; x=1723563449; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=M4YvSew0Q8gKBZ+mOfZdVawRl7Abz3wViJddgQGB2Ks=; b=N7aQegovUp77iUCZNvuGKyJyLv6wUPpKV4qb3fRTiWHOv6YknvJmJjZY47MMv2iyyQ RKXiT3HqBexU5NpxvyBv12Cev+x1DMkQDoeWBmIZ2Xqp5tC0/dfKhtYqyfeB5r2mg/NT KXAi+mhhvV1LYkzDhCSQIB+nRKFYUHM/gynPpyg5V103P9PXYJpI1dojoau4+srLBDzY oJQnBLEmrwWufmlf8WsW6MHmZEmaFSimvnGNKpjDH7KuPusEYpINaZP7T3KOI3791sOV pzpf8atYAaR4i7wa8ATIIzLiEgC7YKqc9wkrmhbuBTpEll0Di1ApHLIW78TYSPfkI1sN 9JIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722958649; x=1723563449; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=M4YvSew0Q8gKBZ+mOfZdVawRl7Abz3wViJddgQGB2Ks=; b=Z8QR8W99V1S0FMVLLMnEbu0LeeqjUD7cKGOOsei4La1LGkhzaMPgdwzTqa6nd2Ue9a lssetXpn45/6S1WMm5LsO63F1fysLi5EVWosNSqbhZn6YiYYQb2jcUA9kyluGebZ/qLc ahYOFJarBr+NJvoD/8y7W/+nIPdvr3wP2KPD6LN4ZfqgCdi58rf02dGx3ImPhulZR/oN Iz7FBDs3EZsSKkaPUd5jN678xItViDF2vvvsQrYfRkvQ+5Cs5dMCyFcTVGpNd3OkefDX YE/IZRWwWBLtCoSRENh9yHjJDiW9Fx+tKeuN4/xBN3FHVKDsMkK6s/yKR4Q0X68zThIW zbUg== X-Gm-Message-State: AOJu0YxAttqT4NbxFIOUqp30OpRBrT+55UywRNJ1uRnfVRZ/WaiD81Kx AP8CsgcCpax8FK3LmS4mc00KvtKPemJnNus5957XKbTEk4PnDImb47OkAMBLEKMAoNkJ+a8RlPb D X-Google-Smtp-Source: AGHT+IGC5nPE557Mj6UIRsBmoOXH3p20xBRU0FRvU1lVP7sTFZ7IUqSUA/LKq6gBZvaEhsxKNjXJBw== X-Received: by 2002:a0d:efc3:0:b0:65f:93c1:fee9 with SMTP id 00721157ae682-6895f60d519mr163826717b3.9.1722958649243; Tue, 06 Aug 2024 08:37:29 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 00721157ae682-68a0fcbe907sm15813377b3.30.2024.08.06.08.37.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Aug 2024 08:37:28 -0700 (PDT) Date: Tue, 6 Aug 2024 11:37:27 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v3 06/19] midx: teach `nth_bitmapped_pack()` about incremental MIDXs Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: In a similar fashion as in previous commits, teach the function `nth_bitmapped_pack()` about incremental MIDXs by translating the given `pack_int_id` from the concatenated lexical order to a MIDX-local lexical position. When accessing the containing MIDX's array of packs, use the local pack ID. Likewise, when reading the 'BTMP' chunk, use the MIDX-local offset when accessing the data within that chunk. (Note that the both the call to prepare_midx_pack() and the assignment of bp->pack_int_id both care about the global pack_int_id, so avoid shadowing the given 'pack_int_id' parameter). Signed-off-by: Taylor Blau --- midx.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/midx.c b/midx.c index 64a051cca1..25350152f1 100644 --- a/midx.c +++ b/midx.c @@ -311,17 +311,19 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m, struct bitmapped_pack *bp, uint32_t pack_int_id) { + uint32_t local_pack_int_id = midx_for_pack(&m, pack_int_id); + if (!m->chunk_bitmapped_packs) return error(_("MIDX does not contain the BTMP chunk")); if (prepare_midx_pack(r, m, pack_int_id)) return error(_("could not load bitmapped pack %"PRIu32), pack_int_id); - bp->p = m->packs[pack_int_id]; + bp->p = m->packs[local_pack_int_id]; bp->bitmap_pos = get_be32((char *)m->chunk_bitmapped_packs + - MIDX_CHUNK_BITMAPPED_PACKS_WIDTH * pack_int_id); + MIDX_CHUNK_BITMAPPED_PACKS_WIDTH * local_pack_int_id); bp->bitmap_nr = get_be32((char *)m->chunk_bitmapped_packs + - MIDX_CHUNK_BITMAPPED_PACKS_WIDTH * pack_int_id + + MIDX_CHUNK_BITMAPPED_PACKS_WIDTH * local_pack_int_id + sizeof(uint32_t)); bp->pack_int_id = pack_int_id; From patchwork Tue Aug 6 15:37:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13755139 Received: from mail-yw1-f175.google.com (mail-yw1-f175.google.com [209.85.128.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DB5C71799B for ; Tue, 6 Aug 2024 15:37:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958655; cv=none; b=TNEAKkwPocD0/LLq/8hHVSuzK0Udw9ekyhrTLytoJfHN3KFqrzoMpevlgRfdI9KjC+Kc0kNXbvpbjdak4IYOCsBjrbvl5ZLuE2nmP54/SH8lxO5WIBecSYbjL0z3bcBz4tz4uPKoK5DKp89t8r9T7mqgEm1SdzVDa323UNFWu1A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958655; c=relaxed/simple; bh=zJpiA6QKm8NFfE3SAZyRzyOdL8EcHTGjp70PTEU+1cE=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=nF2ZQYox/4W0FcSGZBF53GZ9FmPa1PqL1WDsPgvWKD5EAiMkW7NkDsAnDQIwoEz7Y0FGFHkbYvtB0pG4NX8bDkdSRdv1p6/nR76NfvMyJdb1UUrS6ZqpyFKDOJA655hT2rPxqEK4tbQ+wBoPR3bsxkkI7SPTLuS92pZmpzISZiw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=nm8uEm9y; arc=none smtp.client-ip=209.85.128.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="nm8uEm9y" Received: by mail-yw1-f175.google.com with SMTP id 00721157ae682-66ca5e8cc51so6637277b3.1 for ; Tue, 06 Aug 2024 08:37:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1722958652; x=1723563452; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=BW8V281MSbQL4oLIH7m7SiTPBBV9HM8HPjg04kVYmlk=; b=nm8uEm9yMsmtHE2mqB0JtmUxT+teYc6GjePZGXW2+3hwi1Afk8GvKsXrn30e9sGJjx TUXnk68hD0PeItkwhvqDhTrRNUgzXtjwie/8oCFI0iilxo/0Ea/cFnX/LJmUaJAW2Etj 7qKsCWDfCvG7tMBaRp7U6wYrn6Rbu2JtNCzKNBHQb1DlfV3qer5NBATOrJzPHt3VP81u Rggtsb4yOwBouFRzNBCPvRAkk77HI0c3Fxg5zCrkEakqDblZyQHKRmJ6UfG4+vpsuUa6 Di4WCjpQDnUw0J1hYbgcw4J2I1blrSOKCv02yY44+wSMBvwgRL76tT4C3ccCUryMD4xq ceHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722958652; x=1723563452; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=BW8V281MSbQL4oLIH7m7SiTPBBV9HM8HPjg04kVYmlk=; b=dmb8rnzOuLzX9LoOhn/xeD2A0hlf8+YvPVyeGcCEIrsDR13TEvrcZmKrLo2TsbKKK3 niYDOrNNAm7NJYVHeLXs6SCGecvMJqsbcKUKnjRgANt8XsJjVp2Dh5N6XD7X6nTVlDQ7 zCMEeV6MuRsplsZXu36Bi5jRnD+iRhmIJakHHpi/H3CBFBsHvkwIjB61gkG//L3Hwz2y tIOO6W4sKOzBQRfg31CgFAgd1uI4t+qstxoe9aRe95ov2zQjbD4yMBdbEYeM0QqQAmYI EUFwdtEUcFmVwRUuQb0sHjm+2OQxIlFremz6uFTyY5pQyPGuHT+DBnIgAPdeY27Dpi5x bXmA== X-Gm-Message-State: AOJu0YxQ5MjotkbZWmIN9brMzWzO4D4kIuB/RjRBe6wNx6uQNuS0S3ev Wl0jfrtTD2ixdLvfkt3LopxP9xg1hM6ohEllVPVTvMZE0b3ic7lhZcAYcFEM32fTDyRKEVMOCw/ X X-Google-Smtp-Source: AGHT+IGkCK6sueGqkQbZ12dq8XNonzc3Dk4/s5ixWpYJNPpuultzfiwYT4mR43qd4u2weh0h77Af7A== X-Received: by 2002:a81:bb4d:0:b0:63b:d711:e722 with SMTP id 00721157ae682-6895f7faa2dmr184183767b3.1.1722958652295; Tue, 06 Aug 2024 08:37:32 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 00721157ae682-68a09e92aa2sm15890027b3.0.2024.08.06.08.37.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Aug 2024 08:37:31 -0700 (PDT) Date: Tue, 6 Aug 2024 11:37:30 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v3 07/19] midx: introduce `bsearch_one_midx()` Message-ID: <32c3fceada73e8d53f4239eafc1f440c072920c8.1722958596.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The `bsearch_midx()` function will be extended in a following commit to search for the location of a given object ID across all MIDXs in a chain (or the single non-chain MIDX if no chain is available). While most callers will naturally want to use the updated `bsearch_midx()` function, there are a handful of special cases that will want finer control and will only want to search through a single MIDX. For instance, the object abbreviation code, which cares about object IDs near to where we'd expect to find a match in a MIDX. In that case, we want to look at the nearby matches in each layer of the MIDX chain, not just a single one). Split the more fine-grained control out into a separate function called `bsearch_one_midx()` which searches only a single MIDX. At present both `bsearch_midx()` and `bsearch_one_midx()` have identical behavior, but the following commit will rewrite the former to be aware of incremental MIDXs for the remaining non-special case callers. Signed-off-by: Taylor Blau --- midx.c | 17 +++++++-- midx.h | 5 ++- object-name.c | 99 +++++++++++++++++++++++++++------------------------ 3 files changed, 71 insertions(+), 50 deletions(-) diff --git a/midx.c b/midx.c index 25350152f1..bd6e3f26c9 100644 --- a/midx.c +++ b/midx.c @@ -330,10 +330,21 @@ int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m, return 0; } -int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result) +int bsearch_one_midx(const struct object_id *oid, struct multi_pack_index *m, + uint32_t *result) { - return bsearch_hash(oid->hash, m->chunk_oid_fanout, m->chunk_oid_lookup, - the_hash_algo->rawsz, result); + int ret = bsearch_hash(oid->hash, m->chunk_oid_fanout, + m->chunk_oid_lookup, the_hash_algo->rawsz, + result); + if (result) + *result += m->num_objects_in_base; + return ret; +} + +int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, + uint32_t *result) +{ + return bsearch_one_midx(oid, m, result); } struct object_id *nth_midxed_object_oid(struct object_id *oid, diff --git a/midx.h b/midx.h index 020e49f77c..46c53d69ff 100644 --- a/midx.h +++ b/midx.h @@ -90,7 +90,10 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id); int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m, struct bitmapped_pack *bp, uint32_t pack_int_id); -int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result); +int bsearch_one_midx(const struct object_id *oid, struct multi_pack_index *m, + uint32_t *result); +int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, + uint32_t *result); off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos); uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos); struct object_id *nth_midxed_object_oid(struct object_id *oid, diff --git a/object-name.c b/object-name.c index 527b853ac4..739d46f9cf 100644 --- a/object-name.c +++ b/object-name.c @@ -134,28 +134,32 @@ static int match_hash(unsigned len, const unsigned char *a, const unsigned char static void unique_in_midx(struct multi_pack_index *m, struct disambiguate_state *ds) { - uint32_t num, i, first = 0; - const struct object_id *current = NULL; - int len = ds->len > ds->repo->hash_algo->hexsz ? - ds->repo->hash_algo->hexsz : ds->len; - num = m->num_objects; + for (; m; m = m->base_midx) { + uint32_t num, i, first = 0; + const struct object_id *current = NULL; + int len = ds->len > ds->repo->hash_algo->hexsz ? + ds->repo->hash_algo->hexsz : ds->len; - if (!num) - return; + if (!m->num_objects) + continue; - bsearch_midx(&ds->bin_pfx, m, &first); + num = m->num_objects + m->num_objects_in_base; - /* - * At this point, "first" is the location of the lowest object - * with an object name that could match "bin_pfx". See if we have - * 0, 1 or more objects that actually match(es). - */ - for (i = first; i < num && !ds->ambiguous; i++) { - struct object_id oid; - current = nth_midxed_object_oid(&oid, m, i); - if (!match_hash(len, ds->bin_pfx.hash, current->hash)) - break; - update_candidates(ds, current); + bsearch_one_midx(&ds->bin_pfx, m, &first); + + /* + * At this point, "first" is the location of the lowest + * object with an object name that could match + * "bin_pfx". See if we have 0, 1 or more objects that + * actually match(es). + */ + for (i = first; i < num && !ds->ambiguous; i++) { + struct object_id oid; + current = nth_midxed_object_oid(&oid, m, i); + if (!match_hash(len, ds->bin_pfx.hash, current->hash)) + break; + update_candidates(ds, current); + } } } @@ -708,37 +712,40 @@ static int repo_extend_abbrev_len(struct repository *r UNUSED, static void find_abbrev_len_for_midx(struct multi_pack_index *m, struct min_abbrev_data *mad) { - int match = 0; - uint32_t num, first = 0; - struct object_id oid; - const struct object_id *mad_oid; + for (; m; m = m->base_midx) { + int match = 0; + uint32_t num, first = 0; + struct object_id oid; + const struct object_id *mad_oid; - if (!m->num_objects) - return; + if (!m->num_objects) + continue; - num = m->num_objects; - mad_oid = mad->oid; - match = bsearch_midx(mad_oid, m, &first); + num = m->num_objects + m->num_objects_in_base; + mad_oid = mad->oid; + match = bsearch_one_midx(mad_oid, m, &first); - /* - * first is now the position in the packfile where we would insert - * mad->hash if it does not exist (or the position of mad->hash if - * it does exist). Hence, we consider a maximum of two objects - * nearby for the abbreviation length. - */ - mad->init_len = 0; - if (!match) { - if (nth_midxed_object_oid(&oid, m, first)) - extend_abbrev_len(&oid, mad); - } else if (first < num - 1) { - if (nth_midxed_object_oid(&oid, m, first + 1)) - extend_abbrev_len(&oid, mad); + /* + * first is now the position in the packfile where we + * would insert mad->hash if it does not exist (or the + * position of mad->hash if it does exist). Hence, we + * consider a maximum of two objects nearby for the + * abbreviation length. + */ + mad->init_len = 0; + if (!match) { + if (nth_midxed_object_oid(&oid, m, first)) + extend_abbrev_len(&oid, mad); + } else if (first < num - 1) { + if (nth_midxed_object_oid(&oid, m, first + 1)) + extend_abbrev_len(&oid, mad); + } + if (first > 0) { + if (nth_midxed_object_oid(&oid, m, first - 1)) + extend_abbrev_len(&oid, mad); + } + mad->init_len = mad->cur_len; } - if (first > 0) { - if (nth_midxed_object_oid(&oid, m, first - 1)) - extend_abbrev_len(&oid, mad); - } - mad->init_len = mad->cur_len; } static void find_abbrev_len_for_pack(struct packed_git *p, From patchwork Tue Aug 6 15:37:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13755140 Received: from mail-yw1-f169.google.com (mail-yw1-f169.google.com [209.85.128.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7A1EF38FA1 for ; Tue, 6 Aug 2024 15:37:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958657; cv=none; b=UiK0vPWO6STYORqnqKzJfE5B+5SndBfCk4l8eTvW0lXRiVCkd3y0SD4Yg14hcZFjSLgVPwGPMq1EGwR3ySvfwWQHer4RE9ez5W4amrQyvpxZnDRHG2prJIMn6H1FbMRVq1Ihf6IjHv9cTjj/Wa+rtkQfXr+gnktr1ykIe6vRsuw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958657; c=relaxed/simple; bh=xyPhpEP5dPyf3X6sGmGSrgVn88wCTTm8nh5XG6FHn0k=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=EK6zhProfujY5Y8FWyoK2IZuZb5pom2GAPqkRim/cdQ6F3If/x4hKh8hnGaDRfQ3fcIMCjvEbWaG//h9E3tPfC33/ekzvu1LoJLRm/FfdWQdX/ViLCqBZHxcAjJEQqq8U67tXgIXl4L28Byjgud5fyxo2dy6cI1VYdUXKHJVGw0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=FYHSQKNK; arc=none smtp.client-ip=209.85.128.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="FYHSQKNK" Received: by mail-yw1-f169.google.com with SMTP id 00721157ae682-690ad83d4d7so7434267b3.3 for ; Tue, 06 Aug 2024 08:37:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1722958655; x=1723563455; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=YYdP9CgGzHuHysAZLia1dFD2NJ3efZo7Gdh3krm5+Es=; b=FYHSQKNK/51LEbrBkRaJhAaoXMI+I8NzXiyUicy+HwYZJaEmoF/Gipb36thr/sefHl edzxDO1lljT1SUNXhaQG+vI0mnk5QPPVsSMHRa/f6kF6jQAUhAVzh7nAHkbSNmveAQUP lJH7mvdW0j6qXATNrCN9fGYSkET+8xE/5drFBBkzv4ifAmezBdfEU8fyy+lJ2PkX7ukB c9lrVCHasHYntvTBPEi1ucG2fMz5PZoIW9TLL5XUSPXNTG56W+V5ZIt81YCrKLjZVuGd ysAFUWhm9OpmesX6lbg3uQDHAcf5QWZ6GIYvSYeCY+y/JdI16m2UuYVLotGbSevMEIl+ ehXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722958655; x=1723563455; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=YYdP9CgGzHuHysAZLia1dFD2NJ3efZo7Gdh3krm5+Es=; b=iKOe89yCupGHLdejkGipGc5gOqzNGOMliFn1E0h0EO8lFLKy6ZgjnZWPD8q1H8nQjr rTqNCE4xZBAPWdFeKx0lP1Fzzl/GGrcqrGfcAu6+Zx4G4IhfqgMTgZw8BOJSdeVe5Dtg sAwphSSO8aN+YWPcnWnfdWNq5cznTqfSCf85BDaTnNmCV+ScWSTTxE6WWLNWveFAShNM qq7HcbnlWC9nJ62RxXKZBjDhMPpsz6coZUkkeG7aCx1RbLr3/ItxQ+Ek1yCFA+8du3OF qBD4B0JPWF5AOiLmMt5hpiCP7JcGjRPkxryCjcAbxocAt0bSbH25isHirigzyg6rD8O7 7Ckw== X-Gm-Message-State: AOJu0Yz6rYKxzPr/LtiiXpXWfxJJyj3VASHm13B8i1IuzI8gky6Gd6ak FcolET6qhqLw8BCC2m3mvFldHOfLaw1Abkwc3tO9YYBHDPv9sctyzct5iqJqkvmAzKXrtlcewT1 3 X-Google-Smtp-Source: AGHT+IEzRVJlwj/ZVDx/4lfNL8gcS03Wcbo1/PxYZBwCf1dWpmcJrNxH2NieBfKAaDNdwAICl9wyEA== X-Received: by 2002:a81:7702:0:b0:65f:dfd9:b23d with SMTP id 00721157ae682-689601af052mr180247987b3.17.1722958655370; Tue, 06 Aug 2024 08:37:35 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 00721157ae682-68a137b4b37sm15761247b3.130.2024.08.06.08.37.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Aug 2024 08:37:34 -0700 (PDT) Date: Tue, 6 Aug 2024 11:37:33 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v3 08/19] midx: teach `bsearch_midx()` about incremental MIDXs Message-ID: <16db6c98cecb103c9c4aae263dc8b9071be2caea.1722958596.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Now that the special cases callers of `bsearch_midx()` have been dealt with, teach `bsearch_midx()` to handle incremental MIDX chains. The incremental MIDX-aware version of `bsearch_midx()` works by repeatedly searching for a given OID in each layer along the `->base_midx` pointer, stopping either when an exact match is found, or the end of the chain is reached. Signed-off-by: Taylor Blau --- midx.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/midx.c b/midx.c index bd6e3f26c9..83857cbd1e 100644 --- a/midx.c +++ b/midx.c @@ -344,7 +344,10 @@ int bsearch_one_midx(const struct object_id *oid, struct multi_pack_index *m, int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result) { - return bsearch_one_midx(oid, m, result); + for (; m; m = m->base_midx) + if (bsearch_one_midx(oid, m, result)) + return 1; + return 0; } struct object_id *nth_midxed_object_oid(struct object_id *oid, From patchwork Tue Aug 6 15:37:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13755141 Received: from mail-yb1-f170.google.com (mail-yb1-f170.google.com [209.85.219.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 931391803A for ; Tue, 6 Aug 2024 15:37:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958660; cv=none; b=MK8bAInHBjAg0C0ytkZOMPVKOFKeUGM82xjoDvBdntzaG+/U8/UqqgxPEWCJL6Q/FN2TQg52p1d6k7TNagAS6OAqrdq7Luqhn7yx+4iKF6VWiMsP568mKqYir8Tk0SymurnxkzV6bj2g+SsEGwdugVbJp0q5DqEHQg7MQgWCOLQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958660; c=relaxed/simple; bh=Sm902H3OkCI6WvXQvFzvxjLaTw6RNy5olb6it/shaYw=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=jcfPDl5CjtuOrEtGCFeA6vAmplxwBO9ZgbnfPXnGWib3bkBTjxH8469rZFGBchgNJouG5wQ2VLlFQBT2BfWgha2fgspoQzmGaGZNwjPgvJwZ+OFNIvKIN33TMHa1kF9mZCHHnvnY8mQZul/F7CuuF48vacU+MWsSoLoky0SbjkE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=EXXBqBBx; arc=none smtp.client-ip=209.85.219.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="EXXBqBBx" Received: by mail-yb1-f170.google.com with SMTP id 3f1490d57ef6-e08724b2a08so721031276.1 for ; Tue, 06 Aug 2024 08:37:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1722958658; x=1723563458; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=jOQTdvBUzMX58Rlmb47zlKgb68JW4baJQmHV56iFGO8=; b=EXXBqBBxmkK0UnU+FWuUfbCPeBBXb9yOW2KakeEC5HSR849NJPVPCp34LVWBhkmSxK /1TwmHdOLn5w4GVdAeXTUDutzzBKTccYxfuU2es0td9Mo/dpkJ+j80Lyx346qbcsx/nV EU4xQ/Igg79y8JzFghlIsTu7EN8HKwMMGICQP4JW2EWo3wzBjiYr8bMyinHERY6RO10D gPfpztGOixzwzInqdxoqoZ4Tr3iyp5q6J6nL/K+84D6Yim10MmlVcJ1wteLdniD3qtag jYqqDZvwMilYcU8iJmpUDkGa5kIK6Qs+hVilJq9Ju3oaCGsnzZr1gvoIBFLGyelSPNsd 7t8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722958658; x=1723563458; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=jOQTdvBUzMX58Rlmb47zlKgb68JW4baJQmHV56iFGO8=; b=Kin+FGFeVJsVav9tdi1X2dJAZNdipGJF/b376m4pYKB9ALO3da8B21xjZood/+IR9s C1aTB8J1NaHa6jaBDK/YfWbRcVY/trybQW5mbu1EwVRRDJJxRJgpxBibfRNSV8ObOlmu c1es9dD1KixKcsfIYuUjBYgs3ENca4sGD3hCTOdf7soyV40uhWNeCdlawTIueWheIq/g WK62/y82UwDg8E33dCMwIpIq/fHFD9p4VYy2v0Nk4wcs9gdhAttPMB5Z3dLPjh3jmjSx 6D78kLtYvImgGR9FViQvKONPz0yGF8NOTTUrS+XLh1DJAv4eKyqV8z80ALF+fEFhrNFF 3UGQ== X-Gm-Message-State: AOJu0YyVBZnNfoog5n0LYjpT4p7pQgyLQlx20Y1s/6EL2BPOy6/uCbR+ 0hak8OF6Ugialh4R3CZ0YDyfQW/PwBGF6nKyyXWC5lg5RqUAVVJgQeHGrxwg2+PDzGHXS5pNkIf W X-Google-Smtp-Source: AGHT+IEv9jJxnc3Xpfxuz7TP32sCS/YhmSGUdljJDhweOby6hY4K5dzesAHTL6Qef5IN6Y61R+8eng== X-Received: by 2002:a05:6902:1507:b0:e0b:cc1d:3731 with SMTP id 3f1490d57ef6-e0bde2d1938mr19123098276.2.1722958658378; Tue, 06 Aug 2024 08:37:38 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 3f1490d57ef6-e0be559f00fsm1838522276.57.2024.08.06.08.37.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Aug 2024 08:37:38 -0700 (PDT) Date: Tue, 6 Aug 2024 11:37:36 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v3 09/19] midx: teach `nth_midxed_offset()` about incremental MIDXs Message-ID: <761c7c59ba1a245732d86a4e1d6c24baf742a1cb.1722958596.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: In a similar fashion as in previous commits, teach the function `nth_midxed_offset()` about incremental MIDXs. The given object `pos` is used to find the containing MIDX, and translated back into a MIDX-local position by assigning the return value of `midx_for_object()` to it. Signed-off-by: Taylor Blau --- midx.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/midx.c b/midx.c index 83857cbd1e..346e58dec7 100644 --- a/midx.c +++ b/midx.c @@ -369,6 +369,8 @@ off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos) const unsigned char *offset_data; uint32_t offset32; + pos = midx_for_object(&m, pos); + offset_data = m->chunk_object_offsets + (off_t)pos * MIDX_CHUNK_OFFSET_WIDTH; offset32 = get_be32(offset_data + sizeof(uint32_t)); From patchwork Tue Aug 6 15:37:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13755142 Received: from mail-yw1-f180.google.com (mail-yw1-f180.google.com [209.85.128.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7DD341803A for ; Tue, 6 Aug 2024 15:37:42 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958663; cv=none; b=FW9fuTqPWLtUO4sBywecjx7pZ+uRV/VVjDPirfPxf/yGACAClYBSp/4Y/6FlmXDqJ1Basf6wtcLBN3OgSVmIDtaSczJMUneQkler2DJECkPQuonphBOb4jUYShURXvgfBwSZHViExuO5dQiA6mtbeo03eer2WfdLYR/J4trtF98= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958663; c=relaxed/simple; bh=qsYnqXwjxFjNfNwEdrkzSO/piUuRMHat11xsmBQNsr4=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=N9bk79tb/hGgVTNC8Pd5ZxAnj/cDz4HLmHqILV2mvtP76DMni4jeW3stZ5r9Zl91fEdbneLM74Rwc4FlNWVHIeKKIchZ75jFSUmQgOuerGDQFeG8XqDuU6a/BZHW0mI+MHc0MCgposVeuPvjS2GFKatYdDirWAf5wD8Qx7p/1uo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=TJc8YVMP; arc=none smtp.client-ip=209.85.128.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="TJc8YVMP" Received: by mail-yw1-f180.google.com with SMTP id 00721157ae682-66a048806e6so8491077b3.3 for ; Tue, 06 Aug 2024 08:37:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1722958661; x=1723563461; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=qXwVg+Rh+XXZCGzotaQzx5Q3ZfXbxqhxV7KNyAmkvuI=; b=TJc8YVMPGRTb5WBQHw6Am20mVf4L+54+O/X8SHBbLvlaAOxWYFnPNtloTpHjC7P81q CvKYUKb3P/oZttc5iRpPJi7rZLGSe4VM8VO+PqySeVoD4f0BMsz0IF5WztiEAnGi6SAp ODzKfHWOyRiEP97Uv2Q0FHOH58G0wiX8usqeqbWxASXitqdx+VLvz3Dly/E6ryoLE0Rh rOpOKme6f3RLNaz3HpYMRZmVYyyqviT5lAWkc0Ed+0t/RAGo1y+c8ixv+XKA02Clo2fA DomaCQV4CHnU9mqezYaLIY/2jaOyuxrbI1AsVXnqedoTbDZQXTCgj0Jw6lEQlQMHueZO LrHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722958661; x=1723563461; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=qXwVg+Rh+XXZCGzotaQzx5Q3ZfXbxqhxV7KNyAmkvuI=; b=ZamVhRsgum+cC7r8+fiCFpyjiC/Qu7tQH2bg9ei9piJP+HNUD07Ga+tq/jMW8UMXMq aPZw80gWeIeHgTdRJS3bZ8wVx9V5r/+IK9ZbO7GI/T+utzhs1PCpbI64f0zQ+uvVaaFQ C7l2Imcb3z6o9X27SEw50hyiBH4uXKpNN7Xd5lfG81hgGxEstQ/ogZ4bArM7nvQwSjbj e9AbD2He9hMJ8jqVkk/Bb5NAow0czgGTAnDX4u+mOwJA3W6hZj6Vp0OR2loVIT6ScQLa 33O+Pwj+Z3cBkaz/G4wJNz6qqDdenGPUaL5WLbXkMMiy5DApaCG7ugE/GhtHSvXSXVqq xTTg== X-Gm-Message-State: AOJu0YwBEbfP+xn4NqeysFp8LMju8367SirFwVVZ8rN8o5Z40xoWxo0M ZSv1r8hiNe9wu0m5cDQ+l9Mo7gEahfZyDlyftnU0BGnz9OqhfatDzYHXOQkcZYyGvlI3XVtL2Sj e X-Google-Smtp-Source: AGHT+IE/gkeCnsJqGFoctH1O3JaradulARg3f3BbrXWnFBDS8orVwYlXAE2lriqyKxveSTRF93XnJw== X-Received: by 2002:a81:c213:0:b0:65f:ca0b:451c with SMTP id 00721157ae682-6896122b7famr172390287b3.21.1722958661321; Tue, 06 Aug 2024 08:37:41 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 00721157ae682-68a0fdb3b9fsm15959077b3.53.2024.08.06.08.37.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Aug 2024 08:37:40 -0700 (PDT) Date: Tue, 6 Aug 2024 11:37:40 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v3 10/19] midx: teach `fill_midx_entry()` about incremental MIDXs Message-ID: <8366456d29b870533b5a788a372375b11103b8c7.1722958596.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: In a similar fashion as previous commits, teach the `fill_midx_entry()` function to work in a incremental MIDX-aware fashion. This function, unlike others which accept an index into either the lexical order of objects or packs, takes in an object_id, and attempts to fill a caller-provided 'struct pack_entry' with the remaining pieces of information about that object from the MIDX. The function uses `bsearch_midx()` which fills out the frame-local 'pos' variable, recording the given object_id's lexical position within the MIDX chain, if found (if no matching object ID was found, we'll return immediately without filling out the `pack_entry` structure). Once given that position, we jump back through the `->base_midx` pointer to ensure that our `m` points at the MIDX layer which contains the given object_id (and not an ancestor or descendant of it in the chain). Note that we can drop the bounds check "if (pos >= m->num_objects)" because `midx_for_object()` performs this check for us. After that point, we only need to make two special considerations within this function: - First, the pack_int_id returned to us by `nth_midxed_pack_int_id()` is a position in the concatenated lexical order of packs, so we must ensure that we subtract `m->num_packs_in_base` before accessing the MIDX-local `packs` array. - Second, we must avoid translating the `pos` back to a MIDX-local index, since we use it as an argument to `nth_midxed_offset()` which expects a position relative to the concatenated lexical order of objects. Signed-off-by: Taylor Blau --- midx.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/midx.c b/midx.c index 346e58dec7..5e4e6f9b65 100644 --- a/midx.c +++ b/midx.c @@ -407,14 +407,12 @@ int fill_midx_entry(struct repository *r, if (!bsearch_midx(oid, m, &pos)) return 0; - if (pos >= m->num_objects) - return 0; - + midx_for_object(&m, pos); pack_int_id = nth_midxed_pack_int_id(m, pos); if (prepare_midx_pack(r, m, pack_int_id)) return 0; - p = m->packs[pack_int_id]; + p = m->packs[pack_int_id - m->num_packs_in_base]; /* * We are about to tell the caller where they can locate the From patchwork Tue Aug 6 15:37:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13755143 Received: from mail-yw1-f178.google.com (mail-yw1-f178.google.com [209.85.128.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 976951803A for ; Tue, 6 Aug 2024 15:37:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958667; cv=none; b=LNfp8LfPsdqjInlw4FNbu+M/HgjyoylbgqDbGLqAMSLqDwt2nlB+kdF1r277HdveWA1WAfxwBCL+VBrwf+QejGJK5zXGcgheiUHIrIuXHQVTrogCTDQfFL3oI/ruE37NL92m75u1UwLbMXtn4pDxPczaXjP1GSMhuUt5FZDz7Nk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958667; c=relaxed/simple; bh=AQnzizFGFfOl7045WRP7PXi5zvl1mIIXF/bpSF4AwXY=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=n3LIg8o3dovoWpBAYgZ/cTxLJZzPcbA7m6bznSai7XxPnybTXsWQ3HrMLIBMFIxkq6f9i93bUAVdStKghAIebGt679/ho9NuFijAspeIBLOJ0pRhK0NrVwHXl8nDC8rOfID02a9IP+wo1pfDAWKxAxJjwzqEhFjvL2cLQc9Wp8k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=bDaAGnve; arc=none smtp.client-ip=209.85.128.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="bDaAGnve" Received: by mail-yw1-f178.google.com with SMTP id 00721157ae682-690af536546so8787037b3.3 for ; Tue, 06 Aug 2024 08:37:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1722958664; x=1723563464; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=bVroDsdMsGQE63tSMgtYp3l1RZLDlSIT0DRgLmQ0xwY=; b=bDaAGnve0OkNcT/CCKJYk4nBJCbH5p/cUTGmYavijwaQ1O4eVr2jEDK9lG1f5h6taZ 5rVu28PWb5FTb5ECO4wpVO+ZlgW3uHB94fqhI+wTaaE0BCBi1TWcUQMgEMZNvb8zjV05 lKuPT1KdDVouWv/PRLIxn0Oo5EfuzP0LQEEt6ITDI5o+hutH9CJMBSsCSNB0CUTaOMCB AHAt5jEQ9uz6wQ2IRPXFQSPF/Yp92usLn2gcCo2DqkcoJ1zTFpEgj92tRFwyuuUiefo4 TFWaNhQAAbUbj9xiu7ehesizQYFE+VDJW1zf9Uhh6JloK4jwRw35nOqP+JoC/9RNniMj 9tkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722958664; x=1723563464; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=bVroDsdMsGQE63tSMgtYp3l1RZLDlSIT0DRgLmQ0xwY=; b=kEnSnh96NYp0nk+e8XROERkJd0h0l/avKlu1ylN21G8/HrqfnOeM7cGdtPHpjBMsMz 1ODnAZwIpW2Z52CQjwXr1v3/NGqju5gNRXcrPb4xtsAuzvK0sqKUHoI38lhrOPtI6/7s 2+1OCM5cZRuNsYUziLjxPObRXUhIvj5cgzh552eu8quseH2zjqcmABnvAAxx1kLQ8zF9 uUB5Gc3cA2tmWq2/izJlren+FpcJ15zegeo6jFkaz+MH0kaiqTM2Tp7C5ZhTsXl0GEtp 1zeHTTewrKwDyrStZg+Rys9nGhc2/X5he1RWXlPcJ+9SzFqaW8HOnsr8fl/zhsqsFAp0 DSkA== X-Gm-Message-State: AOJu0Yz38lF5fSbtDoRuUwONP2PaW+0cUiP40q71UFXOdxJlya27ZM2j FEjFagkdxnXNemKcIiOTu9wUoG9g9ZzfAfjJ/6QW7bmYHf4QRXejK2CU0ku6P1MyOzZOkHCsgte l X-Google-Smtp-Source: AGHT+IHl+QbsW+GrC9phJuRd9U3vkRSmAH/sUGP/yXyOGryBFKnNBFGjsQeAPBpdd0Y9ijDBNJiCoA== X-Received: by 2002:a81:5c43:0:b0:62f:945a:7bb1 with SMTP id 00721157ae682-689637054a9mr165793827b3.42.1722958664357; Tue, 06 Aug 2024 08:37:44 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 00721157ae682-68a0f418862sm15861567b3.27.2024.08.06.08.37.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Aug 2024 08:37:43 -0700 (PDT) Date: Tue, 6 Aug 2024 11:37:42 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v3 11/19] midx: remove unused `midx_locate_pack()` Message-ID: <909d927c470f24c723ba067fce2016ab80b01885.1722958596.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Commit 307d75bbe6 (midx: implement `midx_locate_pack()`, 2023-12-14) introduced `midx_locate_pack()`, which was described at the time as a complement to the function `midx_contains_pack()` which allowed callers to determine where in the MIDX lexical order a pack appeared, as opposed to whether or not it was simply contained. 307d75bbe6 suggests that future patches would be added which would introduce callers for this new function, but none ever were, meaning the function has gone unused since its introduction. Clean this up by in effect reverting 307d75bbe6, which removes the unused functions and inlines its definition back into `midx_contains_pack()`. (Looking back through the list archives when 307d75bbe6 was written, this was in preparation for this[1] patch from back when we had the concept of "disjoint" packs while developing multi-pack verbatim reuse. That concept was abandoned before the series was merged, but I never dropped what would become 307d75bbe6 from the series, leading to the state prior to this commit). [1]: https://lore.kernel.org/git/3019738b52ba8cd78ea696a3b800fa91e722eb66.1701198172.git.me@ttaylorr.com/ Signed-off-by: Taylor Blau --- midx.c | 13 ++----------- midx.h | 2 -- 2 files changed, 2 insertions(+), 13 deletions(-) diff --git a/midx.c b/midx.c index 5e4e6f9b65..50f131e59a 100644 --- a/midx.c +++ b/midx.c @@ -466,8 +466,7 @@ int cmp_idx_or_pack_name(const char *idx_or_pack_name, return strcmp(idx_or_pack_name, idx_name); } -int midx_locate_pack(struct multi_pack_index *m, const char *idx_or_pack_name, - uint32_t *pos) +int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name) { uint32_t first = 0, last = m->num_packs; @@ -478,11 +477,8 @@ int midx_locate_pack(struct multi_pack_index *m, const char *idx_or_pack_name, current = m->pack_names[mid]; cmp = cmp_idx_or_pack_name(idx_or_pack_name, current); - if (!cmp) { - if (pos) - *pos = mid; + if (!cmp) return 1; - } if (cmp > 0) { first = mid + 1; continue; @@ -493,11 +489,6 @@ int midx_locate_pack(struct multi_pack_index *m, const char *idx_or_pack_name, return 0; } -int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name) -{ - return midx_locate_pack(m, idx_or_pack_name, NULL); -} - int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id) { if (m->preferred_pack_idx == -1) { diff --git a/midx.h b/midx.h index 46c53d69ff..86af7dfc5e 100644 --- a/midx.h +++ b/midx.h @@ -102,8 +102,6 @@ struct object_id *nth_midxed_object_oid(struct object_id *oid, int fill_midx_entry(struct repository *r, const struct object_id *oid, struct pack_entry *e, struct multi_pack_index *m); int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name); -int midx_locate_pack(struct multi_pack_index *m, const char *idx_or_pack_name, - uint32_t *pos); int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id); int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local); From patchwork Tue Aug 6 15:37:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13755144 Received: from mail-yw1-f172.google.com (mail-yw1-f172.google.com [209.85.128.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0D9BE1803A for ; Tue, 6 Aug 2024 15:37:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958670; cv=none; b=HK0lojPq4vZ4zaPH+V4arOOj0XrMro07dDmS+eSXsWnb6OhgnVhq2ezK13ZbkrGOxMCQ0n3/xLbefUtEw+PKdOUiqyUYnBMma0QyCCx0IiSTU7NYZZ/5bdiv91uPq2++LUwcOhEEU4TPwaaDu+Rg8dgi+O4hMl/JkOsjD0HeSi8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958670; c=relaxed/simple; bh=y9r26JRkPthmjfOdeZzbvSJh5wM7rEhc+2H8bT5CxXk=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=b5xf5Q/WVMD9PB5/RGw4bgiS84i/e6q/7/bdbEdjGN6NKerM5YQZBnOAlgu28zcO6VqKkt2wpvC0Xk6FABhqTc27X+1QQsl1ScLfuidvPSsI6SUhcUUCl6TXKHnniPCxNr07B8tiC39+pV9oQr/enKp+v//IV2yxOegIpdSNcNc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=lRDJqkQ4; arc=none smtp.client-ip=209.85.128.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="lRDJqkQ4" Received: by mail-yw1-f172.google.com with SMTP id 00721157ae682-65fe1239f12so7323337b3.0 for ; Tue, 06 Aug 2024 08:37:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1722958668; x=1723563468; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=BOnNBaq/Cg3iM6T2URhiL+L1EIqmj0r+sHLewqbQDJY=; b=lRDJqkQ4a/PMPO65pQLDnE1p/KPBJqqiQwb8J3ukf1ivMXN8hd7F0REejkEmupVWK0 C+w7+FmG+/6eIf2tE57fd+B2cfDFZdfP97LaDgwl4gCAnbIYFF/ABledszfzzgUEuqLE 7jSX8ZVEXTURemzqjNbZy3+N/oid3wOpLlnxtchdF+LSlnL+MBEuHmIVXc7XIfNcxb0C r+ycjrVG0xu7enolF5YO/IaEyk3SLBs3Nne/OvuP4N87xK6pSln1uc1xnJikwlvluE3m SA/pzVh+ERyAdJ1Enp75i19vdaAeuY/StgAm+2NrlyMA+AhWBX3bVhnsIuv2X2JwsiAK A8CQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722958668; x=1723563468; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=BOnNBaq/Cg3iM6T2URhiL+L1EIqmj0r+sHLewqbQDJY=; b=UL1I2nnQMqf64ZDMRGmAcz5R4BmV0JX5+7f77JCgUu0LG19eAHpcfBrf/1FfsDF2Zi B4rbaYY3Y++GSGgsp3WiOed6MD/KPRYKmVfIcRNa9GPueP/UYx2409m3kKcDQeG0Znwe jm41YwyZIBLWhTgoWl6nDJnYlIxQVAC4QqZDXN3gDwGWEWjtRNHk4sBj08OwDzsyUDu6 ZdhPMnM64qk3jI6mssikqupGShAm5WMRfJbAZIybikEOa2L2oGsrgFISENGG1fDhENbB L7zINYHHJBawl6oz4Fb+TJztv4H/hV6QzMRWFjnk75pLuEkuwVCyp7+oW9+S05CkEIt7 wgEg== X-Gm-Message-State: AOJu0YwQ/pyg8DB5kC7OwYUk4FEBG4XuQMbo0diGTIgJAeEaREm//8m4 4wT8U+7I+aDoXLJVLCtuUnqSJnNCTlmk0k1ahIOyMZ8mE3urE7UNk9XernWdQSYIhDq0HX97FVr t X-Google-Smtp-Source: AGHT+IEbWtP0EM5cSRnVmcigh0eyXRFC5xwwSeCIJxqWyuxvpt1EzUiKPZscvWlIJ2C5TpN9Tghx4Q== X-Received: by 2002:a0d:f3c6:0:b0:672:e49d:430e with SMTP id 00721157ae682-6896096854cmr143281287b3.15.1722958667573; Tue, 06 Aug 2024 08:37:47 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 00721157ae682-68a0f419115sm15794327b3.19.2024.08.06.08.37.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Aug 2024 08:37:47 -0700 (PDT) Date: Tue, 6 Aug 2024 11:37:46 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v3 12/19] midx: teach `midx_contains_pack()` about incremental MIDXs Message-ID: <71127601b5d412eff69d6451e328e4b1c74b4caa.1722958596.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Now that the `midx_contains_pack()` versus `midx_locate_pack()` debacle has been cleaned up, teach the former about how to operate in an incremental MIDX-aware world in a similar fashion as in previous commits. Instead of using either of the two `midx_for_object()` or `midx_for_pack()` helpers, this function is split into two: one that determines whether a pack is contained in a single MIDX, and another which calls the former in a loop over all MIDXs. This approach does not require that we change any of the implementation in what is now `midx_contains_pack_1()` as it still operates over a single MIDX. Signed-off-by: Taylor Blau --- midx.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/midx.c b/midx.c index 50f131e59a..454c27b673 100644 --- a/midx.c +++ b/midx.c @@ -466,7 +466,8 @@ int cmp_idx_or_pack_name(const char *idx_or_pack_name, return strcmp(idx_or_pack_name, idx_name); } -int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name) +static int midx_contains_pack_1(struct multi_pack_index *m, + const char *idx_or_pack_name) { uint32_t first = 0, last = m->num_packs; @@ -489,6 +490,14 @@ int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name) return 0; } +int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name) +{ + for (; m; m = m->base_midx) + if (midx_contains_pack_1(m, idx_or_pack_name)) + return 1; + return 0; +} + int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id) { if (m->preferred_pack_idx == -1) { From patchwork Tue Aug 6 15:37:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13755145 Received: from mail-yb1-f179.google.com (mail-yb1-f179.google.com [209.85.219.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E423F1803A for ; Tue, 6 Aug 2024 15:37:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958673; cv=none; b=BpTcsY/pVmRCFoGXso77wlMf6LUHyoJkav/s8tbevLGaUZVJDX4rOVivCC+J0PFLXpUXdGUsWvaXcLy1xDNKv4M5lLr6h9NE42A5472p/rvMXPB1XP2WE7F8Lz1eAjCJBd4TA/drPT8SJfBO8k5TSTEhod2HW3q/kV9oBFtHvf4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958673; c=relaxed/simple; bh=zPQEPQk+mLHSWGCZRtTkh4YWAHIq4jHeXaUMGYZVlBc=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=Evaq6w3IIwafDMF2vrlI7LhwwUASjc2W1V8sea4xrp60qfZFV2rc+2xF+wU+t+F2VO47oXNh5mWpIzmii20b3gLOJNydtvfdfe32qRlg603DpvaVqG2JWUl38BhUJSMgP6oBdgMfanb6m6qiIJ8w6mIvHwO4ohSVwcktYnWZL9Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=qyPQ/hkC; arc=none smtp.client-ip=209.85.219.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="qyPQ/hkC" Received: by mail-yb1-f179.google.com with SMTP id 3f1490d57ef6-e0878971aa9so793202276.0 for ; Tue, 06 Aug 2024 08:37:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1722958671; x=1723563471; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=eOuBwftfA7fRjO8FH2eSlza624bLi0kcVZIjvBjv+eA=; b=qyPQ/hkCkmC9C+WCK6iWNr/U+vNuhgWH2Y2etUn+bvZjoR9DGFK3HWvXm9BestBird HQUGOlgqeF7/C8LPavJX0oyquvEKI4yuba9ZW7NSBWWFCgGcPXqaS/xuRV6/QisoNrPz LdRAwRcXec5vas8WZL9zcu6KmoGnGxlZjm/yzdXeOruqylk/FxJHe6xV4TlGOpJp4G2U aoIo11CcgecXQRGjFz76YFJtvHpOqdbEZcSXOpenXZGciBslqa41nYiQFlpqo/KRe8Qr zjrDc6XHD1M7jhl9rsnOwLBtIMueQlBvdy6vm//6fGZN2lmI6H/NHZfHlPvCSfgCje6H sa5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722958671; x=1723563471; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=eOuBwftfA7fRjO8FH2eSlza624bLi0kcVZIjvBjv+eA=; b=RgX3Iopjc+PCC/tItHg3BQnko7QUZNGG7zhc4m+9wWZeWNfUuPP6Vwjn0YKHc0NHLq /1dgLdr59bXFtoCTDYWVH2qyTxkmYMrfdRP299NfIIatWpddAz1xMWDrhzwFXntVrpzJ 01Yf0FDPvFwGn2LeUFPHLSXU9G6K85BNSzF/A6Uh6ZVbiBipkzAGft2CHRpmND6tG8Wa RW+Il5yWMpitCTSEYRgyqN3S5QkfNykEg+F5KF9DDbo46nwnQ/UPH5FYbXDB5zJQQNEm XHMhP+zpsrg0EuaxpsGB3NPto1RABh1vGAwEZHzQbeQ8uEAgSQ47Qicf6g55EtBZcuWk BphA== X-Gm-Message-State: AOJu0YyfSUjWiJKl9akPHERZ/viML4tQ7aSbupItVd/7HD2Snj13tnca zMu0HPwo8rNdXX1pEyJZx3AUD1nRvFr7zWoVemAkXpNMmROoGOHDF9xUI0lXfTTS3qax3SbK2k9 m X-Google-Smtp-Source: AGHT+IFgW9HUiNU5DlcU0Tvjm33xO+S7+7jG7w8AWmgLBNwP8vg8trxft9xPFO9bYFcNujkxivQowg== X-Received: by 2002:a05:6902:1b04:b0:e0d:19b9:51c9 with SMTP id 3f1490d57ef6-e0d19b962dbmr8024582276.27.1722958670689; Tue, 06 Aug 2024 08:37:50 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 3f1490d57ef6-e0be556426fsm1751207276.47.2024.08.06.08.37.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Aug 2024 08:37:50 -0700 (PDT) Date: Tue, 6 Aug 2024 11:37:49 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v3 13/19] midx: teach `midx_preferred_pack()` about incremental MIDXs Message-ID: <2f98ebb141e7d9231d9f7688b3ebab849e8fa689.1722958596.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The function `midx_preferred_pack()` is used to determine the identity of the preferred pack, which is the identity of a unique pack within the MIDX which is used as a tie-breaker when selecting from which pack to represent an object that appears in multiple packs within the MIDX. Historically we have said that the MIDX's preferred pack has the unique property that all objects from that pack are represented in the MIDX. But that isn't quite true: a more precise statement would be that all objects from that pack *which appear in the MIDX* are selected from that pack. This helps us extend the concept of preferred packs across a MIDX chain, where some object(s) in the preferred pack may appear in other packs in an earlier MIDX layer, in which case those object(s) will not appear in a subsequent MIDX layer from either the preferred pack or any other pack. Extend the concept of preferred packs by using the pack which represents the object at the first position in MIDX pseudo-pack order belonging to the current MIDX layer (i.e., at position 'm->num_objects_in_base'). Signed-off-by: Taylor Blau --- midx.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/midx.c b/midx.c index 454c27b673..918349576f 100644 --- a/midx.c +++ b/midx.c @@ -501,13 +501,16 @@ int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name) int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id) { if (m->preferred_pack_idx == -1) { + uint32_t midx_pos; if (load_midx_revindex(m) < 0) { m->preferred_pack_idx = -2; return -1; } - m->preferred_pack_idx = - nth_midxed_pack_int_id(m, pack_pos_to_midx(m, 0)); + midx_pos = pack_pos_to_midx(m, m->num_objects_in_base); + + m->preferred_pack_idx = nth_midxed_pack_int_id(m, midx_pos); + } else if (m->preferred_pack_idx == -2) return -1; /* no revindex */ From patchwork Tue Aug 6 15:37:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13755146 Received: from mail-yb1-f171.google.com (mail-yb1-f171.google.com [209.85.219.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0DB371803A for ; Tue, 6 Aug 2024 15:37:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958676; cv=none; b=EKoUJf1kmAp2qKBJJLyuhWdtnrsPWudXEVmjjGThvbUg6Pr5n5SQ23aAnC0CXS2WOkXnzFE6YSQqI2AQgklEOPvoWBX/nyeeXO1ERe4n7n1QO6oIgkqQ/gBOwQmdeeKGK9zwCP/rsDX2ETofwfAUwAipNnC/bHjOF7LkTT0Px+0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958676; c=relaxed/simple; bh=Xym2az8nIzWRJRGL1p4CLKPdfGRz5u+uDyzG/rgQtKA=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=CZj17CgWEQ1ai3F2fA1SgkCKccyrQFGrmQJByn5xclyyaL/q+8KsfG9G/3z1xlWUjqo+wchbY8hQ77UOE8HL+ZrCeM2Wcq03W8lEv7xXpxnjmfoCdY8Xp+L9e7wfdDmF9TMaEq8GfWFxAQgUuxQPsOEDWYC8uggaT5jTEPuavEA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=I3Tfi+au; arc=none smtp.client-ip=209.85.219.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="I3Tfi+au" Received: by mail-yb1-f171.google.com with SMTP id 3f1490d57ef6-e0b167da277so759256276.3 for ; Tue, 06 Aug 2024 08:37:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1722958674; x=1723563474; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=tR+XHDDSkiW4aIWdzDSuaHryfMbsYU/x3XE0s8rQEBQ=; b=I3Tfi+auv11NCzcNC/QiBSNrt8f39NyB7WS08L7JZZoFh3ySjXVRlufMb57vec99Ml l0dW8iLZYu1IDjRp2J+Mu/wBU5lhE2Lzxu2GyFTIs/4aKQnRLmFtDlPw6M/U56NQMOmx 5GnfTfj56c0gaCVzHmD9D9RShfrQfplmEI9WI1OdyRcZvH24x1MIVNGwiTxISvjkiG/Z SsC+K4SN32bR8mwn7FyExAQQx3D7xkKtvcUWHweI6warEqnUxJd1yP3x2NiK1tcaxId7 jSMhxzqk01IKjOcS/g6zMzK3BEoh6ppQqKaQZJ37d0cMmYUDtFrOPF8YzPSIOD2bRd6y gnWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722958674; x=1723563474; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=tR+XHDDSkiW4aIWdzDSuaHryfMbsYU/x3XE0s8rQEBQ=; b=kyDCB4Bjh/qLlz5J0/XX1Dfxch0I+uwc8C+ouGD0o4KR5JjvkoZ9lCZzk75ZVEc2nd vJffv22Z0duusiC6QihK9At+jSP1KGRskg3TXgG123bEq35VytWqGUmthe3oAf/sBSOX 8ZkhDKw9E/3kKwkL3ukBapTczipx9bnRQ6n5W/Qi3wufYma+XAmKc6YDaw9cGby9bbPr 5We0Tf655iE/NfbjR17fUnyAYOLbfx7RbxlduCDUS1PXfhonsQfqugHR63uqaGJp+7A3 8u9GavNWMVYbsdP1zPifYl5Lc41qJI+Fw9ifbTu0Zc6Z5rADnu5hMP9OJum61WSjkxTJ nxHA== X-Gm-Message-State: AOJu0YwK+QSGx8zx00Xhqir49jlbF6NJA1miRbJCF/esEF7dOMJhr69t xTfFzZT2+I0Y6F2ehe8ccQpej8ViMERlY8SAVekr9lEaG5ydDA3YO7pCwtF6hWzEmjrKN0a4AQM Q X-Google-Smtp-Source: AGHT+IF038dUDNykiYgC+GEQSarsqeOXFnyBL3m4Owx81JL4Rv5pfnx7sw1hztaVZjy3werrBMvkQw== X-Received: by 2002:a5b:708:0:b0:e0b:c50c:e26a with SMTP id 3f1490d57ef6-e0bde4cad35mr15969452276.35.1722958673682; Tue, 06 Aug 2024 08:37:53 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 3f1490d57ef6-e0be50ef3f8sm1737944276.0.2024.08.06.08.37.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Aug 2024 08:37:53 -0700 (PDT) Date: Tue, 6 Aug 2024 11:37:52 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v3 14/19] midx: teach `midx_fanout_add_midx_fanout()` about incremental MIDXs Message-ID: <550ae2dc933c6d0c45fead9737e405c4025af9ad.1722958596.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The function `midx_fanout_add_midx_fanout()` is used to help construct the fanout table when generating a MIDX by reusing data from an existing MIDX. Prepare this function to work with incremental MIDXs by making a few changes: - The bounds checks need to be adjusted to start object lookups taking into account the number of objects in the previous MIDX layer (i.e., by starting the lookups at position `m->num_objects_in_base` instead of position 0). - Likewise, the bounds checks need to end at `m->num_objects_in_base` objects after `m->num_objects`. - Finally, `midx_fanout_add_midx_fanout()` needs to recur on earlier MIDX layers when dealing with an incremental MIDX chain by calling itself when given a MIDX with a non-NULL `base_midx`. Note that after 0c5a62f14b (midx-write.c: do not read existing MIDX with `packs_to_include`, 2024-06-11), we do not use this function with an existing MIDX (incremental or not) when generating a MIDX with --stdin-packs, and likewise for incremental MIDXs. But it is still used when adding the fanout table from an incremental MIDX when generating a non-incremental MIDX (without --stdin-packs, of course). Signed-off-by: Taylor Blau --- midx-write.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/midx-write.c b/midx-write.c index a77ee73c68..0accbdbb04 100644 --- a/midx-write.c +++ b/midx-write.c @@ -196,7 +196,7 @@ static int nth_midxed_pack_midx_entry(struct multi_pack_index *m, struct pack_midx_entry *e, uint32_t pos) { - if (pos >= m->num_objects) + if (pos >= m->num_objects + m->num_objects_in_base) return 1; nth_midxed_object_oid(&e->oid, m, pos); @@ -247,12 +247,16 @@ static void midx_fanout_add_midx_fanout(struct midx_fanout *fanout, uint32_t cur_fanout, int preferred_pack) { - uint32_t start = 0, end; + uint32_t start = m->num_objects_in_base, end; uint32_t cur_object; + if (m->base_midx) + midx_fanout_add_midx_fanout(fanout, m->base_midx, cur_fanout, + preferred_pack); + if (cur_fanout) - start = ntohl(m->chunk_oid_fanout[cur_fanout - 1]); - end = ntohl(m->chunk_oid_fanout[cur_fanout]); + start += ntohl(m->chunk_oid_fanout[cur_fanout - 1]); + end = m->num_objects_in_base + ntohl(m->chunk_oid_fanout[cur_fanout]); for (cur_object = start; cur_object < end; cur_object++) { if ((preferred_pack > -1) && From patchwork Tue Aug 6 15:37:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13755147 Received: from mail-yb1-f171.google.com (mail-yb1-f171.google.com [209.85.219.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 100CF1803A for ; Tue, 6 Aug 2024 15:37:57 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958679; cv=none; b=Vnd44Vl89e5ATCHPOBoz1H2BCJIPmOn44b2aF0hCpOHo+1H0W4e1bFMoYmw5Ip2ZLEZx8ZgD2DZenQgs6eH7v2Ake/sz5NP19LpHsRvX1MmRuQT/CMWnJUgFAvPzh1yrCDvxX9Xd3zyr8bYVEAQ4z7UdaXTM0UmkW+kyRBAysZY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958679; c=relaxed/simple; bh=Uy8f/30CPr39pMy8j+vmm6iKv4aGmr1jUvukfkAYTyI=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=PmtM/qwAy9t/aGlxNxj1ZuUQO12r/zuXi5Kmlb9btadSJz+EP9fkOnqs+Hi2uMlzsTQubSGGyvPQQIiOEpz+S6svQrNe37xvdYQ/VlEVQJSn1BugMfulbaKd3X5E2i1GTXRI8pYmZqPKRwEARjT+tH1k01f3LjNKgLtcmaRffh8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=C8XP77/F; arc=none smtp.client-ip=209.85.219.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="C8XP77/F" Received: by mail-yb1-f171.google.com with SMTP id 3f1490d57ef6-e02c4983bfaso725219276.2 for ; Tue, 06 Aug 2024 08:37:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1722958677; x=1723563477; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=jWyo0p255wmhMsdmGGrbGOO77FA2SMBg+lsT8ykIVyk=; b=C8XP77/FvFK2MjQ5nokmKbBb/v6w3or+T9l9wHs6Z7rvc7adJ1feUZ6M0tV3YT8g6L l1qBbjfnPB+51EKbmBgXlTnzTTHEu6TJoF7N9Ow7KB0jmIywXgwkDba1rxrDPPxKpZh3 YgzlUiLr8jJQ8HERWA9wTQYoZDp1tC2zqsIpWb/+rchwTA9FJJhjC1cHM3orVmKMUvmo jeTxBDwUHwp/svJA2Xo/H3h1AHypwTtamN+fVo8qrtbJfNirdUkaR/hbIx7uUdpquDnB 3kd2ITId8S5as527nlzBBxLZUMS/qB6k1wsMzShf4Q6hu9Cq/rM2aRH8+EMe9ig63KOf LRKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722958677; x=1723563477; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=jWyo0p255wmhMsdmGGrbGOO77FA2SMBg+lsT8ykIVyk=; b=N4u7MJdL8UrU3jAsNraptUtpr5a9dhBphl7EPFb5J9vNKob9j51zAKAPfoIVXbuA/z QpaIwulQamvGYrUyWGj6eo3xtRWXkGgTp2IA4zJflEnp6d49tsBDp56XSHqw/m/ZTfj3 1eUNnAqprln10433ARKd1r0pybVsiRt4HTiNBCSw2L/YB/RvBJnHoFRurYi/1mSVZvtA Xa5/KJ+QdP1lrO1j6A5l/eoIueg0L+R6uLAURDvzszMi+etgKMcdoY5BBtLqBtvU0Loy TCegaCCuHByFQQuHDyBHw2iW9gpmRx+UXleRpujL7H1jS6uPnt8CJqY+GjlT4kZz1Nbm L/jw== X-Gm-Message-State: AOJu0Yx6VmfwhOuLIPNVJTlf4/0DMmuA/UBL5RQnHcPB7erSlPe7MSWS +rYrsJbrliOqWa5JXCfB5JsKxRQ7EP3QJBQPchQzAB/nE+RNoGheYg1v8LLF1PEgJ3uaQenVClw B X-Google-Smtp-Source: AGHT+IHWofn3Mc+yKK2xpPOOW0T8uNr09TD5NuXKZwSKbcnFVlaPxw57ejcwzXI7tXIh4sKaxYfJwQ== X-Received: by 2002:a05:6902:703:b0:e07:2e88:d88a with SMTP id 3f1490d57ef6-e0bde219350mr21375675276.1.1722958676826; Tue, 06 Aug 2024 08:37:56 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 3f1490d57ef6-e0be55642fdsm1751675276.44.2024.08.06.08.37.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Aug 2024 08:37:56 -0700 (PDT) Date: Tue, 6 Aug 2024 11:37:55 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v3 15/19] midx: support reading incremental MIDX chains Message-ID: <9ae1bc415e9973a403a9bb3adb8a97df3bdfad2a.1722958596.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Now that the MIDX machinery's internals have been taught to understand incremental MIDXs over the previous handful of commits, the MIDX machinery itself can begin reading incremental MIDXs. (Note that while the on-disk format for incremental MIDXs has been defined, the writing end has not been implemented. This will take place in the commit after next.) The core of this change involves following the order specified in the MIDX chain in reverse and opening up MIDXs in the chain one-by-one, adding them to the previous layer's `->base_midx` pointer at each step. In order to implement this, the `load_multi_pack_index()` function is taught to call a new `load_multi_pack_index_chain()` function if loading a non-incremental MIDX failed via `load_multi_pack_index_one()`. When loading a MIDX chain, `load_midx_chain_fd_st()` reads each line in the file one-by-one and dispatches calls to `load_multi_pack_index_one()` to read each layer of the MIDX chain. When a layer was successfully read, it is added to the MIDX chain by calling `add_midx_to_chain()` which validates the contents of the `BASE` chunk, performs some bounds checks on the number of combined packs and objects, and attaches the new MIDX by assigning its `base_midx` pointer to the existing part of the chain. As a supplement to this, introduce a new mode in the test-read-midx test-tool which allows us to read the information for a specific MIDX in the chain by specifying its trailing checksum via the command-line arguments like so: $ test-tool read-midx .git/objects [checksum] Signed-off-by: Taylor Blau --- midx.c | 184 +++++++++++++++++++++++++++++++++++--- midx.h | 7 ++ packfile.c | 5 +- t/helper/test-read-midx.c | 24 +++-- 4 files changed, 201 insertions(+), 19 deletions(-) diff --git a/midx.c b/midx.c index 918349576f..54c06cbb86 100644 --- a/midx.c +++ b/midx.c @@ -91,7 +91,9 @@ static int midx_read_object_offsets(const unsigned char *chunk_start, #define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + the_hash_algo->rawsz) -struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local) +static struct multi_pack_index *load_multi_pack_index_one(const char *object_dir, + const char *midx_name, + int local) { struct multi_pack_index *m = NULL; int fd; @@ -99,31 +101,26 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local size_t midx_size; void *midx_map = NULL; uint32_t hash_version; - struct strbuf midx_name = STRBUF_INIT; uint32_t i; const char *cur_pack_name; struct chunkfile *cf = NULL; - get_midx_filename(&midx_name, object_dir); - - fd = git_open(midx_name.buf); + fd = git_open(midx_name); if (fd < 0) goto cleanup_fail; if (fstat(fd, &st)) { - error_errno(_("failed to read %s"), midx_name.buf); + error_errno(_("failed to read %s"), midx_name); goto cleanup_fail; } midx_size = xsize_t(st.st_size); if (midx_size < MIDX_MIN_SIZE) { - error(_("multi-pack-index file %s is too small"), midx_name.buf); + error(_("multi-pack-index file %s is too small"), midx_name); goto cleanup_fail; } - strbuf_release(&midx_name); - midx_map = xmmap(NULL, midx_size, PROT_READ, MAP_PRIVATE, fd, 0); close(fd); @@ -213,7 +210,6 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local cleanup_fail: free(m); - strbuf_release(&midx_name); free_chunkfile(cf); if (midx_map) munmap(midx_map, midx_size); @@ -222,6 +218,173 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local return NULL; } +void get_midx_chain_dirname(struct strbuf *buf, const char *object_dir) +{ + strbuf_addf(buf, "%s/pack/multi-pack-index.d", object_dir); +} + +void get_midx_chain_filename(struct strbuf *buf, const char *object_dir) +{ + get_midx_chain_dirname(buf, object_dir); + strbuf_addstr(buf, "/multi-pack-index-chain"); +} + +void get_split_midx_filename_ext(struct strbuf *buf, const char *object_dir, + const unsigned char *hash, const char *ext) +{ + get_midx_chain_dirname(buf, object_dir); + strbuf_addf(buf, "/multi-pack-index-%s.%s", hash_to_hex(hash), ext); +} + +static int open_multi_pack_index_chain(const char *chain_file, + int *fd, struct stat *st) +{ + *fd = git_open(chain_file); + if (*fd < 0) + return 0; + if (fstat(*fd, st)) { + close(*fd); + return 0; + } + if (st->st_size < the_hash_algo->hexsz) { + close(*fd); + if (!st->st_size) { + /* treat empty files the same as missing */ + errno = ENOENT; + } else { + warning(_("multi-pack-index chain file too small")); + errno = EINVAL; + } + return 0; + } + return 1; +} + +static int add_midx_to_chain(struct multi_pack_index *midx, + struct multi_pack_index *midx_chain, + struct object_id *oids, + int n) +{ + if (midx_chain) { + if (unsigned_add_overflows(midx_chain->num_packs, + midx_chain->num_packs_in_base)) { + warning(_("pack count in base MIDX too high: %"PRIuMAX), + (uintmax_t)midx_chain->num_packs_in_base); + return 0; + } + if (unsigned_add_overflows(midx_chain->num_objects, + midx_chain->num_objects_in_base)) { + warning(_("object count in base MIDX too high: %"PRIuMAX), + (uintmax_t)midx_chain->num_objects_in_base); + return 0; + } + midx->num_packs_in_base = midx_chain->num_packs + + midx_chain->num_packs_in_base; + midx->num_objects_in_base = midx_chain->num_objects + + midx_chain->num_objects_in_base; + } + + midx->base_midx = midx_chain; + midx->has_chain = 1; + + return 1; +} + +static struct multi_pack_index *load_midx_chain_fd_st(const char *object_dir, + int local, + int fd, struct stat *st, + int *incomplete_chain) +{ + struct multi_pack_index *midx_chain = NULL; + struct strbuf buf = STRBUF_INIT; + struct object_id *layers = NULL; + int valid = 1; + uint32_t i, count; + FILE *fp = xfdopen(fd, "r"); + + count = st->st_size / (the_hash_algo->hexsz + 1); + CALLOC_ARRAY(layers, count); + + for (i = 0; i < count; i++) { + struct multi_pack_index *m; + + if (strbuf_getline_lf(&buf, fp) == EOF) + break; + + if (get_oid_hex(buf.buf, &layers[i])) { + warning(_("invalid multi-pack-index chain: line '%s' " + "not a hash"), + buf.buf); + valid = 0; + break; + } + + valid = 0; + + strbuf_reset(&buf); + get_split_midx_filename_ext(&buf, object_dir, layers[i].hash, + MIDX_EXT_MIDX); + m = load_multi_pack_index_one(object_dir, buf.buf, local); + + if (m) { + if (add_midx_to_chain(m, midx_chain, layers, i)) { + midx_chain = m; + valid = 1; + } else { + close_midx(m); + } + } + if (!valid) { + warning(_("unable to find all multi-pack index files")); + break; + } + } + + free(layers); + fclose(fp); + strbuf_release(&buf); + + *incomplete_chain = !valid; + return midx_chain; +} + +static struct multi_pack_index *load_multi_pack_index_chain(const char *object_dir, + int local) +{ + struct strbuf chain_file = STRBUF_INIT; + struct stat st; + int fd; + struct multi_pack_index *m = NULL; + + get_midx_chain_filename(&chain_file, object_dir); + if (open_multi_pack_index_chain(chain_file.buf, &fd, &st)) { + int incomplete; + /* ownership of fd is taken over by load function */ + m = load_midx_chain_fd_st(object_dir, local, fd, &st, + &incomplete); + } + + strbuf_release(&chain_file); + return m; +} + +struct multi_pack_index *load_multi_pack_index(const char *object_dir, + int local) +{ + struct strbuf midx_name = STRBUF_INIT; + struct multi_pack_index *m; + + get_midx_filename(&midx_name, object_dir); + + m = load_multi_pack_index_one(object_dir, midx_name.buf, local); + if (!m) + m = load_multi_pack_index_chain(object_dir, local); + + strbuf_release(&midx_name); + + return m; +} + void close_midx(struct multi_pack_index *m) { uint32_t i; @@ -230,6 +393,7 @@ void close_midx(struct multi_pack_index *m) return; close_midx(m->next); + close_midx(m->base_midx); munmap((unsigned char *)m->data, m->data_len); diff --git a/midx.h b/midx.h index 86af7dfc5e..94de16a8c4 100644 --- a/midx.h +++ b/midx.h @@ -24,6 +24,7 @@ struct bitmapped_pack; #define MIDX_CHUNKID_OBJECTOFFSETS 0x4f4f4646 /* "OOFF" */ #define MIDX_CHUNKID_LARGEOFFSETS 0x4c4f4646 /* "LOFF" */ #define MIDX_CHUNKID_REVINDEX 0x52494458 /* "RIDX" */ +#define MIDX_CHUNKID_BASE 0x42415345 /* "BASE" */ #define MIDX_CHUNK_OFFSET_WIDTH (2 * sizeof(uint32_t)) #define MIDX_LARGE_OFFSET_NEEDED 0x80000000 @@ -50,6 +51,7 @@ struct multi_pack_index { int preferred_pack_idx; int local; + int has_chain; const unsigned char *chunk_pack_names; size_t chunk_pack_names_len; @@ -80,11 +82,16 @@ struct multi_pack_index { #define MIDX_EXT_REV "rev" #define MIDX_EXT_BITMAP "bitmap" +#define MIDX_EXT_MIDX "midx" const unsigned char *get_midx_checksum(struct multi_pack_index *m); void get_midx_filename(struct strbuf *out, const char *object_dir); void get_midx_filename_ext(struct strbuf *out, const char *object_dir, const unsigned char *hash, const char *ext); +void get_midx_chain_dirname(struct strbuf *buf, const char *object_dir); +void get_midx_chain_filename(struct strbuf *buf, const char *object_dir); +void get_split_midx_filename_ext(struct strbuf *buf, const char *object_dir, + const unsigned char *hash, const char *ext); struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local); int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id); diff --git a/packfile.c b/packfile.c index 813584646f..1eb18e3041 100644 --- a/packfile.c +++ b/packfile.c @@ -880,7 +880,8 @@ static void prepare_pack(const char *full_name, size_t full_name_len, if (!report_garbage) return; - if (!strcmp(file_name, "multi-pack-index")) + if (!strcmp(file_name, "multi-pack-index") || + !strcmp(file_name, "multi-pack-index.d")) return; if (starts_with(file_name, "multi-pack-index") && (ends_with(file_name, ".bitmap") || ends_with(file_name, ".rev"))) @@ -1064,7 +1065,7 @@ struct packed_git *get_all_packs(struct repository *r) prepare_packed_git(r); for (m = r->objects->multi_pack_index; m; m = m->next) { uint32_t i; - for (i = 0; i < m->num_packs; i++) + for (i = 0; i < m->num_packs + m->num_packs_in_base; i++) prepare_midx_pack(r, m, i); } diff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c index 83effc2b5f..69757e94fc 100644 --- a/t/helper/test-read-midx.c +++ b/t/helper/test-read-midx.c @@ -9,8 +9,10 @@ #include "packfile.h" #include "setup.h" #include "gettext.h" +#include "pack-revindex.h" -static int read_midx_file(const char *object_dir, int show_objects) +static int read_midx_file(const char *object_dir, const char *checksum, + int show_objects) { uint32_t i; struct multi_pack_index *m; @@ -21,6 +23,13 @@ static int read_midx_file(const char *object_dir, int show_objects) if (!m) return 1; + if (checksum) { + while (m && strcmp(hash_to_hex(get_midx_checksum(m)), checksum)) + m = m->base_midx; + if (!m) + return 1; + } + printf("header: %08x %d %d %d %d\n", m->signature, m->version, @@ -54,7 +63,8 @@ static int read_midx_file(const char *object_dir, int show_objects) struct pack_entry e; for (i = 0; i < m->num_objects; i++) { - nth_midxed_object_oid(&oid, m, i); + nth_midxed_object_oid(&oid, m, + i + m->num_objects_in_base); fill_midx_entry(the_repository, &oid, &e, m); printf("%s %"PRIu64"\t%s\n", @@ -111,7 +121,7 @@ static int read_midx_bitmapped_packs(const char *object_dir) if (!midx) return 1; - for (i = 0; i < midx->num_packs; i++) { + for (i = 0; i < midx->num_packs + midx->num_packs_in_base; i++) { if (nth_bitmapped_pack(the_repository, midx, &pack, i) < 0) return 1; @@ -127,16 +137,16 @@ static int read_midx_bitmapped_packs(const char *object_dir) int cmd__read_midx(int argc, const char **argv) { - if (!(argc == 2 || argc == 3)) - usage("read-midx [--show-objects|--checksum|--preferred-pack|--bitmap] "); + if (!(argc == 2 || argc == 3 || argc == 4)) + usage("read-midx [--show-objects|--checksum|--preferred-pack|--bitmap] "); if (!strcmp(argv[1], "--show-objects")) - return read_midx_file(argv[2], 1); + return read_midx_file(argv[2], argv[3], 1); else if (!strcmp(argv[1], "--checksum")) return read_midx_checksum(argv[2]); else if (!strcmp(argv[1], "--preferred-pack")) return read_midx_preferred_pack(argv[2]); else if (!strcmp(argv[1], "--bitmap")) return read_midx_bitmapped_packs(argv[2]); - return read_midx_file(argv[1], 0); + return read_midx_file(argv[1], argv[2], 0); } From patchwork Tue Aug 6 15:37:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13755148 Received: from mail-yw1-f182.google.com (mail-yw1-f182.google.com [209.85.128.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 22F9143AD5 for ; Tue, 6 Aug 2024 15:38:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958682; cv=none; b=D7Wl+FSFuQqTyiN3HDdjzHxCVwOfxaJV/+WXKiwjgTZirSMF5NXUrU/NYmmQ+aGa2yiRRlDS6z1EmpO9lXJ2KAy26aIWGu9pql1Nz81K9eF0DDylofW2hfcSr3zid9TwjvT0T6J9Or5X7a546N6YOyt2UuArLXB5Ca33J3z48lc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958682; c=relaxed/simple; bh=cMl4GsdGNvRlSUP9EPtPS8BErGdng6cU/BZWX4GJoJc=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=kFm9jkIMxEuZUaWoNgAMhDfuXFcwN1BpKGef/66VMVIjGki6bx8uX9c0OmdoMsC9UFxA6BfkF8V6sMpR4Hum6cBpr7REa61/jC5wbU2smfVxktz/gXZ6Ezur4B6M3t1wR7LnOxc3GfkkA08unMpXw+NzmanPtW1n/kWQ6nO8aZQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=A8BwSGTq; arc=none smtp.client-ip=209.85.128.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="A8BwSGTq" Received: by mail-yw1-f182.google.com with SMTP id 00721157ae682-66599ca3470so7227457b3.2 for ; Tue, 06 Aug 2024 08:38:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1722958680; x=1723563480; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=W3JV7TWGYNq5kqJPCnyFKrRAe3+NDxsE+gIopkSyuIw=; b=A8BwSGTq0ee7HtnCPLsYmLoLWB47vQOCYMI9ZWp4P7HIB/zANTyYsz4eohTMliN//O HLk5nq/XJb/bj/y94VFkcpVFX4IcK2PMY6lSd7T8qEjkckqGaCGoa717ia0/cnef6jqA 95Skq/Ivc4q1QIBQFEcvUNXG0SiuC8CSLCSGas7vZRY/rEFZ8/JlySXrNGXrBozxTkxf 74l5Pao6bo8C00RTr7hik5eXPKRpkaWtpZVvdqSHlr7A2NO4BKUiOBnqwCugPaNxqxCZ H2gKjE2Bp2USazNlHPKiQu5eOx9a4rE7INv6WWkU1WWDpUbvgnaYgVelOsdj6U4/7tSq Isww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722958680; x=1723563480; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=W3JV7TWGYNq5kqJPCnyFKrRAe3+NDxsE+gIopkSyuIw=; b=HObW5si4xQZKyqqACViBsHCcaLkSLb+dmRZGqYVcT/2UJ04+KczV80vyzI/rFjKifW sZ9QafIWjfspRdS/UQYzsPXnVEWX4K90OrBkLi9kPG+BqwfbT56wfSCGfviaIkrGdzT0 elRQmNehlilQzIG3I5nP6XNk3IXqUbEJX09K1kyhltYPvudYZPWqGq7BZzhivpVPGIxn Ndj0bgE+P7TM9G1DVR0nDU9AhrFjP449bgEjfyMNoltl1ZwfzEK2CLcYPJC+ZJvV5tc+ ZIQmHTjhoDa+mvL6LIoLEzsPSq57S7oGKX18R+viQwzXjnWuZP3makOKKTlPwrml/Uxz qf0g== X-Gm-Message-State: AOJu0YznAOwecLFTb8HSt6An8/xAy2ZQ3uXdHjJ59aVqQ2wojIfRZiU3 LZF+f8j0CqFAcLCSsMw3akIb6WJy3s+Q21Wi56+bpTy7t7JcjTaUSLDuJGawfxeE59nPAR3LVlN W X-Google-Smtp-Source: AGHT+IGZDKIzFP5hJcsjKekoA5LbOA5PYiwSSJV36nOH2i7iX4ond/DJ6k0FaCE2qM6IxgyVyvWNJw== X-Received: by 2002:a0d:c647:0:b0:64a:79f6:2f2d with SMTP id 00721157ae682-689637fcb2emr154411967b3.31.1722958679774; Tue, 06 Aug 2024 08:37:59 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 00721157ae682-68a14157203sm15682237b3.145.2024.08.06.08.37.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Aug 2024 08:37:59 -0700 (PDT) Date: Tue, 6 Aug 2024 11:37:58 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v3 16/19] midx: implement verification support for incremental MIDXs Message-ID: <3d4181df518ece2268534182a11f66b2d84302e9.1722958596.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Teach the verification implementation used by `git multi-pack-index verify` to perform verification for incremental MIDX chains by independently validating each layer within the chain. Signed-off-by: Taylor Blau --- midx.c | 47 ++++++++++++++++++++++++++++++----------------- midx.h | 2 ++ 2 files changed, 32 insertions(+), 17 deletions(-) diff --git a/midx.c b/midx.c index 54c06cbb86..a53d65702d 100644 --- a/midx.c +++ b/midx.c @@ -470,6 +470,13 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, return 0; } +struct packed_git *nth_midxed_pack(struct multi_pack_index *m, + uint32_t pack_int_id) +{ + uint32_t local_pack_int_id = midx_for_pack(&m, pack_int_id); + return m->packs[local_pack_int_id]; +} + #define MIDX_CHUNK_BITMAPPED_PACKS_WIDTH (2 * sizeof(uint32_t)) int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m, @@ -818,6 +825,7 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag uint32_t i; struct progress *progress = NULL; struct multi_pack_index *m = load_multi_pack_index(object_dir, 1); + struct multi_pack_index *curr; verify_midx_error = 0; if (!m) { @@ -840,8 +848,8 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag if (flags & MIDX_PROGRESS) progress = start_delayed_progress(_("Looking for referenced packfiles"), - m->num_packs); - for (i = 0; i < m->num_packs; i++) { + m->num_packs + m->num_packs_in_base); + for (i = 0; i < m->num_packs + m->num_packs_in_base; i++) { if (prepare_midx_pack(r, m, i)) midx_report("failed to load pack in position %d", i); @@ -861,17 +869,20 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag if (flags & MIDX_PROGRESS) progress = start_sparse_progress(_("Verifying OID order in multi-pack-index"), m->num_objects - 1); - for (i = 0; i < m->num_objects - 1; i++) { - struct object_id oid1, oid2; - nth_midxed_object_oid(&oid1, m, i); - nth_midxed_object_oid(&oid2, m, i + 1); + for (curr = m; curr; curr = curr->base_midx) { + for (i = 0; i < m->num_objects - 1; i++) { + struct object_id oid1, oid2; - if (oidcmp(&oid1, &oid2) >= 0) - midx_report(_("oid lookup out of order: oid[%d] = %s >= %s = oid[%d]"), - i, oid_to_hex(&oid1), oid_to_hex(&oid2), i + 1); + nth_midxed_object_oid(&oid1, m, m->num_objects_in_base + i); + nth_midxed_object_oid(&oid2, m, m->num_objects_in_base + i + 1); - midx_display_sparse_progress(progress, i + 1); + if (oidcmp(&oid1, &oid2) >= 0) + midx_report(_("oid lookup out of order: oid[%d] = %s >= %s = oid[%d]"), + i, oid_to_hex(&oid1), oid_to_hex(&oid2), i + 1); + + midx_display_sparse_progress(progress, i + 1); + } } stop_progress(&progress); @@ -881,8 +892,8 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag * each of the objects and only require 1 packfile to be open at a * time. */ - ALLOC_ARRAY(pairs, m->num_objects); - for (i = 0; i < m->num_objects; i++) { + ALLOC_ARRAY(pairs, m->num_objects + m->num_objects_in_base); + for (i = 0; i < m->num_objects + m->num_objects_in_base; i++) { pairs[i].pos = i; pairs[i].pack_int_id = nth_midxed_pack_int_id(m, i); } @@ -896,16 +907,18 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag if (flags & MIDX_PROGRESS) progress = start_sparse_progress(_("Verifying object offsets"), m->num_objects); - for (i = 0; i < m->num_objects; i++) { + for (i = 0; i < m->num_objects + m->num_objects_in_base; i++) { struct object_id oid; struct pack_entry e; off_t m_offset, p_offset; if (i > 0 && pairs[i-1].pack_int_id != pairs[i].pack_int_id && - m->packs[pairs[i-1].pack_int_id]) - { - close_pack_fd(m->packs[pairs[i-1].pack_int_id]); - close_pack_index(m->packs[pairs[i-1].pack_int_id]); + nth_midxed_pack(m, pairs[i-1].pack_int_id)) { + uint32_t pack_int_id = pairs[i-1].pack_int_id; + struct packed_git *p = nth_midxed_pack(m, pack_int_id); + + close_pack_fd(p); + close_pack_index(p); } nth_midxed_object_oid(&oid, m, pairs[i].pos); diff --git a/midx.h b/midx.h index 94de16a8c4..9d30935589 100644 --- a/midx.h +++ b/midx.h @@ -95,6 +95,8 @@ void get_split_midx_filename_ext(struct strbuf *buf, const char *object_dir, struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local); int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id); +struct packed_git *nth_midxed_pack(struct multi_pack_index *m, + uint32_t pack_int_id); int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m, struct bitmapped_pack *bp, uint32_t pack_int_id); int bsearch_one_midx(const struct object_id *oid, struct multi_pack_index *m, From patchwork Tue Aug 6 15:38:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13755149 Received: from mail-yw1-f177.google.com (mail-yw1-f177.google.com [209.85.128.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1624C46444 for ; Tue, 6 Aug 2024 15:38:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958685; cv=none; b=J5eEOWf8+K62q7I2k/kSBMPxcXcOuyjA2c8Gx6QAmjsYeF44upOeHpIt8YhQIXped68TFfeXlis8Qqqe23d5JULHETjrX+4AleT92Eczuoom87A7jXMCnT1O+Fb6aNGUx82x9ejZMLECU8GKPw06/2txOMdxAEEOJcFgDddfciI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958685; c=relaxed/simple; bh=AkkcNlUEyIY/IM7OggFsbSOBe5Zn4MGKI3vQhpx6lUo=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=RbCeFbxOmhj2Py3YVUCBTlh8MZREyFkOPm+zqJg2jDKvm8LPDzjD8A+cNrhHDC0s4T+s3FDAAzo8EnfTG+NOweTD+xRffZX0YKC3CjyU3L2x6ug9vEJvb3GViVv830GzR+3Zim9z3nb6+G5AuFqhI10ghN+L3Xhjh/uT3KTmM3Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=rl4f2N45; arc=none smtp.client-ip=209.85.128.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="rl4f2N45" Received: by mail-yw1-f177.google.com with SMTP id 00721157ae682-651da7c1531so5986227b3.0 for ; Tue, 06 Aug 2024 08:38:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1722958683; x=1723563483; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=+/am1NonnBXhLzCaIUQTlPIkaTQnuwqyi2QfqD+JGZs=; b=rl4f2N45rMWERtzOQd0iHG72xOKdq0x4UkzGnBaq1q41R6nc45fd4p48SfBLZe8Lpp GaZo1vpsdDrNDPRcJ/mkBLEoPQmarSqgzHDCKvt/JXh8v4e/pc9Us/uqcBcdHGFbaqKE JePjxkeUcd45GPc8liw0R78m/7F+NUpOk73zyT0jbDsiY5K0S7TWXnU+43GA0Sqhtj1e jnjy2tQemnQ/grigFybTBbUgwaKHcidtf2LfuwNFUhQpssKbAf136pFKISKinKfIb1z4 ZIMWE1/eOSW2bRMlTVI4BD7GJ4G64LRaZZNj5lvVknQ/BfbF+Xm0zRbSsCkbenDEz2O8 f/dg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722958683; x=1723563483; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=+/am1NonnBXhLzCaIUQTlPIkaTQnuwqyi2QfqD+JGZs=; b=XMdLKrh/xPJ99vFBpeio3WFDRZ8rLs+QQv3uFbaHb86ANi28tbBrx/GTQZB2Fr6qXP sUfvOTOFjzT/Flvg3COC9oGJM0BxypK5PuAPa2oPtmYowxub9h+Bao9zUsib++HYR4Lz 0JADgooxpA4bkFfzbR/6fnhjTl9aWxYWIdSSfAZzB5Okia6uRplS4j/bPZQTk+U60pW1 Pj4TBv2hD6PpDfyNrq/7CmQSIo0ReR2mpKZTCFy2Iki3rMhRTIac/L9/g+2ozSpIH/D6 R7SzNK48LzHfLXW5AeZVLw+q1e/hLeOkWa1LK+Duo8oW7OxtNDVqXIrXC5BEMacWCGXl C16w== X-Gm-Message-State: AOJu0YwXZm3MbF053Mc+5+EtRGTt3j79VmGykqVd4UgMoVZb3Qd/uj8A Y5WlmSsLTj2/xksJw91cUs34pPYy4uU8gFNCZyx32/tpcT2J1QnzCfJNum6WxYYQmwavvfRW/5k O X-Google-Smtp-Source: AGHT+IHBIain+h1ILlczGWyexwiOrvVpxnSMT0aRX0EY+uGB2MTRNeq/SmN0pQsq2WYfcdC4re2lBA== X-Received: by 2002:a0d:c803:0:b0:672:b4dd:eb41 with SMTP id 00721157ae682-6896077655bmr162257347b3.15.1722958682818; Tue, 06 Aug 2024 08:38:02 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 00721157ae682-690b5f3fd19sm9642247b3.91.2024.08.06.08.38.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Aug 2024 08:38:02 -0700 (PDT) Date: Tue, 6 Aug 2024 11:38:01 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v3 17/19] t: retire 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' Message-ID: <3b268f91bf3ec74cee5fc3ceaab19d700e30233e.1722958596.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Two years ago, commit ff1e653c8e2 (midx: respect 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP', 2021-08-31) introduced a new environment variable which caused the test suite to write MIDX bitmaps after any 'git repack' invocation. At the time, this was done to help flush out any bugs with MIDX bitmaps that weren't explicitly covered in the t5326-multi-pack-bitmap.sh script. Two years later, that flag has served us well and is no longer providing meaningful coverage, as the script in t5326 has matured substantially and covers many more interesting cases than it did back when ff1e653c8e2 was originally written. Remove the 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' environment variable as it is no longer serving a useful purpose. More importantly, removing this variable clears the way for us to introduce a new one to help similarly flush out bugs related to incremental MIDX chains. Because these incremental MIDX chains are (for now) incompatible with MIDX bitmaps, we cannot have both. Signed-off-by: Taylor Blau --- builtin/repack.c | 12 ++---------- ci/run-build-and-tests.sh | 1 - midx.h | 2 -- t/README | 4 ---- t/t0410-partial-clone.sh | 2 -- t/t5310-pack-bitmaps.sh | 4 ---- t/t5319-multi-pack-index.sh | 3 +-- t/t5326-multi-pack-bitmaps.sh | 3 +-- t/t5327-multi-pack-bitmaps-rev.sh | 5 ++--- t/t7700-repack.sh | 21 +++++++-------------- 10 files changed, 13 insertions(+), 44 deletions(-) diff --git a/builtin/repack.c b/builtin/repack.c index f0317fa94a..8499bf0e12 100644 --- a/builtin/repack.c +++ b/builtin/repack.c @@ -1217,10 +1217,6 @@ int cmd_repack(int argc, const char **argv, const char *prefix) if (!write_midx && (!(pack_everything & ALL_INTO_ONE) || !is_bare_repository())) write_bitmaps = 0; - } else if (write_bitmaps && - git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0) && - git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0)) { - write_bitmaps = 0; } if (pack_kept_objects < 0) pack_kept_objects = write_bitmaps > 0 && !write_midx; @@ -1518,12 +1514,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix) if (run_update_server_info) update_server_info(0); - if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0)) { - unsigned flags = 0; - if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0)) - flags |= MIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX; - write_midx_file(get_object_directory(), NULL, NULL, flags); - } + if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0)) + write_midx_file(get_object_directory(), NULL, NULL, 0); cleanup: string_list_clear(&names, 1); diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh index 98dda42045..e6fd68630c 100755 --- a/ci/run-build-and-tests.sh +++ b/ci/run-build-and-tests.sh @@ -25,7 +25,6 @@ linux-TEST-vars) export GIT_TEST_COMMIT_GRAPH=1 export GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1 export GIT_TEST_MULTI_PACK_INDEX=1 - export GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=1 export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master export GIT_TEST_NO_WRITE_REV_INDEX=1 export GIT_TEST_CHECKOUT_WORKERS=2 diff --git a/midx.h b/midx.h index 9d30935589..3714cad2cc 100644 --- a/midx.h +++ b/midx.h @@ -29,8 +29,6 @@ struct bitmapped_pack; #define MIDX_LARGE_OFFSET_NEEDED 0x80000000 #define GIT_TEST_MULTI_PACK_INDEX "GIT_TEST_MULTI_PACK_INDEX" -#define GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP \ - "GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP" struct multi_pack_index { struct multi_pack_index *next; diff --git a/t/README b/t/README index 724ee58195..3cee0c05e2 100644 --- a/t/README +++ b/t/README @@ -445,10 +445,6 @@ GIT_TEST_MULTI_PACK_INDEX=, when true, forces the multi-pack- index to be written after every 'git repack' command, and overrides the 'core.multiPackIndex' setting to true. -GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=, when true, sets the -'--bitmap' option on all invocations of 'git multi-pack-index write', -and ignores pack-objects' '--write-bitmap-index'. - GIT_TEST_SIDEBAND_ALL=, when true, overrides the 'uploadpack.allowSidebandAll' setting to true, and when false, forces fetch-pack to not request sideband-all (even if the server advertises diff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh index 2c30c86e7b..34bdb3ab1f 100755 --- a/t/t0410-partial-clone.sh +++ b/t/t0410-partial-clone.sh @@ -5,8 +5,6 @@ test_description='partial clone' . ./test-lib.sh . "$TEST_DIRECTORY"/lib-terminal.sh -# missing promisor objects cause repacks which write bitmaps to fail -GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 # When enabled, some commands will write commit-graphs. This causes fsck # to fail when delete_object() is called because fsck will attempt to # verify the out-of-sync commit graph. diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh index d7fd71360e..a6de7c5764 100755 --- a/t/t5310-pack-bitmaps.sh +++ b/t/t5310-pack-bitmaps.sh @@ -5,10 +5,6 @@ test_description='exercise basic bitmap functionality' . ./test-lib.sh . "$TEST_DIRECTORY"/lib-bitmap.sh -# t5310 deals only with single-pack bitmaps, so don't write MIDX bitmaps in -# their place. -GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 - # Likewise, allow individual tests to control whether or not they use # the boundary-based traversal. sane_unset GIT_TEST_PACK_USE_BITMAP_BOUNDARY_TRAVERSAL diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index ace5ac3b61..8c54fc0655 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -600,8 +600,7 @@ test_expect_success 'repack preserves multi-pack-index when creating packs' ' compare_results_with_midx "after repack" test_expect_success 'multi-pack-index and pack-bitmap' ' - GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \ - git -c repack.writeBitmaps=true repack -ad && + git -c repack.writeBitmaps=true repack -ad && git multi-pack-index write && git rev-list --test-bitmap HEAD ' diff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh index 916da389b6..1cb3e3ff08 100755 --- a/t/t5326-multi-pack-bitmaps.sh +++ b/t/t5326-multi-pack-bitmaps.sh @@ -4,10 +4,9 @@ test_description='exercise basic multi-pack bitmap functionality' . ./test-lib.sh . "${TEST_DIRECTORY}/lib-bitmap.sh" -# We'll be writing our own midx and bitmaps, so avoid getting confused by the +# We'll be writing our own MIDX, so avoid getting confused by the # automatic ones. GIT_TEST_MULTI_PACK_INDEX=0 -GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 # This test exercise multi-pack bitmap functionality where the object order is # stored and read from a special chunk within the MIDX, so use the default diff --git a/t/t5327-multi-pack-bitmaps-rev.sh b/t/t5327-multi-pack-bitmaps-rev.sh index e65e311cd7..23db949c20 100755 --- a/t/t5327-multi-pack-bitmaps-rev.sh +++ b/t/t5327-multi-pack-bitmaps-rev.sh @@ -5,10 +5,9 @@ test_description='exercise basic multi-pack bitmap functionality (.rev files)' . ./test-lib.sh . "${TEST_DIRECTORY}/lib-bitmap.sh" -# We'll be writing our own midx and bitmaps, so avoid getting confused by the -# automatic ones. +# We'll be writing our own MIDX, so avoid getting confused by the automatic +# ones. GIT_TEST_MULTI_PACK_INDEX=0 -GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 # Unlike t5326, this test exercise multi-pack bitmap functionality where the # object order is stored in a separate .rev file. diff --git a/t/t7700-repack.sh b/t/t7700-repack.sh index 127efe99f8..8f34f05087 100755 --- a/t/t7700-repack.sh +++ b/t/t7700-repack.sh @@ -70,14 +70,13 @@ test_expect_success 'objects in packs marked .keep are not repacked' ' test_expect_success 'writing bitmaps via command-line can duplicate .keep objects' ' # build on $oid, $packid, and .keep state from previous - GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 git repack -Adbl && + git repack -Adbl && test_has_duplicate_object true ' test_expect_success 'writing bitmaps via config can duplicate .keep objects' ' # build on $oid, $packid, and .keep state from previous - GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \ - git -c repack.writebitmaps=true repack -Adl && + git -c repack.writebitmaps=true repack -Adl && test_has_duplicate_object true ' @@ -284,8 +283,7 @@ test_expect_success 'repacking fails when missing .pack actually means missing o test_expect_success 'bitmaps are created by default in bare repos' ' git clone --bare .git bare.git && rm -f bare.git/objects/pack/*.bitmap && - GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \ - git -C bare.git repack -ad && + git -C bare.git repack -ad && bitmap=$(ls bare.git/objects/pack/*.bitmap) && test_path_is_file "$bitmap" ' @@ -296,8 +294,7 @@ test_expect_success 'incremental repack does not complain' ' ' test_expect_success 'bitmaps can be disabled on bare repos' ' - GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \ - git -c repack.writeBitmaps=false -C bare.git repack -ad && + git -c repack.writeBitmaps=false -C bare.git repack -ad && bitmap=$(ls bare.git/objects/pack/*.bitmap || :) && test -z "$bitmap" ' @@ -308,8 +305,7 @@ test_expect_success 'no bitmaps created if .keep files present' ' keep=${pack%.pack}.keep && test_when_finished "rm -f \"\$keep\"" && >"$keep" && - GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \ - git -C bare.git repack -ad 2>stderr && + git -C bare.git repack -ad 2>stderr && test_must_be_empty stderr && find bare.git/objects/pack/ -type f -name "*.bitmap" >actual && test_must_be_empty actual @@ -320,8 +316,7 @@ test_expect_success 'auto-bitmaps do not complain if unavailable' ' blob=$(test-tool genrandom big $((1024*1024)) | git -C bare.git hash-object -w --stdin) && git -C bare.git update-ref refs/tags/big $blob && - GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \ - git -C bare.git repack -ad 2>stderr && + git -C bare.git repack -ad 2>stderr && test_must_be_empty stderr && find bare.git/objects/pack -type f -name "*.bitmap" >actual && test_must_be_empty actual @@ -342,9 +337,7 @@ test_expect_success 'repacking with a filter works' ' ' test_expect_success '--filter fails with --write-bitmap-index' ' - test_must_fail \ - env GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \ - git -C bare.git repack -a -d --write-bitmap-index --filter=blob:none + test_must_fail git -C bare.git repack -a -d --write-bitmap-index --filter=blob:none ' test_expect_success 'repacking with two filters works' ' From patchwork Tue Aug 6 15:38:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13755150 Received: from mail-yw1-f180.google.com (mail-yw1-f180.google.com [209.85.128.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AC26B47772 for ; Tue, 6 Aug 2024 15:38:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958688; cv=none; b=eEOv4JlMQKIZLpTTEweA07DQKtrHIrI2fgCueSyF8Vt8SFqQk6iG4xYGr13pJ43iH92X/lm2846ZQ3dLM5iYIFJ0YKs5IcuE+P6hGTTWKl2hnXYV0zdQ3Ltm4ss1tXMHT+JvNZuGUGeui0ooaFDC1ZJof7Pd3zo+a9ns+bQv80w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958688; c=relaxed/simple; bh=7mA99otX3YxgIkm1b3cVceWLhaUOEfbg+9q+Yo9BPX0=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=N9cvfB7GBWYccxEmDBenTUiZDHARy2ISuzcEvY5OPxA2lzs0Rlz6LsArRHLoDaTfLRjpnzxXi2eU6uLatf4ZAY2o1Xphuuw6R/4LPD8kGBqL2ivQtDpomUEGC7qCUWPTNK13H4iPYFi8KIujWoYhzSd7RS+I7qCPPkcyywiFKqA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=Hvsp3wqd; arc=none smtp.client-ip=209.85.128.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="Hvsp3wqd" Received: by mail-yw1-f180.google.com with SMTP id 00721157ae682-66ca536621cso8865837b3.3 for ; Tue, 06 Aug 2024 08:38:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1722958685; x=1723563485; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=SXxwZuCPkk2zaVNnmidTRz6Z3l8hHv1xLJRhIL0foJI=; b=Hvsp3wqdxVoQV1qjEtPI/b3ckCYQbRFUhvqm3hzm37qwGLz/C2i/pSyesWKsCAWOlk iOxjjGFbrW4NO9bfxOmOrR1u9TyGWDkJaGxSbj41mVgE9hoT2HR06G/jxmBkd6JApoG/ LdvKk2zEzR2+Xz7WmlwCY9d8arWJ8IzqhTtYDec144GEPkBslbQ5bNk6Do7iHj8m1F1t rlw6KONSZoxl+019zVxydk0Avxsz0FAL72RWx6u+qiLocJD/FX12PgAo4M84YI1KFcLl wg0DoC+YIfg4NRhuyp3knFn6lVSI6QEATcDY+mj0f0PKYATsc505XXDHZ0Rpbxl5xM64 KMow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722958685; x=1723563485; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=SXxwZuCPkk2zaVNnmidTRz6Z3l8hHv1xLJRhIL0foJI=; b=a0XNDQiZ0OvQMNcTlTp6/6uThoN+9iYjv0Hn0bD56ZPgRkdwGqtmzglXYTz2WJJ13P CF0MN6FC6HFVcgGU8Mk30t9gE/a8cd2RUR+tcQUXFoKjbzBpeLNCf0TSAvRNYyYURu29 /Vp1CKhMa8LzOgAxpfIrCXZj5lwvt7rZN8S1IeEN0SwLeqrNnRJ9riyqs6K5HqH+OFwP s+7j3ctwbUJG2Bksa4tVeyI2UJBUdgmNwMPdetGFofUxmGRoughIYGZAp9L/31kwueN4 8xOB4jRgYBbPXcUIW6+JgNMRWuUNLrxj4Zl7ZkbnFTFmtqe/eyeueZb86JITc5aP4ST9 nlbQ== X-Gm-Message-State: AOJu0YyPJvv1oRQq9oalfamaiQvp8pN/SAy9ebJh2VPhblPOoKu8BY1f dXww6odccQe3Cw7WzggHuykXcbXO/rNjOBJUB4V+y/wgWMtokDyyX7xMHZm/XAu87g25TRyZ3Ri s X-Google-Smtp-Source: AGHT+IF5yRpZybwCKNx555lfdvlHJkojfN3tadsyOOXBiWfLcmORein8qQDsKH0cJPdrCo1J+Y3xxA== X-Received: by 2002:a81:a508:0:b0:647:e079:da73 with SMTP id 00721157ae682-6896058a01dmr144995277b3.10.1722958685697; Tue, 06 Aug 2024 08:38:05 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 00721157ae682-68a137b58c9sm15990047b3.118.2024.08.06.08.38.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Aug 2024 08:38:05 -0700 (PDT) Date: Tue, 6 Aug 2024 11:38:04 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v3 18/19] t/t5313-pack-bounds-checks.sh: prepare for sub-directories Message-ID: <09d74f89423e1a3b5486aa9691ffdb8dea2600f6.1722958596.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Prepare for sub-directories to appear in $GIT_DIR/objects/pack by adjusting the copy, remove, and chmod invocations to perform their behavior recursively. This prepares us for the new $GIT_DIR/objects/pack/multi-pack-index.d directory which will be added in a following commit. Signed-off-by: Taylor Blau --- t/t5313-pack-bounds-checks.sh | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/t/t5313-pack-bounds-checks.sh b/t/t5313-pack-bounds-checks.sh index ceaa6700a2..86fc73f9fb 100755 --- a/t/t5313-pack-bounds-checks.sh +++ b/t/t5313-pack-bounds-checks.sh @@ -7,11 +7,11 @@ TEST_PASSES_SANITIZE_LEAK=true clear_base () { test_when_finished 'restore_base' && - rm -f $base + rm -r -f $base } restore_base () { - cp base-backup/* .git/objects/pack/ + cp -r base-backup/* .git/objects/pack/ } do_pack () { @@ -64,9 +64,9 @@ test_expect_success 'set up base packfile and variables' ' git commit -m base && git repack -ad && base=$(echo .git/objects/pack/*) && - chmod +w $base && + chmod -R +w $base && mkdir base-backup && - cp $base base-backup/ && + cp -r $base base-backup/ && object=$(git rev-parse HEAD:file) ' From patchwork Tue Aug 6 15:38:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13755151 Received: from mail-yw1-f172.google.com (mail-yw1-f172.google.com [209.85.128.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 787F638DC7 for ; Tue, 6 Aug 2024 15:38:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958693; cv=none; b=HA5tiJHjeMkp2/KixedYzof6xsQygQXj2XqZP1utxrT4s9PRfiUXN4E8Q6ltS4cPmpbF2wlbQwQ3QHn7sNRPp5FDg90sMAgwQOnTXgggFvqs2V5kTGGWzmDmc0qS3NVQUv09iRGGLZBsX4PPLMHF7/c2QNBMLsyn76xxP5vbBOk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722958693; c=relaxed/simple; bh=Nfct63PBRSjSkPyz31F3sxoIy/lDr2rYxqAMyfcn84o=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=lKfEJ5WH2I4fV/fTe2YyMFokWeYDomi4m9V1WwuBx9oQ0j6jUyY4NlqEUdnY9gS7yz0AByoEBeXyJH9jEtPqSyNYCZPP7+fPrOVCtYVdEo9kkjyCmzxva5T4lhOWn/yfxYYpFfpAmSvLGxIfVefIFOvQEhZA4e+oouM/7cY9Q5s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=gP6T1oQE; arc=none smtp.client-ip=209.85.128.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="gP6T1oQE" Received: by mail-yw1-f172.google.com with SMTP id 00721157ae682-690aabe2600so6888907b3.0 for ; Tue, 06 Aug 2024 08:38:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1722958689; x=1723563489; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=LyE1MJqiClcKIcidgeH1tIX2TW7IvdDHmyaZAGljyaw=; b=gP6T1oQEPufI2PeWXS3ZupLRbE0Ddl0LX2ihIeSHPgnVbj8yGvpyp64tODdaEebDBq AmP6i4883NlXCaWqEKQlyZSw5gLeJ/q7r9EDGVubutatggh1WEyhIZvYLqU8ZKoufSWL Tcg9qaAKcy68D0p/5NQ6McAPIUjGnwukfw5D+SVYYgLZlHIDdjUnYB+5+V7xJ5GAClng JO1MNFjXXs3ZQXyNwUUDJwjwTtAYI9EbJqzcmJYBQFDfEhlviHwt5+2NmTgwj4SZAIED 7VUwTc/2+KlpyBbr92An+BfSX9s4ShzlqyyC/Lb70j/16X7XnS+St+MnBk0qpzCleFyK b6AQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722958689; x=1723563489; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=LyE1MJqiClcKIcidgeH1tIX2TW7IvdDHmyaZAGljyaw=; b=eZqqNfJ7DG0KMuFQbcYcQtP2zvL2AdNliAi2B3HfyblhNQYSCWDA0loazrlx3LQiUk gY5SLJ/XLEl4xkjqA37FJwegGwo+wINz37keumD3oir7rHikPRiCcKlErr5ICALFINKp bOdM1AbevoPn6gQ9OBOXsf6K9SbVSWxKW7txl4YsEylGHxBra1xcgQndi/ZX3r1V651Z 62TY8eQFXAL0po/cQ42+u+ezxohxuvyHEpSI8p2UsKasRkGTvVttXVtSDYS1YVGnzAZz FQVw07Iu4vDSxDeaXV81ui+RQz14Zzab8+1pNkqmZ4xyagjl9N2MCMD+Bo6jAiJ7YYuo Ml5A== X-Gm-Message-State: AOJu0Yx3KM7uJcG5P7T7InDtblmdOCAK+woPO0rUhvGZThYq//lxpKKO 83a1YA/NgxqD1xcwMH+jEsCFzJG6XdP4tN35+T849LhjoEa54STiMSwa7aHz0e9vj3j+1vlnysy A X-Google-Smtp-Source: AGHT+IElQu8VInq0F2ceK3NbDU0u7AEunWogXLtx80dViyIVnUqAYK70TpaFlONqIp+r24GhPf9kAA== X-Received: by 2002:a81:7c0a:0:b0:64b:69f0:f8f2 with SMTP id 00721157ae682-6895f41dfe5mr150586067b3.3.1722958688679; Tue, 06 Aug 2024 08:38:08 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 00721157ae682-68a141568besm15574037b3.140.2024.08.06.08.38.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 06 Aug 2024 08:38:08 -0700 (PDT) Date: Tue, 6 Aug 2024 11:38:07 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v3 19/19] midx: implement support for writing incremental MIDX chains Message-ID: <5d467d38a8d454228032610b3506040b63eb6369.1722958596.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Now that the rest of the MIDX subsystem and relevant callers have been updated to learn about how to read and process incremental MIDX chains, let's finally update the implementation in `write_midx_internal()` to be able to write incremental MIDX chains. This new feature is available behind the `--incremental` option for the `multi-pack-index` builtin, like so: $ git multi-pack-index write --incremental The implementation for doing so is relatively straightforward, and boils down to a handful of different kinds of changes implemented in this patch: - The `compute_sorted_entries()` function is taught to reject objects which appear in any existing MIDX layer. - Functions like `write_midx_revindex()` are adjusted to write pack_order values which are offset by the number of objects in the base MIDX layer. - The end of `write_midx_internal()` is adjusted to move non-incremental MIDX files when necessary (i.e. when creating an incremental chain with an existing non-incremental MIDX in the repository). There are a handful of other changes that are introduced, like new functions to clear incremental MIDX files that are unrelated to the current chain (using the same "keep_hash" mechanism as in the non-incremental case). The tests explicitly exercising the new incremental MIDX feature are relatively limited for two reasons: 1. Most of the "interesting" behavior is already thoroughly covered in t5319-multi-pack-index.sh, which handles the core logic of reading objects through a MIDX. The new tests in t5334-incremental-multi-pack-index.sh are mostly focused on creating and destroying incremental MIDXs, as well as stitching their results together across layers. 2. A new GIT_TEST environment variable is added called "GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL", which modifies the entire test suite to write incremental MIDXs after repacking when combined with the "GIT_TEST_MULTI_PACK_INDEX" variable. This exercises the long tail of other interesting behavior that is defined implicitly throughout the rest of the CI suite. It is likewise added to the linux-TEST-vars job. Signed-off-by: Taylor Blau --- Documentation/git-multi-pack-index.txt | 11 +- builtin/multi-pack-index.c | 2 + builtin/repack.c | 8 +- ci/run-build-and-tests.sh | 1 + midx-write.c | 314 ++++++++++++++++++++---- midx.c | 62 ++++- midx.h | 4 + packfile.c | 16 +- packfile.h | 4 + t/README | 4 + t/lib-bitmap.sh | 6 +- t/lib-midx.sh | 28 +++ t/t5319-multi-pack-index.sh | 27 +- t/t5326-multi-pack-bitmaps.sh | 1 + t/t5327-multi-pack-bitmaps-rev.sh | 1 + t/t5332-multi-pack-reuse.sh | 2 + t/t5334-incremental-multi-pack-index.sh | 46 ++++ t/t7700-repack.sh | 27 +- 18 files changed, 460 insertions(+), 104 deletions(-) create mode 100755 t/t5334-incremental-multi-pack-index.sh diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt index 3696506eb3..631d5c7d15 100644 --- a/Documentation/git-multi-pack-index.txt +++ b/Documentation/git-multi-pack-index.txt @@ -64,6 +64,12 @@ The file given at `` is expected to be readable, and can contain duplicates. (If a given OID is given more than once, it is marked as preferred if at least one instance of it begins with the special `+` marker). + + --incremental:: + Write an incremental MIDX file containing only objects + and packs not present in an existing MIDX layer. + Migrates non-incremental MIDXs to incremental ones when + necessary. Incompatible with `--bitmap`. -- verify:: @@ -74,6 +80,8 @@ expire:: have no objects referenced by the MIDX (with the exception of `.keep` packs and cruft packs). Rewrite the MIDX file afterward to remove all references to these pack-files. ++ +NOTE: this mode is incompatible with incremental MIDX files. repack:: Create a new pack-file containing objects in small pack-files @@ -95,7 +103,8 @@ repack:: + If `repack.packKeptObjects` is `false`, then any pack-files with an associated `.keep` file will not be selected for the batch to repack. - ++ +NOTE: this mode is incompatible with incremental MIDX files. EXAMPLES -------- diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c index 9cf1a32d65..8805cbbeb3 100644 --- a/builtin/multi-pack-index.c +++ b/builtin/multi-pack-index.c @@ -129,6 +129,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv, MIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX), OPT_BIT(0, "progress", &opts.flags, N_("force progress reporting"), MIDX_PROGRESS), + OPT_BIT(0, "incremental", &opts.flags, + N_("write a new incremental MIDX"), MIDX_WRITE_INCREMENTAL), OPT_BOOL(0, "stdin-packs", &opts.stdin_packs, N_("write multi-pack index containing only given indexes")), OPT_FILENAME(0, "refs-snapshot", &opts.refs_snapshot, diff --git a/builtin/repack.c b/builtin/repack.c index 8499bf0e12..7608430a37 100644 --- a/builtin/repack.c +++ b/builtin/repack.c @@ -1514,8 +1514,12 @@ int cmd_repack(int argc, const char **argv, const char *prefix) if (run_update_server_info) update_server_info(0); - if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0)) - write_midx_file(get_object_directory(), NULL, NULL, 0); + if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0)) { + unsigned flags = 0; + if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL, 0)) + flags |= MIDX_WRITE_INCREMENTAL; + write_midx_file(get_object_directory(), NULL, NULL, flags); + } cleanup: string_list_clear(&names, 1); diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh index e6fd68630c..2e28d02b20 100755 --- a/ci/run-build-and-tests.sh +++ b/ci/run-build-and-tests.sh @@ -25,6 +25,7 @@ linux-TEST-vars) export GIT_TEST_COMMIT_GRAPH=1 export GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1 export GIT_TEST_MULTI_PACK_INDEX=1 + export GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=1 export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master export GIT_TEST_NO_WRITE_REV_INDEX=1 export GIT_TEST_CHECKOUT_WORKERS=2 diff --git a/midx-write.c b/midx-write.c index 0accbdbb04..e74b3d82fa 100644 --- a/midx-write.c +++ b/midx-write.c @@ -17,6 +17,8 @@ #include "refs.h" #include "revision.h" #include "list-objects.h" +#include "path.h" +#include "pack-revindex.h" #define PACK_EXPIRED UINT_MAX #define BITMAP_POS_UNKNOWN (~((uint32_t)0)) @@ -25,7 +27,11 @@ extern int midx_checksum_valid(struct multi_pack_index *m); extern void clear_midx_files_ext(const char *object_dir, const char *ext, - unsigned char *keep_hash); + const char *keep_hash); +extern void clear_incremental_midx_files_ext(const char *object_dir, + const char *ext, + const char **keep_hashes, + uint32_t hashes_nr); extern int cmp_idx_or_pack_name(const char *idx_or_pack_name, const char *idx_name); @@ -86,6 +92,7 @@ struct write_midx_context { size_t nr; size_t alloc; struct multi_pack_index *m; + struct multi_pack_index *base_midx; struct progress *progress; unsigned pack_paths_checked; @@ -99,6 +106,9 @@ struct write_midx_context { int preferred_pack_idx; + int incremental; + uint32_t num_multi_pack_indexes_before; + struct string_list *to_include; }; @@ -122,6 +132,9 @@ static int should_include_pack(const struct write_midx_context *ctx, */ if (ctx->m && midx_contains_pack(ctx->m, file_name)) return 0; + else if (ctx->base_midx && midx_contains_pack(ctx->base_midx, + file_name)) + return 0; else if (ctx->to_include && !string_list_has_string(ctx->to_include, file_name)) return 0; @@ -338,7 +351,7 @@ static void compute_sorted_entries(struct write_midx_context *ctx, for (cur_fanout = 0; cur_fanout < 256; cur_fanout++) { fanout.nr = 0; - if (ctx->m) + if (ctx->m && !ctx->incremental) midx_fanout_add_midx_fanout(&fanout, ctx->m, cur_fanout, ctx->preferred_pack_idx); @@ -364,6 +377,10 @@ static void compute_sorted_entries(struct write_midx_context *ctx, if (cur_object && oideq(&fanout.entries[cur_object - 1].oid, &fanout.entries[cur_object].oid)) continue; + if (ctx->incremental && ctx->base_midx && + midx_has_oid(ctx->base_midx, + &fanout.entries[cur_object].oid)) + continue; ALLOC_GROW(ctx->entries, st_add(ctx->entries_nr, 1), alloc_objects); @@ -547,10 +564,16 @@ static int write_midx_revindex(struct hashfile *f, void *data) { struct write_midx_context *ctx = data; - uint32_t i; + uint32_t i, nr_base; + + if (ctx->incremental && ctx->base_midx) + nr_base = ctx->base_midx->num_objects + + ctx->base_midx->num_objects_in_base; + else + nr_base = 0; for (i = 0; i < ctx->entries_nr; i++) - hashwrite_be32(f, ctx->pack_order[i]); + hashwrite_be32(f, ctx->pack_order[i] + nr_base); return 0; } @@ -579,12 +602,18 @@ static int midx_pack_order_cmp(const void *va, const void *vb) static uint32_t *midx_pack_order(struct write_midx_context *ctx) { struct midx_pack_order_data *data; - uint32_t *pack_order; + uint32_t *pack_order, base_objects = 0; uint32_t i; trace2_region_enter("midx", "midx_pack_order", the_repository); + if (ctx->incremental && ctx->base_midx) + base_objects = ctx->base_midx->num_objects + + ctx->base_midx->num_objects_in_base; + + ALLOC_ARRAY(pack_order, ctx->entries_nr); ALLOC_ARRAY(data, ctx->entries_nr); + for (i = 0; i < ctx->entries_nr; i++) { struct pack_midx_entry *e = &ctx->entries[i]; data[i].nr = i; @@ -596,12 +625,11 @@ static uint32_t *midx_pack_order(struct write_midx_context *ctx) QSORT(data, ctx->entries_nr, midx_pack_order_cmp); - ALLOC_ARRAY(pack_order, ctx->entries_nr); for (i = 0; i < ctx->entries_nr; i++) { struct pack_midx_entry *e = &ctx->entries[data[i].nr]; struct pack_info *pack = &ctx->info[ctx->pack_perm[e->pack_int_id]]; if (pack->bitmap_pos == BITMAP_POS_UNKNOWN) - pack->bitmap_pos = i; + pack->bitmap_pos = i + base_objects; pack->bitmap_nr++; pack_order[i] = data[i].nr; } @@ -649,7 +677,8 @@ static void prepare_midx_packing_data(struct packing_data *pdata, prepare_packing_data(the_repository, pdata); for (i = 0; i < ctx->entries_nr; i++) { - struct pack_midx_entry *from = &ctx->entries[ctx->pack_order[i]]; + uint32_t pos = ctx->pack_order[i]; + struct pack_midx_entry *from = &ctx->entries[pos]; struct object_entry *to = packlist_alloc(pdata, &from->oid); oe_set_in_pack(pdata, to, @@ -897,37 +926,130 @@ static struct multi_pack_index *lookup_multi_pack_index(struct repository *r, static int fill_packs_from_midx(struct write_midx_context *ctx, const char *preferred_pack_name, uint32_t flags) { - uint32_t i; + struct multi_pack_index *m; - for (i = 0; i < ctx->m->num_packs; i++) { - ALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc); + for (m = ctx->m; m; m = m->base_midx) { + uint32_t i; + + for (i = 0; i < m->num_packs; i++) { + ALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc); - if (flags & MIDX_WRITE_REV_INDEX || preferred_pack_name) { /* * If generating a reverse index, need to have * packed_git's loaded to compare their * mtimes and object count. * - * * If a preferred pack is specified, need to * have packed_git's loaded to ensure the chosen * preferred pack has a non-zero object count. */ - if (prepare_midx_pack(the_repository, ctx->m, i)) - return error(_("could not load pack")); + if (flags & MIDX_WRITE_REV_INDEX || + preferred_pack_name) { + if (prepare_midx_pack(the_repository, m, + m->num_packs_in_base + i)) { + error(_("could not load pack")); + return 1; + } - if (open_pack_index(ctx->m->packs[i])) - die(_("could not open index for %s"), - ctx->m->packs[i]->pack_name); + if (open_pack_index(m->packs[i])) + die(_("could not open index for %s"), + m->packs[i]->pack_name); + } + + fill_pack_info(&ctx->info[ctx->nr++], m->packs[i], + m->pack_names[i], + m->num_packs_in_base + i); } - - fill_pack_info(&ctx->info[ctx->nr++], ctx->m->packs[i], - ctx->m->pack_names[i], i); } - return 0; } +static struct { + const char *non_split; + const char *split; +} midx_exts[] = { + {NULL, MIDX_EXT_MIDX}, + {MIDX_EXT_BITMAP, MIDX_EXT_BITMAP}, + {MIDX_EXT_REV, MIDX_EXT_REV}, +}; + +static int link_midx_to_chain(struct multi_pack_index *m) +{ + struct strbuf from = STRBUF_INIT; + struct strbuf to = STRBUF_INIT; + int ret = 0; + size_t i; + + if (!m || m->has_chain) { + /* + * Either no MIDX previously existed, or it was already + * part of a MIDX chain. In both cases, we have nothing + * to link, so return early. + */ + goto done; + } + + for (i = 0; i < ARRAY_SIZE(midx_exts); i++) { + const unsigned char *hash = get_midx_checksum(m); + + get_midx_filename_ext(&from, m->object_dir, hash, + midx_exts[i].non_split); + get_split_midx_filename_ext(&to, m->object_dir, hash, + midx_exts[i].split); + + if (link(from.buf, to.buf) < 0 && errno != ENOENT) { + ret = error_errno(_("unable to link '%s' to '%s'"), + from.buf, to.buf); + goto done; + } + + strbuf_reset(&from); + strbuf_reset(&to); + } + +done: + strbuf_release(&from); + strbuf_release(&to); + return ret; +} + +static void clear_midx_files(const char *object_dir, + const char **hashes, + uint32_t hashes_nr, + unsigned incremental) +{ + /* + * if incremental: + * - remove all non-incremental MIDX files + * - remove any incremental MIDX files not in the current one + * + * if non-incremental: + * - remove all incremental MIDX files + * - remove any non-incremental MIDX files not matching the current + * hash + */ + struct strbuf buf = STRBUF_INIT; + const char *exts[] = { MIDX_EXT_BITMAP, MIDX_EXT_REV, MIDX_EXT_MIDX }; + uint32_t i, j; + + for (i = 0; i < ARRAY_SIZE(exts); i++) { + clear_incremental_midx_files_ext(object_dir, exts[i], + hashes, hashes_nr); + for (j = 0; j < hashes_nr; j++) + clear_midx_files_ext(object_dir, exts[i], hashes[j]); + } + + if (incremental) + get_midx_filename(&buf, object_dir); + else + get_midx_chain_filename(&buf, object_dir); + + if (unlink(buf.buf) && errno != ENOENT) + die_errno(_("failed to clear multi-pack-index at %s"), buf.buf); + + strbuf_release(&buf); +} + static int write_midx_internal(const char *object_dir, struct string_list *packs_to_include, struct string_list *packs_to_drop, @@ -940,42 +1062,66 @@ static int write_midx_internal(const char *object_dir, uint32_t i, start_pack; struct hashfile *f = NULL; struct lock_file lk; + struct tempfile *incr; struct write_midx_context ctx = { 0 }; int bitmapped_packs_concat_len = 0; int pack_name_concat_len = 0; int dropped_packs = 0; int result = 0; + const char **keep_hashes = NULL; struct chunkfile *cf; trace2_region_enter("midx", "write_midx_internal", the_repository); - get_midx_filename(&midx_name, object_dir); + ctx.incremental = !!(flags & MIDX_WRITE_INCREMENTAL); + if (ctx.incremental && (flags & MIDX_WRITE_BITMAP)) + die(_("cannot write incremental MIDX with bitmap")); + + if (ctx.incremental) + strbuf_addf(&midx_name, + "%s/pack/multi-pack-index.d/tmp_midx_XXXXXX", + object_dir); + else + get_midx_filename(&midx_name, object_dir); if (safe_create_leading_directories(midx_name.buf)) die_errno(_("unable to create leading directories of %s"), midx_name.buf); - if (!packs_to_include) { - /* - * Only reference an existing MIDX when not filtering which - * packs to include, since all packs and objects are copied - * blindly from an existing MIDX if one is present. - */ - ctx.m = lookup_multi_pack_index(the_repository, object_dir); - } + if (!packs_to_include || ctx.incremental) { + struct multi_pack_index *m = lookup_multi_pack_index(the_repository, + object_dir); + if (m && !midx_checksum_valid(m)) { + warning(_("ignoring existing multi-pack-index; checksum mismatch")); + m = NULL; + } - if (ctx.m && !midx_checksum_valid(ctx.m)) { - warning(_("ignoring existing multi-pack-index; checksum mismatch")); - ctx.m = NULL; + if (m) { + /* + * Only reference an existing MIDX when not filtering + * which packs to include, since all packs and objects + * are copied blindly from an existing MIDX if one is + * present. + */ + if (ctx.incremental) + ctx.base_midx = m; + else if (!packs_to_include) + ctx.m = m; + } } ctx.nr = 0; - ctx.alloc = ctx.m ? ctx.m->num_packs : 16; + ctx.alloc = ctx.m ? ctx.m->num_packs + ctx.m->num_packs_in_base : 16; ctx.info = NULL; ALLOC_ARRAY(ctx.info, ctx.alloc); - if (ctx.m && fill_packs_from_midx(&ctx, preferred_pack_name, - flags) < 0) { - result = 1; + if (ctx.incremental) { + struct multi_pack_index *m = ctx.base_midx; + while (m) { + ctx.num_multi_pack_indexes_before++; + m = m->base_midx; + } + } else if (ctx.m && fill_packs_from_midx(&ctx, preferred_pack_name, + flags) < 0) { goto cleanup; } @@ -992,7 +1138,8 @@ static int write_midx_internal(const char *object_dir, for_each_file_in_pack_dir(object_dir, add_pack_to_midx, &ctx); stop_progress(&ctx.progress); - if ((ctx.m && ctx.nr == ctx.m->num_packs) && + if ((ctx.m && ctx.nr == ctx.m->num_packs + ctx.m->num_packs_in_base) && + !ctx.incremental && !(packs_to_include || packs_to_drop)) { struct bitmap_index *bitmap_git; int bitmap_exists; @@ -1008,12 +1155,14 @@ static int write_midx_internal(const char *object_dir, * corresponding bitmap (or one wasn't requested). */ if (!want_bitmap) - clear_midx_files_ext(object_dir, ".bitmap", - NULL); + clear_midx_files_ext(object_dir, "bitmap", NULL); goto cleanup; } } + if (ctx.incremental && !ctx.nr) + goto cleanup; /* nothing to do */ + if (preferred_pack_name) { ctx.preferred_pack_idx = -1; @@ -1159,8 +1308,30 @@ static int write_midx_internal(const char *object_dir, pack_name_concat_len += MIDX_CHUNK_ALIGNMENT - (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT); - hold_lock_file_for_update(&lk, midx_name.buf, LOCK_DIE_ON_ERROR); - f = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk)); + if (ctx.incremental) { + struct strbuf lock_name = STRBUF_INIT; + + get_midx_chain_filename(&lock_name, object_dir); + hold_lock_file_for_update(&lk, lock_name.buf, LOCK_DIE_ON_ERROR); + strbuf_release(&lock_name); + + incr = mks_tempfile_m(midx_name.buf, 0444); + if (!incr) { + error(_("unable to create temporary MIDX layer")); + return -1; + } + + if (adjust_shared_perm(get_tempfile_path(incr))) { + error(_("unable to adjust shared permissions for '%s'"), + get_tempfile_path(incr)); + return -1; + } + + f = hashfd(get_tempfile_fd(incr), get_tempfile_path(incr)); + } else { + hold_lock_file_for_update(&lk, midx_name.buf, LOCK_DIE_ON_ERROR); + f = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk)); + } if (ctx.nr - dropped_packs == 0) { error(_("no pack files to index.")); @@ -1253,14 +1424,55 @@ static int write_midx_internal(const char *object_dir, * have been freed in the previous if block. */ - if (ctx.m) + CALLOC_ARRAY(keep_hashes, ctx.num_multi_pack_indexes_before + 1); + + if (ctx.incremental) { + FILE *chainf = fdopen_lock_file(&lk, "w"); + struct strbuf final_midx_name = STRBUF_INIT; + struct multi_pack_index *m = ctx.base_midx; + + if (!chainf) { + error_errno(_("unable to open multi-pack-index chain file")); + return -1; + } + + if (link_midx_to_chain(ctx.base_midx) < 0) + return -1; + + get_split_midx_filename_ext(&final_midx_name, object_dir, + midx_hash, MIDX_EXT_MIDX); + + if (rename_tempfile(&incr, final_midx_name.buf) < 0) { + error_errno(_("unable to rename new multi-pack-index layer")); + return -1; + } + + keep_hashes[ctx.num_multi_pack_indexes_before] = + xstrdup(hash_to_hex(midx_hash)); + + for (i = 0; i < ctx.num_multi_pack_indexes_before; i++) { + uint32_t j = ctx.num_multi_pack_indexes_before - i - 1; + + keep_hashes[j] = xstrdup(hash_to_hex(get_midx_checksum(m))); + m = m->base_midx; + } + + for (i = 0; i < ctx.num_multi_pack_indexes_before + 1; i++) + fprintf(get_lock_file_fp(&lk), "%s\n", keep_hashes[i]); + } else { + keep_hashes[ctx.num_multi_pack_indexes_before] = + xstrdup(hash_to_hex(midx_hash)); + } + + if (ctx.m || ctx.base_midx) close_object_store(the_repository->objects); if (commit_lock_file(&lk) < 0) die_errno(_("could not write multi-pack-index")); - clear_midx_files_ext(object_dir, ".bitmap", midx_hash); - clear_midx_files_ext(object_dir, ".rev", midx_hash); + clear_midx_files(object_dir, keep_hashes, + ctx.num_multi_pack_indexes_before + 1, + ctx.incremental); cleanup: for (i = 0; i < ctx.nr; i++) { @@ -1275,6 +1487,11 @@ static int write_midx_internal(const char *object_dir, free(ctx.entries); free(ctx.pack_perm); free(ctx.pack_order); + if (keep_hashes) { + for (i = 0; i < ctx.num_multi_pack_indexes_before + 1; i++) + free((char *)keep_hashes[i]); + free(keep_hashes); + } strbuf_release(&midx_name); trace2_region_leave("midx", "write_midx_internal", the_repository); @@ -1311,6 +1528,9 @@ int expire_midx_packs(struct repository *r, const char *object_dir, unsigned fla if (!m) return 0; + if (m->base_midx) + die(_("cannot expire packs from an incremental multi-pack-index")); + CALLOC_ARRAY(count, m->num_packs); if (flags & MIDX_PROGRESS) @@ -1485,6 +1705,8 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size, if (!m) return 0; + if (m->base_midx) + die(_("cannot repack an incremental multi-pack-index")); CALLOC_ARRAY(include_pack, m->num_packs); diff --git a/midx.c b/midx.c index a53d65702d..c867b2b6c2 100644 --- a/midx.c +++ b/midx.c @@ -16,7 +16,10 @@ int midx_checksum_valid(struct multi_pack_index *m); void clear_midx_files_ext(const char *object_dir, const char *ext, - unsigned char *keep_hash); + const char *keep_hash); +void clear_incremental_midx_files_ext(const char *object_dir, const char *ext, + char **keep_hashes, + uint32_t hashes_nr); int cmp_idx_or_pack_name(const char *idx_or_pack_name, const char *idx_name); @@ -521,6 +524,11 @@ int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, return 0; } +int midx_has_oid(struct multi_pack_index *m, const struct object_id *oid) +{ + return bsearch_midx(oid, m, NULL); +} + struct object_id *nth_midxed_object_oid(struct object_id *oid, struct multi_pack_index *m, uint32_t n) @@ -723,7 +731,8 @@ int midx_checksum_valid(struct multi_pack_index *m) } struct clear_midx_data { - char *keep; + char **keep; + uint32_t keep_nr; const char *ext; }; @@ -731,32 +740,63 @@ static void clear_midx_file_ext(const char *full_path, size_t full_path_len UNUS const char *file_name, void *_data) { struct clear_midx_data *data = _data; + uint32_t i; if (!(starts_with(file_name, "multi-pack-index-") && ends_with(file_name, data->ext))) return; - if (data->keep && !strcmp(data->keep, file_name)) - return; - + for (i = 0; i < data->keep_nr; i++) { + if (!strcmp(data->keep[i], file_name)) + return; + } if (unlink(full_path)) die_errno(_("failed to remove %s"), full_path); } void clear_midx_files_ext(const char *object_dir, const char *ext, - unsigned char *keep_hash) + const char *keep_hash) { struct clear_midx_data data; memset(&data, 0, sizeof(struct clear_midx_data)); - if (keep_hash) - data.keep = xstrfmt("multi-pack-index-%s%s", - hash_to_hex(keep_hash), ext); + if (keep_hash) { + ALLOC_ARRAY(data.keep, 1); + + data.keep[0] = xstrfmt("multi-pack-index-%s.%s", keep_hash, ext); + data.keep_nr = 1; + } data.ext = ext; for_each_file_in_pack_dir(object_dir, clear_midx_file_ext, &data); + if (keep_hash) + free(data.keep[0]); + free(data.keep); +} + +void clear_incremental_midx_files_ext(const char *object_dir, const char *ext, + char **keep_hashes, + uint32_t hashes_nr) +{ + struct clear_midx_data data; + uint32_t i; + + memset(&data, 0, sizeof(struct clear_midx_data)); + + ALLOC_ARRAY(data.keep, hashes_nr); + for (i = 0; i < hashes_nr; i++) + data.keep[i] = xstrfmt("multi-pack-index-%s.%s", keep_hashes[i], + ext); + data.keep_nr = hashes_nr; + data.ext = ext; + + for_each_file_in_pack_subdir(object_dir, "multi-pack-index.d", + clear_midx_file_ext, &data); + + for (i = 0; i < hashes_nr; i++) + free(data.keep[i]); free(data.keep); } @@ -774,8 +814,8 @@ void clear_midx_file(struct repository *r) if (remove_path(midx.buf)) die(_("failed to clear multi-pack-index at %s"), midx.buf); - clear_midx_files_ext(r->objects->odb->path, ".bitmap", NULL); - clear_midx_files_ext(r->objects->odb->path, ".rev", NULL); + clear_midx_files_ext(r->objects->odb->path, MIDX_EXT_BITMAP, NULL); + clear_midx_files_ext(r->objects->odb->path, MIDX_EXT_REV, NULL); strbuf_release(&midx); } diff --git a/midx.h b/midx.h index 3714cad2cc..42d4f8d149 100644 --- a/midx.h +++ b/midx.h @@ -29,6 +29,8 @@ struct bitmapped_pack; #define MIDX_LARGE_OFFSET_NEEDED 0x80000000 #define GIT_TEST_MULTI_PACK_INDEX "GIT_TEST_MULTI_PACK_INDEX" +#define GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL \ + "GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL" struct multi_pack_index { struct multi_pack_index *next; @@ -77,6 +79,7 @@ struct multi_pack_index { #define MIDX_WRITE_BITMAP (1 << 2) #define MIDX_WRITE_BITMAP_HASH_CACHE (1 << 3) #define MIDX_WRITE_BITMAP_LOOKUP_TABLE (1 << 4) +#define MIDX_WRITE_INCREMENTAL (1 << 5) #define MIDX_EXT_REV "rev" #define MIDX_EXT_BITMAP "bitmap" @@ -101,6 +104,7 @@ int bsearch_one_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result); int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result); +int midx_has_oid(struct multi_pack_index *m, const struct object_id *oid); off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos); uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos); struct object_id *nth_midxed_object_oid(struct object_id *oid, diff --git a/packfile.c b/packfile.c index 1eb18e3041..cf12a539ea 100644 --- a/packfile.c +++ b/packfile.c @@ -815,9 +815,10 @@ static void report_pack_garbage(struct string_list *list) report_helper(list, seen_bits, first, list->nr); } -void for_each_file_in_pack_dir(const char *objdir, - each_file_in_pack_dir_fn fn, - void *data) +void for_each_file_in_pack_subdir(const char *objdir, + const char *subdir, + each_file_in_pack_dir_fn fn, + void *data) { struct strbuf path = STRBUF_INIT; size_t dirnamelen; @@ -826,6 +827,8 @@ void for_each_file_in_pack_dir(const char *objdir, strbuf_addstr(&path, objdir); strbuf_addstr(&path, "/pack"); + if (subdir) + strbuf_addf(&path, "/%s", subdir); dir = opendir(path.buf); if (!dir) { if (errno != ENOENT) @@ -847,6 +850,13 @@ void for_each_file_in_pack_dir(const char *objdir, strbuf_release(&path); } +void for_each_file_in_pack_dir(const char *objdir, + each_file_in_pack_dir_fn fn, + void *data) +{ + for_each_file_in_pack_subdir(objdir, NULL, fn, data); +} + struct prepare_pack_data { struct repository *r; struct string_list *garbage; diff --git a/packfile.h b/packfile.h index eb18ec15db..0f78658229 100644 --- a/packfile.h +++ b/packfile.h @@ -55,6 +55,10 @@ struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path); typedef void each_file_in_pack_dir_fn(const char *full_path, size_t full_path_len, const char *file_name, void *data); +void for_each_file_in_pack_subdir(const char *objdir, + const char *subdir, + each_file_in_pack_dir_fn fn, + void *data); void for_each_file_in_pack_dir(const char *objdir, each_file_in_pack_dir_fn fn, void *data); diff --git a/t/README b/t/README index 3cee0c05e2..44c02d8129 100644 --- a/t/README +++ b/t/README @@ -445,6 +445,10 @@ GIT_TEST_MULTI_PACK_INDEX=, when true, forces the multi-pack- index to be written after every 'git repack' command, and overrides the 'core.multiPackIndex' setting to true. +GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=, when true, sets +the '--incremental' option on all invocations of 'git multi-pack-index +write'. + GIT_TEST_SIDEBAND_ALL=, when true, overrides the 'uploadpack.allowSidebandAll' setting to true, and when false, forces fetch-pack to not request sideband-all (even if the server advertises diff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh index f595937094..62aa6744a6 100644 --- a/t/lib-bitmap.sh +++ b/t/lib-bitmap.sh @@ -1,6 +1,8 @@ # Helpers for scripts testing bitmap functionality; see t5310 for # example usage. +. "$TEST_DIRECTORY"/lib-midx.sh + objdir=.git/objects midx=$objdir/pack/multi-pack-index @@ -264,10 +266,6 @@ have_delta () { test_cmp expect actual } -midx_checksum () { - test-tool read-midx --checksum "$1" -} - # midx_pack_source midx_pack_source () { test-tool read-midx --show-objects .git/objects | grep "^$1 " | cut -f2 diff --git a/t/lib-midx.sh b/t/lib-midx.sh index 1261994744..e38c609604 100644 --- a/t/lib-midx.sh +++ b/t/lib-midx.sh @@ -6,3 +6,31 @@ test_midx_consistent () { test_cmp expect actual && git multi-pack-index --object-dir=$1 verify } + +midx_checksum () { + test-tool read-midx --checksum "$1" +} + +midx_git_two_modes () { + git -c core.multiPackIndex=false $1 >expect && + git -c core.multiPackIndex=true $1 >actual && + if [ "$2" = "sorted" ] + then + sort expect.sorted && + mv expect.sorted expect && + sort actual.sorted && + mv actual.sorted actual + fi && + test_cmp expect actual +} + +compare_results_with_midx () { + MSG=$1 + test_expect_success "check normal git operations: $MSG" ' + midx_git_two_modes "rev-list --objects --all" && + midx_git_two_modes "log --raw" && + midx_git_two_modes "count-objects --verbose" && + midx_git_two_modes "cat-file --batch-all-objects --batch-check" && + midx_git_two_modes "cat-file --batch-all-objects --batch-check --unordered" sorted + ' +} diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index 8c54fc0655..ce1b58c732 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -3,8 +3,11 @@ test_description='multi-pack-indexes' . ./test-lib.sh . "$TEST_DIRECTORY"/lib-chunk.sh +. "$TEST_DIRECTORY"/lib-midx.sh GIT_TEST_MULTI_PACK_INDEX=0 +GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 +GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0 objdir=.git/objects HASH_LEN=$(test_oid rawsz) @@ -107,30 +110,6 @@ test_expect_success 'write midx with one v1 pack' ' midx_read_expect 1 18 4 $objdir ' -midx_git_two_modes () { - git -c core.multiPackIndex=false $1 >expect && - git -c core.multiPackIndex=true $1 >actual && - if [ "$2" = "sorted" ] - then - sort expect.sorted && - mv expect.sorted expect && - sort actual.sorted && - mv actual.sorted actual - fi && - test_cmp expect actual -} - -compare_results_with_midx () { - MSG=$1 - test_expect_success "check normal git operations: $MSG" ' - midx_git_two_modes "rev-list --objects --all" && - midx_git_two_modes "log --raw" && - midx_git_two_modes "count-objects --verbose" && - midx_git_two_modes "cat-file --batch-all-objects --batch-check" && - midx_git_two_modes "cat-file --batch-all-objects --batch-check --unordered" sorted - ' -} - test_expect_success 'write midx with one v2 pack' ' git pack-objects --index-version=2,0x40 $objdir/pack/test &2 && incrpackid=$(git pack-objects --all --unpacked --incremental .git/objects/pack/pack err && + git repack -Adl --write-bitmap-index 2>err && cat >expect <<-EOF && warning: disabling bitmap writing, as some objects are not being packed EOF @@ -533,11 +536,11 @@ test_expect_success 'setup for --write-midx tests' ' test_expect_success '--write-midx unchanged' ' ( cd midx && - GIT_TEST_MULTI_PACK_INDEX=0 git repack && + git repack && test_path_is_missing $midx && test_path_is_missing $midx-*.bitmap && - GIT_TEST_MULTI_PACK_INDEX=0 git repack --write-midx && + git repack --write-midx && test_path_is_file $midx && test_path_is_missing $midx-*.bitmap && @@ -550,7 +553,7 @@ test_expect_success '--write-midx with a new pack' ' cd midx && test_commit loose && - GIT_TEST_MULTI_PACK_INDEX=0 git repack --write-midx && + git repack --write-midx && test_path_is_file $midx && test_path_is_missing $midx-*.bitmap && @@ -561,7 +564,7 @@ test_expect_success '--write-midx with a new pack' ' test_expect_success '--write-midx with -b' ' ( cd midx && - GIT_TEST_MULTI_PACK_INDEX=0 git repack -mb && + git repack -mb && test_path_is_file $midx && test_path_is_file $midx-*.bitmap && @@ -574,7 +577,7 @@ test_expect_success '--write-midx with -d' ' cd midx && test_commit repack && - GIT_TEST_MULTI_PACK_INDEX=0 git repack -Ad --write-midx && + git repack -Ad --write-midx && test_path_is_file $midx && test_path_is_missing $midx-*.bitmap && @@ -587,21 +590,21 @@ test_expect_success 'cleans up MIDX when appropriate' ' cd midx && test_commit repack-2 && - GIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb --write-midx && + git repack -Adb --write-midx && checksum=$(midx_checksum $objdir) && test_path_is_file $midx && test_path_is_file $midx-$checksum.bitmap && test_commit repack-3 && - GIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb --write-midx && + git repack -Adb --write-midx && test_path_is_file $midx && test_path_is_missing $midx-$checksum.bitmap && test_path_is_file $midx-$(midx_checksum $objdir).bitmap && test_commit repack-4 && - GIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb && + git repack -Adb && find $objdir/pack -type f -name "multi-pack-index*" >files && test_must_be_empty files @@ -622,7 +625,6 @@ test_expect_success '--write-midx with preferred bitmap tips' ' git log --format="create refs/tags/%s/%s %H" HEAD >refs && git update-ref --stdin