From patchwork Wed Jul 17 21:11:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13735781 Received: from mail-yw1-f173.google.com (mail-yw1-f173.google.com [209.85.128.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 24A7118757C for ; Wed, 17 Jul 2024 21:12:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250722; cv=none; b=PKHRo7cyeJDtOLqDlratVZR8lK6eQArNTsFdRuuxFns3M3spSTZIfk2q7usuSso53YKeuRnI/kCqlmtyVMTKa77GwiEhq3Th9NB5UvAZWOfMpb7r3Cr3BCzwsC2flvRgxM5meRY1iRlZa+zuR4T7xli9DhgycymoFqgoe3HKees= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250722; c=relaxed/simple; bh=y84aW8EgAuveV+ua5UA+7oj6bQOzKqIUEgy2hYB9/Qg=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=OG25b4/LIRvLy5LWxQRCicM2uhwiuw7Kc1yPthDH4v4d7BAshnUGRkwcpgnecx4dK8S+oyz1eMYWiLpx3z1lnWrHYiqRU9/g96ytDsLf7XpXrPRHlrKIpd+2cnnfLSsyrO1sNyvhofvyqZQHo1sQ/e1MFwPU6T1mwIRyPq6sY0g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=U0+V88sn; arc=none smtp.client-ip=209.85.128.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="U0+V88sn" Received: by mail-yw1-f173.google.com with SMTP id 00721157ae682-65f852c82fcso902407b3.1 for ; Wed, 17 Jul 2024 14:12:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1721250720; x=1721855520; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=8otJcH7ov+/55pOZCRQmfzGpxZD5P+VPTXQntK3Mgrk=; b=U0+V88snjkvEFah5dOE4mk5SMExlrPgYG6+zVpyIlJV+4tqOxzY23ynsx0GjeEFlV8 xoJXpLlZGy7JWRVlZBzXxCcRgJyZv2xvsDL+deqjHNnUHeZyVvKiSnW+WxNNmKxjbGIf OGrt8Ec8BLNyh1mrtZ2iUGiAGu8PP90D+GvGsLnDUQGcLAm60e+rEQOgfyB2pFPxC+tC /LRrh6vKdHUAYgtw8lo+YGMjxj8j+Liezw56vNcBGkdJRahZvhe74m7caWfpl2uR6EaY jsPdLeelmYXkvEk5Mwk0yVVXY/oAyrmRP//kDcmGOGv2c4oJBpSsxK/Cc8090VzgjJ3c H+EA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721250720; x=1721855520; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=8otJcH7ov+/55pOZCRQmfzGpxZD5P+VPTXQntK3Mgrk=; b=dJGJSBa05+anh7cgeRbjVycid8/sMt82P1JDc9cbQR46IBfFmUuhxBZ5TEmiUHF9/I jgxk74k1k9J898scLQxzuRohR2b2pH3TqDsWS390vSczy9sOsnjQ1lmTIZUUVOso5giU SOdYK6BLzsG+tiNjajOOXyDJUL2xzfDboZRB7R1zy25WU+aFIEh9nhLqBwupcCGnirZB eeNH/C+E8YZl+ZqdPFFNfW/OZthQvolHI4ch53y3fyJQcKQm14C/DbuE3X6x89Oqjbvb M3iC86CbDHEYVwQZO5qoBpDaXQghAzkIf1w6kkdn9EXx75OJ//k00peRr66fc7/XIIZU pYBA== X-Gm-Message-State: AOJu0YwGHKLcSmTuvctf9be9yoICZgFGIAn9zxhap04URFKXcchpmhh/ YZdj28uKIwbKfRN/SsDmhl5pIlh5s+trhT1CDEVbakxmQdVufv1CragxgfXgsiyiHMIFlhQBLx4 e X-Google-Smtp-Source: AGHT+IFzjLaZG9WI+jrhJY37YITZ0XXOt7+De37HPp/y62UMeuNhBmWeX4qNUnUsadqsRvn7B3jluw== X-Received: by 2002:a81:c309:0:b0:643:6087:efb8 with SMTP id 00721157ae682-664fd59ef69mr35511397b3.3.1721250719700; Wed, 17 Jul 2024 14:11:59 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 00721157ae682-666038fc601sm763547b3.83.2024.07.17.14.11.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Jul 2024 14:11:59 -0700 (PDT) Date: Wed, 17 Jul 2024 17:11:58 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v2 01/19] Documentation: describe incremental MIDX format Message-ID: <014588b3ecf2e32a6ab1af8e77234dfb2f9ea75a.1721250704.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Prepare to implement incremental multi-pack indexes (MIDXs) over the next several commits by first describing the relevant prerequisites (like a new chunk in the MIDX format, the directory structure for incremental MIDXs, etc.) The format is described in detail in the patch contents below, but the high-level description is as follows. Incremental MIDXs live in $GIT_DIR/objects/pack/multi-pack-index.d, and each `*.midx` within that directory has a single "parent" MIDX, which is the MIDX layer immediately before it in the MIDX chain. The chain order resides in a file 'multi-pack-index-chain' in the same directory. Signed-off-by: Taylor Blau --- Documentation/technical/multi-pack-index.txt | 100 +++++++++++++++++++ 1 file changed, 100 insertions(+) diff --git a/Documentation/technical/multi-pack-index.txt b/Documentation/technical/multi-pack-index.txt index f2221d2b44..d05e3d6dd9 100644 --- a/Documentation/technical/multi-pack-index.txt +++ b/Documentation/technical/multi-pack-index.txt @@ -61,6 +61,106 @@ Design Details - The MIDX file format uses a chunk-based approach (similar to the commit-graph file) that allows optional data to be added. +Incremental multi-pack indexes +------------------------------ + +As repositories grow in size, it becomes more expensive to write a +multi-pack index (MIDX) that includes all packfiles. To accommodate +this, the "incremental multi-pack indexes" feature allows for combining +a "chain" of multi-pack indexes. + +Each individual component of the chain need only contain a small number +of packfiles. Appending to the chain does not invalidate earlier parts +of the chain, so repositories can control how much time is spent +updating the MIDX chain by determining the number of packs in each layer +of the MIDX chain. + +=== Design state + +At present, the incremental multi-pack indexes feature is missing two +important components: + + - The ability to rewrite earlier portions of the MIDX chain (i.e., to + "compact" some collection of adjacent MIDX layers into a single + MIDX). At present the only supported way of shrinking a MIDX chain + is to rewrite the entire chain from scratch without the `--split` + flag. ++ +There are no fundamental limitations that stand in the way of being able +to implement this feature. It is omitted from the initial implementation +in order to reduce the complexity, but will be added later. + + - Support for reachability bitmaps. The classic single MIDX + implementation does support reachability bitmaps (see the section + titled "multi-pack-index reverse indexes" in + linkgit:gitformat-pack[5] for more details). ++ +As above, there are no fundamental limitations that stand in the way of +extending the incremental MIDX format to support reachability bitmaps. +The design below specifically takes this into account, and support for +reachability bitmaps will be added in a future patch series. It is +omitted from this series for the same reason as above. ++ +In brief, to support reachability bitmaps with the incremental MIDX +feature, the concept of the pseudo-pack order is extended across each +layer of the incremental MIDX chain to form a concatenated pseudo-pack +order. This concatenation takes place in the same order as the chain +itself (in other words, the concatenated pseudo-pack order for a chain +`{$H1, $H2, $H3}` would be the pseudo-pack order for `$H1`, followed by +the pseudo-pack order for `$H2`, followed by the pseudo-pack order for +`$H3`). ++ +The layout will then be extended so that each layer of the incremental +MIDX chain can write a `*.bitmap`. The objects in each layer's bitmap +are offset by the number of objects in the previous layers of the chain. + +=== File layout + +Instead of storing a single `multi-pack-index` file (with an optional +`.rev` and `.bitmap` extension) in `$GIT_DIR/objects/pack`, incremental +MIDXs are stored in the following layout: + +---- +$GIT_DIR/objects/pack/multi-pack-index.d/ +$GIT_DIR/objects/pack/multi-pack-index.d/multi-pack-index-chain +$GIT_DIR/objects/pack/multi-pack-index.d/multi-pack-index-$H1.midx +$GIT_DIR/objects/pack/multi-pack-index.d/multi-pack-index-$H2.midx +$GIT_DIR/objects/pack/multi-pack-index.d/multi-pack-index-$H3.midx +---- + +The `multi-pack-index-chain` file contains a list of the incremental +MIDX files in the chain, in order. The above example shows a chain whose +`multi-pack-index-chain` file would contain the following lines: + +---- +$H1 +$H2 +$H3 +---- + +The `multi-pack-index-$H1.midx` file contains the first layer of the +multi-pack-index chain. The `multi-pack-index-$H2.midx` file contains +the second layer of the chain, and so on. + +=== Object positions for incremental MIDXs + +In the original multi-pack-index design, we refer to objects via their +lexicographic position (by object IDs) within the repository's singular +multi-pack-index. In the incremental multi-pack-index design, we refer +to objects via their index into a concatenated lexicographic ordering +among each component in the MIDX chain. + +If `objects_nr()` is a function that returns the number of objects in a +given MIDX layer, then the index of an object at lexicographic position +`i` within, say, $H3 is defined as: + +---- +objects_nr($H2) + objects_nr($H1) + i +---- + +(in the C implementation, this is often computed as `i + +m->num_objects_in_base`). + Future Work ----------- From patchwork Wed Jul 17 21:12:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13735782 Received: from mail-yw1-f170.google.com (mail-yw1-f170.google.com [209.85.128.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 055D8184125 for ; Wed, 17 Jul 2024 21:12:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250725; cv=none; b=LYfd0wlnxdZyET3jZa5M90XMiAUrzzDtqdGM9W+f5aMyBaj8v83F42GVGYhYUv0jZ7+dQsOrdMZnNFFyPL0lkMZhrvso/1kf7WsHn1vcaJFKq0+T4o2GwrPBlT9/cmuEr9ypuMXGiLuHUwYU03O6T8Ab8JVbnrl1h2t5GG9cDcA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250725; c=relaxed/simple; bh=5cEkhm2dm+UbDPXVQwt4lG/X692PMb5K7ykp3DyXCIM=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=QIW8GhCeRhZLf86vsulZcgN6al0ZPgEemOpp9nwYA0AC6SzWtYPhEN0e2DwxsPMhpQ0reXcjCx7WmgBOte3ZIDnRTWgQ2T+wu/dkw0LGib+HoGRz9uyOOSZdW5k6YMRiWowgeaEgRTZLREggs9RpH1Cs1pwgLGLpa5n++r94tds= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=stDgemv5; arc=none smtp.client-ip=209.85.128.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="stDgemv5" Received: by mail-yw1-f170.google.com with SMTP id 00721157ae682-6519528f44fso752757b3.1 for ; Wed, 17 Jul 2024 14:12:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1721250723; x=1721855523; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=JDEorA7r2fVoo3SiQTc1kG4snLi7jhy/6bhDioCYiVw=; b=stDgemv5Zo9+91Adz34+OlOalGFcZSyCxMqphzO2ULme3HohkeZkgXUQZUorjLU3C+ Y5ibib7EqGIvczzvFcwzsyx9PApaGMbG96qNHcBCaFG1OPx0/WDEBOMh5x745wBbS9Sp 69WyQiFGQvYzc00FfnCeuHOFnrKOAIqN/aYRSSxVuG0Rv+k9/0Kmo7iGOu7BCYS6j2e/ 1y/oYXs/fUAYi+1SiKltICXnlksVTkhKp+8N8x0JHjV+b/6uTIOd1XFJnEYfc3w/sCtB m58z9rWAZtbU8UibqCCpJFbdblbYSQTkxJUL+sv8Z3vmRCgqAx0ZlZ7S6G4ZqkH0SMSD 6VrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721250723; x=1721855523; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=JDEorA7r2fVoo3SiQTc1kG4snLi7jhy/6bhDioCYiVw=; b=Nfuo7sq20oa2xb3z76rsfQowvKUUhI3182MH3bmuh0fLPD80uxXfhCgTl4jxxjhVPP N8Qnoxk3T/OirSVQt3zZ1L9dtlc3LgHTfhWN42rPTk+A68iHhIkLaUUnQt4F2y4o+XZQ CLkazYT2nJExOW3MtXUB4Oc0tn5kR3m2D1hzVVeFd2AvDfSaFzPgOuAOup//sYC4NtP/ MGHzDc9jUJ2ui78QWGKwARUC2rprMOYe/wO9VMXtTgr1KP6b8njBskArz67ki60t/XXR p6wdxOqwEu8fpmqD0hJuFnDH8CPy/NlTAR3BYxR4m5bbr95mXHeRDiIR3+Lp2GVzwlTV YkXw== X-Gm-Message-State: AOJu0YyMrPVCXzslW30ks9D3PB36oNemVXCMtNhGKRxfoxTXkcw21hTp r7ub9Beb49X2MU8ggo7WfcpDrZAEO4xYKO/TEuodAaPInYo4mvEXXNEBX8qMU1vWlecfj1Mkm1D u X-Google-Smtp-Source: AGHT+IG9ii2gYa2l1Ru0CDr0U3Z+VjNqFIQMP9sV541+3eNYtaCXRxiz3OpYD+AMNof4e5xWs3BX6w== X-Received: by 2002:a81:dc0d:0:b0:627:956b:dbca with SMTP id 00721157ae682-66603ddb952mr7300427b3.40.1721250722781; Wed, 17 Jul 2024 14:12:02 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 00721157ae682-66602490bd5sm780797b3.59.2024.07.17.14.12.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Jul 2024 14:12:02 -0700 (PDT) Date: Wed, 17 Jul 2024 17:12:01 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v2 02/19] midx: add new fields for incremental MIDX chains Message-ID: <337ebc6de7bdf6ff3b4b09c2bea3df2802174e8b.1721250704.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The incremental MIDX chain feature is designed around the idea of indexing into a concatenated lexicographic ordering of object IDs present in the MIDX. When given an object position, the MIDX machinery needs to be able to locate both (a) which MIDX layer contains the given object, and (b) at what position *within that MIDX layer* that object appears. To do this, three new fields are added to the `struct multi_pack_index`: - struct multi_pack_index *base_midx; - uint32_t num_objects_in_base; - uint32_t num_packs_in_base; These three fields store the pieces of information suggested by their respective field names. In turn, the `num_objects_in_base` and `num_packs_in_base` fields are used to crawl backwards along the `base_midx` pointer to locate the appropriate position for a given object within the MIDX that contains it. The following commits will update various parts of the MIDX machinery (as well as their callers from outside of midx.c and midx-write.c) to be aware and make use of these fields when performing object lookups. Signed-off-by: Taylor Blau --- midx.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/midx.h b/midx.h index 8554f2d616..020e49f77c 100644 --- a/midx.h +++ b/midx.h @@ -63,6 +63,10 @@ struct multi_pack_index { const unsigned char *chunk_revindex; size_t chunk_revindex_len; + struct multi_pack_index *base_midx; + uint32_t num_objects_in_base; + uint32_t num_packs_in_base; + const char **pack_names; struct packed_git **packs; char object_dir[FLEX_ARRAY]; From patchwork Wed Jul 17 21:12:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13735783 Received: from mail-yw1-f182.google.com (mail-yw1-f182.google.com [209.85.128.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E038318A954 for ; Wed, 17 Jul 2024 21:12:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250728; cv=none; b=T21GgEszuJ8One8jAgKmcKpmJgpQ8TNSEVmtPvQx1Aw23u82zdZj8b8pkQ2QpsbUD03W2Blafm+j0bowrfYjxJnyquok5HOntCqjfr+hEAbaJniUD5zwx+Mr8Kj8Oc7hw/od62YuzI8AUVM2ykTBmC1TvlCoGdDPdB3M9kMLJ54= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250728; c=relaxed/simple; bh=Isoh02kTDUBRIwHH6++Z7mL1+1LN7zMmYKYVOZk5s94=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=KArKMm5sXPO4McB0wdczuF5HTkwyUxooA0uGOvPaYz2kaEnlSMVEpKOmlEPohAvwGPIIx7q3xcu3AmfTL7zFpONLAmogZGws2d3FU52ZFwkXlUlMZ8eWu5uaZlWFEjXmky3oXBrzBWo+pZt+wedaUN5Uwa8o2qEc8+M0RNUHa7g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=jqkdhx0J; arc=none smtp.client-ip=209.85.128.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="jqkdhx0J" Received: by mail-yw1-f182.google.com with SMTP id 00721157ae682-65f880c56b1so660517b3.3 for ; Wed, 17 Jul 2024 14:12:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1721250726; x=1721855526; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=kz/8b+wuhxyJnYviG8OTEvKWV/IrYoPoFuxkdnrTjI0=; b=jqkdhx0JH9hczkhwaTIkNhifoSSPlxuCRiTSV3cX7SRENnNL9gcVfw6RWejATyhtyX ulHILzOtrwFWEHkv+x9hslu8dMn17N0cQjLe5LYaLeEilp+fcpYgxpv5P3YI2KIbWsYs tQ51yAY99Sge5j37TQVAiKAqv/pBj0pqHLfXOzH1l6PRv+49QkkkQgDdp1CXROiaeMc0 sb1M3Ad0T8LlAFPl0zraCD5//Z+YfeNrQIHCG3Ewntf5XYjEhLKuxjJv/rjOtuY7Qugz BQMBqty0dM0PPEEbeDJA2i21PIfMbAk9s1bwbLd/foJJaWMI34Y68X755RKGMrM1zT8Z 4e/Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721250726; x=1721855526; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=kz/8b+wuhxyJnYviG8OTEvKWV/IrYoPoFuxkdnrTjI0=; b=w/Sff5DSjJnjAV4CCMVJuLJ+4ZbzVQ3e7aGHctOEvROUlDYsFNTS3rigXLyHem7Qmy tvsKtwP+y0KBsebMJIupsXBXQy2WsHMX7GD0zI81rQhqplewBX7FgGkZNGAGBJbolK9v VeIPNbIbA978Z6YY3+i1FYHD4VAdOFNI1cdiDH02mXMO4lhKxhHb5mMEqQr2iRazvHQ0 ZDsLL6TWKzjXrpO25CDLJH1KK3xS4O3pdIgI3mAR5+ckk+LxklFgZ3LGY9SuVuCXgjPR 5CJJPFqrwBT8ZkqUqi6RRTWlVRztbpEqj04u9MVBAP01JlQkz7EP3laaDzxi/hhCSqYX MagQ== X-Gm-Message-State: AOJu0Yxr31px290qB8k0ZiMJG0HLXlHPCQ5JEpGrjkYLuuh6rOXbIvHy +N/KHcznQwkFpy+x0GBiquF7bc1sSxG1GzcQsdlN7nQLUUJfyO04NsjcaChgJG7043kqApKMco/ Z X-Google-Smtp-Source: AGHT+IH4ftP/IYjMb3vn2qs7pLIqwXMIc9fdeRv9WsAxGsFuTFBG1VFP5SilEpRHU7UhGuomrQNnEQ== X-Received: by 2002:a0d:da07:0:b0:64b:630f:9f85 with SMTP id 00721157ae682-664fe45113fmr32414117b3.12.1721250725741; Wed, 17 Jul 2024 14:12:05 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 00721157ae682-66601fb47cfsm784267b3.32.2024.07.17.14.12.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Jul 2024 14:12:05 -0700 (PDT) Date: Wed, 17 Jul 2024 17:12:04 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v2 03/19] midx: teach `nth_midxed_pack_int_id()` about incremental MIDXs Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The function `nth_midxed_pack_int_id()` takes in a object position in MIDX lexicographic order and returns an identifier of the pack from which that object was selected in the MIDX. Currently, the given object position is an index into the lexicographic order of objects in a single MIDX. Change this position to instead refer into the concatenated lexicographic order of all MIDXs in a MIDX chain. This has two visible effects within the implementation of `prepare_midx_pack()`: - First, the given position is now an index into the concatenated lexicographic order of all MIDXs in the order in which they appear in the MIDX chain. - Second the pack ID returned from this function is now also in the concatenated order of packs among all layers of the MIDX chain in the same order that they appear in the MIDX chain. To do this, introduce the first of two general purpose helpers, this one being `midx_for_object()`. `midx_for_object()` takes a double pointer to a `struct multi_pack_index` as well as an object `pos` in terms of the entire MIDX chain[^1]. The function chases down the '->base_midx' field until it finds the MIDX layer within the chain that contains the given object. It then: - modifies the double pointer to point to the containing MIDX, instead of the tip of the chain, and - returns the MIDX-local position[^2] at which the given object can be found. Use this function within `nth_midxed_pack_int_id()` so that the `pos` it expects is now relative to the entire MIDX chain, and that it returns the appropriate pack position for that object. [^1]: As a reminder, this means that the object is identified among the objects contained in all layers of the incremental MIDX chain, not any particular layer. For example, consider MIDX chain with two individual MIDXs, one with 4 objects and another with 3 objects. If the MIDX with 4 objects appears earlier in the chain, then asking for pack "6" would return the second object in the MIDX with 3 objects. [^2]: Building on the previous example, asking for object 6 in a MIDX chain with (4, 3) objects, respectively, this would set the double pointer to point at the MIDX containing three objects, and would return an index to the second object within that MIDX. Signed-off-by: Taylor Blau --- midx.c | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/midx.c b/midx.c index 3992b05465..39d358da20 100644 --- a/midx.c +++ b/midx.c @@ -242,6 +242,23 @@ void close_midx(struct multi_pack_index *m) free(m); } +static uint32_t midx_for_object(struct multi_pack_index **_m, uint32_t pos) +{ + struct multi_pack_index *m = *_m; + while (m && pos < m->num_objects_in_base) + m = m->base_midx; + + if (!m) + BUG("NULL multi-pack-index for object position: %"PRIu32, pos); + + if (pos >= m->num_objects + m->num_objects_in_base) + die(_("invalid MIDX object position, MIDX is likely corrupt")); + + *_m = m; + + return pos - m->num_objects_in_base; +} + int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id) { struct strbuf pack_name = STRBUF_INIT; @@ -334,8 +351,10 @@ off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos) uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos) { - return get_be32(m->chunk_object_offsets + - (off_t)pos * MIDX_CHUNK_OFFSET_WIDTH); + pos = midx_for_object(&m, pos); + + return m->num_packs_in_base + get_be32(m->chunk_object_offsets + + (off_t)pos * MIDX_CHUNK_OFFSET_WIDTH); } int fill_midx_entry(struct repository *r, From patchwork Wed Jul 17 21:12:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13735784 Received: from mail-yb1-f175.google.com (mail-yb1-f175.google.com [209.85.219.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 252CE18A954 for ; Wed, 17 Jul 2024 21:12:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250731; cv=none; b=bVgbVAKloigweJEroWwx7hvoVWeBkg63VixZqCOrJIoJGPTJKm5fGc3PDYa4inF0PtYcP7evM5ljrYVf3mhFNGmY3PIo5oxh69mn3iZgrBa/RtPadeRtYb7DoHEpKCnxK+ISo0zdyu8SkqPVxvJWeeiJpFxWV3J0waogR6TfGIo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250731; c=relaxed/simple; bh=xvg+l9xogRGcDwTMhuUeQWnrxF90DnU6LQvE/TnbY2o=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=kbtVWaMtXqB4iHBPB5FkpWbGSepzZNvDtYyUtNC4Oo8laBi1lPrjh8sFPoVLkYzlnju9/hoVvPb45QUbZQx6Ygvp8dKN9HUdo3MagQbnXoy3kTo0L8ZRsMv2e/tCyy9zbBeu78Qqyisk5KKMkm6C94Q5ylAYOcXg3AROg2n47Ps= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=cZT1li/p; arc=none smtp.client-ip=209.85.219.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="cZT1li/p" Received: by mail-yb1-f175.google.com with SMTP id 3f1490d57ef6-dff17fd97b3so153901276.2 for ; Wed, 17 Jul 2024 14:12:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1721250729; x=1721855529; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=z4V5h99dsx+XVhq/PbTMbdc8GIFrROpwL4mbb/IAAmo=; b=cZT1li/p+f6h5cLubFTz96Wmi7kiJLcZ3mpO19q+mDCOunnAiTEP0gT0sZcecHGxei wYkTYGsVBFI9gYiyzSknF/PrOqXTaUCkMDq42pLO8U4UEDPvYG9GY98XX58Ch7IcGQ45 TtmAKE0PCURyvyo2O+06fX3wrK58+gX/9T6aJYRpf47gpHeMtL7uUdXpYiC2JH6YyHem M5+lQkY2iwyCnMSnQ4u2+tp13b+hdnC0HpeHuPtnAjRERyRM4ee+lo/p5GGT93QnBkAq qPRprl0/L2kCyviAm9zYOWPW+yNUeebJPDtvV6cnD9ytxbtc+3l/ncUitA4TafPSGHZt C6Sg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721250729; x=1721855529; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=z4V5h99dsx+XVhq/PbTMbdc8GIFrROpwL4mbb/IAAmo=; b=OPbRuCxR8blLsXV5sCKeeXSD2vsGTHl3O215wcfGM5JBMvH+bb6maqwlZhEV1c1txd hCUSqgcQ7KFeA57A7a86BaYRoEGdgaLo1hHcujWXUo0XL0AXa+WY5QmMQfOcWx1m+AY0 sOx9uLJ1wp7v71jvRT3btzgWt4q/jf6N0G3F8iBHJQ5zuo+gMzpzCkza3dTWU6OmwNBM 3tJh3JTPop32ffBLBvGKG5DrW+s6cLaY4z4wdTUB41kQpAVtFNgsBXXbjyiVI1+9NLTZ Az27GytmlLVMP/WrPKXm2JXJdrk8Ds24VtR47H4ZzRd08MeWUiZIVYbdweB8rjHlvKnp aZSg== X-Gm-Message-State: AOJu0YwOedcNqAOTkmJdko3BPtmb12L9JBzsynqHnqAKobyUVouP/4LQ GAXLHJbkPu4oT/I3nctKtW+rr3mMPzjZS6LryKUJTzA0P5H0+BV0uWxTQlnQFZwisRQsSx/MnyH d X-Google-Smtp-Source: AGHT+IGJOywWmYMEFiJD0RMcX1hxMH077rtuFTh+00auVlSHdepQB07OH0tvRE47kV2NkMfsLP6niA== X-Received: by 2002:a05:6902:70f:b0:e03:5ed9:5c2c with SMTP id 3f1490d57ef6-e05fea9fcccmr853962276.11.1721250728799; Wed, 17 Jul 2024 14:12:08 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 3f1490d57ef6-e05feb3fde2sm80068276.43.2024.07.17.14.12.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Jul 2024 14:12:08 -0700 (PDT) Date: Wed, 17 Jul 2024 17:12:07 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v2 04/19] midx: teach `prepare_midx_pack()` about incremental MIDXs Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The function `prepare_midx_pack()` is part of the midx.h API and loads the pack identified by the MIDX-local 'pack_int_id'. This patch prepares that function to be aware of an incremental MIDX world. To do this, introduce the second of the two general purpose helpers mentioned in the previous commit. This commit introduces `midx_for_pack()`, which is the pack-specific analog of `midx_for_object()`, and works in the same fashion. Like `midx_for_object()`, this function chases down the '->base_midx' field until it finds the MIDX layer within the chain that contains the given pack. Use this function within `prepare_midx_pack()` so that the `pack_int_id` it expects is now relative to the entire MIDX chain, and that it prepares the given pack in the appropriate MIDX. Signed-off-by: Taylor Blau --- midx.c | 33 +++++++++++++++++++++++++-------- 1 file changed, 25 insertions(+), 8 deletions(-) diff --git a/midx.c b/midx.c index 39d358da20..da5e0bb940 100644 --- a/midx.c +++ b/midx.c @@ -259,20 +259,37 @@ static uint32_t midx_for_object(struct multi_pack_index **_m, uint32_t pos) return pos - m->num_objects_in_base; } -int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id) +static uint32_t midx_for_pack(struct multi_pack_index **_m, + uint32_t pack_int_id) { - struct strbuf pack_name = STRBUF_INIT; - struct packed_git *p; + struct multi_pack_index *m = *_m; + while (m && pack_int_id < m->num_packs_in_base) + m = m->base_midx; - if (pack_int_id >= m->num_packs) + if (!m) + BUG("NULL multi-pack-index for pack ID: %"PRIu32, pack_int_id); + + if (pack_int_id >= m->num_packs + m->num_packs_in_base) die(_("bad pack-int-id: %u (%u total packs)"), - pack_int_id, m->num_packs); + pack_int_id, m->num_packs + m->num_packs_in_base); - if (m->packs[pack_int_id]) + *_m = m; + + return pack_int_id - m->num_packs_in_base; +} + +int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, + uint32_t pack_int_id) +{ + struct strbuf pack_name = STRBUF_INIT; + struct packed_git *p; + uint32_t local_pack_int_id = midx_for_pack(&m, pack_int_id); + + if (m->packs[local_pack_int_id]) return 0; strbuf_addf(&pack_name, "%s/pack/%s", m->object_dir, - m->pack_names[pack_int_id]); + m->pack_names[local_pack_int_id]); p = add_packed_git(pack_name.buf, pack_name.len, m->local); strbuf_release(&pack_name); @@ -281,7 +298,7 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t return 1; p->multi_pack_index = 1; - m->packs[pack_int_id] = p; + m->packs[local_pack_int_id] = p; install_packed_git(r, p); list_add_tail(&p->mru, &r->objects->packed_git_mru); From patchwork Wed Jul 17 21:12:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13735785 Received: from mail-yw1-f169.google.com (mail-yw1-f169.google.com [209.85.128.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2F23618A954 for ; Wed, 17 Jul 2024 21:12:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250734; cv=none; b=bBX6nNAHcRnHsZKt2gwAoOB4iZ5WOQPaZXXGheZAEJ7MUtH+v1OgKLGFiMGeudfUSTyvW6DzPdEhtIyGh73PuzcTJdDxszBHEuBZBc1CqyjfAIVFVWoQzmave3t5Gr8Ye8uqBLjswig0OYSuQVlAY/pXNmts82CGyTcHb3Eg+z0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250734; c=relaxed/simple; bh=SjLa5uDw7SiFV6E/t2nSyHC/4Amwc5dkMGt89V1lAz0=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=iU85aa3Gec2W+v8R+sELmr+G1gCsM/mCkG/6JmsbDBtvKFbzMpbrOPMTa419+cjwRIspjy5RVwd3OEE+s6R9y0nag2ktIbVNlV96RXDGxU6Xy5CsYN5iB2BEtvkpwytrPVH/8oQtVxLOpBpjkJhHQ35N2usgQ7OjziJtNU56zXQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=V0Wju0c1; arc=none smtp.client-ip=209.85.128.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="V0Wju0c1" Received: by mail-yw1-f169.google.com with SMTP id 00721157ae682-66498fd4f91so627317b3.3 for ; Wed, 17 Jul 2024 14:12:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1721250732; x=1721855532; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=Lnzc86+9vzF4QY7wCGj2g/lNYiSAAm9TogHZViVdr8Y=; b=V0Wju0c1VvNaZrAs2+oB9kdt1ZVECMdcOuMFT2pGBkbFQ1xIRoHZaJ2enx55kZZ8O1 Nnfy6zwrZS80VDsRy4zV41ldPPRAYyK9/Xrsk8yGSiq5exqZTT+b8elqFql1ZtwD5fc7 IWqs8xaMA7b6ZMuAXNOACyO8h0i1WopEyHf2vQnhX31TeaBBy1srmlQSfrRLWUPjwtvP kHHeGEo5RHC675K8Guv0LtAXodfIJS/npuNhusLrW1apaCft+8U4DMi5KdoUHlGVmokF fl74cD2fqNuOE5siV6eQsefg0DgL1RoxyriWpiXIaaZdmLrvc02/9dKAt9JsQUUWi6j2 qkIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721250732; x=1721855532; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=Lnzc86+9vzF4QY7wCGj2g/lNYiSAAm9TogHZViVdr8Y=; b=Ajbakx37btgBqJIZ+vq7nR0EjPBKrQF+2BIcc7LtmKcxC5ACTL9UhHqIf6Man4erj1 F3eKgtWx/KxA/FpNIuYckL9voQMX2rb3D9iPA7EeVk6gQa59gV7FKhty8GLL8NwV+2pg QAxEErWOhMcF6eFlH6joUjhg9WTqdHPiEIYBmfhQA0O5l9bzhO892haTKOWSqRBTxgjv FkoojJ9GRCygddOnIKXH3l1fl5BgMTFcwebhNkG9jk+4DhcqKNxudbgxdaqD8XFVpA5O 8P5VV1Xh6R1/mS0GGFNRJ/BuGCscBQIqM5jnFYlrxpxUTuGBaZklcHzUjZ0Kg7u7SOfc mZYg== X-Gm-Message-State: AOJu0Yx44i6G3hmrX4Av7W2MRG8YeYn85DNih8uLk8YrZM+DQzNkTn2Y jB+aFqkMJOAq3bAKCg86pteEHwqYk7S0pLYIRA4smeWLeHfKoS35XNm1NLGXvIP3IsvHsmkz942 v X-Google-Smtp-Source: AGHT+IFaxWaCSIjydCmcFedj32qWzi1028TX5CQ+Hflbv9Dkzg4h8AiaVpHWxpI1fAeSQBEcuiVy1w== X-Received: by 2002:a81:9c0c:0:b0:64a:7040:2d8a with SMTP id 00721157ae682-664fe25ff8fmr33280707b3.23.1721250731956; Wed, 17 Jul 2024 14:12:11 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 00721157ae682-66603ed5624sm748267b3.127.2024.07.17.14.12.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Jul 2024 14:12:11 -0700 (PDT) Date: Wed, 17 Jul 2024 17:12:10 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v2 05/19] midx: teach `nth_midxed_object_oid()` about incremental MIDXs Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The function `nth_midxed_object_oid()` returns the object ID for a given object position in the MIDX lexicographic order. Teach this function to instead operate over the concatenated lexicographic order defined in an earlier step so that it is able to be used with incremental MIDXs. To do this, we need to both (a) adjust the bounds check for the given 'n', as well as record the MIDX-local position after chasing the `->base_midx` pointer to find the MIDX which contains that object. Signed-off-by: Taylor Blau --- midx.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/midx.c b/midx.c index da5e0bb940..d470a88755 100644 --- a/midx.c +++ b/midx.c @@ -337,9 +337,11 @@ struct object_id *nth_midxed_object_oid(struct object_id *oid, struct multi_pack_index *m, uint32_t n) { - if (n >= m->num_objects) + if (n >= m->num_objects + m->num_objects_in_base) return NULL; + n = midx_for_object(&m, n); + oidread(oid, m->chunk_oid_lookup + st_mult(m->hash_len, n), the_repository->hash_algo); return oid; From patchwork Wed Jul 17 21:12:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13735786 Received: from mail-yw1-f180.google.com (mail-yw1-f180.google.com [209.85.128.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 152D118A954 for ; Wed, 17 Jul 2024 21:12:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250737; cv=none; b=C4H78/UdcWvueNGHLO7OdB4BckxT08zYEPKVIiK//3qVV1lNjUca0nq7aWS1kDCuCzRCD5BpownRpHwD4HLyCPOWONz7Yt7dwixXLhxuJacmdioGC71jvU0vqK0oaGPbQxjYLwWIjMWeQi5pPrpUUK0i8rLF/+hmL0H3gFwWCjc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250737; c=relaxed/simple; bh=rp9H5Ps0slzbmwcdWfNKoeCQ2/c24MjW1R7UEIfykJw=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=HF1v/mQKVKm+XCyBAygF7DlVJu9Kd/PrkHjfG9/91CPIDXqmBGUMD1qMZLVVSkkYobnpScVlmIQx37s0cdMJBlryrVXxpfa07Q57s1ztOMN6yMDWtgPrD6WMHRtv0QQZOjTB1bnMT51Ttu/4O4W57cfDC8jNA7ggiL61LnRjpHQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=XmOHzMNa; arc=none smtp.client-ip=209.85.128.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="XmOHzMNa" Received: by mail-yw1-f180.google.com with SMTP id 00721157ae682-655fa53c64cso657197b3.3 for ; Wed, 17 Jul 2024 14:12:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1721250735; x=1721855535; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=y95yQ3vdOL+plYLZWUmvWiBLqym6EC/GeYFs7CrGkDI=; b=XmOHzMNa/F69VLThhyofNRgOyer7yGpjE2chvoYq4/AyDOfS1PqejINUo7T9fl7IDk pTRmasBci9gM7pzl7ypE54ZsjbW2qFOK6FbSXYqPOZiPgjuALXtA6rqnoDsdiRcEGLsy vbEUK4ht30+uJFGX7EB532EiWJrIquMmIN7o3vyK3XbefTDrOWCIlNVlas7zKN64AqO2 e2gQi/4DKgvwIr0Fd1ug3MzAMlVlhli7RstWTErmwvAfvo8/DwY0iodP3M7xguh0JWmy abhAopPkaLjLtsFMSezbzu/YL2orWIwZ4v3Z3Ku2tah5vB+Xh7iufOoXHoMUSm0wn7Qn t29g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721250735; x=1721855535; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=y95yQ3vdOL+plYLZWUmvWiBLqym6EC/GeYFs7CrGkDI=; b=NLDMbQXu5qLB24r1F/3j4tF/3UiX2YyZ+CPLQUFFvfEVeUP9IS8XAFlquxtqVvYdMJ ayTjmZf2zybfe0U+rsirlL00exqYzY1P96NOZLCMT9WD4J/V4m5G7WkdAj1l4xVoyGnI bWbRItOG7vU776V7KZYKaGhOlFaIlKmh4A38W8vDQXaKm/IX3vF6pIiwn4la5+srCSSh 9vGGR5BjvUaYb0mFwGTZ4t+M7fX3vWL4MLDOcfSCwHkAmHJw6x069VUYPCXNHc1TLagT nUDRpMXekuTRlbvcODRYTxxPZOujKauXFnCg/erACj5mXphdSxxl4cTKyid5nVdV76LS LA2g== X-Gm-Message-State: AOJu0YxeysWIqq7HMEXeaxUqb3KH8J1c8r+zmFhBm6A5NY+X0WGOSApt LdcQrXxJ8+aGLERRQK6r+USJuQba2/J+G9s6HMGrYPMlmNrxLQdN4SuXG3KurirgSdl8Mnp795m i X-Google-Smtp-Source: AGHT+IF439x0TSVzU+wCz6j7xENmkIq7hFzzRxzHSPVhVM2WNwH8rDYQ1MH4+yQUH/UoFJ4qpoA/iQ== X-Received: by 2002:a05:690c:6603:b0:646:5f0b:e54 with SMTP id 00721157ae682-666016fa9f0mr10185167b3.8.1721250734871; Wed, 17 Jul 2024 14:12:14 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 00721157ae682-666038f68cfsm758937b3.90.2024.07.17.14.12.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Jul 2024 14:12:14 -0700 (PDT) Date: Wed, 17 Jul 2024 17:12:13 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v2 06/19] midx: teach `nth_bitmapped_pack()` about incremental MIDXs Message-ID: <650b8c8c21b7e8a6e4f65d9b47185a875f0022bb.1721250704.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: In a similar fashion as in previous commits, teach the function `nth_bitmapped_pack()` about incremental MIDXs by translating the given `pack_int_id` from the concatenated lexical order to a MIDX-local lexical position. When accessing the containing MIDX's array of packs, use the local pack ID. Likewise, when reading the 'BTMP' chunk, use the MIDX-local offset when accessing the data within that chunk. Signed-off-by: Taylor Blau --- midx.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/midx.c b/midx.c index d470a88755..b6c3cd3e59 100644 --- a/midx.c +++ b/midx.c @@ -310,17 +310,19 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m, struct bitmapped_pack *bp, uint32_t pack_int_id) { + uint32_t local_pack_int_id = midx_for_pack(&m, pack_int_id); + if (!m->chunk_bitmapped_packs) return error(_("MIDX does not contain the BTMP chunk")); if (prepare_midx_pack(r, m, pack_int_id)) return error(_("could not load bitmapped pack %"PRIu32), pack_int_id); - bp->p = m->packs[pack_int_id]; + bp->p = m->packs[local_pack_int_id]; bp->bitmap_pos = get_be32((char *)m->chunk_bitmapped_packs + - MIDX_CHUNK_BITMAPPED_PACKS_WIDTH * pack_int_id); + MIDX_CHUNK_BITMAPPED_PACKS_WIDTH * local_pack_int_id); bp->bitmap_nr = get_be32((char *)m->chunk_bitmapped_packs + - MIDX_CHUNK_BITMAPPED_PACKS_WIDTH * pack_int_id + + MIDX_CHUNK_BITMAPPED_PACKS_WIDTH * local_pack_int_id + sizeof(uint32_t)); bp->pack_int_id = pack_int_id; From patchwork Wed Jul 17 21:12:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13735787 Received: from mail-yw1-f181.google.com (mail-yw1-f181.google.com [209.85.128.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 377AA187355 for ; Wed, 17 Jul 2024 21:12:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250740; cv=none; b=MyNapQFo8AQpmESoxpFLTLjHWt6KyNZ/VFdl6CSnwe6BONGfIuH3ct/pN6KL8FJULEy5IM2YkZTmsvWv9IpspSjYBU23mbPIVRXt5ujS7EVn5UIjviFzKGO/gx1phiUwvJ0cxPzY6JYd3E/mx/vNC71N1HCsApmjft2vmm2ehzc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250740; c=relaxed/simple; bh=gkvbmHez6Q+UVDblTIOqKfWNM1AUTI+4TH9Zog1dqso=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=Jy8C3yRMn26uI1XVanBtOtakrF/B32OZM9K+BYV5rR202AS+OldBSgn/JPWk1NCZY5d0B1/2ENObXoOrsUmYN9iaEp5CRG4nHbFG8mP7Mi9qBqbVoj2MYfTgxd/hpufMpAOBQZDUnp9JfaKl0wKj+iKVnX8orpgkDQRomgtt4OE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=jypg1woO; arc=none smtp.client-ip=209.85.128.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="jypg1woO" Received: by mail-yw1-f181.google.com with SMTP id 00721157ae682-65f9708c50dso644697b3.2 for ; Wed, 17 Jul 2024 14:12:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1721250738; x=1721855538; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=j2dfvSv62RBvTUQCe99MddFqLKtY7TcgwCI+/me6fO0=; b=jypg1woOYcckZI8/xE+tlqExsN2IYlgaIAfuR1LX/5lVHHeexZdH1QTm2l5W7KHl3c vNbMPLFBM3HBBXr/J54KRLDoQH1H3UdQASUzNGAYoBRDRFjlIrr7HP9RhVU72mV4Ew1y 30xj3GZjCHpPNAa2C7VvXSJEU0iyj9OVi5EjNAkbxTbvmbDp9EaKdWsJlEfYrXbjXzSq u8/Gyj1NTzwkuB5HPuzjvd5h9Ugka212A3G7LluhUs6PiVrqeaJAkrVQ0+z7/xcvOkC2 y39+pUpSjatf2dJDoeYt8OBdlvGVuHTfuHRv2d71CE81T1SwAKUGSLalu4x6Td9Lp7Ls Xj0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721250738; x=1721855538; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=j2dfvSv62RBvTUQCe99MddFqLKtY7TcgwCI+/me6fO0=; b=jhQlLmnV9UMo7fACAuR4fdblJoK4QqPYS9+w5zV5vnIVUzFMcigtjPLiMDms+5q2MG ultPqSP/flxdgM8/BfPXPBdUzYXhOaNAo/cG606V6/tv3caF/xP8EpZQXzs9Y1VK4nTa wR1IkQDMy3SGLC0pWL3V5ChsVR1JCFTqzyDflMUTURIygsVW2V09bV4Wd1QFeLKTZxJt Y7Z+JfPICKJt1ns4lu8/4Ts5RIDEQ4IFBavz1WL25JsbXRWj/mFQcgjwhKh2dQOabRug F1y73xAfi4R7cqPfL/kvJPEMnbzsbzWcWeKWK1ndLdiEHqok0WTHSNuDzPyh2aGXmiHw 2yHQ== X-Gm-Message-State: AOJu0YzUIbF5yUnV/ObXKN/DgwIqBCNARt+x2aWaqXMXbsnnJoCBUYRV vAdjir7jhFkXfUMuSXC8IgUizethkqNZ/SqOjw688hsupb2OmgAZIraXNtdjHchR3IMOET2xYjd l X-Google-Smtp-Source: AGHT+IHaDbYIW1XjOxMO74kn6QxFuZ6uSj53wRiFFAuZfM4G4nNqTCbaI9HoBB1P2lD/wV1ukJMsIw== X-Received: by 2002:a0d:d182:0:b0:643:4879:3399 with SMTP id 00721157ae682-664fe35ca88mr36499957b3.6.1721250738004; Wed, 17 Jul 2024 14:12:18 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 00721157ae682-66603ce7189sm741987b3.104.2024.07.17.14.12.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Jul 2024 14:12:17 -0700 (PDT) Date: Wed, 17 Jul 2024 17:12:16 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v2 07/19] midx: introduce `bsearch_one_midx()` Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The `bsearch_midx()` function will be extended in a following commit to search for the location of a given object ID across all MIDXs in a chain (or the single non-chain MIDX if no chain is available). While most callers will naturally want to use the updated `bsearch_midx()` function, there are a handful of special cases that will want finer control and will only want to search through a single MIDX. For instance, the object abbreviation code, which cares about object IDs near to where we'd expect to find a match in a MIDX. In that case, we want to look at the nearby matches in each layer of the MIDX chain, not just a single one). Split the more fine-grained control out into a separate function called `bsearch_one_midx()` which searches only a single MIDX. At present both `bsearch_midx()` and `bsearch_one_midx()` have identical behavior, but the following commit will rewrite the former to be aware of incremental MIDXs for the remaining non-special case callers. Signed-off-by: Taylor Blau --- midx.c | 17 +++++++-- midx.h | 5 ++- object-name.c | 99 +++++++++++++++++++++++++++------------------------ 3 files changed, 71 insertions(+), 50 deletions(-) diff --git a/midx.c b/midx.c index b6c3cd3e59..bb3fa43492 100644 --- a/midx.c +++ b/midx.c @@ -329,10 +329,21 @@ int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m, return 0; } -int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result) +int bsearch_one_midx(const struct object_id *oid, struct multi_pack_index *m, + uint32_t *result) { - return bsearch_hash(oid->hash, m->chunk_oid_fanout, m->chunk_oid_lookup, - the_hash_algo->rawsz, result); + int ret = bsearch_hash(oid->hash, m->chunk_oid_fanout, + m->chunk_oid_lookup, the_hash_algo->rawsz, + result); + if (result) + *result += m->num_objects_in_base; + return ret; +} + +int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, + uint32_t *result) +{ + return bsearch_one_midx(oid, m, result); } struct object_id *nth_midxed_object_oid(struct object_id *oid, diff --git a/midx.h b/midx.h index 020e49f77c..46c53d69ff 100644 --- a/midx.h +++ b/midx.h @@ -90,7 +90,10 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id); int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m, struct bitmapped_pack *bp, uint32_t pack_int_id); -int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result); +int bsearch_one_midx(const struct object_id *oid, struct multi_pack_index *m, + uint32_t *result); +int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, + uint32_t *result); off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos); uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos); struct object_id *nth_midxed_object_oid(struct object_id *oid, diff --git a/object-name.c b/object-name.c index 527b853ac4..e23b3e695a 100644 --- a/object-name.c +++ b/object-name.c @@ -134,28 +134,32 @@ static int match_hash(unsigned len, const unsigned char *a, const unsigned char static void unique_in_midx(struct multi_pack_index *m, struct disambiguate_state *ds) { - uint32_t num, i, first = 0; - const struct object_id *current = NULL; - int len = ds->len > ds->repo->hash_algo->hexsz ? - ds->repo->hash_algo->hexsz : ds->len; - num = m->num_objects; + for (; m; m = m->base_midx) { + uint32_t num, i, first = 0; + const struct object_id *current = NULL; + int len = ds->len > ds->repo->hash_algo->hexsz ? + ds->repo->hash_algo->hexsz : ds->len; - if (!num) - return; + num = m->num_objects + m->num_objects_in_base; - bsearch_midx(&ds->bin_pfx, m, &first); + if (!num) + continue; - /* - * At this point, "first" is the location of the lowest object - * with an object name that could match "bin_pfx". See if we have - * 0, 1 or more objects that actually match(es). - */ - for (i = first; i < num && !ds->ambiguous; i++) { - struct object_id oid; - current = nth_midxed_object_oid(&oid, m, i); - if (!match_hash(len, ds->bin_pfx.hash, current->hash)) - break; - update_candidates(ds, current); + bsearch_one_midx(&ds->bin_pfx, m, &first); + + /* + * At this point, "first" is the location of the lowest + * object with an object name that could match + * "bin_pfx". See if we have 0, 1 or more objects that + * actually match(es). + */ + for (i = first; i < num && !ds->ambiguous; i++) { + struct object_id oid; + current = nth_midxed_object_oid(&oid, m, i); + if (!match_hash(len, ds->bin_pfx.hash, current->hash)) + break; + update_candidates(ds, current); + } } } @@ -708,37 +712,40 @@ static int repo_extend_abbrev_len(struct repository *r UNUSED, static void find_abbrev_len_for_midx(struct multi_pack_index *m, struct min_abbrev_data *mad) { - int match = 0; - uint32_t num, first = 0; - struct object_id oid; - const struct object_id *mad_oid; + for (; m; m = m->base_midx) { + int match = 0; + uint32_t num, first = 0; + struct object_id oid; + const struct object_id *mad_oid; - if (!m->num_objects) - return; + if (!m->num_objects) + continue; - num = m->num_objects; - mad_oid = mad->oid; - match = bsearch_midx(mad_oid, m, &first); + num = m->num_objects + m->num_objects_in_base; + mad_oid = mad->oid; + match = bsearch_one_midx(mad_oid, m, &first); - /* - * first is now the position in the packfile where we would insert - * mad->hash if it does not exist (or the position of mad->hash if - * it does exist). Hence, we consider a maximum of two objects - * nearby for the abbreviation length. - */ - mad->init_len = 0; - if (!match) { - if (nth_midxed_object_oid(&oid, m, first)) - extend_abbrev_len(&oid, mad); - } else if (first < num - 1) { - if (nth_midxed_object_oid(&oid, m, first + 1)) - extend_abbrev_len(&oid, mad); + /* + * first is now the position in the packfile where we + * would insert mad->hash if it does not exist (or the + * position of mad->hash if it does exist). Hence, we + * consider a maximum of two objects nearby for the + * abbreviation length. + */ + mad->init_len = 0; + if (!match) { + if (nth_midxed_object_oid(&oid, m, first)) + extend_abbrev_len(&oid, mad); + } else if (first < num - 1) { + if (nth_midxed_object_oid(&oid, m, first + 1)) + extend_abbrev_len(&oid, mad); + } + if (first > 0) { + if (nth_midxed_object_oid(&oid, m, first - 1)) + extend_abbrev_len(&oid, mad); + } + mad->init_len = mad->cur_len; } - if (first > 0) { - if (nth_midxed_object_oid(&oid, m, first - 1)) - extend_abbrev_len(&oid, mad); - } - mad->init_len = mad->cur_len; } static void find_abbrev_len_for_pack(struct packed_git *p, From patchwork Wed Jul 17 21:12:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13735788 Received: from mail-yb1-f180.google.com (mail-yb1-f180.google.com [209.85.219.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3FC92187327 for ; Wed, 17 Jul 2024 21:12:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250743; cv=none; b=fkBbGSKFxy2JOd5jlRB3mKP+NEBPkssJdo1077lgFeeLObMRI4xgWzh6tKx5Iwx28YTwKHhlC3r6VIPb7Up+cV/V1Yju1Orfz1fU00wOujnXTDIfug5J+AvPL5PKsr0Jkn2/hMvQ3f0F6++ImAJzEA4QOxF+opHumh8r0R4gJpo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250743; c=relaxed/simple; bh=Z1uJWiM0bY0+Z/2NRKFSV3TOqCNO8nT/bznc9aElnMI=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=pqxSFKwqeCJ1E1+/GnT8+6qlXN0FumSZL2Fc46SO8ayOryeZBKCzaRD3J5kIzYUVhe8KT3i0P1Of+urVhOzwgvZdaYLIDUPPfYU9XqNtscFN3botdQybCt86olZogUqQ2RsO/wiOhY4xmwaA9LMCq9u2LUSUkwxOcJbNCqfHBV4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=ADoRBpoU; arc=none smtp.client-ip=209.85.219.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="ADoRBpoU" Received: by mail-yb1-f180.google.com with SMTP id 3f1490d57ef6-e05ebd3d607so138646276.3 for ; Wed, 17 Jul 2024 14:12:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1721250741; x=1721855541; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=RpOwbj/ng4c/WJGvfE+VS011anhpk8Rt14+aBR7cXfE=; b=ADoRBpoUW3MXwv9Cqcavhbpp7royes9XXDcyxn4Sm3a7b9laziB9PW6e177GiAFaJh eGek20R+9vUj0Ol9ERt5aLTXMg4AZtR/eFHEKdhqNZh6E2IXJlj/4xPPeagqzQJ2+Zqc 3Un/FXVGYQfdHcpJRXvqzKNWqcDqEMUwSPttLRyxZ7KkMDZUtoo6dEQcd8mkesxdrzDZ i/Ras/aGEJGZIgwouTzCGsfDWF5uR+QJG/ke98z4HKEO3olFemTg3dHvqDMexzEnBvbd b1r0dEmOTVrSArnQX0z6VAu6GWqbXmWJ3+NppA63dyP58XcW8EekEY5zMIHbNmeucWCO j46w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721250741; x=1721855541; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=RpOwbj/ng4c/WJGvfE+VS011anhpk8Rt14+aBR7cXfE=; b=f8oHA9PqmEi1Tc116/ggk2C86Jvxa9oOgp0aAwN8oIQIwRq/3fjjkKzgZUEDjd4qX5 1BsaKofPUqtFqnSoGWS7JryCx1MxBTjBzue2LaSgp4BmqHX4R6Gy5tRVIGCzMHy9G6pK 5v6XVTTmOiDTS6sXZdjVIWdetu/poDDNzlGPUnJ7Ivyg7DDrpibtlL0IOjG7NCWXk2Lp oxHnJr68Oo+/y9KMSzbejXH3TrEtLSZwtIkApwUTReDQLAWFG/D0vGDHIu8cdfannpgU P11yzmFb7p791wLMyQr0v10eAA4mjMKO355L8OF74PmiIa8+P5JxKPIz78ANnnw4Fx8B qb3A== X-Gm-Message-State: AOJu0Yw0R+xbWlAnGjwB9jnx0ag5KDyEdCA9eGNxF8mObVFWZ5xjj3Ru O+mFAemTx3gcTqhNC7aL5jY8YmavhXwsWQiUhLo1qyvysNi/1GSvdD0Tw+tHFEw3LeqPF5Zi3Hf v X-Google-Smtp-Source: AGHT+IEJBSCvkqr6J8jBf4s9PCM8tdVoEvfdTs40Tsw2spXcLvsT2hik9W181nmQcFS1jLSD+Sl9yg== X-Received: by 2002:a05:6902:1205:b0:e03:53a1:ecd7 with SMTP id 3f1490d57ef6-e05feae4d54mr781392276.22.1721250741042; Wed, 17 Jul 2024 14:12:21 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 3f1490d57ef6-e05fea87adasm81027276.34.2024.07.17.14.12.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Jul 2024 14:12:20 -0700 (PDT) Date: Wed, 17 Jul 2024 17:12:19 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v2 08/19] midx: teach `bsearch_midx()` about incremental MIDXs Message-ID: <38bd45bd24436d3404b089dff5c3d85f4026da97.1721250704.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Now that the special cases callers of `bsearch_midx()` have been dealt with, teach `bsearch_midx()` to handle incremental MIDX chains. The incremental MIDX-aware version of `bsearch_midx()` works by repeatedly searching for a given OID in each layer along the `->base_midx` pointer, stopping either when an exact match is found, or the end of the chain is reached. Signed-off-by: Taylor Blau --- midx.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/midx.c b/midx.c index bb3fa43492..cd6e4afde4 100644 --- a/midx.c +++ b/midx.c @@ -343,7 +343,10 @@ int bsearch_one_midx(const struct object_id *oid, struct multi_pack_index *m, int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result) { - return bsearch_one_midx(oid, m, result); + for (; m; m = m->base_midx) + if (bsearch_one_midx(oid, m, result)) + return 1; + return 0; } struct object_id *nth_midxed_object_oid(struct object_id *oid, From patchwork Wed Jul 17 21:12:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13735789 Received: from mail-yw1-f171.google.com (mail-yw1-f171.google.com [209.85.128.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A0CD8180A63 for ; Wed, 17 Jul 2024 21:12:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250746; cv=none; b=irvpM6tw5DXoaLO5jcFpEd8CcM1R0UrqOHDjhZ5+EXVxBvLCbkEp7OdOyWlKzb522vrlIV9EWWFTeWvn4lBr66dKHVhAhIIhmuC0WBQMXlnae4GaAsv54Q84hFoBgvE105l6fCQBFnPuQVIodO/PRXTgqQUA4GVZDBWhndivTQo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250746; c=relaxed/simple; bh=ZotPqOOEp5f2fa3K3TkdKn25ebiG6v+sBu8YiUiogBw=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=fWoNGCeQMX9vAn08q5lxsNGw9MHJttsME2u2EqGMqsxci5WBlqwG2rCqMtG6Opi3/HahKvrYJVtL9UcI/lJwcoee66eH/H8wDCLVS6N3GAAnHdRxRzjLj+WODFU0goY38NFWz/8Xx8BExHGgwEnlFFNz7bLfFr6hqdxVpGyQ5O4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=oU8ZznNX; arc=none smtp.client-ip=209.85.128.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="oU8ZznNX" Received: by mail-yw1-f171.google.com with SMTP id 00721157ae682-661d7e68e89so952147b3.0 for ; Wed, 17 Jul 2024 14:12:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1721250744; x=1721855544; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=3/jpQFS4mTRzksaALfFDBl+2QA3kHR+I+0y1XrKQlBI=; b=oU8ZznNXZXnjiNKywuKHTr0poCW0k5+DnbY0+jXpBkOvCCF90e6P2ywzYcFzRS/Mkn aMJYTqGAp7/mWCsm2ggXZCh3O3dj/rV5pCNoRs0E0jXfShAkG14Aihj+uj7XGo/Yf3m1 Ke004tlq0Rcbrij+NdjpztqNqs7qTPEf04d9DeJliAxS+Y8K0xWwm33w2Ob0G04jLSVu WCf0PTKTQv/Ka1Kj+/AYqkjvfdYyrNpVdo9ZAzSlsF5K6eXSTYRj5O/JUoKi7Ts9NIjr t6XO0w5hbhUQphWJMyjhQOwcNhnHXh3x5pol1x+b7cnL+l+DFZmOr47iFdjwPqw89MfD RrqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721250744; x=1721855544; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=3/jpQFS4mTRzksaALfFDBl+2QA3kHR+I+0y1XrKQlBI=; b=YDooHcQurigj18GdnYe8wA6E3vVH+K8LLH2OuVqBo2aMiSTRIVtpZdIo1LUMQ69HTW Fz9onQa9pkwlosbJfdSfX/ihI+kYy9r873jiV7aBw5y7PfzOriDa6ZPx7F4D7wvEq38u g/OZfVHisYb1VajFegvN7L/Pnx2WZx+GZc2dJl9NnFVNwoL46Q9B3I0LVeQinIaLXwD4 pjMEYvwedJe+V73W18VsvwyGY0rFZz5/fYTS/H1F0ePXxKNZKk4SdEgXcB4SXWdHndbQ sJd67L95LT9hjcMrHbEI9b99vmI+PL7VhCMRLtuHz2r0uDEvkr7uCtspV14QPX3IoqtU HdvQ== X-Gm-Message-State: AOJu0YyL8LpD5jAbaKlj/gucZ7lQ2XgqUL+pu1ME/I2CwCWGI8SB/rYj hjqQDvMdd3b7FqWd4oMog8jftGEaU2H2lkxwoVrlmBcWLYXxjFhdkBovwbVi2UD2aSP+WEgMejD s X-Google-Smtp-Source: AGHT+IFqeCiRiYVviULrTIf5nE0PvU40n5LccpccIWHT2mA4TcNeIvQz1vhdYfqr8yyaP/5sZYKPFg== X-Received: by 2002:a0d:f803:0:b0:630:4fab:a090 with SMTP id 00721157ae682-6660a442443mr4395827b3.22.1721250744307; Wed, 17 Jul 2024 14:12:24 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 00721157ae682-66603ed2393sm755287b3.126.2024.07.17.14.12.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Jul 2024 14:12:23 -0700 (PDT) Date: Wed, 17 Jul 2024 17:12:22 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v2 09/19] midx: teach `nth_midxed_offset()` about incremental MIDXs Message-ID: <342ed560331d95fb0a1d86743dbb999ec7327313.1721250704.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: In a similar fashion as in previous commits, teach the function `nth_midxed_offset()` about incremental MIDXs. The given object `pos` is used to find the containing MIDX, and translated back into a MIDX-local position by assigning the return value of `midx_for_object()` to it. Signed-off-by: Taylor Blau --- midx.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/midx.c b/midx.c index cd6e4afde4..f87df6bede 100644 --- a/midx.c +++ b/midx.c @@ -368,6 +368,8 @@ off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos) const unsigned char *offset_data; uint32_t offset32; + pos = midx_for_object(&m, pos); + offset_data = m->chunk_object_offsets + (off_t)pos * MIDX_CHUNK_OFFSET_WIDTH; offset32 = get_be32(offset_data + sizeof(uint32_t)); From patchwork Wed Jul 17 21:12:25 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13735790 Received: from mail-yw1-f169.google.com (mail-yw1-f169.google.com [209.85.128.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B0F9E180A63 for ; Wed, 17 Jul 2024 21:12:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250750; cv=none; b=uuK1puIqGOC7kaaBq3CguWmcL7vrq5OXfUCVYaeEX51xAijkTX2URbnKo41iWtGMKjFrWn6wIsS3wIxbfXlBriKf8Iduch5maVQDvg03J8p+qdTu3jMrBVdgXGns8V/agyPREJv7qcVFVllbBdap2HfglvCj8rSlrTL4KE+jWXM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250750; c=relaxed/simple; bh=/tHMJPmK6xpqPyULfzyADIx0MY37StOQKNC4IV283bc=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=PlL61fC+ggkeO7yoikGN43sX24Eh30H+NrEV3EToX6rjcX240HktwulPoVgEoorkheYgckLx1GecxrqwX762jar6itoVtqUBzqSNgyWsZQxCUIuZPi4qL8UP6CZHMoojbE98V2bldv2bPHVthKf8FfBClJq3jhl2mTsC8nexIt0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=UZNSauJg; arc=none smtp.client-ip=209.85.128.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="UZNSauJg" Received: by mail-yw1-f169.google.com with SMTP id 00721157ae682-661369ff30aso618077b3.2 for ; Wed, 17 Jul 2024 14:12:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1721250747; x=1721855547; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=MiMFM/estUwkAhNM3IuW0HwUUZc7GFqkIj+rnivbj4A=; b=UZNSauJgLo08dTmFq2zdtiak6sOiA5Z5+3w6IrmfxAt5EHG3GXkRnZcsaWkOhVR13W vtwP/XgpuE7fyH0/U+On1zrNdsxjSrmexEQizO75iIOtSuK9wRoOZb1cELU4jLW4Bgei IVM+xZVDZnniuv7PVVrKHFB/0l3JjLMFaaNWp8APKE/BW+Dl6fDBIZAFE12MLAEgfAHJ +mt2Wmgoxt5uxYetSa2W+EXUZrTnsz3UJeYu+kDqUmhq/yQiahyzQsT+vFG49vJsio1N HbHn7Mpwqf27XlZS118xeov/r06FnLR+A4/QcSMBsTdoBztT0IYimXdp6daRvGa6OpOR gTYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721250747; x=1721855547; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=MiMFM/estUwkAhNM3IuW0HwUUZc7GFqkIj+rnivbj4A=; b=bpUJ/EEiHIi3qzmMCl6j9jSmlcoo/ZnTcdCISbxu4YLZ8wHdKGHpsd/Cs7Pl5upWph usvfoGyssfWJRp+F2jDzgAEGdrVX6cOjh9ml5rZgwiO+6PBkYB7dWoctziKacdKjdSyr C4p1sNjnz7cJixN607ffCcFyWCWA8WB9pc5laUg5Io+G4Kg5nQ12sVfIZA2Kb7Di+9uM yHVjkXCDFSd135qWaKMc+Pg1v4I0DuU6wieZ2/t9Y6rDdLs9BdY7sy3b+lv03oybT3k8 nZjB5OICagegGOx5h5MwYqAhSaNeQ1FPMHQOW2iqsCcPacDlOkC2wjA5I7FOejNmaUhe eFbA== X-Gm-Message-State: AOJu0Yz0TY3MnJSCoPh+1PjGgwHO+nu2co0/7e+ieZMvr4p9VgVAkOcw AIvamlmO/+kcZ9UGHxdks+yfkI8Eww2VLlmczuOEh4Txvy0dpLAkmnLDwLA4EBVXO4n81lB3vQL s X-Google-Smtp-Source: AGHT+IE0ZLOqamqbn08movONf9Oe4thegCBtSBxNvoLaFAM9Yix5a92Z5dSs5QnnCthWi8x3BZfVKg== X-Received: by 2002:a05:690c:2c93:b0:64b:9f5f:67b2 with SMTP id 00721157ae682-66603510ff7mr7178347b3.31.1721250747458; Wed, 17 Jul 2024 14:12:27 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 00721157ae682-666042bea3csm747937b3.136.2024.07.17.14.12.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Jul 2024 14:12:27 -0700 (PDT) Date: Wed, 17 Jul 2024 17:12:25 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v2 10/19] midx: teach `fill_midx_entry()` about incremental MIDXs Message-ID: <2b335c45ae7832b886ef9adccc2530e4ca53d267.1721250704.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: In a similar fashion as previous commits, teach the `fill_midx_entry()` function to work in a incremental MIDX-aware fashion. This function, unlike others which accept an index into either the lexical order of objects or packs, takes in an object_id, and attempts to fill a caller-provided 'struct pack_entry' with the remaining pieces of information about that object from the MIDX. The function uses `bsearch_midx()` which fills out the frame-local 'pos' variable, recording the given object_id's lexical position within the MIDX chain, if found (if no matching object ID was found, we'll return immediately without filling out the `pack_entry` structure). Once given that position, we jump back through the `->base_midx` pointer to ensure that our `m` points at the MIDX layer which contains the given object_id (and not an ancestor or descendant of it in the chain). Note that we can drop the bounds check "if (pos >= m->num_objects)" because `midx_for_object()` performs this check for us. After that point, we only need to make two special considerations within this function: - First, the pack_int_id returned to us by `nth_midxed_pack_int_id()` is a position in the concatenated lexical order of packs, so we must ensure that we subtract `m->num_packs_in_base` before accessing the MIDX-local `packs` array. - Second, we must avoid translating the `pos` back to a MIDX-local index, since we use it as an argument to `nth_midxed_offset()` which expects a position relative to the concatenated lexical order of objects. Signed-off-by: Taylor Blau --- midx.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/midx.c b/midx.c index f87df6bede..dea78474a2 100644 --- a/midx.c +++ b/midx.c @@ -406,14 +406,12 @@ int fill_midx_entry(struct repository *r, if (!bsearch_midx(oid, m, &pos)) return 0; - if (pos >= m->num_objects) - return 0; - + midx_for_object(&m, pos); pack_int_id = nth_midxed_pack_int_id(m, pos); if (prepare_midx_pack(r, m, pack_int_id)) return 0; - p = m->packs[pack_int_id]; + p = m->packs[pack_int_id - m->num_packs_in_base]; /* * We are about to tell the caller where they can locate the From patchwork Wed Jul 17 21:12:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13735791 Received: from mail-yw1-f172.google.com (mail-yw1-f172.google.com [209.85.128.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C979F188CDC for ; Wed, 17 Jul 2024 21:12:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250753; cv=none; b=Um4GTUrwDbgyczZAKFTgfYgnXAIIgVW8N5uhXfDBZZA29+s6zHpUktRvrELB4BPgPJCpPaWOfyOQsQfvIVKlfFIPw8O4N93SFBDoPY5H/zxBukxL6f5yLOOLlD94iqW2fk7KlOS+f3jHsg8QLgZSrnil3WaFOcNqPCMWz4fuMug= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250753; c=relaxed/simple; bh=IqcyzxEercn3gQXpgKMAgTaIphSF5/oTg5WI4onxlU8=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=Qp8+vTBmPQelM7fvxAI/dsiPrexBqVbOk9YaasEWofA2QsKc/XOep0uhsLnoUdN9jh7KxRjxUHhSqGNkeORYF/uQRERWuyj3iQvuyw52hmUne+EMN5XLbYN/VD2YE1px6/LNAHZtm39OO5V0SDfNz2zkFBZLijEf2XcThTy6vss= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=kRgXwcjj; arc=none smtp.client-ip=209.85.128.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="kRgXwcjj" Received: by mail-yw1-f172.google.com with SMTP id 00721157ae682-6512866fa87so774197b3.2 for ; Wed, 17 Jul 2024 14:12:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1721250750; x=1721855550; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=e74Z1x1jZ+hsBKN6Lza6AhIkxE9dD8u6Nwie1Vz8AIo=; b=kRgXwcjjKbOqVIIVe3Z5bmR+z9CzbIfpfoWDofb8PQ/2DfSgnhyKq9ZeQl3i1nyVcf 3IZ6QmiYpW5O2AvqHXZ6677bxZLgL7BSKw+BnYeuWVe/8BED5uZmLrcXIzh6vj4hKSgq qahEZ8cG88vL0j7EFSFQBtJ6BLmoixONWP+AK3//NizJVj6qGJTqYppNBBud5lGLZhUC LPZUtQtAgKKwDP3wOegMNa9AcfRO8gHS/Ti4oEOgtCN/y87L+7Y5uJJte2/FHBWwdli8 clEY/iTUV84EkVkPFicoc2U13dYnZIVKReqNlWtrSzpHrvCQiMBnM0xW+g6Ug3WmmkUb GOhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721250750; x=1721855550; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=e74Z1x1jZ+hsBKN6Lza6AhIkxE9dD8u6Nwie1Vz8AIo=; b=ujlF3B1rwygkdHHbii2sw8t5tP6AxH/9fbVlDroSDI5ExPW0sQsqnEH6CdXSMUpQN8 wol32wzH68YP5X6I0Fvww2/okRlIRYkaopOChbA86YVpjug+TN5P7flMh3F4uUbBPj+H VtzmWxHgEa191ssp7XHz7Jgs49eV7JlgYsZZnXtes/uFbO5i7t8j4+yi4y+qrlsVR93c xh6efaoyK6n7AKvREYehjtYOo/+1zM8G2+N/gW3HY8M9oN2pS5ItHt4AM2++mchplHqD kPwmlC9TxhahBpBMvqm2MX+PixcoykndRtm9/HLeoOs0ZD1LRH2HS0Pnn2pWkR0ztHvS nFBg== X-Gm-Message-State: AOJu0Yx3W8N1MaoAwH7WQQlPZSLhZI4+WXjajdfue7CsW9SmrXDUze/i +Eb48AxHAzy2+sq0FpOTdag9MnLyTJvhPRfcQzSM6KjyUD2zIqo8md3XJBkVe1ccL1pCwVtX68f H X-Google-Smtp-Source: AGHT+IHQIJCHC6rv3uGz2PZBvYjVI/6/QMmeN0xEEpCkwuORZaWeQDkF+HbqUFAjr48HHQIUc2T4Aw== X-Received: by 2002:a05:6902:1548:b0:e02:ed0f:e83b with SMTP id 3f1490d57ef6-e05ed6b96a0mr3432497276.11.1721250750601; Wed, 17 Jul 2024 14:12:30 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 3f1490d57ef6-e05fea86e65sm84425276.35.2024.07.17.14.12.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Jul 2024 14:12:30 -0700 (PDT) Date: Wed, 17 Jul 2024 17:12:29 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v2 11/19] midx: remove unused `midx_locate_pack()` Message-ID: <22de5898f3fec0cfff4289660b5f52aa31533fe1.1721250704.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Commit 307d75bbe6 (midx: implement `midx_locate_pack()`, 2023-12-14) introduced `midx_locate_pack()`, which was described at the time as a complement to the function `midx_contains_pack()` which allowed callers to determine where in the MIDX lexical order a pack appeared, as opposed to whether or not it was simply contained. 307d75bbe6 suggests that future patches would be added which would introduce callers for this new function, but none ever were, meaning the function has gone unused since its introduction. Clean this up by in effect reverting 307d75bbe6, which removes the unused functions and inlines its definition back into `midx_contains_pack()`. (Looking back through the list archives when 307d75bbe6 was written, this was in preparation for this[1] patch from back when we had the concept of "disjoint" packs while developing multi-pack verbatim reuse. That concept was abandoned before the series was merged, but I never dropped what would become 307d75bbe6 from the series, leading to the state prior to this commit). [1]: https://lore.kernel.org/git/3019738b52ba8cd78ea696a3b800fa91e722eb66.1701198172.git.me@ttaylorr.com/ Signed-off-by: Taylor Blau --- midx.c | 13 ++----------- midx.h | 2 -- 2 files changed, 2 insertions(+), 13 deletions(-) diff --git a/midx.c b/midx.c index dea78474a2..59097808a9 100644 --- a/midx.c +++ b/midx.c @@ -465,8 +465,7 @@ int cmp_idx_or_pack_name(const char *idx_or_pack_name, return strcmp(idx_or_pack_name, idx_name); } -int midx_locate_pack(struct multi_pack_index *m, const char *idx_or_pack_name, - uint32_t *pos) +int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name) { uint32_t first = 0, last = m->num_packs; @@ -477,11 +476,8 @@ int midx_locate_pack(struct multi_pack_index *m, const char *idx_or_pack_name, current = m->pack_names[mid]; cmp = cmp_idx_or_pack_name(idx_or_pack_name, current); - if (!cmp) { - if (pos) - *pos = mid; + if (!cmp) return 1; - } if (cmp > 0) { first = mid + 1; continue; @@ -492,11 +488,6 @@ int midx_locate_pack(struct multi_pack_index *m, const char *idx_or_pack_name, return 0; } -int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name) -{ - return midx_locate_pack(m, idx_or_pack_name, NULL); -} - int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id) { if (m->preferred_pack_idx == -1) { diff --git a/midx.h b/midx.h index 46c53d69ff..86af7dfc5e 100644 --- a/midx.h +++ b/midx.h @@ -102,8 +102,6 @@ struct object_id *nth_midxed_object_oid(struct object_id *oid, int fill_midx_entry(struct repository *r, const struct object_id *oid, struct pack_entry *e, struct multi_pack_index *m); int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name); -int midx_locate_pack(struct multi_pack_index *m, const char *idx_or_pack_name, - uint32_t *pos); int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id); int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, int local); From patchwork Wed Jul 17 21:12:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13735792 Received: from mail-yw1-f172.google.com (mail-yw1-f172.google.com [209.85.128.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 104CF18E74A for ; Wed, 17 Jul 2024 21:12:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.172 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250756; cv=none; b=MM48K08pfVDasHIp882zVG1GFpySdlCHHF0cnyF7PeUXHnSV8r+d91gFIbVfqohAFQ8oOM+kbdJqeDPaJeZLp6Bulan1rnEpPbVxZ5TEWUFrBMa9EB9yIHUmvKmP2Y5doviZ4xfPRtmvbnrpMI0DICdRpC3t1RYRElx3ePpUH/8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250756; c=relaxed/simple; bh=snH9jUej4F5TP+BHwD5nYi44yOSVOmO6I+N5WAng/D8=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=DSs3gEa3TwhCE8CCt/4jktdOBgldpKnsD9G/cA+QMAMPonaRbdQOqrcB1ENP6HtTI/MZK8ZMq6T9jmrBSmlaKIF9l5dgrjXG7UhRsu47jWzKvH2AufsHp4jAsEYkqITKieudLqnMYAwV6RQXyx9hNRI3EPZAe1N+HuZhoATq3M8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=Q4p1BE8B; arc=none smtp.client-ip=209.85.128.172 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="Q4p1BE8B" Received: by mail-yw1-f172.google.com with SMTP id 00721157ae682-64789495923so949967b3.0 for ; Wed, 17 Jul 2024 14:12:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1721250754; x=1721855554; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=w+Wi3QtJrRlEnE0BlHCFAXiBSHl4UPUjPzhPjhJGito=; b=Q4p1BE8BwIAJboqKUpsjZpFy4qiylMi1YBuzthLzfwDbfBH9+lenL85cZNrDBSL/A9 dM2diTZKQj4EKi/hV1sH6OA91q/dq4RjSwmW0pvHBH4sDguns4xMkxipt7NcgJbhbpkT VxV+l0eIjOpXoTtKik8k5HYBpXsh4dxZ+c7Jog5zOohdq84AwGj2BgBeNBtqTJkVEPw8 CBvjULiMpGhS3Ujla25r6eEjKjzg4YkuWKiUMa8j+j99ObUHpaslOUblP843J2ziaP3P TVGn9btU5hIltfPyQcwtyVFDTEoi/pRSJZjJCZgw+bdFgxp5ZBHMBkDMKYxQV6elbNDE QoQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721250754; x=1721855554; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=w+Wi3QtJrRlEnE0BlHCFAXiBSHl4UPUjPzhPjhJGito=; b=PC/sTRpx1+R+xpw8NALSTC0GZcqhM/ThgA3T7zErDto4+mOF56Vh+KuGWMPCfvXZEQ 3s5KZCA2IOfIa69iFtr5S4Udjzz8vmggK5CC98aKZdFq9vPtOxEt9UdlkyXeFNQB+g6X s4U2ahZ25VoUEfZI4vHvNonrmwxoyett8ZSk9HnSC6N+a9Mp6JsGF/y4sfhr3jGUggcM dzP8zrwJbbYrQ52zTzYUu7YbN7DD4t+RhquvBrvJtS2B/ZqbG252DVagWsa75w5psx2Q +qur3l32dqvxQmsvdNgNPrestCdFABp8Hp2yTAdKBpVeARGt3UFpsH1yYg+FzKfdaNSi ZsZg== X-Gm-Message-State: AOJu0YwXLq9E7y/KtPKWsXiuNkv1+vV9Fn6NfkGFxVZlYmjUWKs48L/R zsgunWDT+RknZRxxJeWbXwGAn9lqPY8uN2SOLleL7B0m57jfyy2bqFRY/zQxg4YpGnydp1ZbVgh A X-Google-Smtp-Source: AGHT+IHuve/AoLa7CZB54hUu1HxvB2kGwCynBReOZhw3pP9qdNBjcdyVLcCLKxSO5hU1wlPGHKPYbQ== X-Received: by 2002:a05:690c:ec8:b0:64b:3e44:e4f4 with SMTP id 00721157ae682-666016fd79emr10230127b3.7.1721250753753; Wed, 17 Jul 2024 14:12:33 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 00721157ae682-666025869e8sm760327b3.69.2024.07.17.14.12.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Jul 2024 14:12:33 -0700 (PDT) Date: Wed, 17 Jul 2024 17:12:32 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v2 12/19] midx: teach `midx_contains_pack()` about incremental MIDXs Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Now that the `midx_contains_pack()` versus `midx_locate_pack()` debacle has been cleaned up, teach the former about how to operate in an incremental MIDX-aware world in a similar fashion as in previous commits. Instead of using either of the two `midx_for_object()` or `midx_for_pack()` helpers, this function is split into two: one that determines whether a pack is contained in a single MIDX, and another which calls the former in a loop over all MIDXs. This approach does not require that we change any of the implementation in what is now `midx_contains_pack_1()` as it still operates over a single MIDX. Signed-off-by: Taylor Blau --- midx.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/midx.c b/midx.c index 59097808a9..0fa8febb9d 100644 --- a/midx.c +++ b/midx.c @@ -465,7 +465,8 @@ int cmp_idx_or_pack_name(const char *idx_or_pack_name, return strcmp(idx_or_pack_name, idx_name); } -int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name) +static int midx_contains_pack_1(struct multi_pack_index *m, + const char *idx_or_pack_name) { uint32_t first = 0, last = m->num_packs; @@ -488,6 +489,14 @@ int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name) return 0; } +int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name) +{ + for (; m; m = m->base_midx) + if (midx_contains_pack_1(m, idx_or_pack_name)) + return 1; + return 0; +} + int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id) { if (m->preferred_pack_idx == -1) { From patchwork Wed Jul 17 21:12:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13735793 Received: from mail-yw1-f176.google.com (mail-yw1-f176.google.com [209.85.128.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D5AB218E74A for ; Wed, 17 Jul 2024 21:12:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250759; cv=none; b=cyJlVrNxU7Xq7S18QzMPeMUVYMolF0QExTCuzCBWBCjJ23L1DZpPRqm2lV8J7+aZU+YVzAayVcXH+/Rht8SPhNoOTidVIbGWYrVSGs5yIjQlJwX9l7sYhWRfKUoayScgBfaVM+O/9XRClpmonVkC0EYfUXiAOzmeM9AeYAI4NGA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250759; c=relaxed/simple; bh=Q2sbIQhpYSsr+/BXnEOSLnGp0p2vb5wN0mSzNH3FU/w=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=VYHq8Q93meDxT/W6dluSbDLRc3fg85IS4hpT9py9iN6YYKFe28hJkDpC6bsje3l3JFXQ19TiIr1eqv2Zcf4l/gpQZ8/UnIOSnbccqlhQoRM7lftj2Ou17W8WYL7s0XpkUdDDPtrdhP8S5oj0MiR0hgXRSS/KwJ6ut0/1qznIzmM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=3DnXIolB; arc=none smtp.client-ip=209.85.128.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="3DnXIolB" Received: by mail-yw1-f176.google.com with SMTP id 00721157ae682-664b4589b1aso1913247b3.1 for ; Wed, 17 Jul 2024 14:12:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1721250757; x=1721855557; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=Ns4yHYpyM/cO7ZxUx/3QB90dxH93QV0allO65drdbEs=; b=3DnXIolBYFpMQxDqlL0GobDgNPcVIhNDMvGuTnW653BTEJ/Icixb1hU3MXEJWCnAcs ASa73YIPrKzZMb4Dm41DSSbbhIYXwNETDiLNnlKCLA9l1LPdemkDaeVrr4iJDjP+/RVM eVYEz2+HAHi6NloucvfXWlp6usLfsD/ZXF3XsNcKBGMdanP2wRa7zNnHUeFIy8gBYUGX EDYEPtQiEIhRFJP+y2jneR63/J/jgyl120dOj/PbfwW3AbjAiIlTCbbXz8ODCrfnKGdJ 3BEMJt2QHW6csQwgkJvVccf0GBpoS+ogFATXPU1aKDLae5JVIWxXS2qY0KA2uj5CiRGl NEmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721250757; x=1721855557; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=Ns4yHYpyM/cO7ZxUx/3QB90dxH93QV0allO65drdbEs=; b=cHTgOouPdpus7Gq0DKs4U/TGGLhmCIddaNDekIZr1+ulDCqOqv/S03NZPKjqix3PZI niYm1F/PQ27XSELhSV4X7QrMv2X4XK16wUuP7CPOvXfBnKAJrLet14wbFjHAHzOdwaN7 hMrt9WukgrdfMnEE/PH4ie38JSLcCcxhqFMuyvSEjTyXam8ll+vpn+isI0hMgge+oX8j KF02jfyyMH/1ajcWICLU+DdirBWI3Zku2kqfPIPIEpeU4EF97K1moyMokJvYLBOQvMnQ Uohx/r9dFEqnGW2QJwkS3Sc5HKc2um1elSFZu3/msXy6RQDadn2hjdHENUGFka0omGbs 64Yg== X-Gm-Message-State: AOJu0Yxje4geN7JYeFJvgDb680Iat/8PNDF4Vm48qf96zpM3Ra8fnQ1q Z0psgOww+w6JEeJ7woUyEkffbF72MBFw5sjwLF+2GJXpNJCYDrjnWPC2bQdEdI4Xkj3nEUOQgrF B X-Google-Smtp-Source: AGHT+IFvvprnNOCeqo070Agk35Sf2Jd5LV8VkwRb/2sZ+YrLUZ6RJHULc+NQKmVMCR8BAyghyeIykg== X-Received: by 2002:a81:77d5:0:b0:61a:d0d8:75f6 with SMTP id 00721157ae682-6660a738d69mr4421817b3.24.1721250756738; Wed, 17 Jul 2024 14:12:36 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 00721157ae682-666025869basm779987b3.71.2024.07.17.14.12.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Jul 2024 14:12:36 -0700 (PDT) Date: Wed, 17 Jul 2024 17:12:35 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v2 13/19] midx: teach `midx_preferred_pack()` about incremental MIDXs Message-ID: <38b642d40423f2e14dd47d676de54f264b9d6f22.1721250704.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The function `midx_preferred_pack()` is used to determine the identity of the preferred pack, which is the identity of a unique pack within the MIDX which is used as a tie-breaker when selecting from which pack to represent an object that appears in multiple packs within the MIDX. Historically we have said that the MIDX's preferred pack has the unique property that all objects from that pack are represented in the MIDX. But that isn't quite true: a more precise statement would be that all objects from that pack *which appear in the MIDX* are selected from that pack. This helps us extend the concept of preferred packs across a MIDX chain, where some object(s) in the preferred pack may appear in other packs in an earlier MIDX layer, in which case those object(s) will not appear in a subsequent MIDX layer from either the preferred pack or any other pack. Extend the concept of preferred packs by using the pack which represents the object at the first position in MIDX pseudo-pack order belonging to the current MIDX layer (i.e., at position 'm->num_objects_in_base'). Signed-off-by: Taylor Blau --- midx.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/midx.c b/midx.c index 0fa8febb9d..d2dbea41e4 100644 --- a/midx.c +++ b/midx.c @@ -500,13 +500,16 @@ int midx_contains_pack(struct multi_pack_index *m, const char *idx_or_pack_name) int midx_preferred_pack(struct multi_pack_index *m, uint32_t *pack_int_id) { if (m->preferred_pack_idx == -1) { + uint32_t midx_pos; if (load_midx_revindex(m) < 0) { m->preferred_pack_idx = -2; return -1; } - m->preferred_pack_idx = - nth_midxed_pack_int_id(m, pack_pos_to_midx(m, 0)); + midx_pos = pack_pos_to_midx(m, m->num_objects_in_base); + + m->preferred_pack_idx = nth_midxed_pack_int_id(m, midx_pos); + } else if (m->preferred_pack_idx == -2) return -1; /* no revindex */ From patchwork Wed Jul 17 21:12:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13735794 Received: from mail-yw1-f169.google.com (mail-yw1-f169.google.com [209.85.128.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 087281850A0 for ; Wed, 17 Jul 2024 21:12:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.169 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250762; cv=none; b=piO2c8D4R3H23PTxMXIqc5e1Pf2G3nPSQaH+EkEi+ErPy9ysd0s6AJ129YoaSCysNZF0HGMKVOlSlWqMnHT1e1k87rtp/IyLLUVoosmoIbyM9orLI1NClx899BIkpADMbE42OmFpmakKqr2xwAk2S60Cb0Md/vUS2rgNhJYyN9w= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250762; c=relaxed/simple; bh=hk+muef5xEL7uvfylUoJxyYDu+naAELbOYZJY1cPx+s=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=CqQsnrfe0ixsxhu6m7E5ZGEBpWP/X9nZgKLgwBKK8BMMzNHOvpdPqe2OyUI5fvKr3ZqucI+oAE+Fq1TcAa5NA0ANdBdXfkxLsPynHWZPbvhCjrRmEyAMC3ecqmqk3jC9coKlxZaaInVZDPSdvMu6dSD9bxD18DM6egrF7ODJE0U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=q+0qnDlS; arc=none smtp.client-ip=209.85.128.169 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="q+0qnDlS" Received: by mail-yw1-f169.google.com with SMTP id 00721157ae682-6510c0c8e29so903327b3.0 for ; Wed, 17 Jul 2024 14:12:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1721250760; x=1721855560; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=hAakTSQquisNdf+ftCvXBOJ4qymHPREWzr12EPqTS48=; b=q+0qnDlS+Ritx1V1wkrsHuea5ZUw9S40/tVOUShfW1iegqQ1SRTDY9APfxLr+8TsTW m2LRMuTCnUJn0W+1u9Gn1NRkgxFFxmC+2OWyz5esA7Y3X0MP3YL4EckJm6Co9X8miogr BS6svuVvclRLzDFp5aZA1POcqgJ2W0VCSZn+jiYpCvmtnD0i1/YY6az+PxtxWaf9m1a8 /QVsRuMp/cvacKBZ8FXluZFmowZVbCag4TR6TXnad8B2VwRDk6+r8WO5QvGWgbEkuJt3 lcGKy/m9kXLYleVkRp1xUW55wNBcI3RsJ7vYslUzcf5Fu7uRVlF6moEkEEPuEJxz7Ce9 L1fg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721250760; x=1721855560; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=hAakTSQquisNdf+ftCvXBOJ4qymHPREWzr12EPqTS48=; b=mFjCw8ECtLU5vu+cx+2YKt1DNx/4QzgobPwIaw9nqoiIBJxnkXgJybja9JWPy4nmJ3 M19fXBIghCmm15E21wIhOWvntUw3KfX+Qxvv+Lb1Ja39Jpsc0RVCX4daK5et5Nn+Puc6 GFCPV/dEWGtwwZC8Vk6sCeN+3LxgfFa4IaPTtqhT7suxBlOJUVCwfq6sHxUuOCFgSPYP I0Pdr3dd+/3H+Eaq8bvXJn0b+ENpNztKgEqvbyB+rxBZ9iU/GJLgAZw+8HR6TR18UTi1 aay3Gqqbaa12tA4kZzIUkR7wNejQ+3Fv6uUsL+MfcN+8xd/m8ZamSrdZCAEFHb31oOWt RWhQ== X-Gm-Message-State: AOJu0YxPuSNSiF+eEAO3ToxpfkZIzq+0Ea9UaAtK9thJy7iaXVQTPUb2 2ZLOHmC5DwqJSlInwkbAV5gcA1nqj+PLKhh3gvuYPY1CMEIwfFURg+IhPGRAfrCkVwgEvcCa0Cy v X-Google-Smtp-Source: AGHT+IHknfbJAOEqVCpPkr9lgoaRwVzJqGO2kwKeqVObvpqF7uo3GW9kjn7VaotnN04h2Hba9i4Wcg== X-Received: by 2002:a81:87c2:0:b0:664:5957:f7a with SMTP id 00721157ae682-664fe93537emr33740297b3.15.1721250759922; Wed, 17 Jul 2024 14:12:39 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 00721157ae682-6660249148fsm766727b3.61.2024.07.17.14.12.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Jul 2024 14:12:39 -0700 (PDT) Date: Wed, 17 Jul 2024 17:12:38 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v2 14/19] midx: teach `midx_fanout_add_midx_fanout()` about incremental MIDXs Message-ID: <594386da10bc3f3d6364916201438bf453b70098.1721250704.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The function `midx_fanout_add_midx_fanout()` is used to help construct the fanout table when generating a MIDX by reusing data from an existing MIDX. Prepare this function to work with incremental MIDXs by making a few changes: - The bounds checks need to be adjusted to start object lookups taking into account the number of objects in the previous MIDX layer (i.e., by starting the lookups at position `m->num_objects_in_base` instead of position 0). - Likewise, the bounds checks need to end at `m->num_objects_in_base` objects after `m->num_objects`. - Finally, `midx_fanout_add_midx_fanout()` needs to recur on earlier MIDX layers when dealing with an incremental MIDX chain by calling itself when given a MIDX with a non-NULL `base_midx`. Signed-off-by: Taylor Blau --- midx-write.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/midx-write.c b/midx-write.c index 478b42e720..d5275d719b 100644 --- a/midx-write.c +++ b/midx-write.c @@ -196,7 +196,7 @@ static int nth_midxed_pack_midx_entry(struct multi_pack_index *m, struct pack_midx_entry *e, uint32_t pos) { - if (pos >= m->num_objects) + if (pos >= m->num_objects + m->num_objects_in_base) return 1; nth_midxed_object_oid(&e->oid, m, pos); @@ -247,12 +247,16 @@ static void midx_fanout_add_midx_fanout(struct midx_fanout *fanout, uint32_t cur_fanout, int preferred_pack) { - uint32_t start = 0, end; + uint32_t start = m->num_objects_in_base, end; uint32_t cur_object; + if (m->base_midx) + midx_fanout_add_midx_fanout(fanout, m->base_midx, cur_fanout, + preferred_pack); + if (cur_fanout) - start = ntohl(m->chunk_oid_fanout[cur_fanout - 1]); - end = ntohl(m->chunk_oid_fanout[cur_fanout]); + start += ntohl(m->chunk_oid_fanout[cur_fanout - 1]); + end = m->num_objects_in_base + ntohl(m->chunk_oid_fanout[cur_fanout]); for (cur_object = start; cur_object < end; cur_object++) { if ((preferred_pack > -1) && From patchwork Wed Jul 17 21:12:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13735795 Received: from mail-yb1-f180.google.com (mail-yb1-f180.google.com [209.85.219.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 77ED71850B5 for ; Wed, 17 Jul 2024 21:12:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250766; cv=none; b=ksYM5jS2619eyMuXbW+5vu1lbwqjmqQFLK7BIVQwWxVzvf46ykwKQH2qTVV0Kznas50v7/LROh49sOFAeDl0YwtnAN1AjEWJR7dINIKANklgvPlpP/mUM3FPYKDSMd2uT+YaAiq2CNjh/nj/nlGTu0Q3IImR8f3hk/YmAzLpxMU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250766; c=relaxed/simple; bh=yLr7yEfdRPTAOP9xWK++5MH+lz2qZsK4ZUbbbBmDfrc=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=KfizPPdVMnlM+GKxA09l+Y0dclOvJp6aTeLxAs4MBDyIa0cs4XhDCJzfBWNaAtmENxhP6nC3xtGnMxXpNpG4HWfUdAuXz2hbtPzkBtIfEkDNWDoie/5DgGEgxBKPLc3Pt43g6DXZ3K53iauKxoL+voqozm5C+cisybIwPH2qfdo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=htJ/j9jv; arc=none smtp.client-ip=209.85.219.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="htJ/j9jv" Received: by mail-yb1-f180.google.com with SMTP id 3f1490d57ef6-dfdb6122992so129287276.3 for ; Wed, 17 Jul 2024 14:12:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1721250763; x=1721855563; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=LvY5XJ++9riibVqOfO5u/b4URyhIQQ/xzVLmu7ZlHLo=; b=htJ/j9jvMv4drNsY050Aqjgkimf+Py9BnUAcGH56pgxcmxISkJai+duHTNvy7niZjg Vs73jQh75zbE8ycQHpc+zH/HXpC3TAAgMaosJGsn8uJu4zpzt8AKUXRgWk0PqZ5xJXVU ARAV+FnUdbGXbQyoS5ERmbWCfSAbT6F3WjNdhvdWHPFBdHkWhjNeuaPMOwc+nPrLGY/X OQczovy8+xnibGYKIT+m9XQ6SesZ0nAGGjY1MEC3U3VFS+C26zgmNyRwf1wY2+fzp+qp eq8JsNSBXLPagsRLuewvGHOCacmUwpc57tftsza5RdkN7RxviH5KQupxYW7B8oS/QUOC SivQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721250763; x=1721855563; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=LvY5XJ++9riibVqOfO5u/b4URyhIQQ/xzVLmu7ZlHLo=; b=iw0bFG0L8/yhUxXySSZZujk2ihSzvuDAp9OFs45yCwhNsyimLtA0lYy+roc0Wi/3UT fghIG5ju96ORKuskxrEIO7mnuTkgfEAvZg/65h7GLG3xgo6ia38skTtzH9hzYV7ecwiA 9UN/GfaJuDS6yROWKLuBfZVF5BHrUWCWjiBVtdTcmV7uoOyvwZNdLN8zyEuCePLCzpfR UhghlhTDukT/onc8ktVI9KXbkKKwdgMz63qoJdp9agf0ne8tv81P6t6E/eLKp7W0kIFD /+QWAmtTFjOeFAU214kWaHivKQCgudIUe6/sWQmrcUTdy80DBh4pAO3qcEWMQObYsjav tpWg== X-Gm-Message-State: AOJu0Yw4i1u5yrNFxOOF9LDzU4Jk3fYauYA16vSaFO696UEg7HgXYjgi XZjMSdODYFLNVBH0MTmWCqFm36+hTbPuxoqXptzaAc9uGXcT+Zg+dghVJw/nafpbmsWFuWdqSNM V X-Google-Smtp-Source: AGHT+IGth6bVgr6KImxzwXUta9qw8NBYO2lndJeTNBAz5OM289u62Dh1ud03aYEQgCKUJLGnrLuHJw== X-Received: by 2002:a05:6902:1148:b0:e05:78eb:9a87 with SMTP id 3f1490d57ef6-e05ed722b3amr3086626276.36.1721250763081; Wed, 17 Jul 2024 14:12:43 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 3f1490d57ef6-e05fe9f457dsm81940276.10.2024.07.17.14.12.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Jul 2024 14:12:42 -0700 (PDT) Date: Wed, 17 Jul 2024 17:12:41 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v2 15/19] midx: support reading incremental MIDX chains Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Now that the MIDX machinery's internals have been taught to understand incremental MIDXs over the previous handful of commits, the MIDX machinery itself can begin reading incremental MIDXs. (Note that while the on-disk format for incremental MIDXs has been defined, the writing end has not been implemented. This will take place in the commit after next.) The core of this change involves following the order specified in the MIDX chain and opening up MIDXs in the chain one-by-one, adding them to the previous layer's `->base_midx` pointer at each step. In order to implement this, the `load_multi_pack_index()` function is taught to call a new `load_multi_pack_index_chain()` function if loading a non-incremental MIDX failed via `load_multi_pack_index_one()`. When loading a MIDX chain, `load_midx_chain_fd_st()` reads each line in the file one-by-one and dispatches calls to `load_multi_pack_index_one()` to read each layer of the MIDX chain. When a layer was successfully read, it is added to the MIDX chain by calling `add_midx_to_chain()` which validates the contents of the `BASE` chunk, performs some bounds checks on the number of combined packs and objects, and attaches the new MIDX by assigning its `base_midx` pointer to the existing part of the chain. As a supplement to this, introduce a new mode in the test-read-midx test-tool which allows us to read the information for a specific MIDX in the chain by specifying its trailing checksum via the command-line arguments like so: $ test-tool read-midx .git/objects [checksum] Signed-off-by: Taylor Blau --- midx.c | 184 +++++++++++++++++++++++++++++++++++--- midx.h | 7 ++ packfile.c | 5 +- t/helper/test-read-midx.c | 24 +++-- 4 files changed, 201 insertions(+), 19 deletions(-) diff --git a/midx.c b/midx.c index d2dbea41e4..0bfd17c021 100644 --- a/midx.c +++ b/midx.c @@ -91,7 +91,9 @@ static int midx_read_object_offsets(const unsigned char *chunk_start, #define MIDX_MIN_SIZE (MIDX_HEADER_SIZE + the_hash_algo->rawsz) -struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local) +static struct multi_pack_index *load_multi_pack_index_one(const char *object_dir, + const char *midx_name, + int local) { struct multi_pack_index *m = NULL; int fd; @@ -99,31 +101,26 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local size_t midx_size; void *midx_map = NULL; uint32_t hash_version; - struct strbuf midx_name = STRBUF_INIT; uint32_t i; const char *cur_pack_name; struct chunkfile *cf = NULL; - get_midx_filename(&midx_name, object_dir); - - fd = git_open(midx_name.buf); + fd = git_open(midx_name); if (fd < 0) goto cleanup_fail; if (fstat(fd, &st)) { - error_errno(_("failed to read %s"), midx_name.buf); + error_errno(_("failed to read %s"), midx_name); goto cleanup_fail; } midx_size = xsize_t(st.st_size); if (midx_size < MIDX_MIN_SIZE) { - error(_("multi-pack-index file %s is too small"), midx_name.buf); + error(_("multi-pack-index file %s is too small"), midx_name); goto cleanup_fail; } - strbuf_release(&midx_name); - midx_map = xmmap(NULL, midx_size, PROT_READ, MAP_PRIVATE, fd, 0); close(fd); @@ -213,7 +210,6 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local cleanup_fail: free(m); - strbuf_release(&midx_name); free_chunkfile(cf); if (midx_map) munmap(midx_map, midx_size); @@ -222,6 +218,173 @@ struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local return NULL; } +void get_midx_chain_dirname(struct strbuf *buf, const char *object_dir) +{ + strbuf_addf(buf, "%s/pack/multi-pack-index.d", object_dir); +} + +void get_midx_chain_filename(struct strbuf *buf, const char *object_dir) +{ + get_midx_chain_dirname(buf, object_dir); + strbuf_addstr(buf, "/multi-pack-index-chain"); +} + +void get_split_midx_filename_ext(struct strbuf *buf, const char *object_dir, + const unsigned char *hash, const char *ext) +{ + get_midx_chain_dirname(buf, object_dir); + strbuf_addf(buf, "/multi-pack-index-%s.%s", hash_to_hex(hash), ext); +} + +static int open_multi_pack_index_chain(const char *chain_file, + int *fd, struct stat *st) +{ + *fd = git_open(chain_file); + if (*fd < 0) + return 0; + if (fstat(*fd, st)) { + close(*fd); + return 0; + } + if (st->st_size < the_hash_algo->hexsz) { + close(*fd); + if (!st->st_size) { + /* treat empty files the same as missing */ + errno = ENOENT; + } else { + warning(_("multi-pack-index chain file too small")); + errno = EINVAL; + } + return 0; + } + return 1; +} + +static int add_midx_to_chain(struct multi_pack_index *midx, + struct multi_pack_index *midx_chain, + struct object_id *oids, + int n) +{ + if (midx_chain) { + if (unsigned_add_overflows(midx_chain->num_packs, + midx_chain->num_packs_in_base)) { + warning(_("pack count in base MIDX too high: %"PRIuMAX), + (uintmax_t)midx_chain->num_packs_in_base); + return 0; + } + if (unsigned_add_overflows(midx_chain->num_objects, + midx_chain->num_objects_in_base)) { + warning(_("object count in base MIDX too high: %"PRIuMAX), + (uintmax_t)midx_chain->num_objects_in_base); + return 0; + } + midx->num_packs_in_base = midx_chain->num_packs + + midx_chain->num_packs_in_base; + midx->num_objects_in_base = midx_chain->num_objects + + midx_chain->num_objects_in_base; + } + + midx->base_midx = midx_chain; + midx->has_chain = 1; + + return 1; +} + +static struct multi_pack_index *load_midx_chain_fd_st(const char *object_dir, + int local, + int fd, struct stat *st, + int *incomplete_chain) +{ + struct multi_pack_index *midx_chain = NULL; + struct strbuf buf = STRBUF_INIT; + struct object_id *layers = NULL; + int valid = 1; + uint32_t i, count; + FILE *fp = xfdopen(fd, "r"); + + count = st->st_size / (the_hash_algo->hexsz + 1); + CALLOC_ARRAY(layers, count); + + for (i = 0; i < count; i++) { + struct multi_pack_index *m; + + if (strbuf_getline_lf(&buf, fp) == EOF) + break; + + if (get_oid_hex(buf.buf, &layers[i])) { + warning(_("invalid multi-pack-index chain: line '%s' " + "not a hash"), + buf.buf); + valid = 0; + break; + } + + valid = 0; + + strbuf_reset(&buf); + get_split_midx_filename_ext(&buf, object_dir, layers[i].hash, + MIDX_EXT_MIDX); + m = load_multi_pack_index_one(object_dir, buf.buf, local); + + if (m) { + if (add_midx_to_chain(m, midx_chain, layers, i)) { + midx_chain = m; + valid = 1; + } else { + close_midx(m); + } + } + if (!valid) { + warning(_("unable to find all multi-pack index files")); + break; + } + } + + free(layers); + fclose(fp); + strbuf_release(&buf); + + *incomplete_chain = !valid; + return midx_chain; +} + +static struct multi_pack_index *load_multi_pack_index_chain(const char *object_dir, + int local) +{ + struct strbuf chain_file = STRBUF_INIT; + struct stat st; + int fd; + struct multi_pack_index *m = NULL; + + get_midx_chain_filename(&chain_file, object_dir); + if (open_multi_pack_index_chain(chain_file.buf, &fd, &st)) { + int incomplete; + /* ownership of fd is taken over by load function */ + m = load_midx_chain_fd_st(object_dir, local, fd, &st, + &incomplete); + } + + strbuf_release(&chain_file); + return m; +} + +struct multi_pack_index *load_multi_pack_index(const char *object_dir, + int local) +{ + struct strbuf midx_name = STRBUF_INIT; + struct multi_pack_index *m; + + get_midx_filename(&midx_name, object_dir); + + m = load_multi_pack_index_one(object_dir, midx_name.buf, local); + if (!m) + m = load_multi_pack_index_chain(object_dir, local); + + strbuf_release(&midx_name); + + return m; +} + void close_midx(struct multi_pack_index *m) { uint32_t i; @@ -230,6 +393,7 @@ void close_midx(struct multi_pack_index *m) return; close_midx(m->next); + close_midx(m->base_midx); munmap((unsigned char *)m->data, m->data_len); diff --git a/midx.h b/midx.h index 86af7dfc5e..94de16a8c4 100644 --- a/midx.h +++ b/midx.h @@ -24,6 +24,7 @@ struct bitmapped_pack; #define MIDX_CHUNKID_OBJECTOFFSETS 0x4f4f4646 /* "OOFF" */ #define MIDX_CHUNKID_LARGEOFFSETS 0x4c4f4646 /* "LOFF" */ #define MIDX_CHUNKID_REVINDEX 0x52494458 /* "RIDX" */ +#define MIDX_CHUNKID_BASE 0x42415345 /* "BASE" */ #define MIDX_CHUNK_OFFSET_WIDTH (2 * sizeof(uint32_t)) #define MIDX_LARGE_OFFSET_NEEDED 0x80000000 @@ -50,6 +51,7 @@ struct multi_pack_index { int preferred_pack_idx; int local; + int has_chain; const unsigned char *chunk_pack_names; size_t chunk_pack_names_len; @@ -80,11 +82,16 @@ struct multi_pack_index { #define MIDX_EXT_REV "rev" #define MIDX_EXT_BITMAP "bitmap" +#define MIDX_EXT_MIDX "midx" const unsigned char *get_midx_checksum(struct multi_pack_index *m); void get_midx_filename(struct strbuf *out, const char *object_dir); void get_midx_filename_ext(struct strbuf *out, const char *object_dir, const unsigned char *hash, const char *ext); +void get_midx_chain_dirname(struct strbuf *buf, const char *object_dir); +void get_midx_chain_filename(struct strbuf *buf, const char *object_dir); +void get_split_midx_filename_ext(struct strbuf *buf, const char *object_dir, + const unsigned char *hash, const char *ext); struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local); int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id); diff --git a/packfile.c b/packfile.c index 813584646f..1eb18e3041 100644 --- a/packfile.c +++ b/packfile.c @@ -880,7 +880,8 @@ static void prepare_pack(const char *full_name, size_t full_name_len, if (!report_garbage) return; - if (!strcmp(file_name, "multi-pack-index")) + if (!strcmp(file_name, "multi-pack-index") || + !strcmp(file_name, "multi-pack-index.d")) return; if (starts_with(file_name, "multi-pack-index") && (ends_with(file_name, ".bitmap") || ends_with(file_name, ".rev"))) @@ -1064,7 +1065,7 @@ struct packed_git *get_all_packs(struct repository *r) prepare_packed_git(r); for (m = r->objects->multi_pack_index; m; m = m->next) { uint32_t i; - for (i = 0; i < m->num_packs; i++) + for (i = 0; i < m->num_packs + m->num_packs_in_base; i++) prepare_midx_pack(r, m, i); } diff --git a/t/helper/test-read-midx.c b/t/helper/test-read-midx.c index 83effc2b5f..69757e94fc 100644 --- a/t/helper/test-read-midx.c +++ b/t/helper/test-read-midx.c @@ -9,8 +9,10 @@ #include "packfile.h" #include "setup.h" #include "gettext.h" +#include "pack-revindex.h" -static int read_midx_file(const char *object_dir, int show_objects) +static int read_midx_file(const char *object_dir, const char *checksum, + int show_objects) { uint32_t i; struct multi_pack_index *m; @@ -21,6 +23,13 @@ static int read_midx_file(const char *object_dir, int show_objects) if (!m) return 1; + if (checksum) { + while (m && strcmp(hash_to_hex(get_midx_checksum(m)), checksum)) + m = m->base_midx; + if (!m) + return 1; + } + printf("header: %08x %d %d %d %d\n", m->signature, m->version, @@ -54,7 +63,8 @@ static int read_midx_file(const char *object_dir, int show_objects) struct pack_entry e; for (i = 0; i < m->num_objects; i++) { - nth_midxed_object_oid(&oid, m, i); + nth_midxed_object_oid(&oid, m, + i + m->num_objects_in_base); fill_midx_entry(the_repository, &oid, &e, m); printf("%s %"PRIu64"\t%s\n", @@ -111,7 +121,7 @@ static int read_midx_bitmapped_packs(const char *object_dir) if (!midx) return 1; - for (i = 0; i < midx->num_packs; i++) { + for (i = 0; i < midx->num_packs + midx->num_packs_in_base; i++) { if (nth_bitmapped_pack(the_repository, midx, &pack, i) < 0) return 1; @@ -127,16 +137,16 @@ static int read_midx_bitmapped_packs(const char *object_dir) int cmd__read_midx(int argc, const char **argv) { - if (!(argc == 2 || argc == 3)) - usage("read-midx [--show-objects|--checksum|--preferred-pack|--bitmap] "); + if (!(argc == 2 || argc == 3 || argc == 4)) + usage("read-midx [--show-objects|--checksum|--preferred-pack|--bitmap] "); if (!strcmp(argv[1], "--show-objects")) - return read_midx_file(argv[2], 1); + return read_midx_file(argv[2], argv[3], 1); else if (!strcmp(argv[1], "--checksum")) return read_midx_checksum(argv[2]); else if (!strcmp(argv[1], "--preferred-pack")) return read_midx_preferred_pack(argv[2]); else if (!strcmp(argv[1], "--bitmap")) return read_midx_bitmapped_packs(argv[2]); - return read_midx_file(argv[1], 0); + return read_midx_file(argv[1], argv[2], 0); } From patchwork Wed Jul 17 21:12:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13735796 Received: from mail-yw1-f177.google.com (mail-yw1-f177.google.com [209.85.128.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 71CBC18E778 for ; Wed, 17 Jul 2024 21:12:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250768; cv=none; b=OI0KmQ1Bb1rryOzN0MvnEbLW5/582NsR10AZO07nbAycWjMcnF8QhR2qYiVJOJ1GBsqQakmztiSPLPBN40PF8qpLsP97uWbwH7hFE3C3a5htrq84PSYXUsFmJ+FnOLpHYSlcSd7LxX33KVw7eNkuF0zlowZS3b+3jIKkIM9kPlo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250768; c=relaxed/simple; bh=r3UNEMkSy+2vPsAvqc9M6eUeSUSNfSdumOZ80smL+us=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=njZ0J3bB/J1rp927T1hJHRiHt2lKIk7G6AqXLspgMLkAzYxeN+12Ynq/bLU9gcOen/eCGUzOutO0BSfJdp0YvHl33S7PsBputUJm6t67EG2LbpMbeMoovrXUEvPy7Z0sdAcE1dPZIvj/I+GU3FpN/OKhgi6AA+2/mLcikgs1IR8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=UOKQavui; arc=none smtp.client-ip=209.85.128.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="UOKQavui" Received: by mail-yw1-f177.google.com with SMTP id 00721157ae682-64b0d45a7aaso863927b3.0 for ; Wed, 17 Jul 2024 14:12:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1721250766; x=1721855566; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=3/1mIPzJkYTGJfWMxbTvkTLHd6GV5kmJzXmnug6TJBY=; b=UOKQavuiZ+zYZySGTZT4s0C9UAo0q+CWMq7qopO7EneqTLUpj5M1/ieqgtZG2Omn/b SRJORvA0aZr8IFAgFgA3jwNUkteozJb7DwpCnPCNTry5F4rHvkzPSSq5FONN2riZAuzZ eJyXQA5kRbg5k7+4+jm2Ztfh4rcbt4FOOCBWqOu+W9xyFPiYaeRcoCZYLGJjPkXDBb+V CjX43u9J7eCu7TvB+JsLSIUfQz3KfU/HcVaRwN+J2GncmizPTAoNUd8HcgN7zrclN/qf 3wYv1wu8nWlmbkHrZLmRkIrHRrwpHn1VbjtIk/7TRIvIda/BzHEiIJvrHK/ptrbpJ7Jk kQGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721250766; x=1721855566; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=3/1mIPzJkYTGJfWMxbTvkTLHd6GV5kmJzXmnug6TJBY=; b=FyX9g+nS22Pe7aXpIcS8pWUsnCpHOU2YdAGZhQzovholQsIQ4WmW9RnD75uIBQtLJI UA9rLJainpqoc+tEVsiPm0cx0YgMNuOwC4cE9T4+cozM6dQRXAYFePfjkH0Y2XoxzF+y fWo9PzIOGU4g4ANzKZ9EKRrAa5/fxiHg0TMmoqA/bEKN8pZCvW+m3I9GJV5gJ0ERq4FN cMNbjfaplYdzZKIFRrZ4jiDXcbt/OD9Yw5k0ZMXJPEIVyqgEG7GCqW8447IDZVCOwcWR VgJA1CeUQb7OEtj18BfhW7gAYeJYPKm472yYhFGwnKrsCIlKoXKUvv9FyG9OL830JF+q 0nEw== X-Gm-Message-State: AOJu0YzG4hrZAyz6j/y2CU5UK27gZvhJxZuy1x+/dDIOy+SrgRqRqJfj 3HqbicIgXVmbXo57hQJkzLsP+XR0TQr8TlPegk9lxoJBa24t60v9UdlGpa2bxwq5K26tpXSTUiQ B X-Google-Smtp-Source: AGHT+IEfq0C3BTRCzRhjyvWbnsYyyC9A5k8AhXb8yDQ8Gd94ZrzdIojnjbe497JMzAjPqFqO3hpFGw== X-Received: by 2002:a05:690c:2e0a:b0:61e:a26:c1b2 with SMTP id 00721157ae682-66601bd41d2mr7510337b3.17.1721250766157; Wed, 17 Jul 2024 14:12:46 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 00721157ae682-66601fb56c5sm787997b3.48.2024.07.17.14.12.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Jul 2024 14:12:45 -0700 (PDT) Date: Wed, 17 Jul 2024 17:12:44 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v2 16/19] midx: implement verification support for incremental MIDXs Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Teach the verification implementation used by `git multi-pack-index verify` to perform verification for incremental MIDX chains by independently validating each layer within the chain. Signed-off-by: Taylor Blau --- midx.c | 47 ++++++++++++++++++++++++++++++----------------- midx.h | 2 ++ 2 files changed, 32 insertions(+), 17 deletions(-) diff --git a/midx.c b/midx.c index 0bfd17c021..21a9dbe23a 100644 --- a/midx.c +++ b/midx.c @@ -469,6 +469,13 @@ int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, return 0; } +struct packed_git *nth_midxed_pack(struct multi_pack_index *m, + uint32_t pack_int_id) +{ + uint32_t local_pack_int_id = midx_for_pack(&m, pack_int_id); + return m->packs[local_pack_int_id]; +} + #define MIDX_CHUNK_BITMAPPED_PACKS_WIDTH (2 * sizeof(uint32_t)) int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m, @@ -817,6 +824,7 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag uint32_t i; struct progress *progress = NULL; struct multi_pack_index *m = load_multi_pack_index(object_dir, 1); + struct multi_pack_index *curr; verify_midx_error = 0; if (!m) { @@ -839,8 +847,8 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag if (flags & MIDX_PROGRESS) progress = start_delayed_progress(_("Looking for referenced packfiles"), - m->num_packs); - for (i = 0; i < m->num_packs; i++) { + m->num_packs + m->num_packs_in_base); + for (i = 0; i < m->num_packs + m->num_packs_in_base; i++) { if (prepare_midx_pack(r, m, i)) midx_report("failed to load pack in position %d", i); @@ -860,17 +868,20 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag if (flags & MIDX_PROGRESS) progress = start_sparse_progress(_("Verifying OID order in multi-pack-index"), m->num_objects - 1); - for (i = 0; i < m->num_objects - 1; i++) { - struct object_id oid1, oid2; - nth_midxed_object_oid(&oid1, m, i); - nth_midxed_object_oid(&oid2, m, i + 1); + for (curr = m; curr; curr = curr->base_midx) { + for (i = 0; i < m->num_objects - 1; i++) { + struct object_id oid1, oid2; - if (oidcmp(&oid1, &oid2) >= 0) - midx_report(_("oid lookup out of order: oid[%d] = %s >= %s = oid[%d]"), - i, oid_to_hex(&oid1), oid_to_hex(&oid2), i + 1); + nth_midxed_object_oid(&oid1, m, m->num_objects_in_base + i); + nth_midxed_object_oid(&oid2, m, m->num_objects_in_base + i + 1); - midx_display_sparse_progress(progress, i + 1); + if (oidcmp(&oid1, &oid2) >= 0) + midx_report(_("oid lookup out of order: oid[%d] = %s >= %s = oid[%d]"), + i, oid_to_hex(&oid1), oid_to_hex(&oid2), i + 1); + + midx_display_sparse_progress(progress, i + 1); + } } stop_progress(&progress); @@ -880,8 +891,8 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag * each of the objects and only require 1 packfile to be open at a * time. */ - ALLOC_ARRAY(pairs, m->num_objects); - for (i = 0; i < m->num_objects; i++) { + ALLOC_ARRAY(pairs, m->num_objects + m->num_objects_in_base); + for (i = 0; i < m->num_objects + m->num_objects_in_base; i++) { pairs[i].pos = i; pairs[i].pack_int_id = nth_midxed_pack_int_id(m, i); } @@ -895,16 +906,18 @@ int verify_midx_file(struct repository *r, const char *object_dir, unsigned flag if (flags & MIDX_PROGRESS) progress = start_sparse_progress(_("Verifying object offsets"), m->num_objects); - for (i = 0; i < m->num_objects; i++) { + for (i = 0; i < m->num_objects + m->num_objects_in_base; i++) { struct object_id oid; struct pack_entry e; off_t m_offset, p_offset; if (i > 0 && pairs[i-1].pack_int_id != pairs[i].pack_int_id && - m->packs[pairs[i-1].pack_int_id]) - { - close_pack_fd(m->packs[pairs[i-1].pack_int_id]); - close_pack_index(m->packs[pairs[i-1].pack_int_id]); + nth_midxed_pack(m, pairs[i-1].pack_int_id)) { + uint32_t pack_int_id = pairs[i-1].pack_int_id; + struct packed_git *p = nth_midxed_pack(m, pack_int_id); + + close_pack_fd(p); + close_pack_index(p); } nth_midxed_object_oid(&oid, m, pairs[i].pos); diff --git a/midx.h b/midx.h index 94de16a8c4..9d30935589 100644 --- a/midx.h +++ b/midx.h @@ -95,6 +95,8 @@ void get_split_midx_filename_ext(struct strbuf *buf, const char *object_dir, struct multi_pack_index *load_multi_pack_index(const char *object_dir, int local); int prepare_midx_pack(struct repository *r, struct multi_pack_index *m, uint32_t pack_int_id); +struct packed_git *nth_midxed_pack(struct multi_pack_index *m, + uint32_t pack_int_id); int nth_bitmapped_pack(struct repository *r, struct multi_pack_index *m, struct bitmapped_pack *bp, uint32_t pack_int_id); int bsearch_one_midx(const struct object_id *oid, struct multi_pack_index *m, From patchwork Wed Jul 17 21:12:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13735797 Received: from mail-yw1-f181.google.com (mail-yw1-f181.google.com [209.85.128.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 88AD118C178 for ; Wed, 17 Jul 2024 21:12:50 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250772; cv=none; b=eCa62DC/SiGNPx/D7NN4uQHS+lNL4ldBllvB//ZJ2bjnAngFURA9GJ1tfYmylBGoCbVDcCyFCrQQlRPOwonwZkzpav8d6zab/sPB40kXONhZPHMLftLwYnbUy6fITA5Snxxn7yQYH2K3C4Z7m0OSh48HnGifaCyRdTiPR0PgMyU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250772; c=relaxed/simple; bh=PyJMXNw+HWvHfMQ8GqPb1y4cQH+UCtGp1h5WxqX8reA=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=Mg70Y29xbGKNFYWUcT0+Id6uyj7hKQztZ7HZ/O21djYhXVyLWS3St2U/OBl4yoQ6hqwbbuktGfn7cxFSoGzCAgajDo9ohbZPis75DifZB/TT/K5IkUd25AT3rnxruNKQxdUGS5Y11GreY1FDbUTUacg0f55UdHlEpAe3bCeUQ6Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=EqActHdq; arc=none smtp.client-ip=209.85.128.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="EqActHdq" Received: by mail-yw1-f181.google.com with SMTP id 00721157ae682-663dd13c0bbso704587b3.1 for ; Wed, 17 Jul 2024 14:12:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1721250769; x=1721855569; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=3ief6IM70Lb0TN4FVSp89G1ouPFUedSKUZwo7zsBmG4=; b=EqActHdqSQCHd0E28WurEJmk1mwC0+uEcv0pabt9ysI2q/+JbjldiIkXMP/fhPAwFf Kp9Mpg/zChjvGV1PqI4LPKnzUOx8li4p0pNRx4lzwxe525uhns1/sBYfgXr1+XeyEkaG s6v3ii2HPnCHdhUP6CnTFtRDDL8dAps2vyQKgu6MP1b9t+XV2rFSQqlZPQfZHtP609a0 zb2Q53k3cZcwrVd8D52pltNFjWYa2D5ELb/o7u8egmOQiX4wKDEsNQnqjny2WBJGo9ou Kb8Sx0PfbvonQWDnV0ONGuhQSf2Wxc6K17euQKXO/LTBFyzljNqNRyEWvuJbqQTOc3fS 71Wg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721250769; x=1721855569; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=3ief6IM70Lb0TN4FVSp89G1ouPFUedSKUZwo7zsBmG4=; b=OjkgOO1WVII+xnBKBOQ+vsF7SDI0b9f4TDECLQF2JMhjXSo+bTl/dUk/+IP/1V1s+9 uppaUe04JXpzHc1UfN99ykL8Xa0Yd7/wZBhC4ISFxZ7uiUJlr19/YgUvhqr8N5mHmOyZ qmWIlGHopS3mYsJvKroelJaENhCZswewLuSPHvNMwgJWU6QkQa02K+DI9Ol/OS58WJy3 ZmSa5/eYfb1UGTrYpFB/WXbS2OKNgY4bO91CcOnF0kEiEf3flQeUvcHQFzPdUSbGcpWJ NbWGPlQCrYkBZ2H8sPqFtrRKX3Zc2/vs5d3x6LAyeto95Se4Fcgt1lhveHAcQn1pRt+1 B16w== X-Gm-Message-State: AOJu0YzOLaO7elyzIy5m0BGX54PiDSLdwcJwlc+oC/BaH/kwFmOtAoPa lZFoZnsoci3P0rxAFvPhn9jr0daHxdKRPPYoy0I9leHYQwrHpmSJwciS5ARU0Wn8/47ItC3hoCa t X-Google-Smtp-Source: AGHT+IEA5+5QIre+DiyiCT9GKQ/17DE2yyoYlaQ750LKiPK1l460qqRSFh8U9XwNKj/WsoPhz6aLZA== X-Received: by 2002:a05:6902:2081:b0:e02:6e1e:209b with SMTP id 3f1490d57ef6-e05ed797701mr3828756276.38.1721250769230; Wed, 17 Jul 2024 14:12:49 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 3f1490d57ef6-e05feb6554asm82293276.54.2024.07.17.14.12.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Jul 2024 14:12:48 -0700 (PDT) Date: Wed, 17 Jul 2024 17:12:47 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v2 17/19] t: retire 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' Message-ID: <23912425bf7c0106ed388f7712e7bb822572fe5d.1721250704.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Two years ago, commit ff1e653c8e2 (midx: respect 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP', 2021-08-31) introduced a new environment variable which caused the test suite to write MIDX bitmaps after any 'git repack' invocation. At the time, this was done to help flush out any bugs with MIDX bitmaps that weren't explicitly covered in the t5326-multi-pack-bitmap.sh script. Two years later, that flag has served us well and is no longer providing meaningful coverage, as the script in t5326 has matured substantially and covers many more interesting cases than it did back when ff1e653c8e2 was originally written. Remove the 'GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP' environment variable as it is no longer serving a useful purpose. More importantly, removing this variable clears the way for us to introduce a new one to help similarly flush out bugs related to incremental MIDX chains. Because these incremental MIDX chains are (for now) incompatible with MIDX bitmaps, we cannot have both. Signed-off-by: Taylor Blau --- builtin/repack.c | 12 ++---------- ci/run-build-and-tests.sh | 1 - midx.h | 2 -- t/README | 4 ---- t/t0410-partial-clone.sh | 2 -- t/t5310-pack-bitmaps.sh | 4 ---- t/t5319-multi-pack-index.sh | 3 +-- t/t5326-multi-pack-bitmaps.sh | 3 +-- t/t5327-multi-pack-bitmaps-rev.sh | 5 ++--- t/t7700-repack.sh | 21 +++++++-------------- 10 files changed, 13 insertions(+), 44 deletions(-) diff --git a/builtin/repack.c b/builtin/repack.c index f0317fa94a..8499bf0e12 100644 --- a/builtin/repack.c +++ b/builtin/repack.c @@ -1217,10 +1217,6 @@ int cmd_repack(int argc, const char **argv, const char *prefix) if (!write_midx && (!(pack_everything & ALL_INTO_ONE) || !is_bare_repository())) write_bitmaps = 0; - } else if (write_bitmaps && - git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0) && - git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0)) { - write_bitmaps = 0; } if (pack_kept_objects < 0) pack_kept_objects = write_bitmaps > 0 && !write_midx; @@ -1518,12 +1514,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix) if (run_update_server_info) update_server_info(0); - if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0)) { - unsigned flags = 0; - if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP, 0)) - flags |= MIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX; - write_midx_file(get_object_directory(), NULL, NULL, flags); - } + if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0)) + write_midx_file(get_object_directory(), NULL, NULL, 0); cleanup: string_list_clear(&names, 1); diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh index 98dda42045..e6fd68630c 100755 --- a/ci/run-build-and-tests.sh +++ b/ci/run-build-and-tests.sh @@ -25,7 +25,6 @@ linux-TEST-vars) export GIT_TEST_COMMIT_GRAPH=1 export GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1 export GIT_TEST_MULTI_PACK_INDEX=1 - export GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=1 export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master export GIT_TEST_NO_WRITE_REV_INDEX=1 export GIT_TEST_CHECKOUT_WORKERS=2 diff --git a/midx.h b/midx.h index 9d30935589..3714cad2cc 100644 --- a/midx.h +++ b/midx.h @@ -29,8 +29,6 @@ struct bitmapped_pack; #define MIDX_LARGE_OFFSET_NEEDED 0x80000000 #define GIT_TEST_MULTI_PACK_INDEX "GIT_TEST_MULTI_PACK_INDEX" -#define GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP \ - "GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP" struct multi_pack_index { struct multi_pack_index *next; diff --git a/t/README b/t/README index d9e0e07506..e8a11926e4 100644 --- a/t/README +++ b/t/README @@ -469,10 +469,6 @@ GIT_TEST_MULTI_PACK_INDEX=, when true, forces the multi-pack- index to be written after every 'git repack' command, and overrides the 'core.multiPackIndex' setting to true. -GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=, when true, sets the -'--bitmap' option on all invocations of 'git multi-pack-index write', -and ignores pack-objects' '--write-bitmap-index'. - GIT_TEST_SIDEBAND_ALL=, when true, overrides the 'uploadpack.allowSidebandAll' setting to true, and when false, forces fetch-pack to not request sideband-all (even if the server advertises diff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh index 2c30c86e7b..34bdb3ab1f 100755 --- a/t/t0410-partial-clone.sh +++ b/t/t0410-partial-clone.sh @@ -5,8 +5,6 @@ test_description='partial clone' . ./test-lib.sh . "$TEST_DIRECTORY"/lib-terminal.sh -# missing promisor objects cause repacks which write bitmaps to fail -GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 # When enabled, some commands will write commit-graphs. This causes fsck # to fail when delete_object() is called because fsck will attempt to # verify the out-of-sync commit graph. diff --git a/t/t5310-pack-bitmaps.sh b/t/t5310-pack-bitmaps.sh index d7fd71360e..a6de7c5764 100755 --- a/t/t5310-pack-bitmaps.sh +++ b/t/t5310-pack-bitmaps.sh @@ -5,10 +5,6 @@ test_description='exercise basic bitmap functionality' . ./test-lib.sh . "$TEST_DIRECTORY"/lib-bitmap.sh -# t5310 deals only with single-pack bitmaps, so don't write MIDX bitmaps in -# their place. -GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 - # Likewise, allow individual tests to control whether or not they use # the boundary-based traversal. sane_unset GIT_TEST_PACK_USE_BITMAP_BOUNDARY_TRAVERSAL diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index 10d2a6bf92..6e9ee23398 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -600,8 +600,7 @@ test_expect_success 'repack preserves multi-pack-index when creating packs' ' compare_results_with_midx "after repack" test_expect_success 'multi-pack-index and pack-bitmap' ' - GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \ - git -c repack.writeBitmaps=true repack -ad && + git -c repack.writeBitmaps=true repack -ad && git multi-pack-index write && git rev-list --test-bitmap HEAD ' diff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh index 916da389b6..1cb3e3ff08 100755 --- a/t/t5326-multi-pack-bitmaps.sh +++ b/t/t5326-multi-pack-bitmaps.sh @@ -4,10 +4,9 @@ test_description='exercise basic multi-pack bitmap functionality' . ./test-lib.sh . "${TEST_DIRECTORY}/lib-bitmap.sh" -# We'll be writing our own midx and bitmaps, so avoid getting confused by the +# We'll be writing our own MIDX, so avoid getting confused by the # automatic ones. GIT_TEST_MULTI_PACK_INDEX=0 -GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 # This test exercise multi-pack bitmap functionality where the object order is # stored and read from a special chunk within the MIDX, so use the default diff --git a/t/t5327-multi-pack-bitmaps-rev.sh b/t/t5327-multi-pack-bitmaps-rev.sh index e65e311cd7..23db949c20 100755 --- a/t/t5327-multi-pack-bitmaps-rev.sh +++ b/t/t5327-multi-pack-bitmaps-rev.sh @@ -5,10 +5,9 @@ test_description='exercise basic multi-pack bitmap functionality (.rev files)' . ./test-lib.sh . "${TEST_DIRECTORY}/lib-bitmap.sh" -# We'll be writing our own midx and bitmaps, so avoid getting confused by the -# automatic ones. +# We'll be writing our own MIDX, so avoid getting confused by the automatic +# ones. GIT_TEST_MULTI_PACK_INDEX=0 -GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 # Unlike t5326, this test exercise multi-pack bitmap functionality where the # object order is stored in a separate .rev file. diff --git a/t/t7700-repack.sh b/t/t7700-repack.sh index 127efe99f8..8f34f05087 100755 --- a/t/t7700-repack.sh +++ b/t/t7700-repack.sh @@ -70,14 +70,13 @@ test_expect_success 'objects in packs marked .keep are not repacked' ' test_expect_success 'writing bitmaps via command-line can duplicate .keep objects' ' # build on $oid, $packid, and .keep state from previous - GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 git repack -Adbl && + git repack -Adbl && test_has_duplicate_object true ' test_expect_success 'writing bitmaps via config can duplicate .keep objects' ' # build on $oid, $packid, and .keep state from previous - GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \ - git -c repack.writebitmaps=true repack -Adl && + git -c repack.writebitmaps=true repack -Adl && test_has_duplicate_object true ' @@ -284,8 +283,7 @@ test_expect_success 'repacking fails when missing .pack actually means missing o test_expect_success 'bitmaps are created by default in bare repos' ' git clone --bare .git bare.git && rm -f bare.git/objects/pack/*.bitmap && - GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \ - git -C bare.git repack -ad && + git -C bare.git repack -ad && bitmap=$(ls bare.git/objects/pack/*.bitmap) && test_path_is_file "$bitmap" ' @@ -296,8 +294,7 @@ test_expect_success 'incremental repack does not complain' ' ' test_expect_success 'bitmaps can be disabled on bare repos' ' - GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \ - git -c repack.writeBitmaps=false -C bare.git repack -ad && + git -c repack.writeBitmaps=false -C bare.git repack -ad && bitmap=$(ls bare.git/objects/pack/*.bitmap || :) && test -z "$bitmap" ' @@ -308,8 +305,7 @@ test_expect_success 'no bitmaps created if .keep files present' ' keep=${pack%.pack}.keep && test_when_finished "rm -f \"\$keep\"" && >"$keep" && - GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \ - git -C bare.git repack -ad 2>stderr && + git -C bare.git repack -ad 2>stderr && test_must_be_empty stderr && find bare.git/objects/pack/ -type f -name "*.bitmap" >actual && test_must_be_empty actual @@ -320,8 +316,7 @@ test_expect_success 'auto-bitmaps do not complain if unavailable' ' blob=$(test-tool genrandom big $((1024*1024)) | git -C bare.git hash-object -w --stdin) && git -C bare.git update-ref refs/tags/big $blob && - GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \ - git -C bare.git repack -ad 2>stderr && + git -C bare.git repack -ad 2>stderr && test_must_be_empty stderr && find bare.git/objects/pack -type f -name "*.bitmap" >actual && test_must_be_empty actual @@ -342,9 +337,7 @@ test_expect_success 'repacking with a filter works' ' ' test_expect_success '--filter fails with --write-bitmap-index' ' - test_must_fail \ - env GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 \ - git -C bare.git repack -a -d --write-bitmap-index --filter=blob:none + test_must_fail git -C bare.git repack -a -d --write-bitmap-index --filter=blob:none ' test_expect_success 'repacking with two filters works' ' From patchwork Wed Jul 17 21:12:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13735798 Received: from mail-yw1-f182.google.com (mail-yw1-f182.google.com [209.85.128.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6D080186E2C for ; Wed, 17 Jul 2024 21:12:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.182 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250774; cv=none; b=fVmZ0mx94sRPyKV47dt+wVb02gte0ffDrcDnSfRB9uHnD3IuV1gCK/J6Y9tdiT0zytwxPUa8JCkuzswSJN58hL7xyVtNUq+ZJTxTbAU3zpQAe7WtA70Ax9iI7/X1kuNLjlxdDrzFepGlEl4Tj9ydvDJ9hCJB1YGZ+ybprWNeMuo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250774; c=relaxed/simple; bh=GwcA0vjAVcI/IDF1+5n4yk03Ps6ak4milWhk8cNmXUs=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=PWzVIt2qwqBMLiQJnDS558vLZr35xwgkKHl2AMpFtQTSIJVqE+Q7OCosvoZod1Jmfjfc47uRtMsfgvqOPH2wHBVYE8TKJ3XW259ZwVa6stsHiufobb2y3bSd8fN5wy6kyd462MS+fsYsLs2TNeqerwBBdsy43moGRSNVoxsPQ9c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=lHYnEzHL; arc=none smtp.client-ip=209.85.128.182 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="lHYnEzHL" Received: by mail-yw1-f182.google.com with SMTP id 00721157ae682-654cf0a069eso882377b3.1 for ; Wed, 17 Jul 2024 14:12:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1721250772; x=1721855572; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=BLzkIZ0cCpY2SYOsUQYVtbx1+n9FCTW4/QCpVPJfOX4=; b=lHYnEzHLPkslMc2dsu6tyEVv4gvLUrNGu+PqHkPo9Pwiy/FizZZE2Mn0bPTLvFXnYT nKfJiRTzY8Pi5hKG295fEUvxf8eXkW7Q3RcugWPGLhvrlsfeMKbfFJYWPq+YJipo3gb/ ZozERF5HGQrhmJNpsAOFBACZ+ushpN85Cn6nRiKT4IdxZMQzGKNSSuyHve3HK+gcR3Ax HIHZB9310DP0VLswMIXYaqo1WvUec1pHQK8m2NO79un3VOhFDARmapITnZQ/53hiocb8 uF50HUJx5rR1eQL6qXdz9QdZoeQQAn9EN3hZMiwajZt/MaOmbqsnQrczPL6ykDoLniuY tf5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721250772; x=1721855572; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=BLzkIZ0cCpY2SYOsUQYVtbx1+n9FCTW4/QCpVPJfOX4=; b=PDWb/OogQQTHirRDi2Y4VkZTKqzisge/h2aebRZ3uczX6I5SzHYbE2REdze9hNYtJx W5kag00ZFTVqDUpRS8iyh4Gp6O0SoO8vPxUe8MJ+gYezZkaDw4r9KGvjuSDDRJWX9rA2 KEURV9dsNL8v8EMmZbKclqsSNmiBU0XGYVCD694Yogbrez4r92Iurv6Ihm9X0c8n4bcs SHn0ngkc32Teuuuwdj9kRdMMU2yhxSovkd1eN7+21KE+HmRdVkEu/IdHaYQr+WvYoLaf s/IX88S5ZAdWe8ZZzhtfNR0zrYtl+lb7Oubpy3P+Ag3r7csvZOlwd6pp657fv+CciuCR v9Tg== X-Gm-Message-State: AOJu0Yx1MwZzd77al8PJXzxHLGvrM5WKfhJHlh/z5MuB8oCGbviIzNPi EgotSF4LV7lABPC9xsIk82iDqPMFm0VbP+d+hEKSmLYxhvvNwu4FFcswC42oY7DVWeTObYrCqZO S X-Google-Smtp-Source: AGHT+IGU8U+ejzQ4LFiQiPUftvuRXe7UwYSEACMWi9x0Suz1lmuArkFm7SblLpPZr1id+FDakX1wbw== X-Received: by 2002:a0d:f606:0:b0:65f:80bb:e6b2 with SMTP id 00721157ae682-664fe4577aamr34802607b3.14.1721250772231; Wed, 17 Jul 2024 14:12:52 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 00721157ae682-666020a9acdsm780187b3.50.2024.07.17.14.12.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Jul 2024 14:12:51 -0700 (PDT) Date: Wed, 17 Jul 2024 17:12:50 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v2 18/19] t/t5313-pack-bounds-checks.sh: prepare for sub-directories Message-ID: <814da1916d36173ffb18db64bcce8ea489b7da10.1721250704.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Prepare for sub-directories to appear in $GIT_DIR/objects/pack by adjusting the copy, remove, and chmod invocations to perform their behavior recursively. This prepares us for the new $GIT_DIR/objects/pack/multi-pack-index.d directory which will be added in a following commit. Signed-off-by: Taylor Blau --- t/t5313-pack-bounds-checks.sh | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/t/t5313-pack-bounds-checks.sh b/t/t5313-pack-bounds-checks.sh index ceaa6700a2..86fc73f9fb 100755 --- a/t/t5313-pack-bounds-checks.sh +++ b/t/t5313-pack-bounds-checks.sh @@ -7,11 +7,11 @@ TEST_PASSES_SANITIZE_LEAK=true clear_base () { test_when_finished 'restore_base' && - rm -f $base + rm -r -f $base } restore_base () { - cp base-backup/* .git/objects/pack/ + cp -r base-backup/* .git/objects/pack/ } do_pack () { @@ -64,9 +64,9 @@ test_expect_success 'set up base packfile and variables' ' git commit -m base && git repack -ad && base=$(echo .git/objects/pack/*) && - chmod +w $base && + chmod -R +w $base && mkdir base-backup && - cp $base base-backup/ && + cp -r $base base-backup/ && object=$(git rev-parse HEAD:file) ' From patchwork Wed Jul 17 21:12:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13735799 Received: from mail-yw1-f175.google.com (mail-yw1-f175.google.com [209.85.128.175]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D8DD2186E2C for ; Wed, 17 Jul 2024 21:12:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.175 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250779; cv=none; b=X9eh6oxHSsE5gOIc1obnOjGpEIfPTEmsPKbPMIXxX3SRxqo2nwUX+3h2xFUsrqnGNqAz2ToSQZ6RCGsslJ0qESynjdN/DjBd6fK5DVgxvggrxqZI7zpIA36AV7OghJ+ciqx2iSEp/n9J7F/8Ko6mx+9OtaZJ+O0lZK2FYpLiuhA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1721250779; c=relaxed/simple; bh=SUeQTs4V8VhrLNvpBK7PC5XnHkXyswvJRiGNDCluV7s=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=ZHYBei6TyahRaEF8JraP8Kb8GUlkzZnPSkLoZawq8e3CuTavhP6BqJ+//5RLOn93sgisyrYHZmSpnLnb2Q0W00vyh2eb1JSgmKj54tZZgd9sze9RqCDaPdf9Soly2BRtw0uEBN69MkspPmSXl7VZ+GUpTvTisCxZtrl74KuTuYU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=eeoHuY1Q; arc=none smtp.client-ip=209.85.128.175 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="eeoHuY1Q" Received: by mail-yw1-f175.google.com with SMTP id 00721157ae682-6510c0c8e29so905297b3.0 for ; Wed, 17 Jul 2024 14:12:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1721250776; x=1721855576; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=jfc0dYMwmz01FuwUgKbHXLXwoLdCLjeLdLyClyfPaAI=; b=eeoHuY1QDqJJQCFepzndJ0Kxq/x0wHnGFDgZ72U+FW87fzlZJzoVnOIOlWPyPOPp3R fts1MhwDHnxYbIt19wz4tAjllNmjmr/7P/Iw6A96U98swASLK2yx2R9iZ/wOVzygPsk+ Q1mTalVjsF/dr+JL/699fmQR148EM/1VGwHGT6O1zNgFab9XNzlvhptPxcWL3CjSecdS 92su/R3+aSeq8p/rBwLhFCWflQfKFWzp5TLVDNsiTvX7kT3pHS/PLb7h2k6OHHgFInae BqXzgfR65x2qVHksIN4W6Ha1/j417gN7JkIzNxWTAFxX8GsPeTjusS3PBaZ2yHsgDgRN Ankw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721250776; x=1721855576; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=jfc0dYMwmz01FuwUgKbHXLXwoLdCLjeLdLyClyfPaAI=; b=Z3hTznxCao9crxUVesgzQ3uOeTCX8VC5egxKNhxFbLQUs3QAg1e7T2AQXFnNncA3b3 k7ZuHR+NaS92i1BkESVTbdwfWjSaDzoXtqLtbocT4ZoJWOuYhujJ/KkbpWq1mtsHTNum RHWOsAncdg27LsoebYc9JrsTeVLvjP3LHLkk52Oz7+EWJjXQVLQIMpwvsY75aasDo7f4 yfPIs8kO200b+VvWMicIoYYh8s/v68E2xcBZ9JXIKkw2hIoSMm67JGQUFJbBWCuxqwxm 2pcFV+mxgFMxK2wohx11HcBOGMJA0k/mXrr0tZQmlDVNLWZTCInOZZK/6Ux/vBJMTqAI f5tw== X-Gm-Message-State: AOJu0YwUDM94RC5MJfDVaMDe6szsHuCUw7vMnmjSFOMSTUeQwpXE6usS 2fJwrVrPkRphL4cfhSKerRKUDm7d1fPb1K0nN/9Z+umnobOgC91SzBEfIT7WMYMJUoJ5zAfIvdZ 3 X-Google-Smtp-Source: AGHT+IHiToIT4ZAIEhhZqZQjvlLYfGY5Hev/e2eZS52MB4ervKtzk5oPEAQhI1CPguMh/JgsJbhAuw== X-Received: by 2002:a81:87c2:0:b0:664:5957:f7a with SMTP id 00721157ae682-664fe93537emr33747947b3.15.1721250775393; Wed, 17 Jul 2024 14:12:55 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 00721157ae682-66601ada053sm783557b3.23.2024.07.17.14.12.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Jul 2024 14:12:55 -0700 (PDT) Date: Wed, 17 Jul 2024 17:12:53 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Elijah Newren , Junio C Hamano Subject: [PATCH v2 19/19] midx: implement support for writing incremental MIDX chains Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Now that the rest of the MIDX subsystem and relevant callers have been updated to learn about how to read and process incremental MIDX chains, let's finally update the implementation in `write_midx_internal()` to be able to write incremental MIDX chains. This new feature is available behind the `--incremental` option for the `multi-pack-index` builtin, like so: $ git multi-pack-index write --incremental The implementation for doing so is relatively straightforward, and boils down to a handful of different kinds of changes implemented in this patch: - The `compute_sorted_entries()` function is taught to reject objects which appear in any existing MIDX layer. - Functions like `write_midx_revindex()` are adjusted to write pack_order values which are offset by the number of objects in the base MIDX layer. - The end of `write_midx_internal()` is adjusted to move non-incremental MIDX files when necessary (i.e. when creating an incremental chain with an existing non-incremental MIDX in the repository). There are a handful of other changes that are introduced, like new functions to clear incremental MIDX files that are unrelated to the current chain (using the same "keep_hash" mechanism as in the non-incremental case). The tests explicitly exercising the new incremental MIDX feature are relatively limited for two reasons: 1. Most of the "interesting" behavior is already thoroughly covered in t5319-multi-pack-index.sh, which handles the core logic of reading objects through a MIDX. The new tests in t5334-incremental-multi-pack-index.sh are mostly focused on creating and destroying incremental MIDXs, as well as stitching their results together across layers. 2. A new GIT_TEST environment variable is added called "GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL", which modifies the entire test suite to write incremental MIDXs after repacking when combined with the "GIT_TEST_MULTI_PACK_INDEX" variable. This exercises the long tail of other interesting behavior that is defined implicitly throughout the rest of the CI suite. It is likewise added to the linux-TEST-vars job. Signed-off-by: Taylor Blau --- Documentation/git-multi-pack-index.txt | 11 +- builtin/multi-pack-index.c | 2 + builtin/repack.c | 8 +- ci/run-build-and-tests.sh | 1 + midx-write.c | 314 ++++++++++++++++++++---- midx.c | 62 ++++- midx.h | 4 + packfile.c | 16 +- packfile.h | 4 + t/README | 4 + t/lib-bitmap.sh | 6 +- t/lib-midx.sh | 28 +++ t/t5319-multi-pack-index.sh | 27 +- t/t5326-multi-pack-bitmaps.sh | 1 + t/t5327-multi-pack-bitmaps-rev.sh | 1 + t/t5332-multi-pack-reuse.sh | 2 + t/t5334-incremental-multi-pack-index.sh | 46 ++++ t/t7700-repack.sh | 27 +- 18 files changed, 460 insertions(+), 104 deletions(-) create mode 100755 t/t5334-incremental-multi-pack-index.sh diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt index 3696506eb3..631d5c7d15 100644 --- a/Documentation/git-multi-pack-index.txt +++ b/Documentation/git-multi-pack-index.txt @@ -64,6 +64,12 @@ The file given at `` is expected to be readable, and can contain duplicates. (If a given OID is given more than once, it is marked as preferred if at least one instance of it begins with the special `+` marker). + + --incremental:: + Write an incremental MIDX file containing only objects + and packs not present in an existing MIDX layer. + Migrates non-incremental MIDXs to incremental ones when + necessary. Incompatible with `--bitmap`. -- verify:: @@ -74,6 +80,8 @@ expire:: have no objects referenced by the MIDX (with the exception of `.keep` packs and cruft packs). Rewrite the MIDX file afterward to remove all references to these pack-files. ++ +NOTE: this mode is incompatible with incremental MIDX files. repack:: Create a new pack-file containing objects in small pack-files @@ -95,7 +103,8 @@ repack:: + If `repack.packKeptObjects` is `false`, then any pack-files with an associated `.keep` file will not be selected for the batch to repack. - ++ +NOTE: this mode is incompatible with incremental MIDX files. EXAMPLES -------- diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c index 9cf1a32d65..8805cbbeb3 100644 --- a/builtin/multi-pack-index.c +++ b/builtin/multi-pack-index.c @@ -129,6 +129,8 @@ static int cmd_multi_pack_index_write(int argc, const char **argv, MIDX_WRITE_BITMAP | MIDX_WRITE_REV_INDEX), OPT_BIT(0, "progress", &opts.flags, N_("force progress reporting"), MIDX_PROGRESS), + OPT_BIT(0, "incremental", &opts.flags, + N_("write a new incremental MIDX"), MIDX_WRITE_INCREMENTAL), OPT_BOOL(0, "stdin-packs", &opts.stdin_packs, N_("write multi-pack index containing only given indexes")), OPT_FILENAME(0, "refs-snapshot", &opts.refs_snapshot, diff --git a/builtin/repack.c b/builtin/repack.c index 8499bf0e12..7608430a37 100644 --- a/builtin/repack.c +++ b/builtin/repack.c @@ -1514,8 +1514,12 @@ int cmd_repack(int argc, const char **argv, const char *prefix) if (run_update_server_info) update_server_info(0); - if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0)) - write_midx_file(get_object_directory(), NULL, NULL, 0); + if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX, 0)) { + unsigned flags = 0; + if (git_env_bool(GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL, 0)) + flags |= MIDX_WRITE_INCREMENTAL; + write_midx_file(get_object_directory(), NULL, NULL, flags); + } cleanup: string_list_clear(&names, 1); diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh index e6fd68630c..2e28d02b20 100755 --- a/ci/run-build-and-tests.sh +++ b/ci/run-build-and-tests.sh @@ -25,6 +25,7 @@ linux-TEST-vars) export GIT_TEST_COMMIT_GRAPH=1 export GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1 export GIT_TEST_MULTI_PACK_INDEX=1 + export GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=1 export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master export GIT_TEST_NO_WRITE_REV_INDEX=1 export GIT_TEST_CHECKOUT_WORKERS=2 diff --git a/midx-write.c b/midx-write.c index d5275d719b..a94cb28bfd 100644 --- a/midx-write.c +++ b/midx-write.c @@ -17,6 +17,8 @@ #include "refs.h" #include "revision.h" #include "list-objects.h" +#include "path.h" +#include "pack-revindex.h" #define PACK_EXPIRED UINT_MAX #define BITMAP_POS_UNKNOWN (~((uint32_t)0)) @@ -25,7 +27,11 @@ extern int midx_checksum_valid(struct multi_pack_index *m); extern void clear_midx_files_ext(const char *object_dir, const char *ext, - unsigned char *keep_hash); + const char *keep_hash); +extern void clear_incremental_midx_files_ext(const char *object_dir, + const char *ext, + const char **keep_hashes, + uint32_t hashes_nr); extern int cmp_idx_or_pack_name(const char *idx_or_pack_name, const char *idx_name); @@ -86,6 +92,7 @@ struct write_midx_context { size_t nr; size_t alloc; struct multi_pack_index *m; + struct multi_pack_index *base_midx; struct progress *progress; unsigned pack_paths_checked; @@ -99,6 +106,9 @@ struct write_midx_context { int preferred_pack_idx; + int incremental; + uint32_t num_multi_pack_indexes_before; + struct string_list *to_include; }; @@ -122,6 +132,9 @@ static int should_include_pack(const struct write_midx_context *ctx, */ if (ctx->m && midx_contains_pack(ctx->m, file_name)) return 0; + else if (ctx->base_midx && midx_contains_pack(ctx->base_midx, + file_name)) + return 0; else if (ctx->to_include && !string_list_has_string(ctx->to_include, file_name)) return 0; @@ -338,7 +351,7 @@ static void compute_sorted_entries(struct write_midx_context *ctx, for (cur_fanout = 0; cur_fanout < 256; cur_fanout++) { fanout.nr = 0; - if (ctx->m) + if (ctx->m && !ctx->incremental) midx_fanout_add_midx_fanout(&fanout, ctx->m, cur_fanout, ctx->preferred_pack_idx); @@ -364,6 +377,10 @@ static void compute_sorted_entries(struct write_midx_context *ctx, if (cur_object && oideq(&fanout.entries[cur_object - 1].oid, &fanout.entries[cur_object].oid)) continue; + if (ctx->incremental && ctx->base_midx && + midx_has_oid(ctx->base_midx, + &fanout.entries[cur_object].oid)) + continue; ALLOC_GROW(ctx->entries, st_add(ctx->entries_nr, 1), alloc_objects); @@ -547,10 +564,16 @@ static int write_midx_revindex(struct hashfile *f, void *data) { struct write_midx_context *ctx = data; - uint32_t i; + uint32_t i, nr_base; + + if (ctx->incremental && ctx->base_midx) + nr_base = ctx->base_midx->num_objects + + ctx->base_midx->num_objects_in_base; + else + nr_base = 0; for (i = 0; i < ctx->entries_nr; i++) - hashwrite_be32(f, ctx->pack_order[i]); + hashwrite_be32(f, ctx->pack_order[i] + nr_base); return 0; } @@ -579,12 +602,18 @@ static int midx_pack_order_cmp(const void *va, const void *vb) static uint32_t *midx_pack_order(struct write_midx_context *ctx) { struct midx_pack_order_data *data; - uint32_t *pack_order; + uint32_t *pack_order, base_objects = 0; uint32_t i; trace2_region_enter("midx", "midx_pack_order", the_repository); + if (ctx->incremental && ctx->base_midx) + base_objects = ctx->base_midx->num_objects + + ctx->base_midx->num_objects_in_base; + + ALLOC_ARRAY(pack_order, ctx->entries_nr); ALLOC_ARRAY(data, ctx->entries_nr); + for (i = 0; i < ctx->entries_nr; i++) { struct pack_midx_entry *e = &ctx->entries[i]; data[i].nr = i; @@ -596,12 +625,11 @@ static uint32_t *midx_pack_order(struct write_midx_context *ctx) QSORT(data, ctx->entries_nr, midx_pack_order_cmp); - ALLOC_ARRAY(pack_order, ctx->entries_nr); for (i = 0; i < ctx->entries_nr; i++) { struct pack_midx_entry *e = &ctx->entries[data[i].nr]; struct pack_info *pack = &ctx->info[ctx->pack_perm[e->pack_int_id]]; if (pack->bitmap_pos == BITMAP_POS_UNKNOWN) - pack->bitmap_pos = i; + pack->bitmap_pos = i + base_objects; pack->bitmap_nr++; pack_order[i] = data[i].nr; } @@ -649,7 +677,8 @@ static void prepare_midx_packing_data(struct packing_data *pdata, prepare_packing_data(the_repository, pdata); for (i = 0; i < ctx->entries_nr; i++) { - struct pack_midx_entry *from = &ctx->entries[ctx->pack_order[i]]; + uint32_t pos = ctx->pack_order[i]; + struct pack_midx_entry *from = &ctx->entries[pos]; struct object_entry *to = packlist_alloc(pdata, &from->oid); oe_set_in_pack(pdata, to, @@ -897,37 +926,130 @@ static struct multi_pack_index *lookup_multi_pack_index(struct repository *r, static int fill_packs_from_midx(struct write_midx_context *ctx, const char *preferred_pack_name, uint32_t flags) { - uint32_t i; + struct multi_pack_index *m; - for (i = 0; i < ctx->m->num_packs; i++) { - ALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc); + for (m = ctx->m; m; m = m->base_midx) { + uint32_t i; + + for (i = 0; i < m->num_packs; i++) { + ALLOC_GROW(ctx->info, ctx->nr + 1, ctx->alloc); - if (flags & MIDX_WRITE_REV_INDEX || preferred_pack_name) { /* * If generating a reverse index, need to have * packed_git's loaded to compare their * mtimes and object count. * - * * If a preferred pack is specified, need to * have packed_git's loaded to ensure the chosen * preferred pack has a non-zero object count. */ - if (prepare_midx_pack(the_repository, ctx->m, i)) - return error(_("could not load pack")); + if (flags & MIDX_WRITE_REV_INDEX || + preferred_pack_name) { + if (prepare_midx_pack(the_repository, m, + m->num_packs_in_base + i)) { + error(_("could not load pack")); + return 1; + } - if (open_pack_index(ctx->m->packs[i])) - die(_("could not open index for %s"), - ctx->m->packs[i]->pack_name); + if (open_pack_index(m->packs[i])) + die(_("could not open index for %s"), + m->packs[i]->pack_name); + } + + fill_pack_info(&ctx->info[ctx->nr++], m->packs[i], + m->pack_names[i], + m->num_packs_in_base + i); } - - fill_pack_info(&ctx->info[ctx->nr++], ctx->m->packs[i], - ctx->m->pack_names[i], i); } - return 0; } +static struct { + const char *non_split; + const char *split; +} midx_exts[] = { + {NULL, MIDX_EXT_MIDX}, + {MIDX_EXT_BITMAP, MIDX_EXT_BITMAP}, + {MIDX_EXT_REV, MIDX_EXT_REV}, +}; + +static int link_midx_to_chain(struct multi_pack_index *m) +{ + struct strbuf from = STRBUF_INIT; + struct strbuf to = STRBUF_INIT; + int ret = 0; + size_t i; + + if (!m || m->has_chain) { + /* + * Either no MIDX previously existed, or it was already + * part of a MIDX chain. In both cases, we have nothing + * to link, so return early. + */ + goto done; + } + + for (i = 0; i < ARRAY_SIZE(midx_exts); i++) { + const unsigned char *hash = get_midx_checksum(m); + + get_midx_filename_ext(&from, m->object_dir, hash, + midx_exts[i].non_split); + get_split_midx_filename_ext(&to, m->object_dir, hash, + midx_exts[i].split); + + if (link(from.buf, to.buf) < 0 && errno != ENOENT) { + ret = error_errno(_("unable to link '%s' to '%s'"), + from.buf, to.buf); + goto done; + } + + strbuf_reset(&from); + strbuf_reset(&to); + } + +done: + strbuf_release(&from); + strbuf_release(&to); + return ret; +} + +static void clear_midx_files(const char *object_dir, + const char **hashes, + uint32_t hashes_nr, + unsigned incremental) +{ + /* + * if incremental: + * - remove all non-incremental MIDX files + * - remove any incremental MIDX files not in the current one + * + * if non-incremental: + * - remove all incremental MIDX files + * - remove any non-incremental MIDX files not matching the current + * hash + */ + struct strbuf buf = STRBUF_INIT; + const char *exts[] = { MIDX_EXT_BITMAP, MIDX_EXT_REV, MIDX_EXT_MIDX }; + uint32_t i, j; + + for (i = 0; i < ARRAY_SIZE(exts); i++) { + clear_incremental_midx_files_ext(object_dir, exts[i], + hashes, hashes_nr); + for (j = 0; j < hashes_nr; j++) + clear_midx_files_ext(object_dir, exts[i], hashes[j]); + } + + if (incremental) + get_midx_filename(&buf, object_dir); + else + get_midx_chain_filename(&buf, object_dir); + + if (unlink(buf.buf) && errno != ENOENT) + die_errno(_("failed to clear multi-pack-index at %s"), buf.buf); + + strbuf_release(&buf); +} + static int write_midx_internal(const char *object_dir, struct string_list *packs_to_include, struct string_list *packs_to_drop, @@ -940,42 +1062,66 @@ static int write_midx_internal(const char *object_dir, uint32_t i, start_pack; struct hashfile *f = NULL; struct lock_file lk; + struct tempfile *incr; struct write_midx_context ctx = { 0 }; int bitmapped_packs_concat_len = 0; int pack_name_concat_len = 0; int dropped_packs = 0; int result = 0; + const char **keep_hashes = NULL; struct chunkfile *cf; trace2_region_enter("midx", "write_midx_internal", the_repository); - get_midx_filename(&midx_name, object_dir); + ctx.incremental = !!(flags & MIDX_WRITE_INCREMENTAL); + if (ctx.incremental && (flags & MIDX_WRITE_BITMAP)) + die(_("cannot write incremental MIDX with bitmap")); + + if (ctx.incremental) + strbuf_addf(&midx_name, + "%s/pack/multi-pack-index.d/tmp_midx_XXXXXX", + object_dir); + else + get_midx_filename(&midx_name, object_dir); if (safe_create_leading_directories(midx_name.buf)) die_errno(_("unable to create leading directories of %s"), midx_name.buf); - if (!packs_to_include) { - /* - * Only reference an existing MIDX when not filtering which - * packs to include, since all packs and objects are copied - * blindly from an existing MIDX if one is present. - */ - ctx.m = lookup_multi_pack_index(the_repository, object_dir); - } + if (!packs_to_include || ctx.incremental) { + struct multi_pack_index *m = lookup_multi_pack_index(the_repository, + object_dir); + if (m && !midx_checksum_valid(m)) { + warning(_("ignoring existing multi-pack-index; checksum mismatch")); + m = NULL; + } - if (ctx.m && !midx_checksum_valid(ctx.m)) { - warning(_("ignoring existing multi-pack-index; checksum mismatch")); - ctx.m = NULL; + if (m) { + /* + * Only reference an existing MIDX when not filtering + * which packs to include, since all packs and objects + * are copied blindly from an existing MIDX if one is + * present. + */ + if (ctx.incremental) + ctx.base_midx = m; + else if (!packs_to_include) + ctx.m = m; + } } ctx.nr = 0; - ctx.alloc = ctx.m ? ctx.m->num_packs : 16; + ctx.alloc = ctx.m ? ctx.m->num_packs + ctx.m->num_packs_in_base : 16; ctx.info = NULL; ALLOC_ARRAY(ctx.info, ctx.alloc); - if (ctx.m && fill_packs_from_midx(&ctx, preferred_pack_name, - flags) < 0) { - result = 1; + if (ctx.incremental) { + struct multi_pack_index *m = ctx.base_midx; + while (m) { + ctx.num_multi_pack_indexes_before++; + m = m->base_midx; + } + } else if (ctx.m && fill_packs_from_midx(&ctx, preferred_pack_name, + flags) < 0) { goto cleanup; } @@ -992,7 +1138,8 @@ static int write_midx_internal(const char *object_dir, for_each_file_in_pack_dir(object_dir, add_pack_to_midx, &ctx); stop_progress(&ctx.progress); - if ((ctx.m && ctx.nr == ctx.m->num_packs) && + if ((ctx.m && ctx.nr == ctx.m->num_packs + ctx.m->num_packs_in_base) && + !ctx.incremental && !(packs_to_include || packs_to_drop)) { struct bitmap_index *bitmap_git; int bitmap_exists; @@ -1008,12 +1155,14 @@ static int write_midx_internal(const char *object_dir, * corresponding bitmap (or one wasn't requested). */ if (!want_bitmap) - clear_midx_files_ext(object_dir, ".bitmap", - NULL); + clear_midx_files_ext(object_dir, "bitmap", NULL); goto cleanup; } } + if (ctx.incremental && !ctx.nr) + goto cleanup; /* nothing to do */ + if (preferred_pack_name) { ctx.preferred_pack_idx = -1; @@ -1159,8 +1308,30 @@ static int write_midx_internal(const char *object_dir, pack_name_concat_len += MIDX_CHUNK_ALIGNMENT - (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT); - hold_lock_file_for_update(&lk, midx_name.buf, LOCK_DIE_ON_ERROR); - f = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk)); + if (ctx.incremental) { + struct strbuf lock_name = STRBUF_INIT; + + get_midx_chain_filename(&lock_name, object_dir); + hold_lock_file_for_update(&lk, lock_name.buf, LOCK_DIE_ON_ERROR); + strbuf_release(&lock_name); + + incr = mks_tempfile_m(midx_name.buf, 0444); + if (!incr) { + error(_("unable to create temporary MIDX layer")); + return -1; + } + + if (adjust_shared_perm(get_tempfile_path(incr))) { + error(_("unable to adjust shared permissions for '%s'"), + get_tempfile_path(incr)); + return -1; + } + + f = hashfd(get_tempfile_fd(incr), get_tempfile_path(incr)); + } else { + hold_lock_file_for_update(&lk, midx_name.buf, LOCK_DIE_ON_ERROR); + f = hashfd(get_lock_file_fd(&lk), get_lock_file_path(&lk)); + } if (ctx.nr - dropped_packs == 0) { error(_("no pack files to index.")); @@ -1253,14 +1424,55 @@ static int write_midx_internal(const char *object_dir, * have been freed in the previous if block. */ - if (ctx.m) + CALLOC_ARRAY(keep_hashes, ctx.num_multi_pack_indexes_before + 1); + + if (ctx.incremental) { + FILE *chainf = fdopen_lock_file(&lk, "w"); + struct strbuf final_midx_name = STRBUF_INIT; + struct multi_pack_index *m = ctx.base_midx; + + if (!chainf) { + error_errno(_("unable to open multi-pack-index chain file")); + return -1; + } + + if (link_midx_to_chain(ctx.base_midx) < 0) + return -1; + + get_split_midx_filename_ext(&final_midx_name, object_dir, + midx_hash, MIDX_EXT_MIDX); + + if (rename_tempfile(&incr, final_midx_name.buf) < 0) { + error_errno(_("unable to rename new multi-pack-index layer")); + return -1; + } + + keep_hashes[ctx.num_multi_pack_indexes_before] = + xstrdup(hash_to_hex(midx_hash)); + + for (i = 0; i < ctx.num_multi_pack_indexes_before; i++) { + uint32_t j = ctx.num_multi_pack_indexes_before - i - 1; + + keep_hashes[j] = xstrdup(hash_to_hex(get_midx_checksum(m))); + m = m->base_midx; + } + + for (i = 0; i < ctx.num_multi_pack_indexes_before + 1; i++) + fprintf(get_lock_file_fp(&lk), "%s\n", keep_hashes[i]); + } else { + keep_hashes[ctx.num_multi_pack_indexes_before] = + xstrdup(hash_to_hex(midx_hash)); + } + + if (ctx.m || ctx.base_midx) close_object_store(the_repository->objects); if (commit_lock_file(&lk) < 0) die_errno(_("could not write multi-pack-index")); - clear_midx_files_ext(object_dir, ".bitmap", midx_hash); - clear_midx_files_ext(object_dir, ".rev", midx_hash); + clear_midx_files(object_dir, keep_hashes, + ctx.num_multi_pack_indexes_before + 1, + ctx.incremental); cleanup: for (i = 0; i < ctx.nr; i++) { @@ -1275,6 +1487,11 @@ static int write_midx_internal(const char *object_dir, free(ctx.entries); free(ctx.pack_perm); free(ctx.pack_order); + if (keep_hashes) { + for (i = 0; i < ctx.num_multi_pack_indexes_before + 1; i++) + free((char *)keep_hashes[i]); + free(keep_hashes); + } strbuf_release(&midx_name); trace2_region_leave("midx", "write_midx_internal", the_repository); @@ -1311,6 +1528,9 @@ int expire_midx_packs(struct repository *r, const char *object_dir, unsigned fla if (!m) return 0; + if (m->base_midx) + die(_("cannot expire packs from an incremental multi-pack-index")); + CALLOC_ARRAY(count, m->num_packs); if (flags & MIDX_PROGRESS) @@ -1485,6 +1705,8 @@ int midx_repack(struct repository *r, const char *object_dir, size_t batch_size, if (!m) return 0; + if (m->base_midx) + die(_("cannot repack an incremental multi-pack-index")); CALLOC_ARRAY(include_pack, m->num_packs); diff --git a/midx.c b/midx.c index 21a9dbe23a..6372138ba8 100644 --- a/midx.c +++ b/midx.c @@ -16,7 +16,10 @@ int midx_checksum_valid(struct multi_pack_index *m); void clear_midx_files_ext(const char *object_dir, const char *ext, - unsigned char *keep_hash); + const char *keep_hash); +void clear_incremental_midx_files_ext(const char *object_dir, const char *ext, + char **keep_hashes, + uint32_t hashes_nr); int cmp_idx_or_pack_name(const char *idx_or_pack_name, const char *idx_name); @@ -520,6 +523,11 @@ int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, return 0; } +int midx_has_oid(struct multi_pack_index *m, const struct object_id *oid) +{ + return bsearch_midx(oid, m, NULL); +} + struct object_id *nth_midxed_object_oid(struct object_id *oid, struct multi_pack_index *m, uint32_t n) @@ -722,7 +730,8 @@ int midx_checksum_valid(struct multi_pack_index *m) } struct clear_midx_data { - char *keep; + char **keep; + uint32_t keep_nr; const char *ext; }; @@ -730,32 +739,63 @@ static void clear_midx_file_ext(const char *full_path, size_t full_path_len UNUS const char *file_name, void *_data) { struct clear_midx_data *data = _data; + uint32_t i; if (!(starts_with(file_name, "multi-pack-index-") && ends_with(file_name, data->ext))) return; - if (data->keep && !strcmp(data->keep, file_name)) - return; - + for (i = 0; i < data->keep_nr; i++) { + if (!strcmp(data->keep[i], file_name)) + return; + } if (unlink(full_path)) die_errno(_("failed to remove %s"), full_path); } void clear_midx_files_ext(const char *object_dir, const char *ext, - unsigned char *keep_hash) + const char *keep_hash) { struct clear_midx_data data; memset(&data, 0, sizeof(struct clear_midx_data)); - if (keep_hash) - data.keep = xstrfmt("multi-pack-index-%s%s", - hash_to_hex(keep_hash), ext); + if (keep_hash) { + ALLOC_ARRAY(data.keep, 1); + + data.keep[0] = xstrfmt("multi-pack-index-%s.%s", keep_hash, ext); + data.keep_nr = 1; + } data.ext = ext; for_each_file_in_pack_dir(object_dir, clear_midx_file_ext, &data); + if (keep_hash) + free(data.keep[0]); + free(data.keep); +} + +void clear_incremental_midx_files_ext(const char *object_dir, const char *ext, + char **keep_hashes, + uint32_t hashes_nr) +{ + struct clear_midx_data data; + uint32_t i; + + memset(&data, 0, sizeof(struct clear_midx_data)); + + ALLOC_ARRAY(data.keep, hashes_nr); + for (i = 0; i < hashes_nr; i++) + data.keep[i] = xstrfmt("multi-pack-index-%s.%s", keep_hashes[i], + ext); + data.keep_nr = hashes_nr; + data.ext = ext; + + for_each_file_in_pack_subdir(object_dir, "multi-pack-index.d", + clear_midx_file_ext, &data); + + for (i = 0; i < hashes_nr; i++) + free(data.keep[i]); free(data.keep); } @@ -773,8 +813,8 @@ void clear_midx_file(struct repository *r) if (remove_path(midx.buf)) die(_("failed to clear multi-pack-index at %s"), midx.buf); - clear_midx_files_ext(r->objects->odb->path, ".bitmap", NULL); - clear_midx_files_ext(r->objects->odb->path, ".rev", NULL); + clear_midx_files_ext(r->objects->odb->path, MIDX_EXT_BITMAP, NULL); + clear_midx_files_ext(r->objects->odb->path, MIDX_EXT_REV, NULL); strbuf_release(&midx); } diff --git a/midx.h b/midx.h index 3714cad2cc..42d4f8d149 100644 --- a/midx.h +++ b/midx.h @@ -29,6 +29,8 @@ struct bitmapped_pack; #define MIDX_LARGE_OFFSET_NEEDED 0x80000000 #define GIT_TEST_MULTI_PACK_INDEX "GIT_TEST_MULTI_PACK_INDEX" +#define GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL \ + "GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL" struct multi_pack_index { struct multi_pack_index *next; @@ -77,6 +79,7 @@ struct multi_pack_index { #define MIDX_WRITE_BITMAP (1 << 2) #define MIDX_WRITE_BITMAP_HASH_CACHE (1 << 3) #define MIDX_WRITE_BITMAP_LOOKUP_TABLE (1 << 4) +#define MIDX_WRITE_INCREMENTAL (1 << 5) #define MIDX_EXT_REV "rev" #define MIDX_EXT_BITMAP "bitmap" @@ -101,6 +104,7 @@ int bsearch_one_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result); int bsearch_midx(const struct object_id *oid, struct multi_pack_index *m, uint32_t *result); +int midx_has_oid(struct multi_pack_index *m, const struct object_id *oid); off_t nth_midxed_offset(struct multi_pack_index *m, uint32_t pos); uint32_t nth_midxed_pack_int_id(struct multi_pack_index *m, uint32_t pos); struct object_id *nth_midxed_object_oid(struct object_id *oid, diff --git a/packfile.c b/packfile.c index 1eb18e3041..cf12a539ea 100644 --- a/packfile.c +++ b/packfile.c @@ -815,9 +815,10 @@ static void report_pack_garbage(struct string_list *list) report_helper(list, seen_bits, first, list->nr); } -void for_each_file_in_pack_dir(const char *objdir, - each_file_in_pack_dir_fn fn, - void *data) +void for_each_file_in_pack_subdir(const char *objdir, + const char *subdir, + each_file_in_pack_dir_fn fn, + void *data) { struct strbuf path = STRBUF_INIT; size_t dirnamelen; @@ -826,6 +827,8 @@ void for_each_file_in_pack_dir(const char *objdir, strbuf_addstr(&path, objdir); strbuf_addstr(&path, "/pack"); + if (subdir) + strbuf_addf(&path, "/%s", subdir); dir = opendir(path.buf); if (!dir) { if (errno != ENOENT) @@ -847,6 +850,13 @@ void for_each_file_in_pack_dir(const char *objdir, strbuf_release(&path); } +void for_each_file_in_pack_dir(const char *objdir, + each_file_in_pack_dir_fn fn, + void *data) +{ + for_each_file_in_pack_subdir(objdir, NULL, fn, data); +} + struct prepare_pack_data { struct repository *r; struct string_list *garbage; diff --git a/packfile.h b/packfile.h index eb18ec15db..0f78658229 100644 --- a/packfile.h +++ b/packfile.h @@ -55,6 +55,10 @@ struct packed_git *parse_pack_index(unsigned char *sha1, const char *idx_path); typedef void each_file_in_pack_dir_fn(const char *full_path, size_t full_path_len, const char *file_name, void *data); +void for_each_file_in_pack_subdir(const char *objdir, + const char *subdir, + each_file_in_pack_dir_fn fn, + void *data); void for_each_file_in_pack_dir(const char *objdir, each_file_in_pack_dir_fn fn, void *data); diff --git a/t/README b/t/README index e8a11926e4..e93a29de1b 100644 --- a/t/README +++ b/t/README @@ -469,6 +469,10 @@ GIT_TEST_MULTI_PACK_INDEX=, when true, forces the multi-pack- index to be written after every 'git repack' command, and overrides the 'core.multiPackIndex' setting to true. +GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=, when true, sets +the '--incremental' option on all invocations of 'git multi-pack-index +write'. + GIT_TEST_SIDEBAND_ALL=, when true, overrides the 'uploadpack.allowSidebandAll' setting to true, and when false, forces fetch-pack to not request sideband-all (even if the server advertises diff --git a/t/lib-bitmap.sh b/t/lib-bitmap.sh index f595937094..62aa6744a6 100644 --- a/t/lib-bitmap.sh +++ b/t/lib-bitmap.sh @@ -1,6 +1,8 @@ # Helpers for scripts testing bitmap functionality; see t5310 for # example usage. +. "$TEST_DIRECTORY"/lib-midx.sh + objdir=.git/objects midx=$objdir/pack/multi-pack-index @@ -264,10 +266,6 @@ have_delta () { test_cmp expect actual } -midx_checksum () { - test-tool read-midx --checksum "$1" -} - # midx_pack_source midx_pack_source () { test-tool read-midx --show-objects .git/objects | grep "^$1 " | cut -f2 diff --git a/t/lib-midx.sh b/t/lib-midx.sh index 1261994744..e38c609604 100644 --- a/t/lib-midx.sh +++ b/t/lib-midx.sh @@ -6,3 +6,31 @@ test_midx_consistent () { test_cmp expect actual && git multi-pack-index --object-dir=$1 verify } + +midx_checksum () { + test-tool read-midx --checksum "$1" +} + +midx_git_two_modes () { + git -c core.multiPackIndex=false $1 >expect && + git -c core.multiPackIndex=true $1 >actual && + if [ "$2" = "sorted" ] + then + sort expect.sorted && + mv expect.sorted expect && + sort actual.sorted && + mv actual.sorted actual + fi && + test_cmp expect actual +} + +compare_results_with_midx () { + MSG=$1 + test_expect_success "check normal git operations: $MSG" ' + midx_git_two_modes "rev-list --objects --all" && + midx_git_two_modes "log --raw" && + midx_git_two_modes "count-objects --verbose" && + midx_git_two_modes "cat-file --batch-all-objects --batch-check" && + midx_git_two_modes "cat-file --batch-all-objects --batch-check --unordered" sorted + ' +} diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index 6e9ee23398..4b0b5a5c9f 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -3,8 +3,11 @@ test_description='multi-pack-indexes' . ./test-lib.sh . "$TEST_DIRECTORY"/lib-chunk.sh +. "$TEST_DIRECTORY"/lib-midx.sh GIT_TEST_MULTI_PACK_INDEX=0 +GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=0 +GIT_TEST_MULTI_PACK_INDEX_WRITE_INCREMENTAL=0 objdir=.git/objects HASH_LEN=$(test_oid rawsz) @@ -107,30 +110,6 @@ test_expect_success 'write midx with one v1 pack' ' midx_read_expect 1 18 4 $objdir ' -midx_git_two_modes () { - git -c core.multiPackIndex=false $1 >expect && - git -c core.multiPackIndex=true $1 >actual && - if [ "$2" = "sorted" ] - then - sort expect.sorted && - mv expect.sorted expect && - sort actual.sorted && - mv actual.sorted actual - fi && - test_cmp expect actual -} - -compare_results_with_midx () { - MSG=$1 - test_expect_success "check normal git operations: $MSG" ' - midx_git_two_modes "rev-list --objects --all" && - midx_git_two_modes "log --raw" && - midx_git_two_modes "count-objects --verbose" && - midx_git_two_modes "cat-file --batch-all-objects --batch-check" && - midx_git_two_modes "cat-file --batch-all-objects --batch-check --unordered" sorted - ' -} - test_expect_success 'write midx with one v2 pack' ' git pack-objects --index-version=2,0x40 $objdir/pack/test &2 && incrpackid=$(git pack-objects --all --unpacked --incremental .git/objects/pack/pack err && + git repack -Adl --write-bitmap-index 2>err && cat >expect <<-EOF && warning: disabling bitmap writing, as some objects are not being packed EOF @@ -533,11 +536,11 @@ test_expect_success 'setup for --write-midx tests' ' test_expect_success '--write-midx unchanged' ' ( cd midx && - GIT_TEST_MULTI_PACK_INDEX=0 git repack && + git repack && test_path_is_missing $midx && test_path_is_missing $midx-*.bitmap && - GIT_TEST_MULTI_PACK_INDEX=0 git repack --write-midx && + git repack --write-midx && test_path_is_file $midx && test_path_is_missing $midx-*.bitmap && @@ -550,7 +553,7 @@ test_expect_success '--write-midx with a new pack' ' cd midx && test_commit loose && - GIT_TEST_MULTI_PACK_INDEX=0 git repack --write-midx && + git repack --write-midx && test_path_is_file $midx && test_path_is_missing $midx-*.bitmap && @@ -561,7 +564,7 @@ test_expect_success '--write-midx with a new pack' ' test_expect_success '--write-midx with -b' ' ( cd midx && - GIT_TEST_MULTI_PACK_INDEX=0 git repack -mb && + git repack -mb && test_path_is_file $midx && test_path_is_file $midx-*.bitmap && @@ -574,7 +577,7 @@ test_expect_success '--write-midx with -d' ' cd midx && test_commit repack && - GIT_TEST_MULTI_PACK_INDEX=0 git repack -Ad --write-midx && + git repack -Ad --write-midx && test_path_is_file $midx && test_path_is_missing $midx-*.bitmap && @@ -587,21 +590,21 @@ test_expect_success 'cleans up MIDX when appropriate' ' cd midx && test_commit repack-2 && - GIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb --write-midx && + git repack -Adb --write-midx && checksum=$(midx_checksum $objdir) && test_path_is_file $midx && test_path_is_file $midx-$checksum.bitmap && test_commit repack-3 && - GIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb --write-midx && + git repack -Adb --write-midx && test_path_is_file $midx && test_path_is_missing $midx-$checksum.bitmap && test_path_is_file $midx-$(midx_checksum $objdir).bitmap && test_commit repack-4 && - GIT_TEST_MULTI_PACK_INDEX=0 git repack -Adb && + git repack -Adb && find $objdir/pack -type f -name "multi-pack-index*" >files && test_must_be_empty files @@ -622,7 +625,6 @@ test_expect_success '--write-midx with preferred bitmap tips' ' git log --format="create refs/tags/%s/%s %H" HEAD >refs && git update-ref --stdin