From patchwork Fri Apr 11 23:26:20 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 14048820 Received: from mail-qk1-f181.google.com (mail-qk1-f181.google.com [209.85.222.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2C86C2BD595 for ; Fri, 11 Apr 2025 23:26:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.181 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744413984; cv=none; b=BO509BKJUwkgl8NgVt5glDPJy4SCY3SJv+Kb3ZVmwng0YSbA6NJAkUzewzZq0cg44DDEVZqKQE2GVVmJ8TXMdYqwHXxfn/6MBialE+VQ9Jg5UnElF/U54iSgzg2JZT+yb3U+pHgh9x2L0AcRhiPM+01vJNKbIZUJi+s07tTrNSE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744413984; c=relaxed/simple; bh=D5wqJTmEj8TusnWroA8mhFWRRGsOtCK4HO6waA8KeHQ=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=uYzjGW6BLZs2Aj5ayjAtavd8KuJEMuhQJ7MIiwRaXtsBaz7PqmJbfCySSngGx0XkFl93ob19P5uZ1RZix97MKHrwlZVP51p9IgqDyYQ703DQVFO6Td8CPRkadfeiHo7ysujc+An+te/vEdEgx8CYAVim10LvycxYtOG5Qd8KM0g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=pass smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=rlcp39gX; arc=none smtp.client-ip=209.85.222.181 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="rlcp39gX" Received: by mail-qk1-f181.google.com with SMTP id af79cd13be357-7c5675dec99so240431485a.0 for ; Fri, 11 Apr 2025 16:26:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1744413982; x=1745018782; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=9cOiIlsxv55af/sHxUu/IEXqPKIiPF/rBNErHxeqicw=; b=rlcp39gXYfNS5+kEYHRcU6AvbJkxEpWgbQUhi30vsvEoUYYrw4qwU1gS6oVDnZMjT9 MmtNmltQ7TeXw3ua9Fy22IW05yREf9XTICuPD7WYTwoKJn8hK+5o2nCQyR+zbB5WKCXc hNqTNavIFcRG5RYoEApMO5GTuyC75x5T69GQP1/sSJb24TkrR4+LmIvgKWSludhD4454 pEdvOzBdFm95RFDgXS0HTCqim6oj5MVHDi+ns8m/t+JYnc4reFyJligBfxx6Ef0Hrui1 pzo8all1I+wpfGbJQe9CCog9SgK7KPTHUcsxg+KoA8tZeYnK3WuDJfgHr6U/C4g53NVw Z7Ig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744413982; x=1745018782; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=9cOiIlsxv55af/sHxUu/IEXqPKIiPF/rBNErHxeqicw=; b=OrT6F2uBx9v7kUivHafPLQTnjcQE+cj5EL0TH7ZyfF24xuNloCwboj7dfTNUD6spZm 4sfJPMzDqOz2ojX2FLC09ZKDTfhQfSGF5P/V6eirhJusFgdjZ7TjS6eQBp4Fl3C0d9hE hQaDIC/bNsNrgjtg5/R5i2T9hSgKuI7tzsuOzVYIbI0tKfKxLlwI6kNjfzYRz6oKgSOr WTO8W5VF7CAjj+rDurzLljNnx+Y9LxrYjN9MhRyRgFOuOR3SPwqlMMpFUqbRqfoRTrtq HvdT25oKAa9dCDJZNmBxeBnot9BlnJHVKntWjBUsEHkYyo8pdQYmKBhrmaR6DqJvPCiP tGSw== X-Gm-Message-State: AOJu0YxdOsfmtMz+OALbJS07WIehMJuCwm9GjrE+AFAkTlwM8RbWKOnM +JOU/2X47ZlTnL2I30JcFG6jXaEnRCJaVpTxBJG24inRo/UMyXIx+CwuI3J3+LJqnbVbL383d49 gxSw= X-Gm-Gg: ASbGncsieSaCG7YV3bmTLV7uAqHiEQDwjS5x88bhzbQHho7TtJhUmJVOFUXAgz9cxG2 nxf63DMF8rYI5yjKzjzMvlViczGM2cR+mtZeTuYInhbA+GZ3ys81yH3W2w76p7hKDPF7sbyJ2YV HYtjhEGT33AO6aBroAy5LSE1RySMNk6GfxEb3dGZSMDX9WE1n5kO4r43CBnKXDs8+56/2w6kFqo bkwZ8VmHGUewqcZgi+kW1U2e4TMQncCGEOOVvFogox1pSgzbRWhWVPGZM0A19/6841fpq5jXGW0 YzApdGP+UJp46+TGQk6js8irn1iPRNwJ+BNRZiKa9Zdcj0sOSpaevJprWXEjuXuYDkb54+/yMSV z2S92nugeeC52 X-Google-Smtp-Source: AGHT+IHHso+WqKwyTx5n/ORPE4zaGYpPXf5b18pclJ4xpIry1VIYN00a9eo4SGifG+A9J1w7KJ/psQ== X-Received: by 2002:a05:620a:28ca:b0:7c5:57e6:ee87 with SMTP id af79cd13be357-7c7af1caba5mr664230385a.41.1744413981620; Fri, 11 Apr 2025 16:26:21 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with UTF8SMTPSA id af79cd13be357-7c7a8969e66sm327831585a.52.2025.04.11.16.26.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Apr 2025 16:26:21 -0700 (PDT) Date: Fri, 11 Apr 2025 19:26:20 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Elijah Newren , Jeff King , Junio C Hamano Subject: [RFC PATCH 1/8] pack-objects: use standard option incompatibility functions Message-ID: <63fb4dab30a6bf1dee4318341078fe9eb4716537.1744413969.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: pack-objects has a handful of explicit checks for pairs of command-line options which are mutually incompatible. Many of these pre-date a699367bb8 (i18n: factorize more 'incompatible options' messages, 2022-01-31). Convert the explicit checks into die_for_incompatible_opt2() calls, which simplifies the implementation and standardizes pack-objects' output when given incompatible options (e.g., --stdin-packs with --filter gives different output than --keep-unreachable with --unpack-unreachable). There is one minor piece of test fallout in t5331 that expects the old format, which has been corrected. Signed-off-by: Taylor Blau --- builtin/pack-objects.c | 19 ++++++++++--------- t/t5331-pack-objects-stdin.sh | 2 +- 2 files changed, 11 insertions(+), 10 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 6b06d159d2..aaea968ed2 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -4651,9 +4651,10 @@ int cmd_pack_objects(int argc, strvec_push(&rp, "--unpacked"); } - if (exclude_promisor_objects && exclude_promisor_objects_best_effort) - die(_("options '%s' and '%s' cannot be used together"), - "--exclude-promisor-objects", "--exclude-promisor-objects-best-effort"); + die_for_incompatible_opt2(exclude_promisor_objects, + "--exclude-promisor-objects", + exclude_promisor_objects_best_effort, + "--exclude-promisor-objects-best-effort"); if (exclude_promisor_objects) { use_internal_rev_list = 1; fetch_if_missing = 0; @@ -4691,13 +4692,13 @@ int cmd_pack_objects(int argc, if (!pack_to_stdout && thin) die(_("--thin cannot be used to build an indexable pack")); - if (keep_unreachable && unpack_unreachable) - die(_("options '%s' and '%s' cannot be used together"), "--keep-unreachable", "--unpack-unreachable"); + die_for_incompatible_opt2(keep_unreachable, "--keep-unreachable", + unpack_unreachable, "--unpack-unreachable"); if (!rev_list_all || !rev_list_reflog || !rev_list_index) unpack_unreachable_expiration = 0; - if (stdin_packs && filter_options.choice) - die(_("cannot use --filter with --stdin-packs")); + die_for_incompatible_opt2(filter_options.choice, "--filter", + stdin_packs, "--stdin-packs"); if (stdin_packs && use_internal_rev_list) die(_("cannot use internal rev list with --stdin-packs")); @@ -4705,8 +4706,8 @@ int cmd_pack_objects(int argc, if (cruft) { if (use_internal_rev_list) die(_("cannot use internal rev list with --cruft")); - if (stdin_packs) - die(_("cannot use --stdin-packs with --cruft")); + die_for_incompatible_opt2(stdin_packs, "--stdin-packs", + cruft, "--cruft"); } /* diff --git a/t/t5331-pack-objects-stdin.sh b/t/t5331-pack-objects-stdin.sh index b48c0cbe8f..4f5e2733a2 100755 --- a/t/t5331-pack-objects-stdin.sh +++ b/t/t5331-pack-objects-stdin.sh @@ -64,7 +64,7 @@ test_expect_success '--stdin-packs is incompatible with --filter' ' cd stdin-packs && test_must_fail git pack-objects --stdin-packs --stdout \ --filter=blob:none err && - test_grep "cannot use --filter with --stdin-packs" err + test_grep "options .--filter. and .--stdin-packs. cannot be used together" err ) ' From patchwork Fri Apr 11 23:26:23 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 14048821 Received: from mail-qt1-f179.google.com (mail-qt1-f179.google.com [209.85.160.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3B8362BE7B1 for ; Fri, 11 Apr 2025 23:26:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744413987; cv=none; b=WnJEDgkc2E4zL2HVK+7SrCF2NUl0pJFrz0JHGwD8n7kVC9fpG75SMVdB98vRj/d1XZ3XqrxWl3u7NSyz6/qUCTNsO2u5/Q7HflACy3ky5lYpOH+yf9vWqh04oBIwODOj3ZhfsLAhG75CGSlUDVsuBtwb1uXTSDQJmWEHwmznBUc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744413987; c=relaxed/simple; bh=NB2cAdmufL3V6Az6kIuBj/iYNobliVsmUdSypPw0uoQ=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=jTmrVRyQZWaCviR+eWqVTq1EmB2hny2FBkRBaZTCXhH/2McBXBBDMt8kk7lAwQq6Ce/aCcTqEJsCIzaS+xG8D8lMJzOg0zt45fHqbBlZzOYzJ+NNW62oGXIHXrHKmBzb35+Nl9Id9FXmDswzuNGDMo0+mPxulwf1veCk2NeyXUo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=pass smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=JGSHgSbZ; arc=none smtp.client-ip=209.85.160.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="JGSHgSbZ" Received: by mail-qt1-f179.google.com with SMTP id d75a77b69052e-476b89782c3so26158741cf.1 for ; Fri, 11 Apr 2025 16:26:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1744413985; x=1745018785; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=khXxc1vF6BJaAq4IgAvRk9E0N05RXVeO5PHGLG7X7bY=; b=JGSHgSbZAsRS7wKuJfa0LB0bm9ZuvrqkHUdHxmdJ9trfn5c3w4HVwONdCAB1KenFk1 hP+vEWBBQJyth/fZll+r1SeIhj/CO4SANwFQdrBIkJUIw3zO2bzFiGUv2Y1A7TYCr7w8 FyFiOXSM878jGjfgjvBtClEdSGVmpexm0hAs/NxfSfgT1MBdQ/DDnqwA+wNBK9j+2dEt Phdli9+iLDGiLCYsaGw53E5JDYE9li/NaS35MgxtVt9iqnoAZIX+nURDWNDyAozW3UWq Heqbxn6a8qyvmkgbRIXifGC6WAVUskavKX7Q37dfgChvYuxX8BaF1aQug3GnOhXcsfUr bJgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744413985; x=1745018785; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=khXxc1vF6BJaAq4IgAvRk9E0N05RXVeO5PHGLG7X7bY=; b=U5iNrwh1BNDacPUAVesFhc0sOPnbHSY5r1UKv6XNmPVS6sbToTq99+sIUHg7SUeDHL ZjVnex9MXnyExfhbsvevzWGb1BhyInogWyZzfsfaP40kyVzaTvela0ws3uWQE1ErGBmG HmOc+4vPGIzMQnY3nYO3C3g1xHv6JqcqbS5GzvKWbWiHTt3s2NB6iEWr+YTT38LDTTXG r+2ArhMP/ja6F57v9MkaRTfRzwysLyXlhdEn5mI9XJwE16yRtrxu2FlyVpenmKuwR0zf SbqKZoXYGbBRMaq51zpT1cey7IN0djtFNjJ7KRGRfs7N0ZjVysRuT0kfLe0ucP1/EpjE RuBg== X-Gm-Message-State: AOJu0YyDAPpHXh2KtBLUOWCAKm8Ivmy990+YSZdOCsekl/BuJm6hOMBA 577tMZqxnPSmrkBrGYIdZbNaGZo+Y9FwieJG56UhKa6LeTsDaDqpM8DeE/7aj2V+lnWeEp0srHO ZMp4= X-Gm-Gg: ASbGnctofvG/67adiinzETEIatynmR4EmT3Cvom/mT5c7jgYnMVRGQ/esfrbERxL1f7 pdSNqs3yUuvApXOAk8ASB43o6BE7mIdXp1z+WfI7KwVPHDCUP9pcZmKZODMNCLXbuz3elZh+mLs Q6Ta+ALQmb1pglJiAKxqe6961KRP1EcyxQHZZ2HPajRb1+D2GYEOMh3nmgo9NLVvC/xRdeaLgVw uJyoQcSAAQyGreL/yoi6QPbgEMaSybJZDzilP5dD/IEE8Y+FzS21o6WDjks5sbvNx5nZaoyDV2C R/QQYHWRpE3vmFuL3c6rq6wbewLj5T7XHftR1ETp/nydaVJ2lvjphO+efk1Fs6hA+MBCGn9P4Bt 4+fa8yJO+unLn X-Google-Smtp-Source: AGHT+IHLjRHLdIyB/8HkhDkHXAMJbX+IpIiN+CiozKFW4uWfCu6mke8s0kpiNkXgRyPKM3TXX25Qtw== X-Received: by 2002:a05:622a:1649:b0:474:eff7:a478 with SMTP id d75a77b69052e-479775ea817mr75940561cf.46.1744413984684; Fri, 11 Apr 2025 16:26:24 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with UTF8SMTPSA id d75a77b69052e-4796ed9bcdcsm33415141cf.49.2025.04.11.16.26.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Apr 2025 16:26:24 -0700 (PDT) Date: Fri, 11 Apr 2025 19:26:23 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Elijah Newren , Jeff King , Junio C Hamano Subject: [RFC PATCH 2/8] pack-objects: limit scope in 'add_object_entry_from_pack()' Message-ID: <6357633f6d144f24ac561ccba908aea3aed7aaaf.1744413969.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: add_object_entry_from_pack() handles objects from identified packs by checking their type, before adding commit objects as pending in the subsequent traversal used by `--stdin-packs`. There are a couple of quality-of-life refactorings that I noticed while working in this area: - We declare 'revs' (given to us through the miscellaneous context argument) earlier in the "if (p)" conditional than is necessary. - The 'struct object_info' can use a designated initializer to fill in the structures type pointer, since that is the only field that we care about. Signed-off-by: Taylor Blau --- builtin/pack-objects.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index aaea968ed2..540e5eba9e 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -3490,14 +3490,12 @@ static int add_object_entry_from_pack(const struct object_id *oid, return 0; if (p) { - struct rev_info *revs = _data; - struct object_info oi = OBJECT_INFO_INIT; - - oi.typep = &type; + struct object_info oi = { .typep = &type }; if (packed_object_info(the_repository, p, ofs, &oi) < 0) { die(_("could not get type of object %s in pack %s"), oid_to_hex(oid), p->pack_name); } else if (type == OBJ_COMMIT) { + struct rev_info *revs = _data; /* * commits in included packs are used as starting points for the * subsequent revision walk From patchwork Fri Apr 11 23:26:26 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 14048822 Received: from mail-qv1-f53.google.com (mail-qv1-f53.google.com [209.85.219.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 273022BE7D5 for ; Fri, 11 Apr 2025 23:26:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744413990; cv=none; b=Zffvxpyww4b2Ff2nvh2ZAv0zwLIDoTAAuWd64Dm+B7VslH2e5Iu2yNGnRytiWuBZV/RkAvGrXXzEZkiX3z/C1F+SLBdm4Lx1n6S3oX4Hv/z75z+QR8xBpyOvui71pMSSyTBR6YRJL29Xq62DT229wAXq5Fw4Ywh/EY1mZOR+QDQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744413990; c=relaxed/simple; bh=8qX1xwMLsD9aXK/D/8ctPhNyPyHJd3eArgQLAnabp94=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=CvtSop1xCT/+N9GAAMn5qUr4PcEUa2tTqvxfQMMb532zUFt+Y5GhV6kimY25HHWDBPbzdW/39Q7kxql70U4GMrTt9TKFtLlu30gH/XsKiNjQ5SUv4ITRAc6Tf/3N+ajVv1KlvGj+ViMGwMfCznY5j8wxHfCNKSX9UiIq63pWJKg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=pass smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=xPsJJgRz; arc=none smtp.client-ip=209.85.219.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="xPsJJgRz" Received: by mail-qv1-f53.google.com with SMTP id 6a1803df08f44-6e90b8d4686so19070616d6.2 for ; Fri, 11 Apr 2025 16:26:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1744413988; x=1745018788; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=w1i3yzxZd1yu8PCFMzrOIMo/zRGQEeeeMxBao/sAoy8=; b=xPsJJgRzkvak7HF0GQ9mdyvXzsXV/LiAGDUT+rIFkmX4z34VA0Xai+LGFwPZTDKxD/ 5Om4QfYtUSYzzC5GyztJhdjnZx7vm+ohHdAPTbwETxx1o8cLND+AFwTarGRH2bReGZgP Appbkjz6yNE/6pf5C9nHq8vczDBLBjn2w0RKnNKMIci/ls8WtsyT1pFW7y7a/aS9PDYa BiLQSGhUGfOE95+Qsjt71gkK20HmSLym3LZLhhT1jmMbf+srmKOFroPdKU76cgmlljiX RdyxqSRW5bpeBoRoOCUY2DFbcgbu2M3AtYK5jeHGRXdt1OIJM5CSi10Kmvh6RZlUAsnV fJ3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744413988; x=1745018788; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=w1i3yzxZd1yu8PCFMzrOIMo/zRGQEeeeMxBao/sAoy8=; b=KfRvyL4EvSc4uYNzvaTpN6USA7ZRdypZcfovHvEBOnqL3eEoTkOKGlYu7ixi49sa9u 7Nu76JTfHglMNzmTabM8HxQrvSgI9x2l0ZyKgt/2SRhS9u6Hm2iqTKWx/4XlJhVPEIhZ txSH4GoRqKv93/ayH3+P5Xf5T5DTjt5lR3nxYjKwC3LrxgWiYFu8fSK1pDk1HWXCEQrk fYophw4JLT0KIPWA7yrpncjU/lUnEdjXWkC0p111FmTkejMi6mwf78aDkB781bqbZ231 T2gfXCek+iUXyO+Kc3zI1o2MgGqZ3phrTn+7IQKHJHdVlEhCCB3DsPV51uZjjfAfF33H okMg== X-Gm-Message-State: AOJu0YxsP+P6a6ZWyEw+r+n/SOjoItGjRnOO3qjWOjKld1jGOubBRtoa lC2yQwnA9NjRHtkpBuKe/0HZ9jMXQWgBZkbyDmDLhKHTZNzK9DG3e/FyqIX+mml5/Pra5p7eKbv tXzQ= X-Gm-Gg: ASbGncvVOY1fv+C1goH9hY+RFfXA2Zbj3uFwgm8JHSOl1uCIpAQ3043f8CMalJH7mmr rpoFndx+2Zln7wK2MrTaIzZr7q8VVsWQgb9REe9HfPIhItaOWeq0l3ARKg/aNu78YmETLVrSz8O kj04Ym6ZjcBiVKsAriMSP6q5i1ASIY6BKthOCVIimCcA40UftFN7qy45y33ba+oxdtKbTiczecO yvxN2AYLCy1L5Zab1UdKv3SPoB3f3eIwrApzaV2Epx6Ia5Mr0O8fSTRhv1u7F01R5iCQt+vk7FS RobfczktO0RG3cKxgeQ5ciRBaIcTdPwLHBWQSErGx9DEdlKXcRg3x01nbatc/83s1u9v84kIN03 W0TInJ1ZswMRP X-Google-Smtp-Source: AGHT+IH3PR9TgyaxV5DksPqMvOpVqQwpIsBdC+FHppbey1R+uofS/IN45V8qbMKFLNVBPNbCXBRhzQ== X-Received: by 2002:a05:6214:d08:b0:6eb:2f30:55cd with SMTP id 6a1803df08f44-6f23f1dd77emr69803036d6.45.1744413987706; Fri, 11 Apr 2025 16:26:27 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with UTF8SMTPSA id 6a1803df08f44-6f0dea06e15sm42771646d6.83.2025.04.11.16.26.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Apr 2025 16:26:27 -0700 (PDT) Date: Fri, 11 Apr 2025 19:26:26 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Elijah Newren , Jeff King , Junio C Hamano Subject: [RFC PATCH 3/8] pack-objects: factor out handling '--stdin-packs' Message-ID: <43e889b15762b2aed65d0c5fbaef19bcb65e8f14.1744413969.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: At the bottom of cmd_pack_objects() we check which mode the command is running in (e.g., generating a cruft pack, handling '--stdin-packs', using the internal rev-list, etc.) and handle the mode appropriately. The '--stdin-packs' case is handled inline (dating back to its introduction in 339bce27f4 (builtin/pack-objects.c: add '--stdin-packs' option, 2021-02-22)) since it is relatively short. Extract the body of "if (stdin_packs)" into its own function to prepare for the implementation to become lengthier in a following commit. Signed-off-by: Taylor Blau --- builtin/pack-objects.c | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 540e5eba9e..793d245721 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -3672,6 +3672,17 @@ static void read_packs_list_from_stdin(void) string_list_clear(&exclude_packs, 0); } +static void add_unreachable_loose_objects(void); + +static void read_stdin_packs(int rev_list_unpacked) +{ + /* avoids adding objects in excluded packs */ + ignore_packed_keep_in_core = 1; + read_packs_list_from_stdin(); + if (rev_list_unpacked) + add_unreachable_loose_objects(); +} + static void add_cruft_object_entry(const struct object_id *oid, enum object_type type, struct packed_git *pack, off_t offset, const char *name, uint32_t mtime) @@ -3767,7 +3778,6 @@ static void mark_pack_kept_in_core(struct string_list *packs, unsigned keep) } } -static void add_unreachable_loose_objects(void); static void add_objects_in_unpacked_packs(void); static void enumerate_cruft_objects(void) @@ -4773,11 +4783,7 @@ int cmd_pack_objects(int argc, progress_state = start_progress(the_repository, _("Enumerating objects"), 0); if (stdin_packs) { - /* avoids adding objects in excluded packs */ - ignore_packed_keep_in_core = 1; - read_packs_list_from_stdin(); - if (rev_list_unpacked) - add_unreachable_loose_objects(); + read_stdin_packs(rev_list_unpacked); } else if (cruft) { read_cruft_objects(); } else if (!use_internal_rev_list) { From patchwork Fri Apr 11 23:26:29 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 14048823 Received: from mail-qv1-f54.google.com (mail-qv1-f54.google.com [209.85.219.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3B1DC2BE7D5 for ; Fri, 11 Apr 2025 23:26:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744413993; cv=none; b=LaCVDas1uIVSLvdD9fTwSPz3SNpKFjsHJEXTzHoaYmLT6y6lPAzKtkM6ccdGIcKcKQXvHXEPKEBhReVTUQGrCCNat3IJtfhhVu0gFOXJwlbYi4XorA45osc2pmz3AmqxSz5uA7sEKddGLOXHzuPXQ5ZVgO/QqTnWF5yCXvzGUWM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744413993; c=relaxed/simple; bh=c24m+17HE1RXJ10noVgS/Q5FOmgJYbXBD5illFWO6yI=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=BTnJqNEjp0Om0hlVYtYX0qV9CrHGNnCjw2OVJH9uvBwhLnOxwcRm+UF4/JjrQUsZ3gONVsLzlSHO+C/kAeLVAm2EU2I9tFHBSX1Vea6Zl5sP6OZXbmKMCj19OGeUZabJ5l399w9mVCpt1nKy4X0Kw8O0Dyh5TMc2exdZlIWZo8k= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=pass smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=K8nBOKFl; arc=none smtp.client-ip=209.85.219.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="K8nBOKFl" Received: by mail-qv1-f54.google.com with SMTP id 6a1803df08f44-6e8f6970326so24100436d6.0 for ; Fri, 11 Apr 2025 16:26:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1744413991; x=1745018791; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=OJU7WfGVi/UhdTa13q43AKA4Fob2sPhv2kiSw+epyWI=; b=K8nBOKFltF9umyzcxvNF39c4XDBkKish3wxPKL8OOuSwombE56oifyYj8gUZHSCXre ux3LaRQJmTI7W528MHHlxtRIFdHvEPtAMV8vWmdPHaHGJb2VrksFv8aRcfGN7xoqp+MC k99Z11dIvXOUZs/z2A23Ff07vrG6nz//ps5cvUITimem4nmu+h5QnyBxPBTjOqNRTe3M AQFO+ILxyVEp6XvMu2U5UFGOurGbp/0ZC4LaV4fK/OZUSqY12YlfYeQjbjYWu2q4iTkz VkzM9ur2n/Xt0BDRNf9wOlkR5WlwHqlR7YOHgJNm/ENM9auQKlcQHhR1XonOjCkcDstx kjRA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744413991; x=1745018791; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=OJU7WfGVi/UhdTa13q43AKA4Fob2sPhv2kiSw+epyWI=; b=Bb3cp06ZGIWtPOzSwRWoUsRyZgmGo1MaU0C8uKS9KkSxA/PpP78I0w2+iwTzmXQOU8 ySYosLLqg0YFYsOqngcSCB20IpO1dA5ckbmrxtJQrnVnnKKozAmudp66B90HCiPRigZp 9/AKzWDQoGoj0BKxIfmOYpOHWFPAV+TPRYq/KpPh4+yFE2oTVLH+KlgKk5yw7sbtSDjJ K5yc9BRRIC1RmHzCaBbU6Vf7OZSMMqhZb65l+9TurGMDu7dce4y3lrNcLjWzMu1zKfc4 +QOqjqx6nhq2nUbisMGutMaUSx3M25Pb6Y+xiBsjsXlJZ3Vh79G44RnFOteKKdJZAgab P4SA== X-Gm-Message-State: AOJu0YyiIm7jG2EOqP0XziiGxnVOcgancJA3q5KeZHDUdOiAcuzZX72+ BvQcg2uSUxwknGe2bbJxsUhmvN04L1JMzReavd4m5azwosR40uZERnAiqTfK79dHYE4mETOQV5x PGcg= X-Gm-Gg: ASbGncsxemU0rlM87+iW8DzSamoBozXiS7C6atn96ucaYpIN80/AVpKOuIYbc7h8I6V Iz6IBVrMLneGP4jDuwnkvr1HZdSBB1UpYQsLk2xREl+X3uqVcZL0my7znIy6ev4V+LAkG0V7wMn T9Wfyx0dsPs8XlcUVw0/DKm07kXMrXTkU4NHfPfkYERYxDamm0ETR8b44rfQwDnkKeeLCCgZoBg 8r9G8WrxiRErRoU2PqwtlWBv0oOBuZwXhJsAbRH0Ec25S6IxlUmizaEFRhHjE/z/EVZN+d2ZV+C 70a0+zP0Rev/CCwfUJOy+KKcT9Huiwj2tsOfRhHJDGVS+TjisHC4meCWO0ClEZk47RG/WCJIJVR fVFWckC2pfCyE X-Google-Smtp-Source: AGHT+IGPS115Cmr4KUTW68oq6tHPrsumhkSkGLJtAe4Kr0DGy+s9Au3HSEUde31TWNaWJoMUBia8LQ== X-Received: by 2002:a05:6214:5003:b0:6ea:d49f:ddeb with SMTP id 6a1803df08f44-6f23f137ee8mr78976896d6.31.1744413990761; Fri, 11 Apr 2025 16:26:30 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with UTF8SMTPSA id 6a1803df08f44-6f0de95f99bsm43628556d6.25.2025.04.11.16.26.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Apr 2025 16:26:30 -0700 (PDT) Date: Fri, 11 Apr 2025 19:26:29 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Elijah Newren , Jeff King , Junio C Hamano Subject: [RFC PATCH 4/8] pack-objects: declare 'rev_info' for '--stdin-packs' earlier Message-ID: <07a91be3ec07183d440e5ba6d579995d63c6ef85.1744413969.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Once 'read_packs_list_from_stdin()' has called for_each_object_in_pack() on each of the input packs, we do a reachability traversal to discover names for any objects we picked up so we can generate name hash values and hopefully get higher quality deltas as a result. A future commit will change the purpose of this reachability traversal to find and pack objects which are reachable from commits in the input packs, but are packed in an unknown (not included nor excluded) pack. Extract the code which initializes and performs the reachability traversal to take place in the caller, not the callee, which prepares us to share this code for the '--unpacked' case (see the function add_unreachable_loose_objects() for more details). Signed-off-by: Taylor Blau --- builtin/pack-objects.c | 71 +++++++++++++++++++++--------------------- 1 file changed, 36 insertions(+), 35 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 793d245721..1689cddd3a 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -3556,7 +3556,7 @@ static int pack_mtime_cmp(const void *_a, const void *_b) return 0; } -static void read_packs_list_from_stdin(void) +static void read_packs_list_from_stdin(struct rev_info *revs) { struct strbuf buf = STRBUF_INIT; struct string_list include_packs = STRING_LIST_INIT_DUP; @@ -3564,24 +3564,6 @@ static void read_packs_list_from_stdin(void) struct string_list_item *item = NULL; struct packed_git *p; - struct rev_info revs; - - repo_init_revisions(the_repository, &revs, NULL); - /* - * Use a revision walk to fill in the namehash of objects in the include - * packs. To save time, we'll avoid traversing through objects that are - * in excluded packs. - * - * That may cause us to avoid populating all of the namehash fields of - * all included objects, but our goal is best-effort, since this is only - * an optimization during delta selection. - */ - revs.no_kept_objects = 1; - revs.keep_pack_cache_flags |= IN_CORE_KEEP_PACKS; - revs.blob_objects = 1; - revs.tree_objects = 1; - revs.tag_objects = 1; - revs.ignore_missing_links = 1; while (strbuf_getline(&buf, stdin) != EOF) { if (!buf.len) @@ -3651,10 +3633,44 @@ static void read_packs_list_from_stdin(void) struct packed_git *p = item->util; for_each_object_in_pack(p, add_object_entry_from_pack, - &revs, + revs, FOR_EACH_OBJECT_PACK_ORDER); } + strbuf_release(&buf); + string_list_clear(&include_packs, 0); + string_list_clear(&exclude_packs, 0); +} + +static void add_unreachable_loose_objects(void); + +static void read_stdin_packs(int rev_list_unpacked) +{ + struct rev_info revs; + + repo_init_revisions(the_repository, &revs, NULL); + /* + * Use a revision walk to fill in the namehash of objects in the include + * packs. To save time, we'll avoid traversing through objects that are + * in excluded packs. + * + * That may cause us to avoid populating all of the namehash fields of + * all included objects, but our goal is best-effort, since this is only + * an optimization during delta selection. + */ + revs.no_kept_objects = 1; + revs.keep_pack_cache_flags |= IN_CORE_KEEP_PACKS; + revs.blob_objects = 1; + revs.tree_objects = 1; + revs.tag_objects = 1; + revs.ignore_missing_links = 1; + + /* avoids adding objects in excluded packs */ + ignore_packed_keep_in_core = 1; + read_packs_list_from_stdin(&revs); + if (rev_list_unpacked) + add_unreachable_loose_objects(); + if (prepare_revision_walk(&revs)) die(_("revision walk setup failed")); traverse_commit_list(&revs, @@ -3666,21 +3682,6 @@ static void read_packs_list_from_stdin(void) stdin_packs_found_nr); trace2_data_intmax("pack-objects", the_repository, "stdin_packs_hints", stdin_packs_hints_nr); - - strbuf_release(&buf); - string_list_clear(&include_packs, 0); - string_list_clear(&exclude_packs, 0); -} - -static void add_unreachable_loose_objects(void); - -static void read_stdin_packs(int rev_list_unpacked) -{ - /* avoids adding objects in excluded packs */ - ignore_packed_keep_in_core = 1; - read_packs_list_from_stdin(); - if (rev_list_unpacked) - add_unreachable_loose_objects(); } static void add_cruft_object_entry(const struct object_id *oid, enum object_type type, From patchwork Fri Apr 11 23:26:32 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 14048824 Received: from mail-qk1-f173.google.com (mail-qk1-f173.google.com [209.85.222.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5EEBF2BD5A0 for ; Fri, 11 Apr 2025 23:26:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744413998; cv=none; b=iWZp/vShQvJAcZekBVbynEURc0aZPGR+Je8jfTvy0B5J6+bZuwgfJoQ06oYacZGMMiQIr3Kidek7PXzh2OCq/LQx9huRdy+cpV9GGI+97GycuxyQhKUfk1BEVqrZSDirkqUqGpmaHj+PfqOBDYtWQRXVNRLQ4qui5bS8LV2w1vI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744413998; c=relaxed/simple; bh=03m2k/L/wccVCoTux+ddNl66+4qbLVpJXlgNog235u4=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=dZyqtAaYBvDroNxATgJJsfKQbKgNVGQ/4c5AULM2g13hULaiaEt/1n4dsuvLMExhOVQnqJkkQKauAWVTyAV6ib7tGffCNPEOvff+PLmNMKYDos3N1vjLNfFQLCUcr2DqrVjn2R9gUeFMVnCMzz2X/syl1rXB5/iOpT/aEml48ck= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=pass smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=DkzsdWaz; arc=none smtp.client-ip=209.85.222.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="DkzsdWaz" Received: by mail-qk1-f173.google.com with SMTP id af79cd13be357-7c5b2472969so258522185a.1 for ; Fri, 11 Apr 2025 16:26:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1744413994; x=1745018794; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=52sRGsH+u5pDHgJMEAlwVbs/+v0K18dKhSGwVtF+xyU=; b=DkzsdWazTe3sWVb0wYvBZnZEsXEYXLeQ/euAbe+qtmglrFub2X1oxQrWONiHbNiEg/ 7htJEbGF7m3Nl+g4N+mkDoF7sOgUkYSPp24fDjpBWiLbzS/R7TpenQBTuFLyCPhn/gwr Kw4qt8EG40MbKz8V5/ALBqbeuOp+fyx+RmhQCAghaczCKLvbeWuvkDfyLodd90bCPLxf ua8Wdv14m4kP8UPDiYcAOPDrFbe2ge/ukoCrX26TchuZkkU6eItpLutcNzBpT4N0SmxL jZiwZ/hpgAaFskcMp+UjW0zA7MrwMW67aIphpM+vpyLjhKHPWUoDtHeIdfLlNd7cEutZ qJaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744413994; x=1745018794; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=52sRGsH+u5pDHgJMEAlwVbs/+v0K18dKhSGwVtF+xyU=; b=fLX+b0pzRtGXUtKnTGGFIiWfR9k3xeaZMF/60xAH7vcbEnlztEXAyyDO9kANPXqIVs LxLs0CGHr3JtCnbzA0aMlA9Uks0U0bQn/SvfGKEj/oozjSDsrS3y1LShaPDXsa6u8hCW f3E0nTN0oJKAQ9gHGWF6Nj7+UHfRF5gA0NBE3rFEqvwQEySz9ZgedsXiviXRfcsu8ywq QO1K3UamyfF6R9v2fMPROCQ3c6VEJZ8HGTalVabhxL+c+3vR1mxe6Z8q7z7Tw7OX5lIq nWyVW0IfSSnZzbjaLhdAdGsChkDkTUjKg0V3JtgztZt9xHy4SBpwmquRuzlVCPQiWLyd sTgA== X-Gm-Message-State: AOJu0YwHq0ybZGlnGu3pvdoI9xQEbmzg5B4EV7YmemP74Ad/TsMwlRjY egevjQHdnA8DDR49ejxpRo6Wgal9ZZQSA+ERNyIxFOmDQSuHmIlHpBdxJSAiCSpPHXdTObFHQ9k x1hE= X-Gm-Gg: ASbGnct3Y/kl7UDfW7/qQbCW8Ly6S25LlGTgOzaany8scLrpE1sol2+ucJGOuSDFvP7 ai1LKnBpzV+NXa5aKSwABu5AMUsa1Xw+i0iI/azZ0x+nYD/IA5dy9zWsiEJA3cId9Z4sKuDCI09 FfRdnApQ83ae8t7rN4D8nblyM0CxhHkMD0LXVA2PLhxGoT4Yk7xZD0U5OuixVm4AyUf6X33hsVb qMxumc2dUrqFL26H3JW50Esj5RI6c22RQja6KSMAPmAoBXwd0R++EsDZlTZEcFT2sncd6FMvnd7 gmmYZOlde7Jhw3ewWQu0XZ5KtScbpiajWd3troDf1ujgrGYLAJXW7YyS//Aayo3t+cVt6jo7wTh 3NdlBF16uGxAl X-Google-Smtp-Source: AGHT+IGXkCD/IDkF6Eg2JEnGu5+phsatASwdmstwYpAEra1cZOx2gBP+Ml4yQcm4tLQqj3gMgYo3pQ== X-Received: by 2002:a05:620a:2806:b0:7c5:4b91:6a41 with SMTP id af79cd13be357-7c7af1f41f7mr620280485a.42.1744413993748; Fri, 11 Apr 2025 16:26:33 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with UTF8SMTPSA id af79cd13be357-7c7a8a0e5efsm328457485a.108.2025.04.11.16.26.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Apr 2025 16:26:33 -0700 (PDT) Date: Fri, 11 Apr 2025 19:26:32 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Elijah Newren , Jeff King , Junio C Hamano Subject: [RFC PATCH 5/8] pack-objects: perform name-hash traversal for unpacked objects Message-ID: <241f7c87e52c4123d0e00d75f5af201bd37fcfd4.1744413969.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: With '--unpacked', pack-objects adds loose objects (which don't appear in any of the excluded packs from '--stdin-packs') to the output pack without considering them as reachability tips for the name-hash traversal. This was an oversight in the original implementation of '--stdin-packs', since the code which enumerates and adds loose objects to the output pack (`add_unreachable_loose_objects()`) did not have access to the 'rev_info' struct found in `read_packs_list_from_stdin()`. Excluding unpacked objects from that traversal doesn't effect the correctness of the resulting pack, but it does make it harder to discover good deltas for loose objects. Now that the 'rev_info' struct is declared outside of `read_packs_list_from_stdin()`, we can pass it to `add_objects_in_unpacked_packs()` and add any loose objects as tips to the above-mentioned traversal, in theory producing slightly tighter packs as a result. Signed-off-by: Taylor Blau --- builtin/pack-objects.c | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 1689cddd3a..2aa12da4af 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -3642,7 +3642,7 @@ static void read_packs_list_from_stdin(struct rev_info *revs) string_list_clear(&exclude_packs, 0); } -static void add_unreachable_loose_objects(void); +static void add_unreachable_loose_objects(struct rev_info *revs); static void read_stdin_packs(int rev_list_unpacked) { @@ -3669,7 +3669,7 @@ static void read_stdin_packs(int rev_list_unpacked) ignore_packed_keep_in_core = 1; read_packs_list_from_stdin(&revs); if (rev_list_unpacked) - add_unreachable_loose_objects(); + add_unreachable_loose_objects(&revs); if (prepare_revision_walk(&revs)) die(_("revision walk setup failed")); @@ -3788,7 +3788,7 @@ static void enumerate_cruft_objects(void) _("Enumerating cruft objects"), 0); add_objects_in_unpacked_packs(); - add_unreachable_loose_objects(); + add_unreachable_loose_objects(NULL); stop_progress(&progress_state); } @@ -4066,8 +4066,9 @@ static void add_objects_in_unpacked_packs(void) } static int add_loose_object(const struct object_id *oid, const char *path, - void *data UNUSED) + void *data) { + struct rev_info *revs = data; enum object_type type = oid_object_info(the_repository, oid, NULL); if (type < 0) { @@ -4088,6 +4089,10 @@ static int add_loose_object(const struct object_id *oid, const char *path, } else { add_object_entry(oid, type, "", 0); } + + if (revs && type == OBJ_COMMIT) + add_pending_oid(revs, NULL, oid, 0); + return 0; } @@ -4096,11 +4101,10 @@ static int add_loose_object(const struct object_id *oid, const char *path, * add_object_entry will weed out duplicates, so we just add every * loose object we find. */ -static void add_unreachable_loose_objects(void) +static void add_unreachable_loose_objects(struct rev_info *revs) { for_each_loose_file_in_objdir(repo_get_object_directory(the_repository), - add_loose_object, - NULL, NULL, NULL); + add_loose_object, NULL, NULL, revs); } static int has_sha1_pack_kept_or_nonlocal(const struct object_id *oid) @@ -4356,7 +4360,7 @@ static void get_object_list(struct rev_info *revs, int ac, const char **av) if (keep_unreachable) add_objects_in_unpacked_packs(); if (pack_loose_unreachable) - add_unreachable_loose_objects(); + add_unreachable_loose_objects(NULL); if (unpack_unreachable) loosen_unused_packed_objects(); From patchwork Fri Apr 11 23:26:35 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 14048825 Received: from mail-qv1-f48.google.com (mail-qv1-f48.google.com [209.85.219.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 47B8B2BE7A2 for ; Fri, 11 Apr 2025 23:26:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744414000; cv=none; b=YGQNdppCEjb1WbfustiU3paseqrNIQLqJpA1R67wGcFCOm8lwz/i2YeT4ggudd4FyZJM1A+KUw2QrhjiAT4DmhgPkUjtF4k9Ts3TYYv3NYfPz+Kzy1fh+4sl/zLwKWAomtqgWtUBdWqEoyLCgXjV8WulvXdO1ZXk4AyaYPo1QHc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744414000; c=relaxed/simple; bh=+cLEM0zpvYjVGGrsBnpqme76sCTexVUMzbqfcsYAgWo=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=DQzCc7GU628xZRio799nYtfgfBiT31zwnn//RCSZhTrZQRF9nGNV3Y/VUYegHmX0dUJuAP4AmovSMsRYNNqqRwXYvBPTrkISqN3bZ99oN6HVPDmt3bP+HDaipFBiOE1rG3Jm+O6izMrrelqkkAvCzdg/DPhZAj/wY+PaTCq+Oa8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=pass smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=wP6v1lwD; arc=none smtp.client-ip=209.85.219.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="wP6v1lwD" Received: by mail-qv1-f48.google.com with SMTP id 6a1803df08f44-6ecf99dd567so29576686d6.0 for ; Fri, 11 Apr 2025 16:26:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1744413997; x=1745018797; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=xBgJmMJ+G2Cje2oi/p2jzhsYwCCHJR0OCUAsfpMku6w=; b=wP6v1lwDm/YBCd/nWmVo+o6OaSUKGCtux4x6UsqdqAViDgbgJpIWFgZgt3AHLPcq/w fqTdtP0XizhvQ5UC4/kdrlDt0VYhzqgvT5rkHww+9q3xObb/Pxn2/EY303YTNUp6kUhW xAUVi+IXvAd1QxZUOUQWb1FmdMPfRHe10uWh73brNl8cZV6UKrVMnNM3cYGNmB4IiA23 d26f1sOBycrJ/GBnbX9q0FZzsQPtuJBmtNOtv16BcXd4fjLM5VShE4T1bUUhWLoTwUuG p024fvwB+MWELtAVm2/ua3cmz5UwnHMy8MGxceyPeNAnGBifaUbqylcfoO7T7HezNHtC 7lvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744413997; x=1745018797; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=xBgJmMJ+G2Cje2oi/p2jzhsYwCCHJR0OCUAsfpMku6w=; b=GCReZYwTP8uLQcozPrjUYYqfp66y/D0wfJnSNfjUGVI6TAdriF7Twiwk03IfG8Vnxt E8ymoFYUCiK0QdyJd9w07mFqzxnNkKI8ltzAxtqnUSOzIq02L1g2dcaPs+2CJs0E1OvU 0QBDe9cAA8ziz3pglX7WVyaTG7teuSLLQdHlm+Zz1DAj0/ofijoaesrULU7lqe82V9Xz 7kAD5E32PVurT4ykPzy7z84kiU4+5V2gCj0veYM8nnLLSXP+SWymwamBgagGNjWruAHI AYz1EbOAsAOJv/zgyXb7HQ7qV8KSADAZZKA6PFT7BAiT55Vd4qAfREBrslMwLlTUKDN3 4VUA== X-Gm-Message-State: AOJu0Yzik62hyiAc2+3lhFYm4vt/FlfUZRQXKDvlSdCgv0CIhRCWv+2Y qZWZQtspeG3PNYDAaBgWR1tGm/XmMWXjac6SYSy/c2yhclr0PN+BFI7PVK4WuBZWU49ctV91G2D gDj0= X-Gm-Gg: ASbGnctUs8wSj1hI6PGs6Xc/ub8E4jeqz9DarRcQ5tlOekfVJcR/xesi/mT3EPfvK8o psOdVmUWI/NHcxUEozofIurHHZPcf3fFjoYU8dlTwRSm3WCAmU9h78//0CppmqVUAp8muxYEfHR p5YIXVNDKm0pDp8NnB9sdWX0PJtsJvRDSTzuq0Y8GR11o/O/YROlXU6joBuvjT9yHRrcNEVSW/A qgOGs8ukAb8PNSv069qrCM7xUFUwYiDH3kfZ8OYupn+bD6c+Ff6+vJwJO0RgnG95c9u6ajrMGrg LrILGJtNLCLiIY174wAxXOT6N0r2oMsbGJ8nwRUjf3bTZec21cVMnRo4eOKxeLpmbGQbXuWqIld FbGE01IhTC2JOqh6JuY8Rx0o= X-Google-Smtp-Source: AGHT+IEt3kOa11bg+vYyhhBDITTnCmcYLK08DC0WvNKTXt82qM+DCS2/nLz3aNwN719R+6t2MHqXVg== X-Received: by 2002:ad4:5bc9:0:b0:6e8:fee2:aadb with SMTP id 6a1803df08f44-6f230dc06ffmr75618506d6.27.1744413996678; Fri, 11 Apr 2025 16:26:36 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with UTF8SMTPSA id 6a1803df08f44-6f0de95fd90sm42752916d6.11.2025.04.11.16.26.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Apr 2025 16:26:36 -0700 (PDT) Date: Fri, 11 Apr 2025 19:26:35 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Elijah Newren , Jeff King , Junio C Hamano Subject: [RFC PATCH 6/8] pack-objects: introduce '--stdin-packs=follow' Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: When invoked with '--stdin-packs', pack-objects will generate a pack which contains the objects found in the "included" packs, less any objects from "excluded" packs. Packs that exist in the repository but weren't specified as either included or excluded are in practice treated like the latter, at least in the sense that pack-objects won't include objects from those packs. This behavior forces us to include any cruft pack(s) in a repository's multi-pack index for the reasons described in ddee3703b3 (builtin/repack.c: add cruft packs to MIDX during geometric repack, 2022-05-20). The full details are in ddee3703b3, but the gist is if you have a once-unreachable object in a cruft pack which later becomes reachable via one or more commits in a pack generated with '--stdin-packs', you *have* to include that object in the MIDX via the copy in the cruft pack, otherwise we cannot generate reachability bitmaps for any commits which reach that object. This prepares us for new repacking behavior which will "resurrect" objects found in cruft or otherwise unspecified packs when generating new packs. In the context of geometric repacking, this may be used to maintain a sequence of geometrically-repacked packs, the union of which is closed under reachability, even in the case described earlier. Signed-off-by: Taylor Blau --- Documentation/git-pack-objects.adoc | 8 ++- builtin/pack-objects.c | 89 +++++++++++++++++------- t/t5331-pack-objects-stdin.sh | 101 ++++++++++++++++++++++++++++ 3 files changed, 171 insertions(+), 27 deletions(-) diff --git a/Documentation/git-pack-objects.adoc b/Documentation/git-pack-objects.adoc index 7f69ae4855..c894582799 100644 --- a/Documentation/git-pack-objects.adoc +++ b/Documentation/git-pack-objects.adoc @@ -87,13 +87,19 @@ base-name:: reference was included in the resulting packfile. This can be useful to send new tags to native Git clients. ---stdin-packs:: +--stdin-packs[=]:: Read the basenames of packfiles (e.g., `pack-1234abcd.pack`) from the standard input, instead of object names or revision arguments. The resulting pack contains all objects listed in the included packs (those not beginning with `^`), excluding any objects listed in the excluded packs (beginning with `^`). + +When `mode` is "follow", pack objects which are reachable from objects +in the included packs, but appear in packs that are not listed. +Reachable objects which appear in excluded packs are not packed. Useful +for resurrecting once-cruft objects to generate packs which are closed +under reachability up to the excluded packs. ++ Incompatible with `--revs`, or options that imply `--revs` (such as `--all`), with the exception of `--unpacked`, which is compatible. diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 2aa12da4af..6406f4a5b1 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -272,6 +272,12 @@ static struct oidmap configured_exclusions; static struct oidset excluded_by_config; static int name_hash_version = -1; +enum stdin_packs_mode { + STDIN_PACKS_MODE_NONE, + STDIN_PACKS_MODE_STANDARD, + STDIN_PACKS_MODE_FOLLOW, +}; + /** * Check whether the name_hash_version chosen by user input is appropriate, * and also validate whether it is compatible with other features. @@ -3511,32 +3517,43 @@ static int add_object_entry_from_pack(const struct object_id *oid, return 0; } -static void show_commit_pack_hint(struct commit *commit UNUSED, - void *data UNUSED) -{ - /* nothing to do; commits don't have a namehash */ -} - static void show_object_pack_hint(struct object *object, const char *name, - void *data UNUSED) + void *data) { - struct object_entry *oe = packlist_find(&to_pack, &object->oid); - if (!oe) + enum stdin_packs_mode mode = *(enum stdin_packs_mode *)data; + if (mode == STDIN_PACKS_MODE_FOLLOW) { + add_object_entry(&object->oid, object->type, name, 0); + } else { + struct object_entry *oe = packlist_find(&to_pack, &object->oid); + if (!oe) + return; + + /* + * Our 'to_pack' list was constructed by iterating all + * objects packed in included packs, and so doesn't + * have a non-zero hash field that you would typically + * pick up during a reachability traversal. + * + * Make a best-effort attempt to fill in the ->hash + * and ->no_try_delta here using a now in order to + * perhaps improve the delta selection process. + */ + oe->hash = pack_name_hash_fn(name); + oe->no_try_delta = name && no_try_delta(name); + + stdin_packs_hints_nr++; + } +} + +static void show_commit_pack_hint(struct commit *commit, void *data) +{ + enum stdin_packs_mode mode = *(enum stdin_packs_mode *)data; + if (mode == STDIN_PACKS_MODE_FOLLOW) { + show_object_pack_hint((struct object *)commit, "", data); return; + } + /* nothing to do; commits don't have a namehash */ - /* - * Our 'to_pack' list was constructed by iterating all objects packed in - * included packs, and so doesn't have a non-zero hash field that you - * would typically pick up during a reachability traversal. - * - * Make a best-effort attempt to fill in the ->hash and ->no_try_delta - * here using a now in order to perhaps improve the delta selection - * process. - */ - oe->hash = pack_name_hash_fn(name); - oe->no_try_delta = name && no_try_delta(name); - - stdin_packs_hints_nr++; } static int pack_mtime_cmp(const void *_a, const void *_b) @@ -3644,7 +3661,7 @@ static void read_packs_list_from_stdin(struct rev_info *revs) static void add_unreachable_loose_objects(struct rev_info *revs); -static void read_stdin_packs(int rev_list_unpacked) +static void read_stdin_packs(enum stdin_packs_mode mode, int rev_list_unpacked) { struct rev_info revs; @@ -3676,7 +3693,7 @@ static void read_stdin_packs(int rev_list_unpacked) traverse_commit_list(&revs, show_commit_pack_hint, show_object_pack_hint, - NULL); + &mode); trace2_data_intmax("pack-objects", the_repository, "stdin_packs_found", stdin_packs_found_nr); @@ -4467,6 +4484,23 @@ static int is_not_in_promisor_pack(struct commit *commit, void *data) { return is_not_in_promisor_pack_obj((struct object *) commit, data); } +static int parse_stdin_packs_mode(const struct option *opt, const char *arg, + int unset) +{ + enum stdin_packs_mode *mode = opt->value; + + if (unset) + *mode = STDIN_PACKS_MODE_NONE; + else if (!arg || !*arg) + *mode = STDIN_PACKS_MODE_STANDARD; + else if (!strcmp(arg, "follow")) + *mode = STDIN_PACKS_MODE_FOLLOW; + else + die(_("invalid value for '%s': '%s'"), opt->long_name, arg); + + return 0; +} + int cmd_pack_objects(int argc, const char **argv, const char *prefix, @@ -4478,7 +4512,7 @@ int cmd_pack_objects(int argc, struct strvec rp = STRVEC_INIT; int rev_list_unpacked = 0, rev_list_all = 0, rev_list_reflog = 0; int rev_list_index = 0; - int stdin_packs = 0; + enum stdin_packs_mode stdin_packs = STDIN_PACKS_MODE_NONE; struct string_list keep_pack_list = STRING_LIST_INIT_NODUP; struct list_objects_filter_options filter_options = LIST_OBJECTS_FILTER_INIT; @@ -4533,6 +4567,9 @@ int cmd_pack_objects(int argc, OPT_SET_INT_F(0, "indexed-objects", &rev_list_index, N_("include objects referred to by the index"), 1, PARSE_OPT_NONEG), + OPT_CALLBACK_F(0, "stdin-packs", &stdin_packs, N_("mode"), + N_("read packs from stdin"), + PARSE_OPT_OPTARG, parse_stdin_packs_mode), OPT_BOOL(0, "stdin-packs", &stdin_packs, N_("read packs from stdin")), OPT_BOOL(0, "stdout", &pack_to_stdout, @@ -4788,7 +4825,7 @@ int cmd_pack_objects(int argc, progress_state = start_progress(the_repository, _("Enumerating objects"), 0); if (stdin_packs) { - read_stdin_packs(rev_list_unpacked); + read_stdin_packs(stdin_packs, rev_list_unpacked); } else if (cruft) { read_cruft_objects(); } else if (!use_internal_rev_list) { diff --git a/t/t5331-pack-objects-stdin.sh b/t/t5331-pack-objects-stdin.sh index 4f5e2733a2..f97d2d1b71 100755 --- a/t/t5331-pack-objects-stdin.sh +++ b/t/t5331-pack-objects-stdin.sh @@ -236,4 +236,105 @@ test_expect_success 'pack-objects --stdin with packfiles from main and alternate test_cmp expected-objects actual-objects ' +packdir=.git/objects/pack + +objects_in_packs () { + for p in "$@" + do + git show-index <"$packdir/pack-$p.idx" || return 1 + done >objects.raw && + + cut -d' ' -f2 objects.raw | sort && + rm -f objects.raw +} + +test_expect_success 'setup for --stdin-packs=follow' ' + git init stdin-packs--follow && + ( + cd stdin-packs--follow && + + for c in A B C D + do + test_commit "$c" || return 1 + done && + + A="$(echo A | git pack-objects --revs $packdir/pack)" && + B="$(echo A..B | git pack-objects --revs $packdir/pack)" && + C="$(echo B..C | git pack-objects --revs $packdir/pack)" && + D="$(echo C..D | git pack-objects --revs $packdir/pack)" && + + git prune-packed + ) +' + +test_expect_success '--stdin-packs=follow walks into unknown packs' ' + test_when_finished "rm -fr repo" && + + git init repo && + ( + cd repo && + + for c in A B C D + do + test_commit "$c" || return 1 + done && + + A="$(echo A | git pack-objects --revs $packdir/pack)" && + B="$(echo A..B | git pack-objects --revs $packdir/pack)" && + C="$(echo B..C | git pack-objects --revs $packdir/pack)" && + D="$(echo C..D | git pack-objects --revs $packdir/pack)" && + + git prune-packed && + + cat >in <<-EOF && + pack-$B.pack + ^pack-$C.pack + pack-$D.pack + EOF + + # With just --stdin-packs, pack "A" is unknown to us, so + # only objects from packs "B" and "D" are included in + # the output pack. + P=$(git pack-objects --stdin-packs $packdir/pack expect && + objects_in_packs $P >actual && + test_cmp expect actual && + + # But with --stdin-packs=follow, objects from both + # included packs reach objects from the unknown pack, so + # objects from pack "A" is included in the output pack + # in addition to the above. + P=$(git pack-objects --stdin-packs=follow $packdir/pack expect && + objects_in_packs $P >actual && + test_cmp expect actual && + + test_commit E && + # And with --unpacked, we will pick up objects from unknown + # packs that are reachable from loose objects. Loose object E + # reaches objects in pack A, but there are three excluded packs + # in between. + # + # The resulting pack should include objects reachable from E + # that are not present in packs B, C, or D, along with those + # present in pack A. + cat >in <<-EOF && + ^pack-$B.pack + ^pack-$C.pack + ^pack-$D.pack + EOF + + P=$(git pack-objects --stdin-packs=follow --unpacked \ + $packdir/pack expect.raw && + sort expect.raw >expect && + objects_in_packs $P >actual && + test_cmp expect actual + ) +' + test_done From patchwork Fri Apr 11 23:26:38 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 14048826 Received: from mail-qv1-f46.google.com (mail-qv1-f46.google.com [209.85.219.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0ADD52BEC2F for ; Fri, 11 Apr 2025 23:26:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744414002; cv=none; b=LbhHJMkmdjla0cWXMF8iVTC2K4GyELTzHr2pekZS4jl6KzSg7lTnY5FjII+zzsA1UAulD6RQ6BMcMEsrzv1iqMF112cWsr0eqXrJcMnu7fikiyxUGv5pT0T8ynIvsCUh5jJNdHWM2b3B9HLGazRunYQPbzrzRiZAOJtwdO8tHOE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744414002; c=relaxed/simple; bh=uOjyfBEFUgbsE82MMrqan29h4MUfesQlHI344S4+POU=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=hzP2wCvatOBEcexVBKjulOORbiXjhNWc4E0nkVa5SgeExsifUCUmXU/tpUS580Qv/b08P0iBSmIf0mOIvXcvilttg59WkerLjdNa4XV7M9J/7XmlBH3U6+gAnFek8nWm86Jbh0DRO7EdRbhlj+mhst4Rciax0wdA1ZWmgIlubZk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=pass smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=N5W4MUNW; arc=none smtp.client-ip=209.85.219.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="N5W4MUNW" Received: by mail-qv1-f46.google.com with SMTP id 6a1803df08f44-6ecfa716ec1so23525356d6.2 for ; Fri, 11 Apr 2025 16:26:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1744414000; x=1745018800; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=q+ZuHGNzHCOrF3eSw6ZAuxYMp6vmVpe8Kpb1d1Bd1Ko=; b=N5W4MUNW18H5mtTMu+fHXQTW7GahVG1ZXgJnKqyWq7xoPEF2wM1vkKmPSyUVr4ujbh r2hJPrKb4Vaeb21Hpuem5kHLdfp/2l2i8RJkYo5oxnnhvRHLIs7gHFrN9aQ8YJhWWbnf avtfYmFPT89J0Csh4HFdNT9s4iOyua/yDeX816UbIvghdFUKFGrctRo66Krnyn8jBx3j eqA9o/9xE/KQrfrH2R9dVK8o/TlYvzXk4Omwl01qNWx5VmNoBJlxrLHpWxzUk8PuTZiX q4zpYTefB/hekkIj3G9GAsAjJ8lKq0Iwwcu4In6xU/Oo1rNy6sLb6Mwhe3jaZQRYQ11j CNfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744414000; x=1745018800; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=q+ZuHGNzHCOrF3eSw6ZAuxYMp6vmVpe8Kpb1d1Bd1Ko=; b=XdcY2USJvKsvrbUrS4KVO1Xeo/VdzUbNEE9HGAHXf1Pz2VawbZgqWDoS4zLcsqJbVF bKQRa8nS3g7hl01iacHhhheKgVAYoJXXQ9G00BGcO60J3ObhlpLPc/jyiKUi0Kha8DVI hLo4yHdlXbilH39RQl7/FtYQUBQqf9Mvn3wCtmiKasohi9UCSAm8Nv6wskTqPKjGMVNP VO2Rrf/ily6BXo+tNaG/u8u8zJ6PK8XdRJMvQihGQ9ekPYRuCdybzSKEBADf85zKozxf 4erDDGarfBPYFwTsIAFYx7KzrwNOxGOvu9cdkcNhT1zSGRQQhenrTYLnO0U3oeFw0obG FI9A== X-Gm-Message-State: AOJu0YyDVK5J5er89fcFBtS9omqyCbytaJUSi6J4kapMAos7jLCO1Rvh oWHdrYYxRQeH0kj06feC4QmmV1W6bG9XP0Q5jd3LnDnIdYzULv/gMKIlfEV3w+lMUxHry2ZX03A 0btg= X-Gm-Gg: ASbGncunwI5wQYFN4T7rXQ/1+hgrp+XKCwMZpUjRNjMU3LaTa+tvavJkhH5YUfm1Wk1 sQrS1wT9PFqhUlmBLJhEyA3pQrijb/xssOLB8pwfthPmPgwLBGu6vPy1pUCe3GGU0NDZKpIda4x 5lNM3KvC+S2djW6aL5tR+4q/SK3+kWciwlS1HRQB2wQmenFRwXWQcvSgSr/yA1p17FDqhUuWflM I5PEl/4C1oUZeyyPptCMnCk2FO2eyudcEHmHaKKjPWxltNxhLRvfeVxoOFWhC+pquvRAqFeYDva rYMNTBN4F/+FqvDeGAebzaEUfB1l5wOFqXzqZYIgdQ8H/x/sYANecD2SCkDvXR4oetnItaBMwtQ HTeAI1omGOifY X-Google-Smtp-Source: AGHT+IEKcIyRJNhqvNzarB2VNv3vaaok28U8O82rpkzVrRP1W5vsct636sPYVhhKT9I8ZsDcKcEQlw== X-Received: by 2002:a05:6214:226d:b0:6e6:6103:f708 with SMTP id 6a1803df08f44-6f23f13ae3amr77797766d6.38.1744413999601; Fri, 11 Apr 2025 16:26:39 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with UTF8SMTPSA id 6a1803df08f44-6f0dea215adsm43005696d6.120.2025.04.11.16.26.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Apr 2025 16:26:39 -0700 (PDT) Date: Fri, 11 Apr 2025 19:26:38 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Elijah Newren , Jeff King , Junio C Hamano Subject: [RFC PATCH 7/8] repack: keep track of existing MIDX'd packs Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The following commit will want to condition whether or not it generates a pack during geometric repacking with '--stdin-packs=follow' based on whether or not the existing MIDX has a cruft pack in it. Keep track of that in the 'existing_packs' struct by adding an additional flag bit to denote which packs appear in a MIDX. Signed-off-by: Taylor Blau --- builtin/repack.c | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/builtin/repack.c b/builtin/repack.c index f3330ade7b..bc47bede7b 100644 --- a/builtin/repack.c +++ b/builtin/repack.c @@ -31,6 +31,7 @@ #define DELETE_PACK 1 #define RETAIN_PACK 2 +#define PACK_IN_MIDX 4 static int pack_everything; static int delta_base_offset = 1; @@ -161,6 +162,11 @@ static int pack_is_retained(struct string_list_item *item) return (uintptr_t)item->util & RETAIN_PACK; } +static void pack_mark_in_midx(struct string_list_item *item) +{ + item->util = (void*)((uintptr_t)item->util | PACK_IN_MIDX); +} + static void mark_packs_for_deletion_1(struct string_list *names, struct string_list *list) { @@ -264,6 +270,7 @@ static void collect_pack_filenames(struct existing_packs *existing, for (p = get_all_packs(the_repository); p; p = p->next) { int i; const char *base; + struct string_list_item *item; if (!p->pack_local) continue; @@ -279,11 +286,17 @@ static void collect_pack_filenames(struct existing_packs *existing, strbuf_strip_suffix(&buf, ".pack"); if ((extra_keep->nr > 0 && i < extra_keep->nr) || p->pack_keep) - string_list_append(&existing->kept_packs, buf.buf); + item = string_list_append(&existing->kept_packs, + buf.buf); else if (p->is_cruft) - string_list_append(&existing->cruft_packs, buf.buf); + item = string_list_append(&existing->cruft_packs, + buf.buf); else - string_list_append(&existing->non_kept_packs, buf.buf); + item = string_list_append(&existing->non_kept_packs, + buf.buf); + + if (p->multi_pack_index) + pack_mark_in_midx(item); } string_list_sort(&existing->kept_packs); From patchwork Fri Apr 11 23:26:41 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 14048827 Received: from mail-qk1-f173.google.com (mail-qk1-f173.google.com [209.85.222.173]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 822772BE7B2 for ; Fri, 11 Apr 2025 23:26:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.173 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744414008; cv=none; b=PKEe7/BscStdl4XjBEhG8GfJ9ke36y6UxshPLqsuWY3VWHFg5Oyth31iGAI0aK4zB4tBdBj9H7bUQtoGHnIZ2oRnNzaOjA9rqI5LsfknioTwyhOfDMHMG7K36hMZUHyBuHvAT3J9Q1tfsU5aCcAVwKIOpVNXUzxOWtFPIo1Kx0U= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744414008; c=relaxed/simple; bh=iYTq8z4Do9InaaWs07s3vzJ15SHr+9gOn8aU8/2HXfk=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=jpZyEO/X08sKeh9v5WBJCEjTAqq9a9pBPaKWN/+E3bH0HWermQAXu0q3+0pkF0PsaWJfc1qoGXMNchGSXQlD34cWjc0cbBzwFNiX4Lzyni3yK1Os3kr3UiiDx8AmQbfbz+EieFE2jgrut7Azn5XvR7qS1lxQkurljcFu34smaDE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=pass smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=hXLxEt2N; arc=none smtp.client-ip=209.85.222.173 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="hXLxEt2N" Received: by mail-qk1-f173.google.com with SMTP id af79cd13be357-7c5e2fe5f17so246472685a.3 for ; Fri, 11 Apr 2025 16:26:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1744414003; x=1745018803; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=bUXVQ12qLeJU0lDR9nMEcVH0cv/grsLe1vQQ3MHfHs8=; b=hXLxEt2NylSUMuwGp8HmZOX9LzzrVOLJtU/g1X8P457OrMsub71w0WtMJvqBpYxxIa DJU0xur6vSY1MY2+VKH5/fHoxsPr2QV7Dlqr50nGwR9oEGGgVVVxcUEfrl2D9/JLCvxM a4dSYiNQrRRQ6vE+QA1O/6DMx3zdhGHm40QgForgHf5OO4A2lnZJFkWITh5NRFlfQaIE 5OVhn8qpB5B/ghOSGRErJs8ZdaQBZ9GlvnRr2IgR/z268dqIBeXgA3jhTMx4eH16LoGe 8kxCGpDGb+SyTqbbSt1j6YUo7K66TpylQ8XQK/2AvwQohwiI3OfnJQCFQn1hiwSJzE4d W7PQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1744414003; x=1745018803; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=bUXVQ12qLeJU0lDR9nMEcVH0cv/grsLe1vQQ3MHfHs8=; b=uqIWrDejeBaHKyttVdpF9C1UKt9LxsoWfHIM/HgSgH9Pvlz5rj0tKDyDemvULHgXX6 v5hR8M/Aa7fHE8vmOgcNcBrEKPlvQ1RPCkr7vsOjkLOcHO+N1J6eyUcf2lY5hLhDldfM vjx4kfMggCbdF/9tW6FSpLwRV2k6DqLXuwKjv+mp6UZFjt6lUm9zL8hjbNZON2pjK3Du sECKoYKjkv2jOnJEvHLIdeDcwNeSZ0sQW9t06L4Myhe2Yaek/NgkHOYNzgi6Z+R4gUHk BI7dog7jiKtJNLh9w5J+gcc8QeeDwKo39tPjqEv2rfGlN/yCzpVNi1RrH7pQHJepEpv8 94Qw== X-Gm-Message-State: AOJu0YyNBm+a0QfE0AkoKlSIL4gnXivwSubLEceISDfgEUDO7KK9GWkk 4giebeuPK5J4M1FgOsyIbBuKyNFSIS0iX+96t4vR/X4VIX/4r9VNDPK9kOlRhM0E6c7pfuHncEN zCF0= X-Gm-Gg: ASbGncvlwBf1UFmKaxOCLpHLO/RexrrXolfOnDSNNA3kfWV2Zj8MU/6N+dw1S5vb/Wb 2vyDsLi6ZzIYp6NXJkymhNuXW3zhx1ljPPH1MQIwV7hHtFxqiAALg1apdwRP8dDVg4PTGPmVlW4 YZsLdJHUjapjuoxRxoYvj6wh9XFrApxVQ1EJEHCTDK9Cfvw5pKqM1SSoPLnzB3gSOemPVP4AQ1z QWPql0/4Ef1EcWdLM19yL4YSVxpd8SA4jZCnIXnyilnyWmIS160lfmrVxrhaOsQdnkT/ewfsurb b1Pwl8C04QaRx5lhZPhaKEANrtmu7zARet/6+X3qdYqHScAHis6vJukAjDQBmcf/eu4pJpN9fzP 7fSHa/KsEYk9QG6rhmPyPSU8= X-Google-Smtp-Source: AGHT+IFlIA5QnoipZ9sGYi3KjbbcwGlYX8WPoi6NkIc+b452Cb1U+KgrAPVuzVoXlGvQQxfKFqmSog== X-Received: by 2002:a05:620a:444e:b0:7c7:a543:dcfe with SMTP id af79cd13be357-7c7af0e2360mr671924985a.32.1744414002714; Fri, 11 Apr 2025 16:26:42 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with UTF8SMTPSA id af79cd13be357-7c7a89f95b3sm327498685a.84.2025.04.11.16.26.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Apr 2025 16:26:42 -0700 (PDT) Date: Fri, 11 Apr 2025 19:26:41 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Elijah Newren , Jeff King , Junio C Hamano Subject: [RFC PATCH 8/8] repack: exclude cruft pack(s) from the MIDX where possible Message-ID: <19b69c124696c31c04a09f56ef03bf8ca9c61c4f.1744413969.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: In ddee3703b3 (builtin/repack.c: add cruft packs to MIDX during geometric repack, 2022-05-20), repack began adding cruft pack(s) to the MIDX with '--write-midx' to ensure that the resulting MIDX was always closed under reachability in order to generate reachability bitmaps. Suppose you have a once-unreachable object packed in a cruft pack, which later on becomes reachable from one or more objects in a geometrically repacked pack. That once-unreachable object *won't* appear in the new pack, since the cruft pack was specified as neither included nor excluded to 'pack-objects --stdin-packs'. If the bitmap selection process picks one or more commits which reach the once-unreachable objects, commit ddee3703b3 ensures that the MIDX will be closed under reachability. Without it, we would fail to generate a MIDX bitmap. ddee3703b3 alludes to the fact that this is sub-optimal by saying [...] it's desirable to avoid including cruft packs in the MIDX because it causes the MIDX to store a bunch of objects which are likely to get thrown away. , which is true, but hides an even larger problem. If repositories rarely prune their unreachable objects and/or have many of them, the MIDX must keep track of a large number of objects which bloats the MIDX and slows down object lookup. This is doubly unfortunate because the vast majority of objects in cruft pack(s) are unlikely to be read, but object reads that go through the MIDX have to search through them anyway. This patch causes geometrically-repacked packs to contain a copy of any once-unreachable object(s) with 'git pack-objects --stdin-packs=follow', allowing us to avoid including any cruft packs in the MIDX. This is because a sequence of geometrically-repacked packs that were all generated with '--stdin-packs=follow' are guaranteed to have their union be closed under reachability. Note that you cannot guarantee that a collection of packs is closed under reachability if not all of them were generated with following as above. One tell-tale sign that not all geometrically-repacked packs in the MIDX were generated with following is to see if there is a cruft pack already in the MIDX. If there is, then starting to generate packs with following during geometric repacking won't work, since it's open to the same race as described above. But if you're starting from scratch (e.g., building the first MIDX after an all-into-one '--cruft' repack), then you can guarantee that the union of subsequently generated packs from geometric repacking *is* closed under reachability. Detect when this is the case and avoid including cruft packs in the MIDX where possible. Signed-off-by: Taylor Blau --- builtin/repack.c | 78 ++++++++++++++++++++++++++++++++--------- t/t7704-repack-cruft.sh | 70 ++++++++++++++++++++++++++++++++++++ 2 files changed, 131 insertions(+), 17 deletions(-) diff --git a/builtin/repack.c b/builtin/repack.c index bc47bede7b..d0b88f12f6 100644 --- a/builtin/repack.c +++ b/builtin/repack.c @@ -167,6 +167,21 @@ static void pack_mark_in_midx(struct string_list_item *item) item->util = (void*)((uintptr_t)item->util | PACK_IN_MIDX); } +static int pack_is_in_midx(struct string_list_item *item) +{ + return (uintptr_t)item->util & PACK_IN_MIDX; +} + +static int existing_has_cruft_in_midx(struct existing_packs *existing) +{ + struct string_list_item *item; + for_each_string_list_item(item, &existing->cruft_packs) { + if (pack_is_in_midx(item)) + return 1; + } + return 0; +} + static void mark_packs_for_deletion_1(struct string_list *names, struct string_list *list) { @@ -821,26 +836,52 @@ static void midx_included_packs(struct string_list *include, } } - for_each_string_list_item(item, &existing->cruft_packs) { + if (existing_has_cruft_in_midx(existing)) { /* - * When doing a --geometric repack, there is no need to check - * for deleted packs, since we're by definition not doing an - * ALL_INTO_ONE repack (hence no packs will be deleted). - * Otherwise we must check for and exclude any packs which are - * enqueued for deletion. + * If we had one or more cruft pack(s) present in the + * MIDX before the repack, keep them as they may be + * required to form a reachability closure if the MIDX + * is bitmapped. * - * So we could omit the conditional below in the --geometric - * case, but doing so is unnecessary since no packs are marked - * as pending deletion (since we only call - * `mark_packs_for_deletion()` when doing an all-into-one - * repack). + * A cruft pack can be required to form a reachability + * closure if the MIDX is bitmapped and one or more of + * its selected commits reaches a once-cruft object that + * was later made reachable. */ - if (pack_is_marked_for_deletion(item)) - continue; + for_each_string_list_item(item, &existing->cruft_packs) { + /* + * When doing a --geometric repack, there is no + * need to check for deleted packs, since we're + * by definition not doing an ALL_INTO_ONE + * repack (hence no packs will be deleted). + * Otherwise we must check for and exclude any + * packs which are enqueued for deletion. + * + * So we could omit the conditional below in the + * --geometric case, but doing so is unnecessary + * since no packs are marked as pending + * deletion (since we only call + * `mark_packs_for_deletion()` when doing an + * all-into-one repack). + */ + if (pack_is_marked_for_deletion(item)) + continue; - strbuf_reset(&buf); - strbuf_addf(&buf, "%s.idx", item->string); - string_list_insert(include, buf.buf); + strbuf_reset(&buf); + strbuf_addf(&buf, "%s.idx", item->string); + string_list_insert(include, buf.buf); + } + } else { + /* + * Modern versions of Git will write new copies of + * once-cruft objects when doing a --geometric repack. + * + * If the MIDX has no cruft pack, new packs written + * during a --geometric repack will not rely on the + * cruft pack to form a reachability closure, so we can + * avoid including them in the MIDX in that case. + */ + ; } strbuf_release(&buf); @@ -1369,7 +1410,10 @@ int cmd_repack(int argc, !(pack_everything & PACK_CRUFT)) strvec_push(&cmd.args, "--pack-loose-unreachable"); } else if (geometry.split_factor) { - strvec_push(&cmd.args, "--stdin-packs"); + if (existing_has_cruft_in_midx(&existing)) + strvec_push(&cmd.args, "--stdin-packs"); + else + strvec_push(&cmd.args, "--stdin-packs=follow"); strvec_push(&cmd.args, "--unpacked"); } else { strvec_push(&cmd.args, "--unpacked"); diff --git a/t/t7704-repack-cruft.sh b/t/t7704-repack-cruft.sh index 8aebfb45f5..33ac58a3a5 100755 --- a/t/t7704-repack-cruft.sh +++ b/t/t7704-repack-cruft.sh @@ -724,4 +724,74 @@ test_expect_success 'cruft repack respects --quiet' ' ) ' +test_expect_success 'repack --write-midx excludes cruft where possible' ' + git init exclude-cruft-when-possible && + ( + cd exclude-cruft-when-possible && + + test_commit one && + + test_commit --no-tag two && + two="$(git rev-parse HEAD)" && + test_commit --no-tag three && + three="$(git rev-parse HEAD)" && + git reset --hard one && + + git reflog expire --all --expire=all && + + git repack --cruft -d && + ls $packdir/pack-*.idx | sort >packs.before && + + git merge $two && + test_commit four && + git repack -d --geometric=2 --write-midx --write-bitmap-index && + ls $packdir/pack-*.idx | sort >packs.after && + + comm -13 packs.before packs.after >packs.new && + test_line_count = 1 packs.new && + + git rev-list --objects --no-object-names one..four >expect.raw && + sort expect.raw >expect && + + git show-index <$(cat packs.new) >actual.raw && + cut -d" " -f2 actual.raw | sort >actual && + + test_cmp expect actual && + + test-tool read-midx --show-objects $objdir >actual.raw && + grep "\.pack$" actual.raw | cut -d" " -f1 | sort >actual.objects && + git rev-list --objects --no-object-names HEAD >expect.raw && + sort expect.raw >expect.objects && + + test_cmp expect.objects actual.objects && + + cruft="$(basename $(ls $packdir/*.mtimes))" && + grep "^pack-" actual.raw >actual.packs && + ! test_grep "${cruft%.mtimes}.idx" actual.packs + ) +' + +test_expect_success 'repack --write-midx includes cruft when necessary' ' + ( + cd exclude-cruft-when-possible && + + ls $packdir/pack-*.idx | sort >packs.all && + grep -o "pack-.*\.idx$" packs.all >in && + + git multi-pack-index write --stdin-packs --bitmap actual.raw && + grep "\.pack$" actual.raw | cut -d" " -f1 | sort >actual.objects && + git cat-file --batch-all-objects --batch-check="%(objectname)" \ + >expect.objects && + test_cmp expect.objects actual.objects && + + grep "^pack-" actual.raw >actual.packs && + test_line_count = "$(($(wc -l