From patchwork Tue May 14 18:47:44 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 10943579 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 060FD1515 for ; Tue, 14 May 2019 18:48:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EC5A128869 for ; Tue, 14 May 2019 18:48:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E037C28872; Tue, 14 May 2019 18:48:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7E3EE28869 for ; Tue, 14 May 2019 18:48:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727553AbfENSsB (ORCPT ); Tue, 14 May 2019 14:48:01 -0400 Received: from mail-yb1-f193.google.com ([209.85.219.193]:34877 "EHLO mail-yb1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726201AbfENSsA (ORCPT ); Tue, 14 May 2019 14:48:00 -0400 Received: by mail-yb1-f193.google.com with SMTP id k202so47776ybk.2 for ; Tue, 14 May 2019 11:48:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=mAF6FHEC0v1bwPVDs4U8j0qt4ZT7mzmVDQS5eGWgfnI=; b=ubh7ZtnTzgEZRXiwwLSigmMZ6kr5b78uX3cu3k9ijx8DvhtTwjLIxecZHf8/yDPBH4 4tUPnOY6uR5wZkIXItii7n9bB+oiPtFYsH4ILlh+B1f+f0VHCb8cE7CXsRkTXphkaMFD odPZOKS6875SUmhF3lhTwjET1VJu8CNoNzS/eZWFpqRJ9ogJqCg/wKblebXxSbSrW4Wg yzk1y/kqOKeda7ds+ww4QK/EnQZbcBqfKW0eKGdSt9xtMTyPZJ/4TFajRGSjWFXQl/Ph 7lsEuH+5Jtu6fcQ7hxjphQlbfle3q7UrS8RCzB2b6zNq9gcX5TKLGjCB44+MJiLJFPFj f/Zw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=mAF6FHEC0v1bwPVDs4U8j0qt4ZT7mzmVDQS5eGWgfnI=; b=LTcxaV0oqHfIVJPqPL0Hr2G/lBPBWN47JmpBtJB42Af3ih0TEtuwS3aE+Qgjxesv8o EgWVLgb00NXa208drGzRhGbjbMFT0jNWQjb2HVbt8aq/sSIrtypcmB61o5rbQdOpgPJt +QbWl+mHqJQswa1rwPkElXXrFF9EvtfWbIGX+u3D6rIhCY8WrHIlMte7c/Qqe/vBNXBr b7LRyrd75exjQVZjRh43dSJBpw3IAoa9OmdeXHTFgpIcr+ODwlNi/zC6RZ7lm8NOWw0I p5RWEpavBopnjGiINPuGDlDj+IMc6Xcy0gkOJlEFvcXKCcpbrl4HDNVdbp31QcfX7+hh 2c7Q== X-Gm-Message-State: APjAAAWdxrRinG/2tbOgavTvOIO7B9U5/zOJXoxz8JwcHB+/5DglWFqX /RGy7PXqWYapVTzamDBPhOWCm9PS58s= X-Google-Smtp-Source: APXvYqzriCk9B68663mlRzAZ2INF/PnljILVDzGL1FNHts2GJoHN65v9Yg/hlSVkW6Bh2cRA0e9qpg== X-Received: by 2002:a25:5456:: with SMTP id i83mr17582718ybb.301.1557859679830; Tue, 14 May 2019 11:47:59 -0700 (PDT) Received: from linux.mshome.net ([98.122.173.75]) by smtp.gmail.com with ESMTPSA id u17sm3438627ywu.94.2019.05.14.11.47.58 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 14 May 2019 11:47:58 -0700 (PDT) From: Derrick Stolee X-Google-Original-From: Derrick Stolee To: git@vger.kernel.org Cc: peff@peff.net, jrnieder@gmail.com, gitster@pobox.com, avarab@gmail.com, dstolee@microsoft.com Subject: [PATCH v6 01/11] repack: refactor pack deletion for future use Date: Tue, 14 May 2019 18:47:44 +0000 Message-Id: <20190514184754.3196-2-dstolee@microsoft.com> X-Mailer: git-send-email 2.22.0.rc0 In-Reply-To: <20190514184754.3196-1-dstolee@microsoft.com> References: <20190424151428.170316-1-dstolee@microsoft.com> <20190514184754.3196-1-dstolee@microsoft.com> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The repack builtin deletes redundant pack-files and their associated .idx, .promisor, .bitmap, and .keep files. We will want to re-use this logic in the future for other types of repack, so pull the logic into 'unlink_pack_path()' in packfile.c. The 'ignore_keep' parameter is enabled for the use in repack, but will be important for a future caller. Signed-off-by: Derrick Stolee --- builtin/repack.c | 14 ++------------ packfile.c | 28 ++++++++++++++++++++++++++++ packfile.h | 7 +++++++ 3 files changed, 37 insertions(+), 12 deletions(-) diff --git a/builtin/repack.c b/builtin/repack.c index 67f8978043..1f9e6fad1b 100644 --- a/builtin/repack.c +++ b/builtin/repack.c @@ -129,19 +129,9 @@ static void get_non_kept_pack_filenames(struct string_list *fname_list, static void remove_redundant_pack(const char *dir_name, const char *base_name) { - const char *exts[] = {".pack", ".idx", ".keep", ".bitmap", ".promisor"}; - int i; struct strbuf buf = STRBUF_INIT; - size_t plen; - - strbuf_addf(&buf, "%s/%s", dir_name, base_name); - plen = buf.len; - - for (i = 0; i < ARRAY_SIZE(exts); i++) { - strbuf_setlen(&buf, plen); - strbuf_addstr(&buf, exts[i]); - unlink(buf.buf); - } + strbuf_addf(&buf, "%s/%s.pack", dir_name, base_name); + unlink_pack_path(buf.buf, 1); strbuf_release(&buf); } diff --git a/packfile.c b/packfile.c index 060de420d1..683ce5674c 100644 --- a/packfile.c +++ b/packfile.c @@ -352,6 +352,34 @@ void close_all_packs(struct raw_object_store *o) } } +void unlink_pack_path(const char *pack_name, int force_delete) +{ + static const char *exts[] = {".pack", ".idx", ".keep", ".bitmap", ".promisor"}; + int i; + struct strbuf buf = STRBUF_INIT; + size_t plen; + + strbuf_addstr(&buf, pack_name); + strip_suffix_mem(buf.buf, &buf.len, ".pack"); + plen = buf.len; + + if (!force_delete) { + strbuf_addstr(&buf, ".keep"); + if (!access(buf.buf, F_OK)) { + strbuf_release(&buf); + return; + } + } + + for (i = 0; i < ARRAY_SIZE(exts); i++) { + strbuf_setlen(&buf, plen); + strbuf_addstr(&buf, exts[i]); + unlink(buf.buf); + } + + strbuf_release(&buf); +} + /* * The LRU pack is the one with the oldest MRU window, preferring packs * with no used windows, or the oldest mtime if it has no windows allocated. diff --git a/packfile.h b/packfile.h index 12baa6118a..09f5222113 100644 --- a/packfile.h +++ b/packfile.h @@ -94,6 +94,13 @@ void unuse_pack(struct pack_window **); void clear_delta_base_cache(void); struct packed_git *add_packed_git(const char *path, size_t path_len, int local); +/* + * Unlink the .pack and associated extension files. + * Does not unlink if 'force_delete' is false and the pack-file is + * marked as ".keep". + */ +extern void unlink_pack_path(const char *pack_name, int force_delete); + /* * Make sure that a pointer access into an mmap'd index file is within bounds, * and can provide at least 8 bytes of data. From patchwork Tue May 14 18:47:45 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 10943583 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 102C0933 for ; Tue, 14 May 2019 18:48:05 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 010D828869 for ; Tue, 14 May 2019 18:48:05 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E936828872; Tue, 14 May 2019 18:48:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9A02528869 for ; Tue, 14 May 2019 18:48:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727565AbfENSsD (ORCPT ); Tue, 14 May 2019 14:48:03 -0400 Received: from mail-yb1-f193.google.com ([209.85.219.193]:36926 "EHLO mail-yb1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727549AbfENSsC (ORCPT ); Tue, 14 May 2019 14:48:02 -0400 Received: by mail-yb1-f193.google.com with SMTP id p134so47948ybc.4 for ; Tue, 14 May 2019 11:48:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=G6kBqGumlgMWg//8WJMHQRmX3RpWhL8fUfwG0BHKvvg=; b=T6ANq0WDsB6Xe71HiBzVEsRzQ0XAR8HjI/EBc5bqzAmCDIoe3rePwH0WYmu/AscI5i wYNlgjoMZlNWvUTg5sONx5dXliJXtCzaIFp0WUZpOlvA3ETmhCeBbTG8KQpHNO+zUQ4M MhXvtnrRzCJf9pbpJHqUQ08fQeLk74Ja9x/pbnUvoC3fa6OqJqHM3orkmk8Iz3Wltl4o G7Fo/yfgPJ8LGH+JUFxVKi4/qP9CbCwqOfz8vqqydJY9VPiacIruytfvEVCquCLmi0CJ cIg/2Lp7V9YatRkE0KigUfukHcdTDXovcPuJeVgRBSWZP9wxz7DIOtrfgUAvbuPbBOie E3ew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=G6kBqGumlgMWg//8WJMHQRmX3RpWhL8fUfwG0BHKvvg=; b=IVY3Rb80jOvpvbUMpOFJRK61MIqMgBqkWd6teSVp9946G69oWM8zWIgDWGpvcLCKcz vTM4yZLfBhiWqUG3OZhDi//uPJN/HNTDFPowkUU3RW91EpmeSCNiWetjD05fBN6ylxo5 obz+4C9eHHD4B+9Urrg+BQm+L/vvOZUROJYfxAuHq7I8mmTwDxe8cGSjjOZGzQ3mxu/7 fLnl1O1V1yHoyTeTR1j/pu7Nsy96e2jlhzwCo84qyJfK5aRcrxLdfLsOL5HHuDWKJa7i qh7cb1f5w9ITdd0K0eQEWlRURwyrKSe5PGNtJTKi9QvpVk4ljlCusKF0XLlRuGmeqnSW BOVw== X-Gm-Message-State: APjAAAX+02Lt/6xBr2eBIT45z6d8KGsjevggB5iwGMSjbd4cdmgqP/Y1 yyLMCJF6AGwk3rt5/XIUy5oBgG1687k= X-Google-Smtp-Source: APXvYqzFj81IsD/whP/R8oskpTRV9RPM4B1eVHsGLV3t/Fzp2MZ34PVgbKqFVkCDSvntnz3+OHeeGQ== X-Received: by 2002:a25:449:: with SMTP id 70mr73885ybe.25.1557859680804; Tue, 14 May 2019 11:48:00 -0700 (PDT) Received: from linux.mshome.net ([98.122.173.75]) by smtp.gmail.com with ESMTPSA id u17sm3438627ywu.94.2019.05.14.11.47.59 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 14 May 2019 11:48:00 -0700 (PDT) From: Derrick Stolee X-Google-Original-From: Derrick Stolee To: git@vger.kernel.org Cc: peff@peff.net, jrnieder@gmail.com, gitster@pobox.com, avarab@gmail.com, dstolee@microsoft.com, Stefan Beller , =?utf-8?q?Szeder?= =?utf-8?q?_G=C3=A1bor?= Subject: [PATCH v6 02/11] Docs: rearrange subcommands for multi-pack-index Date: Tue, 14 May 2019 18:47:45 +0000 Message-Id: <20190514184754.3196-3-dstolee@microsoft.com> X-Mailer: git-send-email 2.22.0.rc0 In-Reply-To: <20190514184754.3196-1-dstolee@microsoft.com> References: <20190424151428.170316-1-dstolee@microsoft.com> <20190514184754.3196-1-dstolee@microsoft.com> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP We will add new subcommands to the multi-pack-index, and that will make the documentation a bit messier. Clean up the 'verb' descriptions by renaming the concept to 'subcommand' and removing the reference to the object directory. Helped-by: Stefan Beller Helped-by: Szeder Gábor Signed-off-by: Derrick Stolee --- Documentation/git-multi-pack-index.txt | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt index f7778a2c85..1af406aca2 100644 --- a/Documentation/git-multi-pack-index.txt +++ b/Documentation/git-multi-pack-index.txt @@ -9,7 +9,7 @@ git-multi-pack-index - Write and verify multi-pack-indexes SYNOPSIS -------- [verse] -'git multi-pack-index' [--object-dir=] +'git multi-pack-index' [--object-dir=] DESCRIPTION ----------- @@ -23,13 +23,13 @@ OPTIONS `/packs/multi-pack-index` for the current MIDX file, and `/packs` for the pack-files to index. +The following subcommands are available: + write:: - When given as the verb, write a new MIDX file to - `/packs/multi-pack-index`. + Write a new MIDX file. verify:: - When given as the verb, verify the contents of the MIDX file - at `/packs/multi-pack-index`. + Verify the contents of the MIDX file. EXAMPLES From patchwork Tue May 14 18:47:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 10943589 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 034F21390 for ; Tue, 14 May 2019 18:48:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E920B2886F for ; Tue, 14 May 2019 18:48:09 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DD56328872; Tue, 14 May 2019 18:48:09 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5DE3328869 for ; Tue, 14 May 2019 18:48:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727589AbfENSsG (ORCPT ); Tue, 14 May 2019 14:48:06 -0400 Received: from mail-yb1-f194.google.com ([209.85.219.194]:34878 "EHLO mail-yb1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726201AbfENSsD (ORCPT ); Tue, 14 May 2019 14:48:03 -0400 Received: by mail-yb1-f194.google.com with SMTP id k202so47808ybk.2 for ; Tue, 14 May 2019 11:48:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=4s0Z+0C5dUqbeM2oTRwfFmJd/GWgOCJKYiUE6/Z08Rk=; b=idKUOjPKk2hOB5lAbzCwSJIDAf8TxPN/6USVMbjl0oGSPYS609pLSCzddtOs5pMzQe ufugn+GZIEAgfxlPqlohaOOksQ9402fBjzgpJgAZX2e2lFMkiBIplCGVloR9JopvgO7Z dC6T2S9nweN9kHz7my42FxN+rlayez+5IbbZKZc8PYh19fAOx+AT/asDWO1jeCynwcJB 9GDJdnUZ4CeRZCKvixyzIIO5hg9IGXjUtiIep4kid36HjaFpkPeVq5fKqz3rhG5Ud83F u1u6LG/UdNacqlMuavOXMLs4WuakDCSi/A+KqY6spm3p9UzCRpnx0ez4UlU+acZjiNbq nrBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4s0Z+0C5dUqbeM2oTRwfFmJd/GWgOCJKYiUE6/Z08Rk=; b=j6a0amOPFkcN5lbqWo87Tp59IomMhCFEQAIwCR+ciCwTWJVOvJgb0qMInaXL3UnOOb 40YsT38IbgkJurGtCZvRZROHRfQPI96A+/BOc/2Nx6cFNSyd0R8av/Rbrq1wU3Cf8OZ1 rynyvDHDxa+oJwNI4q7W4JceT4W25AVJd+TYiVWfheJrDwpL9kZJ6nm61RLt/I96dMOK de+xAf7qaC8DjAYbh3ylUXb4QjaLZvkdmkTw2bhccwEDOujcXKsmNgsiutKoLhD5GLrq 4FT+CAabSFU3zKdJJ59m/qhB0Y2Qwi1kPLedxjfwgHwoeSAHpET70CqzqU/Rsd9/lGAw /z8Q== X-Gm-Message-State: APjAAAWJLSxu7xJ2mIXUhODjVtz368eINJe+DZk/LoNhy97lesLagZ79 mRmCBamBfnKy99G/Ts6g1cFeOUo9iuk= X-Google-Smtp-Source: APXvYqyVZjp1BcS9Edntofvk/mqjsTlPGSMgwlp3+We5QJv1AfUkJIVatqqZLggrQ253Igmcxvrkyw== X-Received: by 2002:a5b:74c:: with SMTP id s12mr18300994ybq.436.1557859681704; Tue, 14 May 2019 11:48:01 -0700 (PDT) Received: from linux.mshome.net ([98.122.173.75]) by smtp.gmail.com with ESMTPSA id u17sm3438627ywu.94.2019.05.14.11.48.00 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 14 May 2019 11:48:01 -0700 (PDT) From: Derrick Stolee X-Google-Original-From: Derrick Stolee To: git@vger.kernel.org Cc: peff@peff.net, jrnieder@gmail.com, gitster@pobox.com, avarab@gmail.com, dstolee@microsoft.com Subject: [PATCH v6 03/11] multi-pack-index: prepare for 'expire' subcommand Date: Tue, 14 May 2019 18:47:46 +0000 Message-Id: <20190514184754.3196-4-dstolee@microsoft.com> X-Mailer: git-send-email 2.22.0.rc0 In-Reply-To: <20190514184754.3196-1-dstolee@microsoft.com> References: <20190424151428.170316-1-dstolee@microsoft.com> <20190514184754.3196-1-dstolee@microsoft.com> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The multi-pack-index tracks objects in a collection of pack-files. Only one copy of each object is indexed, using the modified time of the pack-files to determine tie-breakers. It is possible to have a pack-file with no referenced objects because all objects have a duplicate in a newer pack-file. Introduce a new 'expire' subcommand to the multi-pack-index builtin. This subcommand will delete these unused pack-files and rewrite the multi-pack-index to no longer refer to those files. More details about the specifics will follow as the method is implemented. Add a test that verifies the 'expire' subcommand is correctly wired, but will still be valid when the verb is implemented. Specifically, create a set of packs that should all have referenced objects and should not be removed during an 'expire' operation. The packs are created carefully to ensure they have a specific order when sorted by size. This will be important in a later test. Signed-off-by: Derrick Stolee --- Documentation/git-multi-pack-index.txt | 5 +++ builtin/multi-pack-index.c | 4 ++- midx.c | 5 +++ midx.h | 1 + t/t5319-multi-pack-index.sh | 49 ++++++++++++++++++++++++++ 5 files changed, 63 insertions(+), 1 deletion(-) diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt index 1af406aca2..6186c4c936 100644 --- a/Documentation/git-multi-pack-index.txt +++ b/Documentation/git-multi-pack-index.txt @@ -31,6 +31,11 @@ write:: verify:: Verify the contents of the MIDX file. +expire:: + Delete the pack-files that are tracked by the MIDX file, but + have no objects referenced by the MIDX. Rewrite the MIDX file + afterward to remove all references to these pack-files. + EXAMPLES -------- diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c index 72dfd3dadc..ad10d40512 100644 --- a/builtin/multi-pack-index.c +++ b/builtin/multi-pack-index.c @@ -6,7 +6,7 @@ #include "trace2.h" static char const * const builtin_multi_pack_index_usage[] = { - N_("git multi-pack-index [--object-dir=] (write|verify)"), + N_("git multi-pack-index [--object-dir=] (write|verify|expire)"), NULL }; @@ -47,6 +47,8 @@ int cmd_multi_pack_index(int argc, const char **argv, return write_midx_file(opts.object_dir); if (!strcmp(argv[0], "verify")) return verify_midx_file(the_repository, opts.object_dir); + if (!strcmp(argv[0], "expire")) + return expire_midx_packs(the_repository, opts.object_dir); die(_("unrecognized verb: %s"), argv[0]); } diff --git a/midx.c b/midx.c index e7e1fe4d65..3b7da1a360 100644 --- a/midx.c +++ b/midx.c @@ -1140,3 +1140,8 @@ int verify_midx_file(struct repository *r, const char *object_dir) return verify_midx_error; } + +int expire_midx_packs(struct repository *r, const char *object_dir) +{ + return 0; +} diff --git a/midx.h b/midx.h index 3eb29731f2..505f1431b7 100644 --- a/midx.h +++ b/midx.h @@ -50,6 +50,7 @@ int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, i int write_midx_file(const char *object_dir); void clear_midx_file(struct repository *r); int verify_midx_file(struct repository *r, const char *object_dir); +int expire_midx_packs(struct repository *r, const char *object_dir); void close_midx(struct multi_pack_index *m); diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index 1ebf19ec3c..1b2d32f475 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -363,4 +363,53 @@ test_expect_success 'verify incorrect 64-bit offset' ' "incorrect object offset" ' +test_expect_success 'setup expire tests' ' + mkdir dup && + ( + cd dup && + git init && + test-tool genrandom "data" 4096 >large_file.txt && + git update-index --add large_file.txt && + for i in $(test_seq 1 20) + do + test_commit $i + done && + git branch A HEAD && + git branch B HEAD~8 && + git branch C HEAD~13 && + git branch D HEAD~16 && + git branch E HEAD~18 && + git pack-objects --revs .git/objects/pack/pack-A <<-EOF && + refs/heads/A + ^refs/heads/B + EOF + git pack-objects --revs .git/objects/pack/pack-B <<-EOF && + refs/heads/B + ^refs/heads/C + EOF + git pack-objects --revs .git/objects/pack/pack-C <<-EOF && + refs/heads/C + ^refs/heads/D + EOF + git pack-objects --revs .git/objects/pack/pack-D <<-EOF && + refs/heads/D + ^refs/heads/E + EOF + git pack-objects --revs .git/objects/pack/pack-E <<-EOF && + refs/heads/E + EOF + git multi-pack-index write + ) +' + +test_expect_success 'expire does not remove any packs' ' + ( + cd dup && + ls .git/objects/pack >expect && + git multi-pack-index expire && + ls .git/objects/pack >actual && + test_cmp expect actual + ) +' + test_done From patchwork Tue May 14 18:47:47 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 10943601 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 633941390 for ; Tue, 14 May 2019 18:48:18 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 55F402886F for ; Tue, 14 May 2019 18:48:18 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4A4D528899; Tue, 14 May 2019 18:48:18 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D24522886F for ; Tue, 14 May 2019 18:48:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727575AbfENSsG (ORCPT ); Tue, 14 May 2019 14:48:06 -0400 Received: from mail-yb1-f193.google.com ([209.85.219.193]:39255 "EHLO mail-yb1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727564AbfENSsD (ORCPT ); Tue, 14 May 2019 14:48:03 -0400 Received: by mail-yb1-f193.google.com with SMTP id x5so47844ybn.6 for ; Tue, 14 May 2019 11:48:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=pszLTzp5Z406tJwGmtSnZ3WOd8K6Ad+41tRmaMknIvU=; b=Mtxkwb4zqZqNues35pZQ4nG+TV6OQMMB1JSO2apn85LI2WbUH6T6m/69PaOfMlAjcZ H5zCjvM4JblyKFytkCPD466nOqjgCX1AUOo6DbDRJ7hVU9fj30ROtnVaelvFnghVPUGA Ajf6txA8np2cZs7aJ/FvGaapGMrEw7ke+FFgx8tfNYMmlVDFXyBChkJiEdLzAP/zYGkM Wirj8SQogCXe5umLY0mwaYPprph1C1qNL2SgDUT3jQGlBF6w6F0ECSUqxi5NSHjxqyvK /PShrtMQv7QImjoGmSNNnmSIoeqi5qqpDXrZXtx1SBj3GbHXr0doZulvPjAPal282017 00IA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=pszLTzp5Z406tJwGmtSnZ3WOd8K6Ad+41tRmaMknIvU=; b=t0h4O6uckEN544qFAsBxgHyTCfXrxQGmfbPN95v+q0VIxJIWBdPZr6CHFliB/69P4n z3vNjy3lkoi55jNwemB2gAK1uoeNuMJpHPfQBrrThSsgIsEt6ruJ/gz1OLH0ZZ7p6vcE WPTxZfeCaGomLtAnnmQhaBlgLN2mB0diYvaLm+6bzTswhYsxxPPI/78BGtJ3C1APrQTe uLdpPOEKaBjUdTsEBVyrkzrHQdkiw0UMg7YOWHyf7BnvsRbTeg4vFD3S4Z8cSCuUGgGE 8zSNIxFJbSsOOCljfg8R3J7ruFSYNSQp84UVdq5wR/fxFmhpqv3fUREdZfGuhBzVwgnd nTMg== X-Gm-Message-State: APjAAAW3kUpE9Mj/82caXNOoQY8GvNbzkBVtIEPnphGhVjjtrQGqkPQ0 dlj0GR1XMqBgWRlchCFeQNJocddgCXU= X-Google-Smtp-Source: APXvYqzEygPmt32XJ3odliOfBhBdGHHlI6ub8VLqZrWLzooUuHewlQxf+7L96h7hbiIE5IILahCDoA== X-Received: by 2002:a25:f506:: with SMTP id a6mr18246992ybe.33.1557859682748; Tue, 14 May 2019 11:48:02 -0700 (PDT) Received: from linux.mshome.net ([98.122.173.75]) by smtp.gmail.com with ESMTPSA id u17sm3438627ywu.94.2019.05.14.11.48.01 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 14 May 2019 11:48:02 -0700 (PDT) From: Derrick Stolee X-Google-Original-From: Derrick Stolee To: git@vger.kernel.org Cc: peff@peff.net, jrnieder@gmail.com, gitster@pobox.com, avarab@gmail.com, dstolee@microsoft.com Subject: [PATCH v6 04/11] midx: simplify computation of pack name lengths Date: Tue, 14 May 2019 18:47:47 +0000 Message-Id: <20190514184754.3196-5-dstolee@microsoft.com> X-Mailer: git-send-email 2.22.0.rc0 In-Reply-To: <20190514184754.3196-1-dstolee@microsoft.com> References: <20190424151428.170316-1-dstolee@microsoft.com> <20190514184754.3196-1-dstolee@microsoft.com> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Before writing the multi-pack-index, we compute the length of the pack-index names concatenated together. This forms the data in the pack name chunk, and we precompute it to compute chunk offsets. The value is also modified to fit alignment needs. Previously, this computation was coupled with adding packs from the existing multi-pack-index and the remaining packs in the object dir not already covered by the multi-pack-index. In anticipation of this becoming more complicated with the 'expire' subcommand, simplify the computation by centralizing it to a single loop before writing the file. Signed-off-by: Derrick Stolee --- midx.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/midx.c b/midx.c index 3b7da1a360..62404620ad 100644 --- a/midx.c +++ b/midx.c @@ -433,7 +433,6 @@ struct pack_list { uint32_t nr; uint32_t alloc_list; uint32_t alloc_names; - size_t pack_name_concat_len; struct multi_pack_index *m; }; @@ -468,7 +467,6 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len, } packs->names[packs->nr] = xstrdup(file_name); - packs->pack_name_concat_len += strlen(file_name) + 1; packs->nr++; } } @@ -812,6 +810,7 @@ int write_midx_file(const char *object_dir) uint32_t nr_entries, num_large_offsets = 0; struct pack_midx_entry *entries = NULL; int large_offsets_needed = 0; + int pack_name_concat_len = 0; midx_name = get_midx_filename(object_dir); if (safe_create_leading_directories(midx_name)) { @@ -827,7 +826,6 @@ int write_midx_file(const char *object_dir) packs.alloc_names = packs.alloc_list; packs.list = NULL; packs.names = NULL; - packs.pack_name_concat_len = 0; ALLOC_ARRAY(packs.list, packs.alloc_list); ALLOC_ARRAY(packs.names, packs.alloc_names); @@ -838,7 +836,6 @@ int write_midx_file(const char *object_dir) packs.list[packs.nr] = NULL; packs.names[packs.nr] = xstrdup(packs.m->pack_names[i]); - packs.pack_name_concat_len += strlen(packs.names[packs.nr]) + 1; packs.nr++; } } @@ -848,10 +845,6 @@ int write_midx_file(const char *object_dir) if (packs.m && packs.nr == packs.m->num_packs) goto cleanup; - if (packs.pack_name_concat_len % MIDX_CHUNK_ALIGNMENT) - packs.pack_name_concat_len += MIDX_CHUNK_ALIGNMENT - - (packs.pack_name_concat_len % MIDX_CHUNK_ALIGNMENT); - ALLOC_ARRAY(pack_perm, packs.nr); sort_packs_by_name(packs.names, packs.nr, pack_perm); @@ -864,6 +857,13 @@ int write_midx_file(const char *object_dir) large_offsets_needed = 1; } + for (i = 0; i < packs.nr; i++) + pack_name_concat_len += strlen(packs.names[i]) + 1; + + if (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT) + pack_name_concat_len += MIDX_CHUNK_ALIGNMENT - + (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT); + hold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR); f = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf); FREE_AND_NULL(midx_name); @@ -881,7 +881,7 @@ int write_midx_file(const char *object_dir) cur_chunk++; chunk_ids[cur_chunk] = MIDX_CHUNKID_OIDFANOUT; - chunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + packs.pack_name_concat_len; + chunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + pack_name_concat_len; cur_chunk++; chunk_ids[cur_chunk] = MIDX_CHUNKID_OIDLOOKUP; From patchwork Tue May 14 18:47:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 10943587 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6EFDD933 for ; Tue, 14 May 2019 18:48:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5F3DE28869 for ; Tue, 14 May 2019 18:48:08 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5320A28895; Tue, 14 May 2019 18:48:08 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 33B8028869 for ; Tue, 14 May 2019 18:48:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727581AbfENSsG (ORCPT ); Tue, 14 May 2019 14:48:06 -0400 Received: from mail-yb1-f196.google.com ([209.85.219.196]:40533 "EHLO mail-yb1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727549AbfENSsF (ORCPT ); Tue, 14 May 2019 14:48:05 -0400 Received: by mail-yb1-f196.google.com with SMTP id q17so47941ybg.7 for ; Tue, 14 May 2019 11:48:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=+b+B0hPRxslRGZUTnk8feyDiLOkqteu4EnGF2GSbZv0=; b=d4b5JQjw6A41F/F26izAa2hedTOIkHJA6eKuaNqdHtjQ2lvZzcSxTOmufSZYNihcM8 hk4uxe8IenerWV0KgzucpEMpkTSkzMQapUEAxcpqt08oC9I1G6gG3Pm/LC6AhccgXEuE 9Xqzc2fMhCIrjr0j3ynKCbDENsf+Kan1HdOR0hXfWT++XdqjSDgMRH4Okqn+5hdELwQ6 cttPlGuXUc/Ydruxj+zCLLLcdTOWl1MDYEjB1jaw2tULZ1VCeAJSv3M8fjbZxQYdTUgj FxIJSHzfeCbnqjQYu8e5fXJvbTadS6cLvO92ZnFuOtzmqaiLq9uK/7aBHfd+1FyI0+qb wTBg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+b+B0hPRxslRGZUTnk8feyDiLOkqteu4EnGF2GSbZv0=; b=fZ8vvn5aNNz4P/4TVARgJcVoB51UULg6DCOnQOOoU/05gbXzrIB1uM6la8EALH9IHP vuGcVXOgTlPZM5Y0fkILHwx6zLiOIeri0poda3PuGKLtxYZ/wYLR+MFrPdSiNrSAM+b/ FMwi8jU10COlSnU4A+Ew8UvN7JIcrnSuO1jwSZxTgV7ZxcMcWGOOoUKn3b5h8XZZ3qFq 2KN7Vg5gz+lxWOxzvO73TLJGJYYF7Tdf5ccb71lqKj31zIOQjquJKzjgW9Zguoe1VPxr RaKVFdPdnqXbE9ag0dmN5q/dD9zAr/2CIW/bB7eCSwcyqoyW2pvlWfGu9HmapTMMpjtv gX4A== X-Gm-Message-State: APjAAAW3RE6YdY15pt9eMYD6yF+tgOw/GYwCGCBWVqln8NTsE7Z3CbUV IZpYByWMWyrtQGy8zA/3waMyPR48LBw= X-Google-Smtp-Source: APXvYqzxUgjmpcLhTXX+XpYgoU9Z5vMZ+SxBYi8LeP5JtnTL/lAZI/K5IQxBNjI4kGxiLgDqotSfGw== X-Received: by 2002:a25:3796:: with SMTP id e144mr18616379yba.173.1557859683646; Tue, 14 May 2019 11:48:03 -0700 (PDT) Received: from linux.mshome.net ([98.122.173.75]) by smtp.gmail.com with ESMTPSA id u17sm3438627ywu.94.2019.05.14.11.48.02 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 14 May 2019 11:48:03 -0700 (PDT) From: Derrick Stolee X-Google-Original-From: Derrick Stolee To: git@vger.kernel.org Cc: peff@peff.net, jrnieder@gmail.com, gitster@pobox.com, avarab@gmail.com, dstolee@microsoft.com Subject: [PATCH v6 05/11] midx: refactor permutation logic and pack sorting Date: Tue, 14 May 2019 18:47:48 +0000 Message-Id: <20190514184754.3196-6-dstolee@microsoft.com> X-Mailer: git-send-email 2.22.0.rc0 In-Reply-To: <20190514184754.3196-1-dstolee@microsoft.com> References: <20190424151428.170316-1-dstolee@microsoft.com> <20190514184754.3196-1-dstolee@microsoft.com> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP In anticipation of the expire subcommand, refactor the way we sort the packfiles by name. This will greatly simplify our approach to dropping expired packs from the list. First, create 'struct pack_info' to replace 'struct pack_pair'. This struct contains the necessary information about a pack, including its name, a pointer to its packfile struct (if not already in the multi-pack-index), and the original pack-int-id. Second, track the pack information using an array of pack_info structs in the pack_list struct. This simplifies the logic around the multiple arrays we were tracking in that struct. Finally, update get_sorted_entries() to not permute the pack-int-id and instead supply the permutation to write_midx_object_offsets(). This requires sorting the packs after get_sorted_entries(). Signed-off-by: Derrick Stolee --- midx.c | 156 +++++++++++++++++++++++++-------------------------------- 1 file changed, 69 insertions(+), 87 deletions(-) diff --git a/midx.c b/midx.c index 62404620ad..6d4b84e243 100644 --- a/midx.c +++ b/midx.c @@ -427,12 +427,23 @@ static size_t write_midx_header(struct hashfile *f, return MIDX_HEADER_SIZE; } +struct pack_info { + uint32_t orig_pack_int_id; + char *pack_name; + struct packed_git *p; +}; + +static int pack_info_compare(const void *_a, const void *_b) +{ + struct pack_info *a = (struct pack_info *)_a; + struct pack_info *b = (struct pack_info *)_b; + return strcmp(a->pack_name, b->pack_name); +} + struct pack_list { - struct packed_git **list; - char **names; + struct pack_info *info; uint32_t nr; - uint32_t alloc_list; - uint32_t alloc_names; + uint32_t alloc; struct multi_pack_index *m; }; @@ -445,66 +456,32 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len, if (packs->m && midx_contains_pack(packs->m, file_name)) return; - ALLOC_GROW(packs->list, packs->nr + 1, packs->alloc_list); - ALLOC_GROW(packs->names, packs->nr + 1, packs->alloc_names); + ALLOC_GROW(packs->info, packs->nr + 1, packs->alloc); - packs->list[packs->nr] = add_packed_git(full_path, - full_path_len, - 0); + packs->info[packs->nr].p = add_packed_git(full_path, + full_path_len, + 0); - if (!packs->list[packs->nr]) { + if (!packs->info[packs->nr].p) { warning(_("failed to add packfile '%s'"), full_path); return; } - if (open_pack_index(packs->list[packs->nr])) { + if (open_pack_index(packs->info[packs->nr].p)) { warning(_("failed to open pack-index '%s'"), full_path); - close_pack(packs->list[packs->nr]); - FREE_AND_NULL(packs->list[packs->nr]); + close_pack(packs->info[packs->nr].p); + FREE_AND_NULL(packs->info[packs->nr].p); return; } - packs->names[packs->nr] = xstrdup(file_name); + packs->info[packs->nr].pack_name = xstrdup(file_name); + packs->info[packs->nr].orig_pack_int_id = packs->nr; packs->nr++; } } -struct pack_pair { - uint32_t pack_int_id; - char *pack_name; -}; - -static int pack_pair_compare(const void *_a, const void *_b) -{ - struct pack_pair *a = (struct pack_pair *)_a; - struct pack_pair *b = (struct pack_pair *)_b; - return strcmp(a->pack_name, b->pack_name); -} - -static void sort_packs_by_name(char **pack_names, uint32_t nr_packs, uint32_t *perm) -{ - uint32_t i; - struct pack_pair *pairs; - - ALLOC_ARRAY(pairs, nr_packs); - - for (i = 0; i < nr_packs; i++) { - pairs[i].pack_int_id = i; - pairs[i].pack_name = pack_names[i]; - } - - QSORT(pairs, nr_packs, pack_pair_compare); - - for (i = 0; i < nr_packs; i++) { - pack_names[i] = pairs[i].pack_name; - perm[pairs[i].pack_int_id] = i; - } - - free(pairs); -} - struct pack_midx_entry { struct object_id oid; uint32_t pack_int_id; @@ -530,7 +507,6 @@ static int midx_oid_compare(const void *_a, const void *_b) } static int nth_midxed_pack_midx_entry(struct multi_pack_index *m, - uint32_t *pack_perm, struct pack_midx_entry *e, uint32_t pos) { @@ -538,7 +514,7 @@ static int nth_midxed_pack_midx_entry(struct multi_pack_index *m, return 1; nth_midxed_object_oid(&e->oid, m, pos); - e->pack_int_id = pack_perm[nth_midxed_pack_int_id(m, pos)]; + e->pack_int_id = nth_midxed_pack_int_id(m, pos); e->offset = nth_midxed_offset(m, pos); /* consider objects in midx to be from "old" packs */ @@ -572,8 +548,7 @@ static void fill_pack_entry(uint32_t pack_int_id, * of a packfile containing the object). */ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m, - struct packed_git **p, - uint32_t *perm, + struct pack_info *info, uint32_t nr_packs, uint32_t *nr_objects) { @@ -584,7 +559,7 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m, uint32_t start_pack = m ? m->num_packs : 0; for (cur_pack = start_pack; cur_pack < nr_packs; cur_pack++) - total_objects += p[cur_pack]->num_objects; + total_objects += info[cur_pack].p->num_objects; /* * As we de-duplicate by fanout value, we expect the fanout @@ -609,7 +584,7 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m, for (cur_object = start; cur_object < end; cur_object++) { ALLOC_GROW(entries_by_fanout, nr_fanout + 1, alloc_fanout); - nth_midxed_pack_midx_entry(m, perm, + nth_midxed_pack_midx_entry(m, &entries_by_fanout[nr_fanout], cur_object); nr_fanout++; @@ -620,12 +595,12 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m, uint32_t start = 0, end; if (cur_fanout) - start = get_pack_fanout(p[cur_pack], cur_fanout - 1); - end = get_pack_fanout(p[cur_pack], cur_fanout); + start = get_pack_fanout(info[cur_pack].p, cur_fanout - 1); + end = get_pack_fanout(info[cur_pack].p, cur_fanout); for (cur_object = start; cur_object < end; cur_object++) { ALLOC_GROW(entries_by_fanout, nr_fanout + 1, alloc_fanout); - fill_pack_entry(perm[cur_pack], p[cur_pack], cur_object, &entries_by_fanout[nr_fanout]); + fill_pack_entry(cur_pack, info[cur_pack].p, cur_object, &entries_by_fanout[nr_fanout]); nr_fanout++; } } @@ -654,7 +629,7 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m, } static size_t write_midx_pack_names(struct hashfile *f, - char **pack_names, + struct pack_info *info, uint32_t num_packs) { uint32_t i; @@ -662,14 +637,14 @@ static size_t write_midx_pack_names(struct hashfile *f, size_t written = 0; for (i = 0; i < num_packs; i++) { - size_t writelen = strlen(pack_names[i]) + 1; + size_t writelen = strlen(info[i].pack_name) + 1; - if (i && strcmp(pack_names[i], pack_names[i - 1]) <= 0) + if (i && strcmp(info[i].pack_name, info[i - 1].pack_name) <= 0) BUG("incorrect pack-file order: %s before %s", - pack_names[i - 1], - pack_names[i]); + info[i - 1].pack_name, + info[i].pack_name); - hashwrite(f, pack_names[i], writelen); + hashwrite(f, info[i].pack_name, writelen); written += writelen; } @@ -740,6 +715,7 @@ static size_t write_midx_oid_lookup(struct hashfile *f, unsigned char hash_len, } static size_t write_midx_object_offsets(struct hashfile *f, int large_offset_needed, + uint32_t *perm, struct pack_midx_entry *objects, uint32_t nr_objects) { struct pack_midx_entry *list = objects; @@ -749,7 +725,7 @@ static size_t write_midx_object_offsets(struct hashfile *f, int large_offset_nee for (i = 0; i < nr_objects; i++) { struct pack_midx_entry *obj = list++; - hashwrite_be32(f, obj->pack_int_id); + hashwrite_be32(f, perm[obj->pack_int_id]); if (large_offset_needed && obj->offset >> 31) hashwrite_be32(f, MIDX_LARGE_OFFSET_NEEDED | nr_large_offset++); @@ -822,20 +798,17 @@ int write_midx_file(const char *object_dir) packs.m = load_multi_pack_index(object_dir, 1); packs.nr = 0; - packs.alloc_list = packs.m ? packs.m->num_packs : 16; - packs.alloc_names = packs.alloc_list; - packs.list = NULL; - packs.names = NULL; - ALLOC_ARRAY(packs.list, packs.alloc_list); - ALLOC_ARRAY(packs.names, packs.alloc_names); + packs.alloc = packs.m ? packs.m->num_packs : 16; + packs.info = NULL; + ALLOC_ARRAY(packs.info, packs.alloc); if (packs.m) { for (i = 0; i < packs.m->num_packs; i++) { - ALLOC_GROW(packs.list, packs.nr + 1, packs.alloc_list); - ALLOC_GROW(packs.names, packs.nr + 1, packs.alloc_names); + ALLOC_GROW(packs.info, packs.nr + 1, packs.alloc); - packs.list[packs.nr] = NULL; - packs.names[packs.nr] = xstrdup(packs.m->pack_names[i]); + packs.info[packs.nr].orig_pack_int_id = i; + packs.info[packs.nr].pack_name = xstrdup(packs.m->pack_names[i]); + packs.info[packs.nr].p = NULL; packs.nr++; } } @@ -845,10 +818,7 @@ int write_midx_file(const char *object_dir) if (packs.m && packs.nr == packs.m->num_packs) goto cleanup; - ALLOC_ARRAY(pack_perm, packs.nr); - sort_packs_by_name(packs.names, packs.nr, pack_perm); - - entries = get_sorted_entries(packs.m, packs.list, pack_perm, packs.nr, &nr_entries); + entries = get_sorted_entries(packs.m, packs.info, packs.nr, &nr_entries); for (i = 0; i < nr_entries; i++) { if (entries[i].offset > 0x7fffffff) @@ -857,8 +827,21 @@ int write_midx_file(const char *object_dir) large_offsets_needed = 1; } + QSORT(packs.info, packs.nr, pack_info_compare); + + /* + * pack_perm stores a permutation between pack-int-ids from the + * previous multi-pack-index to the new one we are writing: + * + * pack_perm[old_id] = new_id + */ + ALLOC_ARRAY(pack_perm, packs.nr); + for (i = 0; i < packs.nr; i++) { + pack_perm[packs.info[i].orig_pack_int_id] = i; + } + for (i = 0; i < packs.nr; i++) - pack_name_concat_len += strlen(packs.names[i]) + 1; + pack_name_concat_len += strlen(packs.info[i].pack_name) + 1; if (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT) pack_name_concat_len += MIDX_CHUNK_ALIGNMENT - @@ -929,7 +912,7 @@ int write_midx_file(const char *object_dir) switch (chunk_ids[i]) { case MIDX_CHUNKID_PACKNAMES: - written += write_midx_pack_names(f, packs.names, packs.nr); + written += write_midx_pack_names(f, packs.info, packs.nr); break; case MIDX_CHUNKID_OIDFANOUT: @@ -941,7 +924,7 @@ int write_midx_file(const char *object_dir) break; case MIDX_CHUNKID_OBJECTOFFSETS: - written += write_midx_object_offsets(f, large_offsets_needed, entries, nr_entries); + written += write_midx_object_offsets(f, large_offsets_needed, pack_perm, entries, nr_entries); break; case MIDX_CHUNKID_LARGEOFFSETS: @@ -964,15 +947,14 @@ int write_midx_file(const char *object_dir) cleanup: for (i = 0; i < packs.nr; i++) { - if (packs.list[i]) { - close_pack(packs.list[i]); - free(packs.list[i]); + if (packs.info[i].p) { + close_pack(packs.info[i].p); + free(packs.info[i].p); } - free(packs.names[i]); + free(packs.info[i].pack_name); } - free(packs.list); - free(packs.names); + free(packs.info); free(entries); free(pack_perm); free(midx_name); From patchwork Tue May 14 18:47:49 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 10943585 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EFE9E1515 for ; Tue, 14 May 2019 18:48:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E027F28869 for ; Tue, 14 May 2019 18:48:08 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D41A928872; Tue, 14 May 2019 18:48:08 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1CF652886F for ; Tue, 14 May 2019 18:48:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727591AbfENSsH (ORCPT ); Tue, 14 May 2019 14:48:07 -0400 Received: from mail-yb1-f196.google.com ([209.85.219.196]:34364 "EHLO mail-yb1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727567AbfENSsG (ORCPT ); Tue, 14 May 2019 14:48:06 -0400 Received: by mail-yb1-f196.google.com with SMTP id b204so48201ybg.1 for ; Tue, 14 May 2019 11:48:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BPp9fCH2wUNothO2MalPsuHTFS1lyFTuflr4dcfE8oA=; b=BfLRwy+Es9YNoWAQ1RhgRJQ4ZgWfKdC2jFmoqZoHVRgId5mtV95FhJdQdGXcvUG9Qx didgVcAstRrpObflCOG2x4M65m0i0PmcWDoambtwT6PlfQyLQR4ca55M/Eoo66D6g4/p ayNsfAx/7Aw3SgKZh56cgSEPEpDt69yhyL0G4sGC0Uj33BcgKKQAYPemW25KELdpF2wT 3hFFTR/xyP787TwnEGQXDYoNYzWbVhib3yyZjs8//qB1d/lF1Wk9zImIyoiWNgbyqBNx 2a8AvWgkJCoIuO0x6CtNcJNxvwaCvT0QE6LyQhg5Tlnj3KJ77nF17mHOb2p074IjLf/a yvrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BPp9fCH2wUNothO2MalPsuHTFS1lyFTuflr4dcfE8oA=; b=fQA6qYJGJBtMS97KccBRY4v9OIYOUU/CgImsKhzs91PrOnw+9jfL1RC7//v8MDssLM /zAL0SsyKIdnnDVAtJpFxYYiNDUt7IwRRQwHNvlAl0N3wPLtOWRXTq0XidHS0nyhgAD9 oUaZTGeMK5Bh1d0pYk5872ePe5okGlmeHVkpePUubBKSsvHMvbuml1yQii45sDsdaD7+ t993UX1k7Ln6FgyMfJyMTK0ecH1cn/EUETwKcPKAmPvYlNBK2f+hQG8+fGYEqzBLGxXw cTEE5fELqXQ81BxQl/oQk8UE5hDGdXdUpNYjcp0mCz7Kr3iuJ+wh0APep6VZIGN+GKA1 5+Cg== X-Gm-Message-State: APjAAAV56WtKFkl4OHIh/2sAKk4izi8kkFeoPrh+cvI7w8y9jc9ro2BK s/6gchKG9K25uYnyZxVZ+FzPHxtFfww= X-Google-Smtp-Source: APXvYqw5HB3egK+/Ks26UyEws1YR5pRxgiu1owZ0KAzFMIX+Yc/vyQt+/vMxcPyueEJZQ1oWC6TgqQ== X-Received: by 2002:a25:54d5:: with SMTP id i204mr18613183ybb.124.1557859684933; Tue, 14 May 2019 11:48:04 -0700 (PDT) Received: from linux.mshome.net ([98.122.173.75]) by smtp.gmail.com with ESMTPSA id u17sm3438627ywu.94.2019.05.14.11.48.03 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 14 May 2019 11:48:04 -0700 (PDT) From: Derrick Stolee X-Google-Original-From: Derrick Stolee To: git@vger.kernel.org Cc: peff@peff.net, jrnieder@gmail.com, gitster@pobox.com, avarab@gmail.com, dstolee@microsoft.com Subject: [PATCH v6 06/11] multi-pack-index: implement 'expire' subcommand Date: Tue, 14 May 2019 18:47:49 +0000 Message-Id: <20190514184754.3196-7-dstolee@microsoft.com> X-Mailer: git-send-email 2.22.0.rc0 In-Reply-To: <20190514184754.3196-1-dstolee@microsoft.com> References: <20190424151428.170316-1-dstolee@microsoft.com> <20190514184754.3196-1-dstolee@microsoft.com> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The 'git multi-pack-index expire' subcommand looks at the existing mult-pack-index, counts the number of objects referenced in each pack-file, deletes the pack-fils with no referenced objects, and rewrites the multi-pack-index to no longer reference those packs. Refactor the write_midx_file() method to call write_midx_internal() which now takes an existing 'struct multi_pack_index' and a list of pack-files to drop (as specified by the names of their pack- indexes). As we write the new multi-pack-index, we drop those file names from the list of known pack-files. The expire_midx_packs() method removes the unreferenced pack-files after carefully closing the packs to avoid open handles. Test that a new pack-file that covers the contents of two other pack-files leads to those pack-files being deleted during the expire subcommand. Be sure to read the multi-pack-index to ensure it no longer references those packs. Signed-off-by: Derrick Stolee --- midx.c | 119 +++++++++++++++++++++++++++++++++--- t/t5319-multi-pack-index.sh | 20 ++++++ 2 files changed, 129 insertions(+), 10 deletions(-) diff --git a/midx.c b/midx.c index 6d4b84e243..9b0b4c1520 100644 --- a/midx.c +++ b/midx.c @@ -34,6 +34,8 @@ #define MIDX_CHUNK_LARGE_OFFSET_WIDTH (sizeof(uint64_t)) #define MIDX_LARGE_OFFSET_NEEDED 0x80000000 +#define PACK_EXPIRED UINT_MAX + static char *get_midx_filename(const char *object_dir) { return xstrfmt("%s/pack/multi-pack-index", object_dir); @@ -431,6 +433,7 @@ struct pack_info { uint32_t orig_pack_int_id; char *pack_name; struct packed_git *p; + unsigned expired : 1; }; static int pack_info_compare(const void *_a, const void *_b) @@ -478,6 +481,7 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len, packs->info[packs->nr].pack_name = xstrdup(file_name); packs->info[packs->nr].orig_pack_int_id = packs->nr; + packs->info[packs->nr].expired = 0; packs->nr++; } } @@ -637,13 +641,17 @@ static size_t write_midx_pack_names(struct hashfile *f, size_t written = 0; for (i = 0; i < num_packs; i++) { - size_t writelen = strlen(info[i].pack_name) + 1; + size_t writelen; + + if (info[i].expired) + continue; if (i && strcmp(info[i].pack_name, info[i - 1].pack_name) <= 0) BUG("incorrect pack-file order: %s before %s", info[i - 1].pack_name, info[i].pack_name); + writelen = strlen(info[i].pack_name) + 1; hashwrite(f, info[i].pack_name, writelen); written += writelen; } @@ -725,6 +733,11 @@ static size_t write_midx_object_offsets(struct hashfile *f, int large_offset_nee for (i = 0; i < nr_objects; i++) { struct pack_midx_entry *obj = list++; + if (perm[obj->pack_int_id] == PACK_EXPIRED) + BUG("object %s is in an expired pack with int-id %d", + oid_to_hex(&obj->oid), + obj->pack_int_id); + hashwrite_be32(f, perm[obj->pack_int_id]); if (large_offset_needed && obj->offset >> 31) @@ -771,7 +784,8 @@ static size_t write_midx_large_offsets(struct hashfile *f, uint32_t nr_large_off return written; } -int write_midx_file(const char *object_dir) +static int write_midx_internal(const char *object_dir, struct multi_pack_index *m, + struct string_list *packs_to_drop) { unsigned char cur_chunk, num_chunks = 0; char *midx_name; @@ -787,6 +801,8 @@ int write_midx_file(const char *object_dir) struct pack_midx_entry *entries = NULL; int large_offsets_needed = 0; int pack_name_concat_len = 0; + int dropped_packs = 0; + int result = 0; midx_name = get_midx_filename(object_dir); if (safe_create_leading_directories(midx_name)) { @@ -795,7 +811,10 @@ int write_midx_file(const char *object_dir) midx_name); } - packs.m = load_multi_pack_index(object_dir, 1); + if (m) + packs.m = m; + else + packs.m = load_multi_pack_index(object_dir, 1); packs.nr = 0; packs.alloc = packs.m ? packs.m->num_packs : 16; @@ -809,13 +828,14 @@ int write_midx_file(const char *object_dir) packs.info[packs.nr].orig_pack_int_id = i; packs.info[packs.nr].pack_name = xstrdup(packs.m->pack_names[i]); packs.info[packs.nr].p = NULL; + packs.info[packs.nr].expired = 0; packs.nr++; } } for_each_file_in_pack_dir(object_dir, add_pack_to_midx, &packs); - if (packs.m && packs.nr == packs.m->num_packs) + if (packs.m && packs.nr == packs.m->num_packs && !packs_to_drop) goto cleanup; entries = get_sorted_entries(packs.m, packs.info, packs.nr, &nr_entries); @@ -829,6 +849,34 @@ int write_midx_file(const char *object_dir) QSORT(packs.info, packs.nr, pack_info_compare); + if (packs_to_drop && packs_to_drop->nr) { + int drop_index = 0; + int missing_drops = 0; + + for (i = 0; i < packs.nr && drop_index < packs_to_drop->nr; i++) { + int cmp = strcmp(packs.info[i].pack_name, + packs_to_drop->items[drop_index].string); + + if (!cmp) { + drop_index++; + packs.info[i].expired = 1; + } else if (cmp > 0) { + error(_("did not see pack-file %s to drop"), + packs_to_drop->items[drop_index].string); + drop_index++; + missing_drops++; + i--; + } else { + packs.info[i].expired = 0; + } + } + + if (missing_drops) { + result = 1; + goto cleanup; + } + } + /* * pack_perm stores a permutation between pack-int-ids from the * previous multi-pack-index to the new one we are writing: @@ -837,11 +885,18 @@ int write_midx_file(const char *object_dir) */ ALLOC_ARRAY(pack_perm, packs.nr); for (i = 0; i < packs.nr; i++) { - pack_perm[packs.info[i].orig_pack_int_id] = i; + if (packs.info[i].expired) { + dropped_packs++; + pack_perm[packs.info[i].orig_pack_int_id] = PACK_EXPIRED; + } else { + pack_perm[packs.info[i].orig_pack_int_id] = i - dropped_packs; + } } - for (i = 0; i < packs.nr; i++) - pack_name_concat_len += strlen(packs.info[i].pack_name) + 1; + for (i = 0; i < packs.nr; i++) { + if (!packs.info[i].expired) + pack_name_concat_len += strlen(packs.info[i].pack_name) + 1; + } if (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT) pack_name_concat_len += MIDX_CHUNK_ALIGNMENT - @@ -857,7 +912,7 @@ int write_midx_file(const char *object_dir) cur_chunk = 0; num_chunks = large_offsets_needed ? 5 : 4; - written = write_midx_header(f, num_chunks, packs.nr); + written = write_midx_header(f, num_chunks, packs.nr - dropped_packs); chunk_ids[cur_chunk] = MIDX_CHUNKID_PACKNAMES; chunk_offsets[cur_chunk] = written + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH; @@ -958,7 +1013,12 @@ int write_midx_file(const char *object_dir) free(entries); free(pack_perm); free(midx_name); - return 0; + return result; +} + +int write_midx_file(const char *object_dir) +{ + return write_midx_internal(object_dir, NULL, NULL); } void clear_midx_file(struct repository *r) @@ -1125,5 +1185,44 @@ int verify_midx_file(struct repository *r, const char *object_dir) int expire_midx_packs(struct repository *r, const char *object_dir) { - return 0; + uint32_t i, *count, result = 0; + struct string_list packs_to_drop = STRING_LIST_INIT_DUP; + struct multi_pack_index *m = load_multi_pack_index(object_dir, 1); + + if (!m) + return 0; + + count = xcalloc(m->num_packs, sizeof(uint32_t)); + for (i = 0; i < m->num_objects; i++) { + int pack_int_id = nth_midxed_pack_int_id(m, i); + count[pack_int_id]++; + } + + for (i = 0; i < m->num_packs; i++) { + char *pack_name; + + if (count[i]) + continue; + + if (prepare_midx_pack(r, m, i)) + continue; + + if (m->packs[i]->pack_keep) + continue; + + pack_name = xstrdup(m->packs[i]->pack_name); + close_pack(m->packs[i]); + + string_list_insert(&packs_to_drop, m->pack_names[i]); + unlink_pack_path(pack_name, 0); + free(pack_name); + } + + free(count); + + if (packs_to_drop.nr) + result = write_midx_internal(object_dir, m, &packs_to_drop); + + string_list_clear(&packs_to_drop, 0); + return result; } diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index 1b2d32f475..12570fe7ac 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -412,4 +412,24 @@ test_expect_success 'expire does not remove any packs' ' ) ' +test_expect_success 'expire removes unreferenced packs' ' + ( + cd dup && + git pack-objects --revs .git/objects/pack/pack-combined <<-EOF && + refs/heads/A + ^refs/heads/C + EOF + git multi-pack-index write && + ls .git/objects/pack | grep -v -e pack-[AB] >expect && + git multi-pack-index expire && + ls .git/objects/pack >actual && + test_cmp expect actual && + ls .git/objects/pack/ | grep idx >expect-idx && + test-tool read-midx .git/objects | grep idx >actual-midx && + test_cmp expect-idx actual-midx && + git multi-pack-index verify && + git fsck + ) +' + test_done From patchwork Tue May 14 18:47:50 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 10943593 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B02C11515 for ; Tue, 14 May 2019 18:48:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A1D7A28869 for ; Tue, 14 May 2019 18:48:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9637328872; Tue, 14 May 2019 18:48:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E4C4F28869 for ; Tue, 14 May 2019 18:48:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727602AbfENSsJ (ORCPT ); Tue, 14 May 2019 14:48:09 -0400 Received: from mail-yw1-f65.google.com ([209.85.161.65]:45539 "EHLO mail-yw1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727595AbfENSsI (ORCPT ); Tue, 14 May 2019 14:48:08 -0400 Received: by mail-yw1-f65.google.com with SMTP id w18so154936ywa.12 for ; Tue, 14 May 2019 11:48:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=axdepppcWwepaUXDh1Cn0fOU3ssNWmrVISP9q6cCias=; b=CGz1/ttGDix5pQ5AqQXQ/OLcoEfY0BJ+a2C0Gp/84BmVTOQANTII8qqoEQltsmkea7 IFvD8vrE8jp8v5zZ26N0Kg4uYFUa+vgefO//x2FGarKRJJqnkk1oX9xpbSo2Upy8DcBM n9HxcD7Vnd11uvIyWS513juD67/HgQsCwntdqcqjAGW2drC/mgMkn974auhfkAKheYED 9J03mFgAemK/fzt2JnxC6F+Key9nLxgq4AE7Ta8lHl+N0Y83r12gjLi2mIyd1ZrC/xtY KDfisBPaTyRCRPa3SbZkyfgjsuSgKB/fkgi83RthYmvUia4QtqFIAydpLigpYu71/KbR QhTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=axdepppcWwepaUXDh1Cn0fOU3ssNWmrVISP9q6cCias=; b=brV2g2cYBSaPEuHc1QRH5CLEB18+2LNdWc84QBvMEqgKTMMXU9bRaK8TTd2eRW7eWT GU/n0FmroGq8r9QSLoTIcZTkUA8rifGSiDlSeuz7RTweAaOAFPCI8EjET7o8bwt5c0Fq PG4FMyqgZB2FrB4sVkjgsBpDXxXUQlf1fz9JN7vC3mxKkEYyGcSfhEhF5O5Opp9Olo2R 19bDwTFmmcCzsxUR1b1TWc6oxKGORJTKzA0OnrxJmnV+zZaiPYFzHhBMW0RVsUNyCYxU 65VPTIGTZCGwOMqeUwnvFrU7Pa1A5noo04SICwBRcRZASUUfavfLgFVCZIrLHFwicslr gUwg== X-Gm-Message-State: APjAAAUCn0ZCULpDgMW9Tu6kVuLY4m64qVG8ksJ2xqfBj/6gP8UDJlhb XbIWHjjbWBb2FZK+/qT7ShLOhx8YTqM= X-Google-Smtp-Source: APXvYqzujqUUrW/JZE+CdAj5iY86RXBe096QF3Av/SKUz4tpQ1Zo5x3K3GgoxcfSFJ23AnQTdMLPsA== X-Received: by 2002:a81:9ad3:: with SMTP id r202mr17439223ywg.139.1557859686461; Tue, 14 May 2019 11:48:06 -0700 (PDT) Received: from linux.mshome.net ([98.122.173.75]) by smtp.gmail.com with ESMTPSA id u17sm3438627ywu.94.2019.05.14.11.48.04 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 14 May 2019 11:48:05 -0700 (PDT) From: Derrick Stolee X-Google-Original-From: Derrick Stolee To: git@vger.kernel.org Cc: peff@peff.net, jrnieder@gmail.com, gitster@pobox.com, avarab@gmail.com, dstolee@microsoft.com Subject: [PATCH v6 07/11] multi-pack-index: prepare 'repack' subcommand Date: Tue, 14 May 2019 18:47:50 +0000 Message-Id: <20190514184754.3196-8-dstolee@microsoft.com> X-Mailer: git-send-email 2.22.0.rc0 In-Reply-To: <20190514184754.3196-1-dstolee@microsoft.com> References: <20190424151428.170316-1-dstolee@microsoft.com> <20190514184754.3196-1-dstolee@microsoft.com> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP In an environment where the multi-pack-index is useful, it is due to many pack-files and an inability to repack the object store into a single pack-file. However, it is likely that many of these pack-files are rather small, and could be repacked into a slightly larger pack-file without too much effort. It may also be important to ensure the object store is highly available and the repack operation does not interrupt concurrent git commands. Introduce a 'repack' subcommand to 'git multi-pack-index' that takes a '--batch-size' option. The subcommand will inspect the multi-pack-index for referenced pack-files whose size is smaller than the batch size, until collecting a list of pack-files whose sizes sum to larger than the batch size. Then, a new pack-file will be created containing the objects from those pack-files that are referenced by the multi-pack-index. The resulting pack is likely to actually be smaller than the batch size due to compression and the fact that there may be objects in the pack- files that have duplicate copies in other pack-files. The current change introduces the command-line arguments, and we add a test that ensures we parse these options properly. Since we specify a small batch size, we will guarantee that future implementations do not change the list of pack-files. In addition, we hard-code the modified times of the packs in the pack directory to ensure the list of packs sorted by modified time matches the order if sorted by size (ascending). This will be important in a future test. Signed-off-by: Derrick Stolee --- Documentation/git-multi-pack-index.txt | 17 +++++++++++++++++ builtin/multi-pack-index.c | 12 ++++++++++-- midx.c | 5 +++++ midx.h | 1 + t/t5319-multi-pack-index.sh | 20 +++++++++++++++++++- 5 files changed, 52 insertions(+), 3 deletions(-) diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt index 6186c4c936..233b2b7862 100644 --- a/Documentation/git-multi-pack-index.txt +++ b/Documentation/git-multi-pack-index.txt @@ -36,6 +36,23 @@ expire:: have no objects referenced by the MIDX. Rewrite the MIDX file afterward to remove all references to these pack-files. +repack:: + Create a new pack-file containing objects in small pack-files + referenced by the multi-pack-index. If the size given by the + `--batch-size=` argument is zero, then create a pack + containing all objects referenced by the multi-pack-index. For + a non-zero batch size, Select the pack-files by examining packs + from oldest-to-newest, computing the "expected size" by counting + the number of objects in the pack referenced by the + multi-pack-index, then divide by the total number of objects in + the pack and multiply by the pack size. We select packs with + expected size below the batch size until the set of packs have + total expected size at least the batch size. If the total size + does not reach the batch size, then do nothing. If a new pack- + file is created, rewrite the multi-pack-index to reference the + new pack-file. A later run of 'git multi-pack-index expire' will + delete the pack-files that were part of this batch. + EXAMPLES -------- diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c index ad10d40512..b1ea1a6aa1 100644 --- a/builtin/multi-pack-index.c +++ b/builtin/multi-pack-index.c @@ -6,12 +6,13 @@ #include "trace2.h" static char const * const builtin_multi_pack_index_usage[] = { - N_("git multi-pack-index [--object-dir=] (write|verify|expire)"), + N_("git multi-pack-index [--object-dir=] (write|verify|expire|repack --batch-size=)"), NULL }; static struct opts_multi_pack_index { const char *object_dir; + unsigned long batch_size; } opts; int cmd_multi_pack_index(int argc, const char **argv, @@ -20,6 +21,8 @@ int cmd_multi_pack_index(int argc, const char **argv, static struct option builtin_multi_pack_index_options[] = { OPT_FILENAME(0, "object-dir", &opts.object_dir, N_("object directory containing set of packfile and pack-index pairs")), + OPT_MAGNITUDE(0, "batch-size", &opts.batch_size, + N_("during repack, collect pack-files of smaller size into a batch that is larger than this size")), OPT_END(), }; @@ -43,6 +46,11 @@ int cmd_multi_pack_index(int argc, const char **argv, trace2_cmd_mode(argv[0]); + if (!strcmp(argv[0], "repack")) + return midx_repack(the_repository, opts.object_dir, (size_t)opts.batch_size); + if (opts.batch_size) + die(_("--batch-size option is only for 'repack' subcommand")); + if (!strcmp(argv[0], "write")) return write_midx_file(opts.object_dir); if (!strcmp(argv[0], "verify")) @@ -50,5 +58,5 @@ int cmd_multi_pack_index(int argc, const char **argv, if (!strcmp(argv[0], "expire")) return expire_midx_packs(the_repository, opts.object_dir); - die(_("unrecognized verb: %s"), argv[0]); + die(_("unrecognized subcommand: %s"), argv[0]); } diff --git a/midx.c b/midx.c index 9b0b4c1520..fbed8a8adb 100644 --- a/midx.c +++ b/midx.c @@ -1226,3 +1226,8 @@ int expire_midx_packs(struct repository *r, const char *object_dir) string_list_clear(&packs_to_drop, 0); return result; } + +int midx_repack(struct repository *r, const char *object_dir, size_t batch_size) +{ + return 0; +} diff --git a/midx.h b/midx.h index 505f1431b7..f0ae656b5d 100644 --- a/midx.h +++ b/midx.h @@ -51,6 +51,7 @@ int write_midx_file(const char *object_dir); void clear_midx_file(struct repository *r); int verify_midx_file(struct repository *r, const char *object_dir); int expire_midx_packs(struct repository *r, const char *object_dir); +int midx_repack(struct repository *r, const char *object_dir, size_t batch_size); void close_midx(struct multi_pack_index *m); diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index 12570fe7ac..133d5b7068 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -398,7 +398,8 @@ test_expect_success 'setup expire tests' ' git pack-objects --revs .git/objects/pack/pack-E <<-EOF && refs/heads/E EOF - git multi-pack-index write + git multi-pack-index write && + cp -r .git/objects/pack .git/objects/pack-backup ) ' @@ -432,4 +433,21 @@ test_expect_success 'expire removes unreferenced packs' ' ) ' +test_expect_success 'repack with minimum size does not alter existing packs' ' + ( + cd dup && + rm -rf .git/objects/pack && + mv .git/objects/pack-backup .git/objects/pack && + touch -m -t 201901010000 .git/objects/pack/pack-D* && + touch -m -t 201901010001 .git/objects/pack/pack-C* && + touch -m -t 201901010002 .git/objects/pack/pack-B* && + touch -m -t 201901010003 .git/objects/pack/pack-A* && + ls .git/objects/pack >expect && + MINSIZE=$(ls -l .git/objects/pack/*pack | awk "{print \$5;}" | sort -n | head -n 1) && + git multi-pack-index repack --batch-size=$MINSIZE && + ls .git/objects/pack >actual && + test_cmp expect actual + ) +' + test_done From patchwork Tue May 14 18:47:51 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 10943599 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 89C921515 for ; Tue, 14 May 2019 18:48:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7B8732886F for ; Tue, 14 May 2019 18:48:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6FEBE28895; Tue, 14 May 2019 18:48:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A86CB28872 for ; Tue, 14 May 2019 18:48:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727620AbfENSsO (ORCPT ); Tue, 14 May 2019 14:48:14 -0400 Received: from mail-yb1-f194.google.com ([209.85.219.194]:34881 "EHLO mail-yb1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727588AbfENSsI (ORCPT ); Tue, 14 May 2019 14:48:08 -0400 Received: by mail-yb1-f194.google.com with SMTP id k202so47928ybk.2 for ; Tue, 14 May 2019 11:48:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=bi1gxvreZD1UTXAfaDQsqcM/4QEp40JUTb7ZDMhd0UA=; b=XhRxY/zdDfv5cuhxgZ4NRoVBO7u7tKwwnG4o1JnxnqNMhqo5MJRMYOoxTVyBpjVbAW yt4ovoVi8zXMKEH1OOc93esO6FE/WvVS6okSPuZaf1ipbcpdHoWLbztDoXQQ1vzPSfPy G+qXC8Pd+Vq+Af3C3Zbwl7/KhvaScGRnQzhn8M5wp3OVySrX7GthgrZZcSubNhkqWe5f k3W2vW9VP4DoMhVmryoS3myJsm5AHTnjXKkFpWP5yeVFMHfofxzs2maE0+ZxTw7MaPQI hHlJTamk+i0pecLE7//LNZnwxWaodXIu9RmK2WzPY2/2L0XklqDAqJRfMTLOxf2pmEHR LtwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=bi1gxvreZD1UTXAfaDQsqcM/4QEp40JUTb7ZDMhd0UA=; b=OBZJcymdUbUIb7tGsQE6Vvv4eWEgvKRdJILPESbAB9fLsc127aagjt8Us3iifeT1lu n/sIaHNY8Js1cqJTguzbHq4tmfLfpS1MvzoZqiyxkT98G2oBUQObCtDKPsOAIg/Lzr19 ZTF+VtvbrhPgjiD+Un4PQ3oBo5hr24eoFZZq0e7J1r2GeItHSK1lI7eoqsZNdWm9ptVN mwhuYmcQ5jO/qN2ulaWkWjEqEsOe1kZPjzZhdSpZqtOPHOZxeHmvMKeqDci+x6Dc+dsA YKvyl1qhSIXykZ3huELxpNCsQd2gCv4QbOBXYSvYOapZYkTJA9hxZpXTLpPUjwxlTtqP +aLw== X-Gm-Message-State: APjAAAUp0PjM/6ZTFnhigqQc7ZruLtkEAAKWwmqdbNBC/i5M4tCaeK1v z+35jTSVfO9BFaj78j1+/0BTrzgMOgk= X-Google-Smtp-Source: APXvYqzEBIrWmbItE7WtSJGWpZGoLlu2SZJoTtzDRHWzO+thbRWjxCee0Y1FfwTNLDcl6DbYsTQbSA== X-Received: by 2002:a25:5456:: with SMTP id i83mr17582941ybb.301.1557859687436; Tue, 14 May 2019 11:48:07 -0700 (PDT) Received: from linux.mshome.net ([98.122.173.75]) by smtp.gmail.com with ESMTPSA id u17sm3438627ywu.94.2019.05.14.11.48.06 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 14 May 2019 11:48:06 -0700 (PDT) From: Derrick Stolee X-Google-Original-From: Derrick Stolee To: git@vger.kernel.org Cc: peff@peff.net, jrnieder@gmail.com, gitster@pobox.com, avarab@gmail.com, dstolee@microsoft.com Subject: [PATCH v6 08/11] midx: implement midx_repack() Date: Tue, 14 May 2019 18:47:51 +0000 Message-Id: <20190514184754.3196-9-dstolee@microsoft.com> X-Mailer: git-send-email 2.22.0.rc0 In-Reply-To: <20190514184754.3196-1-dstolee@microsoft.com> References: <20190424151428.170316-1-dstolee@microsoft.com> <20190514184754.3196-1-dstolee@microsoft.com> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP To repack with a non-zero batch-size, first sort all pack-files by their modified time. Second, walk those pack-files from oldest to newest, compute their expected size, and add the packs to a list if they are smaller than the given batch-size. Stop when the total expected size is at least the batch size. If the batch size is zero, select all packs in the multi-pack-index. Finally, collect the objects from the multi-pack-index that are in the selected packs and send them to 'git pack-objects'. Write a new multi-pack-index that includes the new pack. Using a batch size of zero is very similar to a standard 'git repack' command, except that we do not delete the old packs and instead rely on the new multi-pack-index to prevent new processes from reading the old packs. This does not disrupt other Git processes that are currently reading the old packs based on the old multi-pack-index. While first designing a 'git multi-pack-index repack' operation, I started by collecting the batches based on the actual size of the objects instead of the size of the pack-files. This allows repacking a large pack-file that has very few referencd objects. However, this came at a significant cost of parsing pack-files instead of simply reading the multi-pack-index and getting the file information for the pack-files. The "expected size" version provides similar behavior, but could skip a pack-file if the average object size is much larger than the actual size of the referenced objects, or can create a large pack if the actual size of the referenced objects is larger than the expected size. Signed-off-by: Derrick Stolee --- midx.c | 151 +++++++++++++++++++++++++++++++++++- t/t5319-multi-pack-index.sh | 28 +++++++ 2 files changed, 178 insertions(+), 1 deletion(-) diff --git a/midx.c b/midx.c index fbed8a8adb..d649644420 100644 --- a/midx.c +++ b/midx.c @@ -9,6 +9,7 @@ #include "midx.h" #include "progress.h" #include "trace2.h" +#include "run-command.h" #define MIDX_SIGNATURE 0x4d494458 /* "MIDX" */ #define MIDX_VERSION 1 @@ -1227,7 +1228,155 @@ int expire_midx_packs(struct repository *r, const char *object_dir) return result; } -int midx_repack(struct repository *r, const char *object_dir, size_t batch_size) +struct repack_info { + timestamp_t mtime; + uint32_t referenced_objects; + uint32_t pack_int_id; +}; + +static int compare_by_mtime(const void *a_, const void *b_) { + const struct repack_info *a, *b; + + a = (const struct repack_info *)a_; + b = (const struct repack_info *)b_; + + if (a->mtime < b->mtime) + return -1; + if (a->mtime > b->mtime) + return 1; + return 0; +} + +static int fill_included_packs_all(struct multi_pack_index *m, + unsigned char *include_pack) +{ + uint32_t i; + + for (i = 0; i < m->num_packs; i++) + include_pack[i] = 1; + + return m->num_packs < 2; +} + +static int fill_included_packs_batch(struct repository *r, + struct multi_pack_index *m, + unsigned char *include_pack, + size_t batch_size) +{ + uint32_t i, packs_to_repack; + size_t total_size; + struct repack_info *pack_info = xcalloc(m->num_packs, sizeof(struct repack_info)); + + for (i = 0; i < m->num_packs; i++) { + pack_info[i].pack_int_id = i; + + if (prepare_midx_pack(r, m, i)) + continue; + + pack_info[i].mtime = m->packs[i]->mtime; + } + + for (i = 0; batch_size && i < m->num_objects; i++) { + uint32_t pack_int_id = nth_midxed_pack_int_id(m, i); + pack_info[pack_int_id].referenced_objects++; + } + + QSORT(pack_info, m->num_packs, compare_by_mtime); + + total_size = 0; + packs_to_repack = 0; + for (i = 0; total_size < batch_size && i < m->num_packs; i++) { + int pack_int_id = pack_info[i].pack_int_id; + struct packed_git *p = m->packs[pack_int_id]; + size_t expected_size; + + if (!p) + continue; + if (open_pack_index(p) || !p->num_objects) + continue; + + expected_size = (size_t)(p->pack_size + * pack_info[i].referenced_objects); + expected_size /= p->num_objects; + + if (expected_size >= batch_size) + continue; + + packs_to_repack++; + total_size += expected_size; + include_pack[pack_int_id] = 1; + } + + free(pack_info); + + if (total_size < batch_size || packs_to_repack < 2) + return 1; + return 0; } + +int midx_repack(struct repository *r, const char *object_dir, size_t batch_size) +{ + int result = 0; + uint32_t i; + unsigned char *include_pack; + struct child_process cmd = CHILD_PROCESS_INIT; + struct strbuf base_name = STRBUF_INIT; + struct multi_pack_index *m = load_multi_pack_index(object_dir, 1); + + if (!m) + return 0; + + include_pack = xcalloc(m->num_packs, sizeof(unsigned char)); + + if (batch_size) { + if (fill_included_packs_batch(r, m, include_pack, batch_size)) + goto cleanup; + } else if (fill_included_packs_all(m, include_pack)) + goto cleanup; + + argv_array_push(&cmd.args, "pack-objects"); + + strbuf_addstr(&base_name, object_dir); + strbuf_addstr(&base_name, "/pack/pack"); + argv_array_push(&cmd.args, base_name.buf); + strbuf_release(&base_name); + + cmd.git_cmd = 1; + cmd.in = cmd.out = -1; + + if (start_command(&cmd)) { + error(_("could not start pack-objects")); + result = 1; + goto cleanup; + } + + for (i = 0; i < m->num_objects; i++) { + struct object_id oid; + uint32_t pack_int_id = nth_midxed_pack_int_id(m, i); + + if (!include_pack[pack_int_id]) + continue; + + nth_midxed_object_oid(&oid, m, i); + xwrite(cmd.in, oid_to_hex(&oid), the_hash_algo->hexsz); + xwrite(cmd.in, "\n", 1); + } + close(cmd.in); + + if (finish_command(&cmd)) { + error(_("could not finish pack-objects")); + result = 1; + goto cleanup; + } + + result = write_midx_internal(object_dir, m, NULL); + m = NULL; + +cleanup: + if (m) + close_midx(m); + free(include_pack); + return result; +} diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index 133d5b7068..6e47e5d0b2 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -450,4 +450,32 @@ test_expect_success 'repack with minimum size does not alter existing packs' ' ) ' +test_expect_success 'repack creates a new pack' ' + ( + cd dup && + ls .git/objects/pack/*idx >idx-list && + test_line_count = 5 idx-list && + THIRD_SMALLEST_SIZE=$(ls -l .git/objects/pack/*pack | awk "{print \$5;}" | sort -n | head -n 3 | tail -n 1) && + BATCH_SIZE=$(($THIRD_SMALLEST_SIZE + 1)) && + git multi-pack-index repack --batch-size=$BATCH_SIZE && + ls .git/objects/pack/*idx >idx-list && + test_line_count = 6 idx-list && + test-tool read-midx .git/objects | grep idx >midx-list && + test_line_count = 6 midx-list + ) +' + +test_expect_success 'expire removes repacked packs' ' + ( + cd dup && + ls -al .git/objects/pack/*pack && + ls -S .git/objects/pack/*pack | head -n 4 >expect && + git multi-pack-index expire && + ls -S .git/objects/pack/*pack >actual && + test_cmp expect actual && + test-tool read-midx .git/objects | grep idx >midx-list && + test_line_count = 4 midx-list + ) +' + test_done From patchwork Tue May 14 18:47:52 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 10943595 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 25AE3933 for ; Tue, 14 May 2019 18:48:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 17F8D28869 for ; Tue, 14 May 2019 18:48:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0C05328872; Tue, 14 May 2019 18:48:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A86C228869 for ; Tue, 14 May 2019 18:48:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727604AbfENSsK (ORCPT ); Tue, 14 May 2019 14:48:10 -0400 Received: from mail-yw1-f65.google.com ([209.85.161.65]:36067 "EHLO mail-yw1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727599AbfENSsJ (ORCPT ); Tue, 14 May 2019 14:48:09 -0400 Received: by mail-yw1-f65.google.com with SMTP id q185so207232ywe.3 for ; Tue, 14 May 2019 11:48:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=EGsmOKPIL5P0IKvn2T3Nwt3q7pHwGfvyAH8FIELiDbU=; b=LxaNeJSOhkxTvq7pyLzoTdlrJP6hKsKDzxBcFPR23T1+2p7/eGkfqaw1Vnh++In8X5 vWSjl5sZjTV9FlQU+Ux2LWPG5FXfoYbNaXCDnPOx7N3WWnYbA1YnbO06w8Mx9bc92Pm8 hE/yzMmLE4fuVKg+OQSDL2mK0Wm0bJdWm75HroZoFm8JYBpZVlOxWij5UgQutTje7FEN cESIB4OmRTqrVMlEvXaqQrprf+3FbTVV8wb6q4ipA1Zv9qyD1cKzh5qngMdVkohhaRxc xrLCkP1XKcO3gMjXnDY+FJO8gOKiLoMWbrAgPVNZN/ZYMVC+poHmThUSNwruRsLuyie/ JWUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=EGsmOKPIL5P0IKvn2T3Nwt3q7pHwGfvyAH8FIELiDbU=; b=DugfYExAv7wcFlVTLeUaODMMmz8oJI0WfIXuMn+iandIC2d6rFALFalUcqg/vbgSlw P1L/PiFV1/5I553DU53IJ+BNCPmCbNz+Z+ORq5REpIABV3V3qmT/hTFM1zPGukWQZyMd Tc4PL2jGHvSY+uplcaGtUbWsFRGXtgMcGwTYEK8LdLW8AAyE/iQfBMcH2xH82+O2X/II mVPgqXcFE9lSBWZgC7VVhQRoz0r3fwfuziuzEyRynSPJ7tXOZiQLs5ESfVn8Drn5iOyz qnuKvxtacCAqaU1XW0DARrmr8ejTMBw3023U9U+rFocl6SCgD3fLZjzhU9aewXSNxN6t HS6A== X-Gm-Message-State: APjAAAUNl6o29VvW2HOkH0c+cLlULcw/a5pVFZe+XE7sB38aEVXLfqJ3 BqOtcgnMvSXK0adMTiHeuEtyd5wxLeo= X-Google-Smtp-Source: APXvYqwOP0RqUkPJWFUjOOF9FeJ39LbSwNY4kKgIDEtib8gr8GnMBxEviwBXWYo3K/ZJcEIqBquLiA== X-Received: by 2002:a0d:dd8c:: with SMTP id g134mr18581807ywe.84.1557859688274; Tue, 14 May 2019 11:48:08 -0700 (PDT) Received: from linux.mshome.net ([98.122.173.75]) by smtp.gmail.com with ESMTPSA id u17sm3438627ywu.94.2019.05.14.11.48.07 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 14 May 2019 11:48:07 -0700 (PDT) From: Derrick Stolee X-Google-Original-From: Derrick Stolee To: git@vger.kernel.org Cc: peff@peff.net, jrnieder@gmail.com, gitster@pobox.com, avarab@gmail.com, dstolee@microsoft.com Subject: [PATCH v6 09/11] multi-pack-index: test expire while adding packs Date: Tue, 14 May 2019 18:47:52 +0000 Message-Id: <20190514184754.3196-10-dstolee@microsoft.com> X-Mailer: git-send-email 2.22.0.rc0 In-Reply-To: <20190514184754.3196-1-dstolee@microsoft.com> References: <20190424151428.170316-1-dstolee@microsoft.com> <20190514184754.3196-1-dstolee@microsoft.com> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP During development of the multi-pack-index expire subcommand, a version went out that improperly computed the pack order if a new pack was introduced while other packs were being removed. Part of the subtlety of the bug involved the new pack being placed before other packs that already existed in the multi-pack-index. Add a test to t5319-multi-pack-index.sh that catches this issue. The test adds new packs that cause another pack to be expired, and creates new packs that are lexicographically sorted before and after the existing packs. Signed-off-by: Derrick Stolee --- t/t5319-multi-pack-index.sh | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index 6e47e5d0b2..8e04ce2821 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -478,4 +478,36 @@ test_expect_success 'expire removes repacked packs' ' ) ' +test_expect_success 'expire works when adding new packs' ' + ( + cd dup && + git pack-objects --revs .git/objects/pack/pack-combined <<-EOF && + refs/heads/A + ^refs/heads/B + EOF + git pack-objects --revs .git/objects/pack/pack-combined <<-EOF && + refs/heads/B + ^refs/heads/C + EOF + git pack-objects --revs .git/objects/pack/pack-combined <<-EOF && + refs/heads/C + ^refs/heads/D + EOF + git multi-pack-index write && + git pack-objects --revs .git/objects/pack/a-pack <<-EOF && + refs/heads/D + ^refs/heads/E + EOF + git multi-pack-index write && + git pack-objects --revs .git/objects/pack/z-pack <<-EOF && + refs/heads/E + EOF + git multi-pack-index expire && + ls .git/objects/pack/ | grep idx >expect && + test-tool read-midx .git/objects | grep idx >actual && + test_cmp expect actual && + git multi-pack-index verify + ) +' + test_done From patchwork Tue May 14 18:47:53 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 10943603 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9B1321390 for ; Tue, 14 May 2019 18:48:22 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8CE1328869 for ; Tue, 14 May 2019 18:48:22 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8150C28872; Tue, 14 May 2019 18:48:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2DDAF28869 for ; Tue, 14 May 2019 18:48:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727612AbfENSsL (ORCPT ); Tue, 14 May 2019 14:48:11 -0400 Received: from mail-yb1-f196.google.com ([209.85.219.196]:41321 "EHLO mail-yb1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727603AbfENSsK (ORCPT ); Tue, 14 May 2019 14:48:10 -0400 Received: by mail-yb1-f196.google.com with SMTP id a13so48059ybl.8 for ; Tue, 14 May 2019 11:48:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=vfk1TRfxOcZlsxokNPrgCse88p6CBjQ+EJwegADPK5w=; b=MDIfteL4dYNcZEDOpimPyafl571sjcGgVbYq6qQjV+bm5fwbGGBNfTpzGHmwJX570I e2gNXy5UdcCW0cVaSBLt2ufMT6KlcpPaSVzg7FJrduc1SWPVpDj5Aoevw1GavFea8vTP k+beKF6ztJPgiu1041iwC/rwFXc0lEHyYR+12tV+YduPK+CzzMf0UNvzFRS2hapC8NQ+ FB42euzXuDHfA2iICLxiSvhqR+KaZvS+lDuD7zPrP3m/CVTMUIbLmjl47SOLf+jwWFuc 2Xnmb/Prq/K2JMOvzdoajfK02rFSi0ya0oPl4V8unP7t1lUqD6Z8w4FCSdPyUNhUw3qX oshA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=vfk1TRfxOcZlsxokNPrgCse88p6CBjQ+EJwegADPK5w=; b=YCpzWtVbdPxeqrmgvfYTYBCLgrIWR3wkYTtG1p4bG2JbMCAoW0awyg0FRlC0sWN8S9 OVybz0qGgRrdmVPGrWigvwmGotrVWFK8hZkVOO4Sr0EMCnDZiggnEnORBA8dB4KOPjKZ 9lrn1I/pzIjTmJ/g+B6ua5ez+54txk8I1zvlXF6cLlKHAjnhBaT2gFqXEYUXUruoADn4 THd/ItN7K0QV5ZlZqJqYzXMNlD753jF+wjXdPDPaIUyB8KZGk6/+LHqWuA48CBQzqlDW vk4uE/CRTNLsHH4MidK4ji3u2YzThJ49BuwK9znZqadRft335Re4UNx2Vp/B0UfTEzYy r+jg== X-Gm-Message-State: APjAAAXgNoovJ7cfa2aGiXXmlgl/ocUI6R4YjdgMshUWZt5N4uw6vTRI Z7hDsSupreVqVIG3ium9CTj1UMFAfB0= X-Google-Smtp-Source: APXvYqyxN+0Lh705SCTt1WnnevvPhWcU19C5uutbKNWRdOiWs0LOq1pbDaZmP9yLfZyAxMrw0BZuiQ== X-Received: by 2002:a25:ec4:: with SMTP id 187mr16706408ybo.280.1557859689354; Tue, 14 May 2019 11:48:09 -0700 (PDT) Received: from linux.mshome.net ([98.122.173.75]) by smtp.gmail.com with ESMTPSA id u17sm3438627ywu.94.2019.05.14.11.48.08 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 14 May 2019 11:48:08 -0700 (PDT) From: Derrick Stolee X-Google-Original-From: Derrick Stolee To: git@vger.kernel.org Cc: peff@peff.net, jrnieder@gmail.com, gitster@pobox.com, avarab@gmail.com, dstolee@microsoft.com Subject: [PATCH v6 10/11] midx: add test that 'expire' respects .keep files Date: Tue, 14 May 2019 18:47:53 +0000 Message-Id: <20190514184754.3196-11-dstolee@microsoft.com> X-Mailer: git-send-email 2.22.0.rc0 In-Reply-To: <20190514184754.3196-1-dstolee@microsoft.com> References: <20190424151428.170316-1-dstolee@microsoft.com> <20190514184754.3196-1-dstolee@microsoft.com> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The 'git multi-pack-index expire' subcommand may delete packs that are not needed from the perspective of the multi-pack-index. If a pack has a .keep file, then we should not delete that pack. Add a test that ensures we preserve a pack that would otherwise be expired. First, create a new pack that contains every object in the repo, then add it to the multi-pack-index. Then create a .keep file for a pack starting with "a-pack" that was added in the previous test. Finally, expire and verify that the pack remains and the other packs were expired. Signed-off-by: Derrick Stolee --- t/t5319-multi-pack-index.sh | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index 8e04ce2821..c288901401 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -510,4 +510,22 @@ test_expect_success 'expire works when adding new packs' ' ) ' +test_expect_success 'expire respects .keep files' ' + ( + cd dup && + git pack-objects --revs .git/objects/pack/pack-all <<-EOF && + refs/heads/A + EOF + git multi-pack-index write && + PACKA=$(ls .git/objects/pack/a-pack*\.pack | sed s/\.pack\$//) && + touch $PACKA.keep && + git multi-pack-index expire && + ls -S .git/objects/pack/a-pack* | grep $PACKA >a-pack-files && + test_line_count = 3 a-pack-files && + test-tool read-midx .git/objects | grep idx >midx-list && + test_line_count = 2 midx-list + ) +' + + test_done From patchwork Tue May 14 18:47:54 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 10943597 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 04D25933 for ; Tue, 14 May 2019 18:48:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E6FF12886F for ; Tue, 14 May 2019 18:48:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D57F428898; Tue, 14 May 2019 18:48:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 016582886F for ; Tue, 14 May 2019 18:48:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727617AbfENSsN (ORCPT ); Tue, 14 May 2019 14:48:13 -0400 Received: from mail-yb1-f195.google.com ([209.85.219.195]:36118 "EHLO mail-yb1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727599AbfENSsL (ORCPT ); Tue, 14 May 2019 14:48:11 -0400 Received: by mail-yb1-f195.google.com with SMTP id m10so48253ybk.3 for ; Tue, 14 May 2019 11:48:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=xOCUZ94x/1qoVIO+uvPKWNBVbcVLSLgCya5+r0aWF3Q=; b=p/JDMbTF+k/aWepe5LNG+Kt+TKgqO3MFFf9YCv7CzPc+TI4ItaqPLPHNVNkN4G8BTN P4tSKtE+IjUwdWM4vXtnzwUfCuMl+ayC4eULC2LcLgfv+dp3rLaUG18o18rLUl4j7CYH NyAL7QzxIgCns9tFFfnCehey+Nv6SJvKy+uejJ3tihpD09b94ESZOpPDIzZCCGCOXe6c RtTx6lkOw9F9GGjsQodyeSEI33GxIu1V4sVObFq7iSYnpydGKtPIOtmW9tCVU0YaEVc4 a2jhMLR/SjInsEMO2pcmyqmpjRRCKxB+Nz74dvocOBECxLw2xDDNzJc0RiWXArgnjnmb Yynw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=xOCUZ94x/1qoVIO+uvPKWNBVbcVLSLgCya5+r0aWF3Q=; b=jFWVKZcFAZANj/ZWwBOmayW9bPlD0hDcfYUqID3JjsLC899a9wF5NwW1GO+LN7YSYI DgkPQe3DVkP9hq1KlKdtQFXhIpmyIu1ofijvn2KoygOvcjAw32AnKxGNE836ce3Ld1Db jW47zBPL73gGo2SD4IROgOOZ+YEgz1ebbU92UBz5OiQCMn3lh0wBJNnvxY3iHFMyPqPB RHy9VnmGmVYb/J782Al3DUwEVNQV50cQ8yEHOLNDRk0nIamuwMr2NibjcInWE1N6QMdv ghyJOIgo7g3aIRt7xckhuvwtmChJh9CNpKe81DRvNKeaGf2eP9YLhkYX56RZQHuTWTwN VGAQ== X-Gm-Message-State: APjAAAUG2rFPekVXy1scB0FHG6vG+2KHI+IPvEG2T1l9Ha7cjh7z0r+1 J/mYxGJqOMbjQsBUQtwAtEl1hrGWQmA= X-Google-Smtp-Source: APXvYqwR5Q3cjFr2+PKegOJgbfn9clfIG5iVFOVGlMmL9amY1Oh68XmUeGYTJ1xTdYFo9k0H2jRfcw== X-Received: by 2002:a25:db45:: with SMTP id g66mr19216895ybf.509.1557859690231; Tue, 14 May 2019 11:48:10 -0700 (PDT) Received: from linux.mshome.net ([98.122.173.75]) by smtp.gmail.com with ESMTPSA id u17sm3438627ywu.94.2019.05.14.11.48.09 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 14 May 2019 11:48:09 -0700 (PDT) From: Derrick Stolee X-Google-Original-From: Derrick Stolee To: git@vger.kernel.org Cc: peff@peff.net, jrnieder@gmail.com, gitster@pobox.com, avarab@gmail.com, dstolee@microsoft.com Subject: [PATCH v6 11/11] t5319-multi-pack-index.sh: test batch size zero Date: Tue, 14 May 2019 18:47:54 +0000 Message-Id: <20190514184754.3196-12-dstolee@microsoft.com> X-Mailer: git-send-email 2.22.0.rc0 In-Reply-To: <20190514184754.3196-1-dstolee@microsoft.com> References: <20190424151428.170316-1-dstolee@microsoft.com> <20190514184754.3196-1-dstolee@microsoft.com> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The 'git multi-pack-index repack' command can take a batch size of zero, which creates a new pack-file containing all objects in the multi-pack-index. The first 'repack' command will create one new pack-file, and an 'expire' command after that will delete the old pack-files, as they no longer contain any referenced objects in the multi-pack-index. We must remove the .keep file that was added in the previous test in order to expire that pack-file. Also test that a 'repack' will do nothing if there is only one pack-file. Signed-off-by: Derrick Stolee --- t/t5319-multi-pack-index.sh | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index c288901401..79bfaeafa9 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -527,5 +527,24 @@ test_expect_success 'expire respects .keep files' ' ) ' +test_expect_success 'repack --batch-size=0 repacks everything' ' + ( + cd dup && + rm .git/objects/pack/*.keep && + ls .git/objects/pack/*idx >idx-list && + test_line_count = 2 idx-list && + git multi-pack-index repack --batch-size=0 && + ls .git/objects/pack/*idx >idx-list && + test_line_count = 3 idx-list && + test-tool read-midx .git/objects | grep idx >midx-list && + test_line_count = 3 midx-list && + git multi-pack-index expire && + ls -al .git/objects/pack/*idx >idx-list && + test_line_count = 1 idx-list && + git multi-pack-index repack --batch-size=0 && + ls -al .git/objects/pack/*idx >new-idx-list && + test_cmp idx-list new-idx-list + ) +' test_done