From patchwork Wed Apr 24 15:14:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 10915011 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F1DBE1708 for ; Wed, 24 Apr 2019 15:14:47 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E091A28A8B for ; Wed, 24 Apr 2019 15:14:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D264728A95; Wed, 24 Apr 2019 15:14:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 58F282898B for ; Wed, 24 Apr 2019 15:14:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730567AbfDXPOq (ORCPT ); Wed, 24 Apr 2019 11:14:46 -0400 Received: from mail-qt1-f194.google.com ([209.85.160.194]:34941 "EHLO mail-qt1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730492AbfDXPOp (ORCPT ); Wed, 24 Apr 2019 11:14:45 -0400 Received: by mail-qt1-f194.google.com with SMTP id l17so1510874qtp.2 for ; Wed, 24 Apr 2019 08:14:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=uJmisl9eTZ2db8prImrjM+CSFfGcBImygH6QMGsr3Tc=; b=Eb4nX2eL9cs498Trws8kiJvNVD3d4gdV406ARQht0UqFOb6g2wJ1xlt4ouYUY5R7ck OjVHi15nXc4zWlLCgEe6NMj202rKtSxHEZqeQzTuD3hfrjKATLBJHtcI/vQxuc5S2XWg Vqz2l5ZdRVNjdKd3y+diSM7Vr0kouNrIjUNiMkVZ5GbjZACyowY38qiHNHQg6EtEoRim uwXttUCvJIEwH0BPVoAfp1xpFs+MlNXETqjNCkFTAtq4xIJi8+1cC913lRnW3Z6AoAln tGzCfPWfrQ5K4vhMoZu1sCE6ZR7bNB2eliDQywQGAD5/oYoK8HhKST1t2JJ6ZE4A2vsP gQ2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=uJmisl9eTZ2db8prImrjM+CSFfGcBImygH6QMGsr3Tc=; b=Ow0VDpkQ57hAsr1cbs4CKCOY/Q3YSmtZThayeEnUb1lz0M8UXLPTBxBUJDASRq/o3e g21PP9u1+YCadL7dcZ11Q7/qaNHiS+hXKgaS30D7jCdcU/4GzB5xl08s3Hu/EnO5f41e +DJWrrf3vxgDnFPd8MqQ+uQg/tCxvojL4YjEvE1P+Unf1NWSJjv2aQR41g56+3gjpKtf fWkoWb/Z8Pz7j2Ktkv79wfXJb9c+7NNoavfe+/CNCVSBHaf33xeC7fbjgUDRyS3H6xHo iPjvFxXDMOWnSZaPbBLIodyhgkZCGzQ+mXau7oZzLUStqHCiz6Kb/zi9B9XKc6K2u7to rB9A== X-Gm-Message-State: APjAAAUVAWSLPN8uSZeGZXDrr2RBgQVpQCLTcFyl4bmYGM5e+fT63GGB ddIvngvYLDkQTgGFS6/Idrsr8uwJ X-Google-Smtp-Source: APXvYqyFZUrCPCo59qaoBXTGOjmH9vzzADHhHZeHe0deVR8Mv+HMAs45dxmnQTMJYNEXP3/m3wTKLQ== X-Received: by 2002:a0c:b6c8:: with SMTP id h8mr25594828qve.67.1556118884548; Wed, 24 Apr 2019 08:14:44 -0700 (PDT) Received: from stolee-gitdev.corp.microsoft.com ([2001:4898:8010:2:9efc:3578:ef3e:58d5]) by smtp.gmail.com with ESMTPSA id j129sm9671005qkd.51.2019.04.24.08.14.43 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 24 Apr 2019 08:14:43 -0700 (PDT) From: Derrick Stolee X-Google-Original-From: Derrick Stolee To: git@vger.kernel.org Cc: peff@peff.net, jrnieder@gmail.com, avarab@gmail.com, gitster@pobox.com, Derrick Stolee Subject: [PATCH v5 01/11] repack: refactor pack deletion for future use Date: Wed, 24 Apr 2019 11:14:18 -0400 Message-Id: <20190424151428.170316-2-dstolee@microsoft.com> X-Mailer: git-send-email 2.21.0.1096.g1c91fdc207 In-Reply-To: <20190424151428.170316-1-dstolee@microsoft.com> References: <20190424151428.170316-1-dstolee@microsoft.com> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The repack builtin deletes redundant pack-files and their associated .idx, .promisor, .bitmap, and .keep files. We will want to re-use this logic in the future for other types of repack, so pull the logic into 'unlink_pack_path()' in packfile.c. The 'ignore_keep' parameter is enabled for the use in repack, but will be important for a future caller. Signed-off-by: Derrick Stolee --- builtin/repack.c | 14 ++------------ packfile.c | 28 ++++++++++++++++++++++++++++ packfile.h | 7 +++++++ 3 files changed, 37 insertions(+), 12 deletions(-) diff --git a/builtin/repack.c b/builtin/repack.c index 45583683ee..3d445b34b4 100644 --- a/builtin/repack.c +++ b/builtin/repack.c @@ -129,19 +129,9 @@ static void get_non_kept_pack_filenames(struct string_list *fname_list, static void remove_redundant_pack(const char *dir_name, const char *base_name) { - const char *exts[] = {".pack", ".idx", ".keep", ".bitmap", ".promisor"}; - int i; struct strbuf buf = STRBUF_INIT; - size_t plen; - - strbuf_addf(&buf, "%s/%s", dir_name, base_name); - plen = buf.len; - - for (i = 0; i < ARRAY_SIZE(exts); i++) { - strbuf_setlen(&buf, plen); - strbuf_addstr(&buf, exts[i]); - unlink(buf.buf); - } + strbuf_addf(&buf, "%s/%s.pack", dir_name, base_name); + unlink_pack_path(buf.buf, 1); strbuf_release(&buf); } diff --git a/packfile.c b/packfile.c index d1e6683ffe..bacecb4d0d 100644 --- a/packfile.c +++ b/packfile.c @@ -352,6 +352,34 @@ void close_all_packs(struct raw_object_store *o) } } +void unlink_pack_path(const char *pack_name, int force_delete) +{ + static const char *exts[] = {".pack", ".idx", ".keep", ".bitmap", ".promisor"}; + int i; + struct strbuf buf = STRBUF_INIT; + size_t plen; + + strbuf_addstr(&buf, pack_name); + strip_suffix_mem(buf.buf, &buf.len, ".pack"); + plen = buf.len; + + if (!force_delete) { + strbuf_addstr(&buf, ".keep"); + if (!access(buf.buf, F_OK)) { + strbuf_release(&buf); + return; + } + } + + for (i = 0; i < ARRAY_SIZE(exts); i++) { + strbuf_setlen(&buf, plen); + strbuf_addstr(&buf, exts[i]); + unlink(buf.buf); + } + + strbuf_release(&buf); +} + /* * The LRU pack is the one with the oldest MRU window, preferring packs * with no used windows, or the oldest mtime if it has no windows allocated. diff --git a/packfile.h b/packfile.h index 6c4037605d..5b7bcdb1dd 100644 --- a/packfile.h +++ b/packfile.h @@ -86,6 +86,13 @@ extern void unuse_pack(struct pack_window **); extern void clear_delta_base_cache(void); extern struct packed_git *add_packed_git(const char *path, size_t path_len, int local); +/* + * Unlink the .pack and associated extension files. + * Does not unlink if 'force_delete' is false and the pack-file is + * marked as ".keep". + */ +extern void unlink_pack_path(const char *pack_name, int force_delete); + /* * Make sure that a pointer access into an mmap'd index file is within bounds, * and can provide at least 8 bytes of data. From patchwork Wed Apr 24 15:14:19 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 10915015 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9BA2514DB for ; Wed, 24 Apr 2019 15:14:49 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8B5C0287C1 for ; Wed, 24 Apr 2019 15:14:49 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7F590287E6; Wed, 24 Apr 2019 15:14:49 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 24F0F287C1 for ; Wed, 24 Apr 2019 15:14:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730588AbfDXPOr (ORCPT ); Wed, 24 Apr 2019 11:14:47 -0400 Received: from mail-qt1-f193.google.com ([209.85.160.193]:42268 "EHLO mail-qt1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730578AbfDXPOq (ORCPT ); Wed, 24 Apr 2019 11:14:46 -0400 Received: by mail-qt1-f193.google.com with SMTP id p20so20669692qtc.9 for ; Wed, 24 Apr 2019 08:14:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=cQGIytjRh9X3ZKArMmtc546ORvU00+dVpAORoSUQA4E=; b=I+J+3SoQdBX2zSho6n7lpjLai0lJOxcsS34NYQZs5TCdXa5iN6NRG2DozBR19Zit7O qWEfQAR/RTwtb81woKNtLnmw6PIJgzkf40jChyRo6ob1+Yv7DLeLMrIcO82f1paZFtoZ pX22e0tmAusTO8BTzV4UZD7aDhOtqRzzjaJS/iMKV8SWdclQmQUroOAos5N85MGMiRtR bSQLYlrjdYAXX/H+hyzrr3lK1mASK6zVSBuH1BcgBOwhw/wxBWGmAEflsAoFbN5aHDGK yOewhNEZgQ4SrSxKpxpT9Nc6NvEn1iFxhrBWuCUwR6UIYJPObIhaJiQMXY+WD0axw6uY 552w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=cQGIytjRh9X3ZKArMmtc546ORvU00+dVpAORoSUQA4E=; b=V933RgDTybnrBwxZTBVVGfIkG09TH+UC7T329OI/d3+RrxXJTJsyb71Fb/gdXHsAno shG4uzTeuOFyeaf8/Tu7XmTFndk5aIza8WxSkOD3q/gArY8kxI+nC4b4NxSy+Xsjbsr1 USkd6MqGXdOjNKd4AxO9NMoS2MfX3+GH6I77lE9KGJfQQo60Ec0ZRxSOHFGX2AISXDhZ +04pAkM/RMIZ3mZmkC+T6vzdySvV8CeLrzNz8npk1nXtg0G3re0rHiq3BwwSXuUpVzDZ /Lr5nleUAz0MNwfOnTSeAnz9KHfYolRn9FvzONbLtEfzTV+CpKyQdBciek6g1t/m6oD+ 3PPg== X-Gm-Message-State: APjAAAU6srGHl2UEtUvljc0ZnwNXKp2dsTADhS4Pqm7mYBi1D3bCUQf3 /oKz+8cutM6nq8FaJnwFNPe/d6ib X-Google-Smtp-Source: APXvYqyT/ZpmRw537ODcWYvfObSOFdMvHeNC/Lgp0prSbcxLzcx8uYAW8mNcd0LErg2+nf6pFAl1nA== X-Received: by 2002:a0c:fdc7:: with SMTP id g7mr6545251qvs.230.1556118885502; Wed, 24 Apr 2019 08:14:45 -0700 (PDT) Received: from stolee-gitdev.corp.microsoft.com ([2001:4898:8010:2:9efc:3578:ef3e:58d5]) by smtp.gmail.com with ESMTPSA id j129sm9671005qkd.51.2019.04.24.08.14.44 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 24 Apr 2019 08:14:45 -0700 (PDT) From: Derrick Stolee X-Google-Original-From: Derrick Stolee To: git@vger.kernel.org Cc: peff@peff.net, jrnieder@gmail.com, avarab@gmail.com, gitster@pobox.com, Derrick Stolee , Stefan Beller , =?utf-8?q?Szeder_G=C3=A1bor?= Subject: [PATCH v5 02/11] Docs: rearrange subcommands for multi-pack-index Date: Wed, 24 Apr 2019 11:14:19 -0400 Message-Id: <20190424151428.170316-3-dstolee@microsoft.com> X-Mailer: git-send-email 2.21.0.1096.g1c91fdc207 In-Reply-To: <20190424151428.170316-1-dstolee@microsoft.com> References: <20190424151428.170316-1-dstolee@microsoft.com> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP We will add new subcommands to the multi-pack-index, and that will make the documentation a bit messier. Clean up the 'verb' descriptions by renaming the concept to 'subcommand' and removing the reference to the object directory. Helped-by: Stefan Beller Helped-by: Szeder Gábor Signed-off-by: Derrick Stolee --- Documentation/git-multi-pack-index.txt | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt index f7778a2c85..1af406aca2 100644 --- a/Documentation/git-multi-pack-index.txt +++ b/Documentation/git-multi-pack-index.txt @@ -9,7 +9,7 @@ git-multi-pack-index - Write and verify multi-pack-indexes SYNOPSIS -------- [verse] -'git multi-pack-index' [--object-dir=] +'git multi-pack-index' [--object-dir=] DESCRIPTION ----------- @@ -23,13 +23,13 @@ OPTIONS `/packs/multi-pack-index` for the current MIDX file, and `/packs` for the pack-files to index. +The following subcommands are available: + write:: - When given as the verb, write a new MIDX file to - `/packs/multi-pack-index`. + Write a new MIDX file. verify:: - When given as the verb, verify the contents of the MIDX file - at `/packs/multi-pack-index`. + Verify the contents of the MIDX file. EXAMPLES From patchwork Wed Apr 24 15:14:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 10915017 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 72A1614DB for ; Wed, 24 Apr 2019 15:14:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 634BB28803 for ; Wed, 24 Apr 2019 15:14:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5793C288C3; Wed, 24 Apr 2019 15:14:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CB1D228803 for ; Wed, 24 Apr 2019 15:14:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730607AbfDXPOs (ORCPT ); Wed, 24 Apr 2019 11:14:48 -0400 Received: from mail-qt1-f194.google.com ([209.85.160.194]:37613 "EHLO mail-qt1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730492AbfDXPOr (ORCPT ); Wed, 24 Apr 2019 11:14:47 -0400 Received: by mail-qt1-f194.google.com with SMTP id z16so20680776qtn.4 for ; Wed, 24 Apr 2019 08:14:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=6H+N929r8R60/wZq6VTPfiFZ5jBhmCGMJNLiY+0rvx0=; b=Y2yRmwICC08rFGnRGI082ysJbPvNaQ/qbBpihWLzlxvDJabM2LfZO4SonRmXlRQR7P TJPTLVi5PfJe1qUyVerP0a3FUX50HmD5k0OTh+RpFEmSO+hpNL5KT7a8Jb+1NlWQTAXt wV8SthO4VSyXANEaHpHBNndSfrAmE5F+BHpETVt5n/Ygdyfry5azmgN7r+7daD8ZTLm3 ocy8zmQH7K/KpdVjCvkC4lM6pKxyA8Qv7e3zATiW348+IegJiNnRf7HjxwcEJ8GdDqwj Lv0Cov298DLmzA9AQpvwYY86PgMJkZRTO/322iiO9z/r6R36wGT4pAJIVsSykcqTQ5tP sd9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6H+N929r8R60/wZq6VTPfiFZ5jBhmCGMJNLiY+0rvx0=; b=hw/CNjqNAHFXwAefbvQUlAzcw+LSVaN7du3Gqs3UlxOnb0yidtkhIoSYUWFHUYSmyW BWSLRx453irXZElW/ZxCynzn/N9KtgSMj/bBEecWoHh2mG7o9i5vCiyJT+nByMl/OTt1 QrKHVHejqXx5OJPYnxQRKXEQ91hg2wtmByUr79tTEr1WyGNkpM86fUIFCw9g71mresL2 FJhNoZzDbT+KrEGanEoimRCppXFB83eJEBKvOsABl0iBAg3pFTAdI7TmBN1HkppsGrBP Zt5cQ5n3Ap4EjASim4syRIRb85CGFXD7izwSKafZNgPxfpCo9JM0NKn+j+nh4NieS44U Wj3A== X-Gm-Message-State: APjAAAXB+yFTr1p4Auu/4sFCRUeYcoTCaCErt2ys7rL/uDyp7DQhbm8F +IFF0+SPRarQdZwlXzGog+cF+S00 X-Google-Smtp-Source: APXvYqx7WgDbqMcYY/J76bKcOjIaITsj2NKaBtRiCJUy2hWTngJqVvHyxACWAa38kVoEf2CBSczB0g== X-Received: by 2002:ac8:740f:: with SMTP id p15mr20311369qtq.244.1556118886437; Wed, 24 Apr 2019 08:14:46 -0700 (PDT) Received: from stolee-gitdev.corp.microsoft.com ([2001:4898:8010:2:9efc:3578:ef3e:58d5]) by smtp.gmail.com with ESMTPSA id j129sm9671005qkd.51.2019.04.24.08.14.45 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 24 Apr 2019 08:14:46 -0700 (PDT) From: Derrick Stolee X-Google-Original-From: Derrick Stolee To: git@vger.kernel.org Cc: peff@peff.net, jrnieder@gmail.com, avarab@gmail.com, gitster@pobox.com, Derrick Stolee Subject: [PATCH v5 03/11] multi-pack-index: prepare for 'expire' subcommand Date: Wed, 24 Apr 2019 11:14:20 -0400 Message-Id: <20190424151428.170316-4-dstolee@microsoft.com> X-Mailer: git-send-email 2.21.0.1096.g1c91fdc207 In-Reply-To: <20190424151428.170316-1-dstolee@microsoft.com> References: <20190424151428.170316-1-dstolee@microsoft.com> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The multi-pack-index tracks objects in a collection of pack-files. Only one copy of each object is indexed, using the modified time of the pack-files to determine tie-breakers. It is possible to have a pack-file with no referenced objects because all objects have a duplicate in a newer pack-file. Introduce a new 'expire' subcommand to the multi-pack-index builtin. This subcommand will delete these unused pack-files and rewrite the multi-pack-index to no longer refer to those files. More details about the specifics will follow as the method is implemented. Add a test that verifies the 'expire' subcommand is correctly wired, but will still be valid when the verb is implemented. Specifically, create a set of packs that should all have referenced objects and should not be removed during an 'expire' operation. The packs are created carefully to ensure they have a specific order when sorted by size. This will be important in a later test. Signed-off-by: Derrick Stolee --- Documentation/git-multi-pack-index.txt | 5 +++ builtin/multi-pack-index.c | 4 ++- midx.c | 5 +++ midx.h | 1 + t/t5319-multi-pack-index.sh | 49 ++++++++++++++++++++++++++ 5 files changed, 63 insertions(+), 1 deletion(-) diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt index 1af406aca2..6186c4c936 100644 --- a/Documentation/git-multi-pack-index.txt +++ b/Documentation/git-multi-pack-index.txt @@ -31,6 +31,11 @@ write:: verify:: Verify the contents of the MIDX file. +expire:: + Delete the pack-files that are tracked by the MIDX file, but + have no objects referenced by the MIDX. Rewrite the MIDX file + afterward to remove all references to these pack-files. + EXAMPLES -------- diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c index fca70f8e4f..145de3a46c 100644 --- a/builtin/multi-pack-index.c +++ b/builtin/multi-pack-index.c @@ -5,7 +5,7 @@ #include "midx.h" static char const * const builtin_multi_pack_index_usage[] = { - N_("git multi-pack-index [--object-dir=] (write|verify)"), + N_("git multi-pack-index [--object-dir=] (write|verify|expire)"), NULL }; @@ -44,6 +44,8 @@ int cmd_multi_pack_index(int argc, const char **argv, return write_midx_file(opts.object_dir); if (!strcmp(argv[0], "verify")) return verify_midx_file(opts.object_dir); + if (!strcmp(argv[0], "expire")) + return expire_midx_packs(opts.object_dir); die(_("unrecognized verb: %s"), argv[0]); } diff --git a/midx.c b/midx.c index 730ff84dff..bb825ef816 100644 --- a/midx.c +++ b/midx.c @@ -1025,3 +1025,8 @@ int verify_midx_file(const char *object_dir) return verify_midx_error; } + +int expire_midx_packs(const char *object_dir) +{ + return 0; +} diff --git a/midx.h b/midx.h index 774f652530..e3a2b740b5 100644 --- a/midx.h +++ b/midx.h @@ -49,6 +49,7 @@ int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, i int write_midx_file(const char *object_dir); void clear_midx_file(struct repository *r); int verify_midx_file(const char *object_dir); +int expire_midx_packs(const char *object_dir); void close_midx(struct multi_pack_index *m); diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index 70926b5bc0..a8528f7da0 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -348,4 +348,53 @@ test_expect_success 'verify incorrect 64-bit offset' ' "incorrect object offset" ' +test_expect_success 'setup expire tests' ' + mkdir dup && + ( + cd dup && + git init && + test-tool genrandom "data" 4096 >large_file.txt && + git update-index --add large_file.txt && + for i in $(test_seq 1 20) + do + test_commit $i + done && + git branch A HEAD && + git branch B HEAD~8 && + git branch C HEAD~13 && + git branch D HEAD~16 && + git branch E HEAD~18 && + git pack-objects --revs .git/objects/pack/pack-A <<-EOF && + refs/heads/A + ^refs/heads/B + EOF + git pack-objects --revs .git/objects/pack/pack-B <<-EOF && + refs/heads/B + ^refs/heads/C + EOF + git pack-objects --revs .git/objects/pack/pack-C <<-EOF && + refs/heads/C + ^refs/heads/D + EOF + git pack-objects --revs .git/objects/pack/pack-D <<-EOF && + refs/heads/D + ^refs/heads/E + EOF + git pack-objects --revs .git/objects/pack/pack-E <<-EOF && + refs/heads/E + EOF + git multi-pack-index write + ) +' + +test_expect_success 'expire does not remove any packs' ' + ( + cd dup && + ls .git/objects/pack >expect && + git multi-pack-index expire && + ls .git/objects/pack >actual && + test_cmp expect actual + ) +' + test_done From patchwork Wed Apr 24 15:14:21 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 10915019 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8DE34922 for ; Wed, 24 Apr 2019 15:14:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7DBF2288DA for ; Wed, 24 Apr 2019 15:14:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 724E22881A; Wed, 24 Apr 2019 15:14:51 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 03E1E288D2 for ; Wed, 24 Apr 2019 15:14:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730628AbfDXPOt (ORCPT ); Wed, 24 Apr 2019 11:14:49 -0400 Received: from mail-qt1-f196.google.com ([209.85.160.196]:44499 "EHLO mail-qt1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730578AbfDXPOs (ORCPT ); Wed, 24 Apr 2019 11:14:48 -0400 Received: by mail-qt1-f196.google.com with SMTP id s10so9878691qtc.11 for ; Wed, 24 Apr 2019 08:14:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=N6SUva1olGMN+39zFmsX7r5s+yiam+eS1v58g+8eTRI=; b=GYMq7SrJlW3rlP91RiZWdrtluUUdv4BabRtkVwbpbCJwmQNeo61JOuq5dxR3l7jxk8 yttQ0ARrHzDEeItDiRBWhRDL1Mb4yyxPCDwgkiizGJttwVn1O3ODhsUuD7uwS2q+g8/f JVF84s1rfpEsVJokmN+n8o871VbPyL40fx4av/R4FvIOMD8ogaUyad6dvsA34mdMbVJK 9FfALqZK+5HexTQlrzAiyjfSvv4uqX3mhXaYfkvo9zsfZELSpNXntxi4skqmbYkx/XQo b1tKl6VaUycwCEFUiS7ZFkXcummpG/yzVLXJFzm3WIkkm7AoDIONonkJ56AA1WIJgYIT yJaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=N6SUva1olGMN+39zFmsX7r5s+yiam+eS1v58g+8eTRI=; b=cZmFqoQrFOYRPpaHohwlP1CROEtqA8fGxhtFmL06k9/jco3KftqYJ8JX349dEct7TF ZK6Irkl5EoKhA8+bBU1G2f4ZoY6ybVJPRUVoqDZy/pidIMzJKKFXSzFIYqDrKRdAwYzF Th61/mo7xAE03RV2MsS/+FdWgQvAo5DLNlPfQkhLaoEBQ0asQ7wSVgyh40f+dTxqXZBe 1ldK2AUPipBIZfATYp1iqSrqNaytlDZqQ/9nu8WPdPVxjsaFSfKI5uVPWp3xO0J0cj82 pVqGmGgm12KIkIJe5/rnyst4HUpVwGQqmHoRujT5W/dFUcOaiO5awzH0q56L+k5N3RVp oOTg== X-Gm-Message-State: APjAAAUwaW3Uf1uIreWcu3ogtQoLvf7lCJfdrA3eNz1EWfrxY7T7bsN3 9Tju2WXGHdGW39hy4DVrKTt+uMn3 X-Google-Smtp-Source: APXvYqxUEbPqu51GDx4NIGiFlDoiVTaF7kO/5vUvLV1WJLTdg6MWaRd3+yuechsWuDwuku6NUM/VIA== X-Received: by 2002:ac8:240a:: with SMTP id c10mr25837869qtc.217.1556118887492; Wed, 24 Apr 2019 08:14:47 -0700 (PDT) Received: from stolee-gitdev.corp.microsoft.com ([2001:4898:8010:2:9efc:3578:ef3e:58d5]) by smtp.gmail.com with ESMTPSA id j129sm9671005qkd.51.2019.04.24.08.14.46 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 24 Apr 2019 08:14:46 -0700 (PDT) From: Derrick Stolee X-Google-Original-From: Derrick Stolee To: git@vger.kernel.org Cc: peff@peff.net, jrnieder@gmail.com, avarab@gmail.com, gitster@pobox.com, Derrick Stolee Subject: [PATCH v5 04/11] midx: simplify computation of pack name lengths Date: Wed, 24 Apr 2019 11:14:21 -0400 Message-Id: <20190424151428.170316-5-dstolee@microsoft.com> X-Mailer: git-send-email 2.21.0.1096.g1c91fdc207 In-Reply-To: <20190424151428.170316-1-dstolee@microsoft.com> References: <20190424151428.170316-1-dstolee@microsoft.com> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Before writing the multi-pack-index, we compute the length of the pack-index names concatenated together. This forms the data in the pack name chunk, and we precompute it to compute chunk offsets. The value is also modified to fit alignment needs. Previously, this computation was coupled with adding packs from the existing multi-pack-index and the remaining packs in the object dir not already covered by the multi-pack-index. In anticipation of this becoming more complicated with the 'expire' subcommand, simplify the computation by centralizing it to a single loop before writing the file. Signed-off-by: Derrick Stolee --- midx.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/midx.c b/midx.c index bb825ef816..f087bbbe82 100644 --- a/midx.c +++ b/midx.c @@ -383,7 +383,6 @@ struct pack_list { uint32_t nr; uint32_t alloc_list; uint32_t alloc_names; - size_t pack_name_concat_len; struct multi_pack_index *m; }; @@ -418,7 +417,6 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len, } packs->names[packs->nr] = xstrdup(file_name); - packs->pack_name_concat_len += strlen(file_name) + 1; packs->nr++; } } @@ -762,6 +760,7 @@ int write_midx_file(const char *object_dir) uint32_t nr_entries, num_large_offsets = 0; struct pack_midx_entry *entries = NULL; int large_offsets_needed = 0; + int pack_name_concat_len = 0; midx_name = get_midx_filename(object_dir); if (safe_create_leading_directories(midx_name)) { @@ -777,7 +776,6 @@ int write_midx_file(const char *object_dir) packs.alloc_names = packs.alloc_list; packs.list = NULL; packs.names = NULL; - packs.pack_name_concat_len = 0; ALLOC_ARRAY(packs.list, packs.alloc_list); ALLOC_ARRAY(packs.names, packs.alloc_names); @@ -788,7 +786,6 @@ int write_midx_file(const char *object_dir) packs.list[packs.nr] = NULL; packs.names[packs.nr] = xstrdup(packs.m->pack_names[i]); - packs.pack_name_concat_len += strlen(packs.names[packs.nr]) + 1; packs.nr++; } } @@ -798,10 +795,6 @@ int write_midx_file(const char *object_dir) if (packs.m && packs.nr == packs.m->num_packs) goto cleanup; - if (packs.pack_name_concat_len % MIDX_CHUNK_ALIGNMENT) - packs.pack_name_concat_len += MIDX_CHUNK_ALIGNMENT - - (packs.pack_name_concat_len % MIDX_CHUNK_ALIGNMENT); - ALLOC_ARRAY(pack_perm, packs.nr); sort_packs_by_name(packs.names, packs.nr, pack_perm); @@ -814,6 +807,13 @@ int write_midx_file(const char *object_dir) large_offsets_needed = 1; } + for (i = 0; i < packs.nr; i++) + pack_name_concat_len += strlen(packs.names[i]) + 1; + + if (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT) + pack_name_concat_len += MIDX_CHUNK_ALIGNMENT - + (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT); + hold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR); f = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf); FREE_AND_NULL(midx_name); @@ -831,7 +831,7 @@ int write_midx_file(const char *object_dir) cur_chunk++; chunk_ids[cur_chunk] = MIDX_CHUNKID_OIDFANOUT; - chunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + packs.pack_name_concat_len; + chunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + pack_name_concat_len; cur_chunk++; chunk_ids[cur_chunk] = MIDX_CHUNKID_OIDLOOKUP; From patchwork Wed Apr 24 15:14:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 10915033 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 479EC1708 for ; Wed, 24 Apr 2019 15:15:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 35F8722B27 for ; Wed, 24 Apr 2019 15:15:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2A88B2875F; Wed, 24 Apr 2019 15:15:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 56E04287D3 for ; Wed, 24 Apr 2019 15:15:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730666AbfDXPOx (ORCPT ); Wed, 24 Apr 2019 11:14:53 -0400 Received: from mail-qt1-f194.google.com ([209.85.160.194]:41911 "EHLO mail-qt1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730635AbfDXPOv (ORCPT ); Wed, 24 Apr 2019 11:14:51 -0400 Received: by mail-qt1-f194.google.com with SMTP id f25so11940294qtc.8 for ; Wed, 24 Apr 2019 08:14:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=1hCNOD38lYtwGqR5yX23Tw57UX3TBgFmYr7bhpahtZA=; b=VbNJ4imvpXfx5X5lK3hz1FwNvq4bsKNX/OX22Sk8qY+ZlZF2cY/Ys75Xeqg7woGT9D gUMi/jDcnLfqyv28iy4Ex1jW4/hTV9F+KkyosBu6gY79F/1FvpFCLiLgjNnVwhbTOBSr GugrbQx4k8AYBUZr2/PdCAo4JDsRE03WvZvoHadpKhoat67sejVip8eiDUrOhFE1+GCY aC/uW9qNuV+iuGk05hH0yJEcryQYb0V+Jw8cG3EbAaPGUvR5KLpIarNclgvDHTxLVmqp JFJl7SdRUi6QtIkelDFaD4fTLIU0Qf4jNVGH2Y+ETTs8CwErFrNsiW7h8c9tw31qTwSB 9lmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=1hCNOD38lYtwGqR5yX23Tw57UX3TBgFmYr7bhpahtZA=; b=C4aEOW5mov7pcGI2y87XKjQZnGswyFU+Zo2+sRUnkvuCPY3g4SG/rcnw04VcfsSvws VqSz7JW7yseDv+7fOGrxxkAGYFoa6wOtbCvoXhxVm2sBR6tgJ+DHR7dD7KFVyLxLFRdM J/cEhxzbVHBakcec5kM0iOvqabxOp6ki6nLIf8wz/roUz8wx6Ntye4+8f28qgIWlDAMq c1vo2yKnlkFXL2Hp0QwQFI+jX2iu/hiXXCmyoJbBUeytMZb4xAPoPP3uVOMRj/ZaAVi+ QlydTFlseg9CdawP87Lc0GSOFGnPONlvQ2gzAZU9yUWbTb50X4sr8BCB8io2S0aKbL6R zQxg== X-Gm-Message-State: APjAAAXIEyzrYJ/TqydSB/EeyR5KIF7DfI04gtgG4g64oTKTKJLAWvb2 3PvbpCCqSzSNrdAE08i9fPd9tXrN X-Google-Smtp-Source: APXvYqwa99d5H7xGgF9bdwYP/hY6cOlMt4fYa7k1Laeba1G5A/wfHNArBGn4gPGgQa+zHdDs+y+rYQ== X-Received: by 2002:aed:3a44:: with SMTP id n62mr26173803qte.147.1556118888638; Wed, 24 Apr 2019 08:14:48 -0700 (PDT) Received: from stolee-gitdev.corp.microsoft.com ([2001:4898:8010:2:9efc:3578:ef3e:58d5]) by smtp.gmail.com with ESMTPSA id j129sm9671005qkd.51.2019.04.24.08.14.47 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 24 Apr 2019 08:14:48 -0700 (PDT) From: Derrick Stolee X-Google-Original-From: Derrick Stolee To: git@vger.kernel.org Cc: peff@peff.net, jrnieder@gmail.com, avarab@gmail.com, gitster@pobox.com, Derrick Stolee Subject: [PATCH v5 05/11] midx: refactor permutation logic and pack sorting Date: Wed, 24 Apr 2019 11:14:22 -0400 Message-Id: <20190424151428.170316-6-dstolee@microsoft.com> X-Mailer: git-send-email 2.21.0.1096.g1c91fdc207 In-Reply-To: <20190424151428.170316-1-dstolee@microsoft.com> References: <20190424151428.170316-1-dstolee@microsoft.com> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP In anticipation of the expire subcommand, refactor the way we sort the packfiles by name. This will greatly simplify our approach to dropping expired packs from the list. First, create 'struct pack_info' to replace 'struct pack_pair'. This struct contains the necessary information about a pack, including its name, a pointer to its packfile struct (if not already in the multi-pack-index), and the original pack-int-id. Second, track the pack information using an array of pack_info structs in the pack_list struct. This simplifies the logic around the multiple arrays we were tracking in that struct. Finally, update get_sorted_entries() to not permute the pack-int-id and instead supply the permutation to write_midx_object_offsets(). This requires sorting the packs after get_sorted_entries(). Signed-off-by: Derrick Stolee --- midx.c | 156 +++++++++++++++++++++++++-------------------------------- 1 file changed, 69 insertions(+), 87 deletions(-) diff --git a/midx.c b/midx.c index f087bbbe82..95c39106b2 100644 --- a/midx.c +++ b/midx.c @@ -377,12 +377,23 @@ static size_t write_midx_header(struct hashfile *f, return MIDX_HEADER_SIZE; } +struct pack_info { + uint32_t orig_pack_int_id; + char *pack_name; + struct packed_git *p; +}; + +static int pack_info_compare(const void *_a, const void *_b) +{ + struct pack_info *a = (struct pack_info *)_a; + struct pack_info *b = (struct pack_info *)_b; + return strcmp(a->pack_name, b->pack_name); +} + struct pack_list { - struct packed_git **list; - char **names; + struct pack_info *info; uint32_t nr; - uint32_t alloc_list; - uint32_t alloc_names; + uint32_t alloc; struct multi_pack_index *m; }; @@ -395,66 +406,32 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len, if (packs->m && midx_contains_pack(packs->m, file_name)) return; - ALLOC_GROW(packs->list, packs->nr + 1, packs->alloc_list); - ALLOC_GROW(packs->names, packs->nr + 1, packs->alloc_names); + ALLOC_GROW(packs->info, packs->nr + 1, packs->alloc); - packs->list[packs->nr] = add_packed_git(full_path, - full_path_len, - 0); + packs->info[packs->nr].p = add_packed_git(full_path, + full_path_len, + 0); - if (!packs->list[packs->nr]) { + if (!packs->info[packs->nr].p) { warning(_("failed to add packfile '%s'"), full_path); return; } - if (open_pack_index(packs->list[packs->nr])) { + if (open_pack_index(packs->info[packs->nr].p)) { warning(_("failed to open pack-index '%s'"), full_path); - close_pack(packs->list[packs->nr]); - FREE_AND_NULL(packs->list[packs->nr]); + close_pack(packs->info[packs->nr].p); + FREE_AND_NULL(packs->info[packs->nr].p); return; } - packs->names[packs->nr] = xstrdup(file_name); + packs->info[packs->nr].pack_name = xstrdup(file_name); + packs->info[packs->nr].orig_pack_int_id = packs->nr; packs->nr++; } } -struct pack_pair { - uint32_t pack_int_id; - char *pack_name; -}; - -static int pack_pair_compare(const void *_a, const void *_b) -{ - struct pack_pair *a = (struct pack_pair *)_a; - struct pack_pair *b = (struct pack_pair *)_b; - return strcmp(a->pack_name, b->pack_name); -} - -static void sort_packs_by_name(char **pack_names, uint32_t nr_packs, uint32_t *perm) -{ - uint32_t i; - struct pack_pair *pairs; - - ALLOC_ARRAY(pairs, nr_packs); - - for (i = 0; i < nr_packs; i++) { - pairs[i].pack_int_id = i; - pairs[i].pack_name = pack_names[i]; - } - - QSORT(pairs, nr_packs, pack_pair_compare); - - for (i = 0; i < nr_packs; i++) { - pack_names[i] = pairs[i].pack_name; - perm[pairs[i].pack_int_id] = i; - } - - free(pairs); -} - struct pack_midx_entry { struct object_id oid; uint32_t pack_int_id; @@ -480,7 +457,6 @@ static int midx_oid_compare(const void *_a, const void *_b) } static int nth_midxed_pack_midx_entry(struct multi_pack_index *m, - uint32_t *pack_perm, struct pack_midx_entry *e, uint32_t pos) { @@ -488,7 +464,7 @@ static int nth_midxed_pack_midx_entry(struct multi_pack_index *m, return 1; nth_midxed_object_oid(&e->oid, m, pos); - e->pack_int_id = pack_perm[nth_midxed_pack_int_id(m, pos)]; + e->pack_int_id = nth_midxed_pack_int_id(m, pos); e->offset = nth_midxed_offset(m, pos); /* consider objects in midx to be from "old" packs */ @@ -522,8 +498,7 @@ static void fill_pack_entry(uint32_t pack_int_id, * of a packfile containing the object). */ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m, - struct packed_git **p, - uint32_t *perm, + struct pack_info *info, uint32_t nr_packs, uint32_t *nr_objects) { @@ -534,7 +509,7 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m, uint32_t start_pack = m ? m->num_packs : 0; for (cur_pack = start_pack; cur_pack < nr_packs; cur_pack++) - total_objects += p[cur_pack]->num_objects; + total_objects += info[cur_pack].p->num_objects; /* * As we de-duplicate by fanout value, we expect the fanout @@ -559,7 +534,7 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m, for (cur_object = start; cur_object < end; cur_object++) { ALLOC_GROW(entries_by_fanout, nr_fanout + 1, alloc_fanout); - nth_midxed_pack_midx_entry(m, perm, + nth_midxed_pack_midx_entry(m, &entries_by_fanout[nr_fanout], cur_object); nr_fanout++; @@ -570,12 +545,12 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m, uint32_t start = 0, end; if (cur_fanout) - start = get_pack_fanout(p[cur_pack], cur_fanout - 1); - end = get_pack_fanout(p[cur_pack], cur_fanout); + start = get_pack_fanout(info[cur_pack].p, cur_fanout - 1); + end = get_pack_fanout(info[cur_pack].p, cur_fanout); for (cur_object = start; cur_object < end; cur_object++) { ALLOC_GROW(entries_by_fanout, nr_fanout + 1, alloc_fanout); - fill_pack_entry(perm[cur_pack], p[cur_pack], cur_object, &entries_by_fanout[nr_fanout]); + fill_pack_entry(cur_pack, info[cur_pack].p, cur_object, &entries_by_fanout[nr_fanout]); nr_fanout++; } } @@ -604,7 +579,7 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m, } static size_t write_midx_pack_names(struct hashfile *f, - char **pack_names, + struct pack_info *info, uint32_t num_packs) { uint32_t i; @@ -612,14 +587,14 @@ static size_t write_midx_pack_names(struct hashfile *f, size_t written = 0; for (i = 0; i < num_packs; i++) { - size_t writelen = strlen(pack_names[i]) + 1; + size_t writelen = strlen(info[i].pack_name) + 1; - if (i && strcmp(pack_names[i], pack_names[i - 1]) <= 0) + if (i && strcmp(info[i].pack_name, info[i - 1].pack_name) <= 0) BUG("incorrect pack-file order: %s before %s", - pack_names[i - 1], - pack_names[i]); + info[i - 1].pack_name, + info[i].pack_name); - hashwrite(f, pack_names[i], writelen); + hashwrite(f, info[i].pack_name, writelen); written += writelen; } @@ -690,6 +665,7 @@ static size_t write_midx_oid_lookup(struct hashfile *f, unsigned char hash_len, } static size_t write_midx_object_offsets(struct hashfile *f, int large_offset_needed, + uint32_t *perm, struct pack_midx_entry *objects, uint32_t nr_objects) { struct pack_midx_entry *list = objects; @@ -699,7 +675,7 @@ static size_t write_midx_object_offsets(struct hashfile *f, int large_offset_nee for (i = 0; i < nr_objects; i++) { struct pack_midx_entry *obj = list++; - hashwrite_be32(f, obj->pack_int_id); + hashwrite_be32(f, perm[obj->pack_int_id]); if (large_offset_needed && obj->offset >> 31) hashwrite_be32(f, MIDX_LARGE_OFFSET_NEEDED | nr_large_offset++); @@ -772,20 +748,17 @@ int write_midx_file(const char *object_dir) packs.m = load_multi_pack_index(object_dir, 1); packs.nr = 0; - packs.alloc_list = packs.m ? packs.m->num_packs : 16; - packs.alloc_names = packs.alloc_list; - packs.list = NULL; - packs.names = NULL; - ALLOC_ARRAY(packs.list, packs.alloc_list); - ALLOC_ARRAY(packs.names, packs.alloc_names); + packs.alloc = packs.m ? packs.m->num_packs : 16; + packs.info = NULL; + ALLOC_ARRAY(packs.info, packs.alloc); if (packs.m) { for (i = 0; i < packs.m->num_packs; i++) { - ALLOC_GROW(packs.list, packs.nr + 1, packs.alloc_list); - ALLOC_GROW(packs.names, packs.nr + 1, packs.alloc_names); + ALLOC_GROW(packs.info, packs.nr + 1, packs.alloc); - packs.list[packs.nr] = NULL; - packs.names[packs.nr] = xstrdup(packs.m->pack_names[i]); + packs.info[packs.nr].orig_pack_int_id = i; + packs.info[packs.nr].pack_name = xstrdup(packs.m->pack_names[i]); + packs.info[packs.nr].p = NULL; packs.nr++; } } @@ -795,10 +768,7 @@ int write_midx_file(const char *object_dir) if (packs.m && packs.nr == packs.m->num_packs) goto cleanup; - ALLOC_ARRAY(pack_perm, packs.nr); - sort_packs_by_name(packs.names, packs.nr, pack_perm); - - entries = get_sorted_entries(packs.m, packs.list, pack_perm, packs.nr, &nr_entries); + entries = get_sorted_entries(packs.m, packs.info, packs.nr, &nr_entries); for (i = 0; i < nr_entries; i++) { if (entries[i].offset > 0x7fffffff) @@ -807,8 +777,21 @@ int write_midx_file(const char *object_dir) large_offsets_needed = 1; } + QSORT(packs.info, packs.nr, pack_info_compare); + + /* + * pack_perm stores a permutation between pack-int-ids from the + * previous multi-pack-index to the new one we are writing: + * + * pack_perm[old_id] = new_id + */ + ALLOC_ARRAY(pack_perm, packs.nr); + for (i = 0; i < packs.nr; i++) { + pack_perm[packs.info[i].orig_pack_int_id] = i; + } + for (i = 0; i < packs.nr; i++) - pack_name_concat_len += strlen(packs.names[i]) + 1; + pack_name_concat_len += strlen(packs.info[i].pack_name) + 1; if (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT) pack_name_concat_len += MIDX_CHUNK_ALIGNMENT - @@ -879,7 +862,7 @@ int write_midx_file(const char *object_dir) switch (chunk_ids[i]) { case MIDX_CHUNKID_PACKNAMES: - written += write_midx_pack_names(f, packs.names, packs.nr); + written += write_midx_pack_names(f, packs.info, packs.nr); break; case MIDX_CHUNKID_OIDFANOUT: @@ -891,7 +874,7 @@ int write_midx_file(const char *object_dir) break; case MIDX_CHUNKID_OBJECTOFFSETS: - written += write_midx_object_offsets(f, large_offsets_needed, entries, nr_entries); + written += write_midx_object_offsets(f, large_offsets_needed, pack_perm, entries, nr_entries); break; case MIDX_CHUNKID_LARGEOFFSETS: @@ -914,15 +897,14 @@ int write_midx_file(const char *object_dir) cleanup: for (i = 0; i < packs.nr; i++) { - if (packs.list[i]) { - close_pack(packs.list[i]); - free(packs.list[i]); + if (packs.info[i].p) { + close_pack(packs.info[i].p); + free(packs.info[i].p); } - free(packs.names[i]); + free(packs.info[i].pack_name); } - free(packs.list); - free(packs.names); + free(packs.info); free(entries); free(pack_perm); free(midx_name); From patchwork Wed Apr 24 15:14:23 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 10915031 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EB7CA13B5 for ; Wed, 24 Apr 2019 15:15:02 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DA3B722B27 for ; Wed, 24 Apr 2019 15:15:02 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CEA462875F; Wed, 24 Apr 2019 15:15:02 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5F148288DA for ; Wed, 24 Apr 2019 15:15:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730687AbfDXPOx (ORCPT ); Wed, 24 Apr 2019 11:14:53 -0400 Received: from mail-qk1-f196.google.com ([209.85.222.196]:39417 "EHLO mail-qk1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730578AbfDXPOv (ORCPT ); Wed, 24 Apr 2019 11:14:51 -0400 Received: by mail-qk1-f196.google.com with SMTP id f125so6220978qke.6 for ; Wed, 24 Apr 2019 08:14:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=4jTy5aVKxvyn00jTuCVcNHfAC4aqPCtc8Po4P8/coac=; b=IptQ08mBAkuKYtEXwqAMFxAaCdU7FsKrf0n0lpOKqtBHRn/7hKt1EobbO1w8M3xZdD XVTjtCFm36/ejnANKHhrTfKzqX4eKujmk2W6/+Ehp/lnENaKSqlEHgXkUvmTVWK0fTSo /KoebdyarGVPMXTbTpSBiEhqANFt3CsVwlo15cXzR5egAuQ26xPwOpIxjxc0AZN7+zvQ xC/WxIbOL6k9dR3GoUy1bhEskIZgb5LmT61bIs/0GPfsMV01DuAMSNXVcIkyKCdyFwbT a8OL2uKH1kJSZ4Tp7SgewccmlrTVujMz3pHUL9FjH4z7W2mev65NfkdEMl+exz9owcwT JflQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4jTy5aVKxvyn00jTuCVcNHfAC4aqPCtc8Po4P8/coac=; b=mh75rOdsal6OvKCSTNJuuLd53cnkaqekeTqauo3BJV326YgnnWEWQOl5duFlt4zolB FDZSJBnX4lvYtjgkHYr8uydVgsWUl02zjPCQAWv0XujH+HGQXAGPs7JBLdPpXgRr9HrR c/QeCcRnUqVDMOFtZ6JRrGDHtSlZ3FtBitCcipMpq7xYk9hmLg8f7VmwybN0fD7erskm QG3BHVKSnPUs6j8/TYEKGvlD3KCBo4xnmNpH1sissy6T+oQcEoCZY9PTziayMjcafq+0 4QxuP/LQGx8LJukEwkeYQz3RwjvcIgftpKw6ELC+gW1zEFDMqG1JVsGGHsCmTETb8+Mx rTzA== X-Gm-Message-State: APjAAAUgdVHJF72GyQK3X3b4Ss26PfhtvU1QWY0gyRjEIjmSX4V0cLmh dw9gtevjaD6O6IZhcQ2griGjEmV5 X-Google-Smtp-Source: APXvYqxX14uhC6vdLWMetlkdtLUbAdbnEU1IzRRiVqPt111Ir1DeE6RZjFcrQOtP8DDtMV7Q5+70Bg== X-Received: by 2002:ae9:e204:: with SMTP id c4mr14963728qkc.16.1556118889858; Wed, 24 Apr 2019 08:14:49 -0700 (PDT) Received: from stolee-gitdev.corp.microsoft.com ([2001:4898:8010:2:9efc:3578:ef3e:58d5]) by smtp.gmail.com with ESMTPSA id j129sm9671005qkd.51.2019.04.24.08.14.48 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 24 Apr 2019 08:14:49 -0700 (PDT) From: Derrick Stolee X-Google-Original-From: Derrick Stolee To: git@vger.kernel.org Cc: peff@peff.net, jrnieder@gmail.com, avarab@gmail.com, gitster@pobox.com, Derrick Stolee Subject: [PATCH v5 06/11] multi-pack-index: implement 'expire' subcommand Date: Wed, 24 Apr 2019 11:14:23 -0400 Message-Id: <20190424151428.170316-7-dstolee@microsoft.com> X-Mailer: git-send-email 2.21.0.1096.g1c91fdc207 In-Reply-To: <20190424151428.170316-1-dstolee@microsoft.com> References: <20190424151428.170316-1-dstolee@microsoft.com> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The 'git multi-pack-index expire' subcommand looks at the existing mult-pack-index, counts the number of objects referenced in each pack-file, deletes the pack-fils with no referenced objects, and rewrites the multi-pack-index to no longer reference those packs. Refactor the write_midx_file() method to call write_midx_internal() which now takes an existing 'struct multi_pack_index' and a list of pack-files to drop (as specified by the names of their pack- indexes). As we write the new multi-pack-index, we drop those file names from the list of known pack-files. The expire_midx_packs() method removes the unreferenced pack-files after carefully closing the packs to avoid open handles. Test that a new pack-file that covers the contents of two other pack-files leads to those pack-files being deleted during the expire subcommand. Be sure to read the multi-pack-index to ensure it no longer references those packs. Signed-off-by: Derrick Stolee --- midx.c | 120 +++++++++++++++++++++++++++++++++--- t/t5319-multi-pack-index.sh | 20 ++++++ 2 files changed, 130 insertions(+), 10 deletions(-) diff --git a/midx.c b/midx.c index 95c39106b2..299e9b2e8f 100644 --- a/midx.c +++ b/midx.c @@ -33,6 +33,8 @@ #define MIDX_CHUNK_LARGE_OFFSET_WIDTH (sizeof(uint64_t)) #define MIDX_LARGE_OFFSET_NEEDED 0x80000000 +#define PACK_EXPIRED UINT_MAX + static char *get_midx_filename(const char *object_dir) { return xstrfmt("%s/pack/multi-pack-index", object_dir); @@ -381,6 +383,7 @@ struct pack_info { uint32_t orig_pack_int_id; char *pack_name; struct packed_git *p; + unsigned expired : 1; }; static int pack_info_compare(const void *_a, const void *_b) @@ -428,6 +431,7 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len, packs->info[packs->nr].pack_name = xstrdup(file_name); packs->info[packs->nr].orig_pack_int_id = packs->nr; + packs->info[packs->nr].expired = 0; packs->nr++; } } @@ -587,13 +591,17 @@ static size_t write_midx_pack_names(struct hashfile *f, size_t written = 0; for (i = 0; i < num_packs; i++) { - size_t writelen = strlen(info[i].pack_name) + 1; + size_t writelen; + + if (info[i].expired) + continue; if (i && strcmp(info[i].pack_name, info[i - 1].pack_name) <= 0) BUG("incorrect pack-file order: %s before %s", info[i - 1].pack_name, info[i].pack_name); + writelen = strlen(info[i].pack_name) + 1; hashwrite(f, info[i].pack_name, writelen); written += writelen; } @@ -675,6 +683,11 @@ static size_t write_midx_object_offsets(struct hashfile *f, int large_offset_nee for (i = 0; i < nr_objects; i++) { struct pack_midx_entry *obj = list++; + if (perm[obj->pack_int_id] == PACK_EXPIRED) + BUG("object %s is in an expired pack with int-id %d", + oid_to_hex(&obj->oid), + obj->pack_int_id); + hashwrite_be32(f, perm[obj->pack_int_id]); if (large_offset_needed && obj->offset >> 31) @@ -721,7 +734,8 @@ static size_t write_midx_large_offsets(struct hashfile *f, uint32_t nr_large_off return written; } -int write_midx_file(const char *object_dir) +static int write_midx_internal(const char *object_dir, struct multi_pack_index *m, + struct string_list *packs_to_drop) { unsigned char cur_chunk, num_chunks = 0; char *midx_name; @@ -737,6 +751,8 @@ int write_midx_file(const char *object_dir) struct pack_midx_entry *entries = NULL; int large_offsets_needed = 0; int pack_name_concat_len = 0; + int dropped_packs = 0; + int result = 0; midx_name = get_midx_filename(object_dir); if (safe_create_leading_directories(midx_name)) { @@ -745,7 +761,10 @@ int write_midx_file(const char *object_dir) midx_name); } - packs.m = load_multi_pack_index(object_dir, 1); + if (m) + packs.m = m; + else + packs.m = load_multi_pack_index(object_dir, 1); packs.nr = 0; packs.alloc = packs.m ? packs.m->num_packs : 16; @@ -759,13 +778,14 @@ int write_midx_file(const char *object_dir) packs.info[packs.nr].orig_pack_int_id = i; packs.info[packs.nr].pack_name = xstrdup(packs.m->pack_names[i]); packs.info[packs.nr].p = NULL; + packs.info[packs.nr].expired = 0; packs.nr++; } } for_each_file_in_pack_dir(object_dir, add_pack_to_midx, &packs); - if (packs.m && packs.nr == packs.m->num_packs) + if (packs.m && packs.nr == packs.m->num_packs && !packs_to_drop) goto cleanup; entries = get_sorted_entries(packs.m, packs.info, packs.nr, &nr_entries); @@ -779,6 +799,34 @@ int write_midx_file(const char *object_dir) QSORT(packs.info, packs.nr, pack_info_compare); + if (packs_to_drop && packs_to_drop->nr) { + int drop_index = 0; + int missing_drops = 0; + + for (i = 0; i < packs.nr && drop_index < packs_to_drop->nr; i++) { + int cmp = strcmp(packs.info[i].pack_name, + packs_to_drop->items[drop_index].string); + + if (!cmp) { + drop_index++; + packs.info[i].expired = 1; + } else if (cmp > 0) { + error(_("did not see pack-file %s to drop"), + packs_to_drop->items[drop_index].string); + drop_index++; + missing_drops++; + i--; + } else { + packs.info[i].expired = 0; + } + } + + if (missing_drops) { + result = 1; + goto cleanup; + } + } + /* * pack_perm stores a permutation between pack-int-ids from the * previous multi-pack-index to the new one we are writing: @@ -787,11 +835,18 @@ int write_midx_file(const char *object_dir) */ ALLOC_ARRAY(pack_perm, packs.nr); for (i = 0; i < packs.nr; i++) { - pack_perm[packs.info[i].orig_pack_int_id] = i; + if (packs.info[i].expired) { + dropped_packs++; + pack_perm[packs.info[i].orig_pack_int_id] = PACK_EXPIRED; + } else { + pack_perm[packs.info[i].orig_pack_int_id] = i - dropped_packs; + } } - for (i = 0; i < packs.nr; i++) - pack_name_concat_len += strlen(packs.info[i].pack_name) + 1; + for (i = 0; i < packs.nr; i++) { + if (!packs.info[i].expired) + pack_name_concat_len += strlen(packs.info[i].pack_name) + 1; + } if (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT) pack_name_concat_len += MIDX_CHUNK_ALIGNMENT - @@ -807,7 +862,7 @@ int write_midx_file(const char *object_dir) cur_chunk = 0; num_chunks = large_offsets_needed ? 5 : 4; - written = write_midx_header(f, num_chunks, packs.nr); + written = write_midx_header(f, num_chunks, packs.nr - dropped_packs); chunk_ids[cur_chunk] = MIDX_CHUNKID_PACKNAMES; chunk_offsets[cur_chunk] = written + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH; @@ -908,7 +963,12 @@ int write_midx_file(const char *object_dir) free(entries); free(pack_perm); free(midx_name); - return 0; + return result; +} + +int write_midx_file(const char *object_dir) +{ + return write_midx_internal(object_dir, NULL, NULL); } void clear_midx_file(struct repository *r) @@ -1010,5 +1070,45 @@ int verify_midx_file(const char *object_dir) int expire_midx_packs(const char *object_dir) { - return 0; + uint32_t i, *count, result = 0; + struct string_list packs_to_drop = STRING_LIST_INIT_DUP; + struct multi_pack_index *m = load_multi_pack_index(object_dir, 1); + + if (!m) + return 0; + + count = xcalloc(m->num_packs, sizeof(uint32_t)); + for (i = 0; i < m->num_objects; i++) { + int pack_int_id = nth_midxed_pack_int_id(m, i); + count[pack_int_id]++; + } + + for (i = 0; i < m->num_packs; i++) { + char *pack_name; + + if (count[i]) + continue; + + if (prepare_midx_pack(m, i)) + continue; + + if (m->packs[i]->pack_keep) + continue; + + pack_name = xstrdup(m->packs[i]->pack_name); + close_pack(m->packs[i]); + FREE_AND_NULL(m->packs[i]); + + string_list_insert(&packs_to_drop, m->pack_names[i]); + unlink_pack_path(pack_name, 0); + free(pack_name); + } + + free(count); + + if (packs_to_drop.nr) + result = write_midx_internal(object_dir, m, &packs_to_drop); + + string_list_clear(&packs_to_drop, 0); + return result; } diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index a8528f7da0..65e85debec 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -397,4 +397,24 @@ test_expect_success 'expire does not remove any packs' ' ) ' +test_expect_success 'expire removes unreferenced packs' ' + ( + cd dup && + git pack-objects --revs .git/objects/pack/pack-combined <<-EOF && + refs/heads/A + ^refs/heads/C + EOF + git multi-pack-index write && + ls .git/objects/pack | grep -v -e pack-[AB] >expect && + git multi-pack-index expire && + ls .git/objects/pack >actual && + test_cmp expect actual && + ls .git/objects/pack/ | grep idx >expect-idx && + test-tool read-midx .git/objects | grep idx >actual-midx && + test_cmp expect-idx actual-midx && + git multi-pack-index verify && + git fsck + ) +' + test_done From patchwork Wed Apr 24 15:14:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 10915021 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BD08F922 for ; Wed, 24 Apr 2019 15:14:55 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id ABEDD2074F for ; Wed, 24 Apr 2019 15:14:55 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A003B2875F; Wed, 24 Apr 2019 15:14:55 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F1DFE2074F for ; Wed, 24 Apr 2019 15:14:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727873AbfDXPOy (ORCPT ); Wed, 24 Apr 2019 11:14:54 -0400 Received: from mail-qt1-f193.google.com ([209.85.160.193]:40134 "EHLO mail-qt1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730637AbfDXPOw (ORCPT ); Wed, 24 Apr 2019 11:14:52 -0400 Received: by mail-qt1-f193.google.com with SMTP id y49so1967675qta.7 for ; Wed, 24 Apr 2019 08:14:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=hyWYfa/VjVaccdyj+N5244gjx8A8lRlicFoJTJE9VxY=; b=ivfUZBI4xwgSKlSxC2R+gbmnNtnETnI4wORjE7MAT9O8qtaADzWwa2VYOslwxEZM/m 5eUUEsWQ5EwJL+Ycp2yOT8YNnt8YYMWHp5gAPTDyL2BcJw1SEYLBj5EACd3MduJFHyDa I1T3b0KkL/dc4w7mk1az4JIGIJj5e2zyGoWijuAL7XLzFtK2lc9deNE1MAHxppxQ/PNq zXdBQFYqyO0+OV0oBe40QPKUPCMKEsSNZiBzxltSMwPBokTzOdUjKRbRq1vdLyA25g6e z5CRxk1+TTBTOcgHmwQJEKNBb7XNyAxx05IE5OewiE9Jy3anxkWrY9NmgIHKGCRwhDof CwGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=hyWYfa/VjVaccdyj+N5244gjx8A8lRlicFoJTJE9VxY=; b=Yus/OMc6/oxu7g+nQOx0QMpN5Sv7Cia+E3N/Vruvq6aLd3QdugRU+3SRH+P0J6YP+E HtY6RyQsbIBKkjgk0Z0Vlv/GagUC1PFOqhElQt7T4jwcPWB1tymVdGNtj6C9vjD2RkKf rFrBhaPUMp05aiYZalzIE93knYPaG21P+iED0m5f0w7uBRSU1OSDgOGukyTwpSIWJv8u eDNX//j5GiQ2l1Z+PZTR0NzFqDFamaK6PCtBdmxjIQb+mD/ARxm5dZRCf3WohTe/hXPZ fYYm2MtpyXnxo7IB21f7fQbEsOahJP+o7DvRjNiFTtHEksYW9T03c8zVYNNiBc5xw5Mi 3pxQ== X-Gm-Message-State: APjAAAXTzLkb52I2vhDFHOIjVhDaGK1dyQDUmkapxWSoMtWDGS7d80ft +o9FC9g/6fcwmiXcg2Mq+ecWJdMh X-Google-Smtp-Source: APXvYqy5VbwDly2NZFP6jpoFPF+Td8Mu+4LennR8yKOk4t0z3yRWR7u3W4pw+pyo5KsoNiFjYg50rg== X-Received: by 2002:ac8:88f:: with SMTP id v15mr27169034qth.382.1556118890903; Wed, 24 Apr 2019 08:14:50 -0700 (PDT) Received: from stolee-gitdev.corp.microsoft.com ([2001:4898:8010:2:9efc:3578:ef3e:58d5]) by smtp.gmail.com with ESMTPSA id j129sm9671005qkd.51.2019.04.24.08.14.49 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 24 Apr 2019 08:14:50 -0700 (PDT) From: Derrick Stolee X-Google-Original-From: Derrick Stolee To: git@vger.kernel.org Cc: peff@peff.net, jrnieder@gmail.com, avarab@gmail.com, gitster@pobox.com, Derrick Stolee Subject: [PATCH v5 07/11] multi-pack-index: prepare 'repack' subcommand Date: Wed, 24 Apr 2019 11:14:24 -0400 Message-Id: <20190424151428.170316-8-dstolee@microsoft.com> X-Mailer: git-send-email 2.21.0.1096.g1c91fdc207 In-Reply-To: <20190424151428.170316-1-dstolee@microsoft.com> References: <20190424151428.170316-1-dstolee@microsoft.com> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP In an environment where the multi-pack-index is useful, it is due to many pack-files and an inability to repack the object store into a single pack-file. However, it is likely that many of these pack-files are rather small, and could be repacked into a slightly larger pack-file without too much effort. It may also be important to ensure the object store is highly available and the repack operation does not interrupt concurrent git commands. Introduce a 'repack' subcommand to 'git multi-pack-index' that takes a '--batch-size' option. The subcommand will inspect the multi-pack-index for referenced pack-files whose size is smaller than the batch size, until collecting a list of pack-files whose sizes sum to larger than the batch size. Then, a new pack-file will be created containing the objects from those pack-files that are referenced by the multi-pack-index. The resulting pack is likely to actually be smaller than the batch size due to compression and the fact that there may be objects in the pack- files that have duplicate copies in other pack-files. The current change introduces the command-line arguments, and we add a test that ensures we parse these options properly. Since we specify a small batch size, we will guarantee that future implementations do not change the list of pack-files. In addition, we hard-code the modified times of the packs in the pack directory to ensure the list of packs sorted by modified time matches the order if sorted by size (ascending). This will be important in a future test. Signed-off-by: Derrick Stolee --- Documentation/git-multi-pack-index.txt | 17 +++++++++++++++++ builtin/multi-pack-index.c | 12 ++++++++++-- midx.c | 5 +++++ midx.h | 1 + t/t5319-multi-pack-index.sh | 20 +++++++++++++++++++- 5 files changed, 52 insertions(+), 3 deletions(-) diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt index 6186c4c936..233b2b7862 100644 --- a/Documentation/git-multi-pack-index.txt +++ b/Documentation/git-multi-pack-index.txt @@ -36,6 +36,23 @@ expire:: have no objects referenced by the MIDX. Rewrite the MIDX file afterward to remove all references to these pack-files. +repack:: + Create a new pack-file containing objects in small pack-files + referenced by the multi-pack-index. If the size given by the + `--batch-size=` argument is zero, then create a pack + containing all objects referenced by the multi-pack-index. For + a non-zero batch size, Select the pack-files by examining packs + from oldest-to-newest, computing the "expected size" by counting + the number of objects in the pack referenced by the + multi-pack-index, then divide by the total number of objects in + the pack and multiply by the pack size. We select packs with + expected size below the batch size until the set of packs have + total expected size at least the batch size. If the total size + does not reach the batch size, then do nothing. If a new pack- + file is created, rewrite the multi-pack-index to reference the + new pack-file. A later run of 'git multi-pack-index expire' will + delete the pack-files that were part of this batch. + EXAMPLES -------- diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c index 145de3a46c..c66239de33 100644 --- a/builtin/multi-pack-index.c +++ b/builtin/multi-pack-index.c @@ -5,12 +5,13 @@ #include "midx.h" static char const * const builtin_multi_pack_index_usage[] = { - N_("git multi-pack-index [--object-dir=] (write|verify|expire)"), + N_("git multi-pack-index [--object-dir=] (write|verify|expire|repack --batch-size=)"), NULL }; static struct opts_multi_pack_index { const char *object_dir; + unsigned long batch_size; } opts; int cmd_multi_pack_index(int argc, const char **argv, @@ -19,6 +20,8 @@ int cmd_multi_pack_index(int argc, const char **argv, static struct option builtin_multi_pack_index_options[] = { OPT_FILENAME(0, "object-dir", &opts.object_dir, N_("object directory containing set of packfile and pack-index pairs")), + OPT_MAGNITUDE(0, "batch-size", &opts.batch_size, + N_("during repack, collect pack-files of smaller size into a batch that is larger than this size")), OPT_END(), }; @@ -40,6 +43,11 @@ int cmd_multi_pack_index(int argc, const char **argv, return 1; } + if (!strcmp(argv[0], "repack")) + return midx_repack(opts.object_dir, (size_t)opts.batch_size); + if (opts.batch_size) + die(_("--batch-size option is only for 'repack' subcommand")); + if (!strcmp(argv[0], "write")) return write_midx_file(opts.object_dir); if (!strcmp(argv[0], "verify")) @@ -47,5 +55,5 @@ int cmd_multi_pack_index(int argc, const char **argv, if (!strcmp(argv[0], "expire")) return expire_midx_packs(opts.object_dir); - die(_("unrecognized verb: %s"), argv[0]); + die(_("unrecognized subcommand: %s"), argv[0]); } diff --git a/midx.c b/midx.c index 299e9b2e8f..768a7dff73 100644 --- a/midx.c +++ b/midx.c @@ -1112,3 +1112,8 @@ int expire_midx_packs(const char *object_dir) string_list_clear(&packs_to_drop, 0); return result; } + +int midx_repack(const char *object_dir, size_t batch_size) +{ + return 0; +} diff --git a/midx.h b/midx.h index e3a2b740b5..394a21ee96 100644 --- a/midx.h +++ b/midx.h @@ -50,6 +50,7 @@ int write_midx_file(const char *object_dir); void clear_midx_file(struct repository *r); int verify_midx_file(const char *object_dir); int expire_midx_packs(const char *object_dir); +int midx_repack(const char *object_dir, size_t batch_size); void close_midx(struct multi_pack_index *m); diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index 65e85debec..26ae8b3f62 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -383,7 +383,8 @@ test_expect_success 'setup expire tests' ' git pack-objects --revs .git/objects/pack/pack-E <<-EOF && refs/heads/E EOF - git multi-pack-index write + git multi-pack-index write && + cp -r .git/objects/pack .git/objects/pack-backup ) ' @@ -417,4 +418,21 @@ test_expect_success 'expire removes unreferenced packs' ' ) ' +test_expect_success 'repack with minimum size does not alter existing packs' ' + ( + cd dup && + rm -rf .git/objects/pack && + mv .git/objects/pack-backup .git/objects/pack && + touch -m -t 201901010000 .git/objects/pack/pack-D* && + touch -m -t 201901010001 .git/objects/pack/pack-C* && + touch -m -t 201901010002 .git/objects/pack/pack-B* && + touch -m -t 201901010003 .git/objects/pack/pack-A* && + ls .git/objects/pack >expect && + MINSIZE=$(ls -l .git/objects/pack/*pack | awk "{print \$5;}" | sort -n | head -n 1) && + git multi-pack-index repack --batch-size=$MINSIZE && + ls .git/objects/pack >actual && + test_cmp expect actual + ) +' + test_done From patchwork Wed Apr 24 15:14:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 10915023 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C96BD14DB for ; Wed, 24 Apr 2019 15:14:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B99992873C for ; Wed, 24 Apr 2019 15:14:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id ADCCD2875F; Wed, 24 Apr 2019 15:14:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0C4642074F for ; Wed, 24 Apr 2019 15:14:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730714AbfDXPOy (ORCPT ); Wed, 24 Apr 2019 11:14:54 -0400 Received: from mail-qt1-f196.google.com ([209.85.160.196]:42290 "EHLO mail-qt1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730639AbfDXPOy (ORCPT ); Wed, 24 Apr 2019 11:14:54 -0400 Received: by mail-qt1-f196.google.com with SMTP id p20so20670187qtc.9 for ; Wed, 24 Apr 2019 08:14:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=GL6ULwmZKVMA5OmtyuKJyT1/0Xil5I1+Y6Be+Xs99Ww=; b=kTxEhRvUmZw/0vEqe1DYvQsxlpxwY50B3UxBcQc3trDtyJ/9Nx1YHLzkxe20s7Klr6 VjV9S6VL0Qb2JVYbr2t4Bf/+DaP6sdyif/6GGXaw8T2i8EPAseq/SyvKuSddLqyHehG5 j2+h2uX0IfJJMxySOyhoDrLNLzuwKvLOL+CcuT7+jIGZYxRutJQMH9VzrBXKKlfZcQyC UTcvAd844/6WSY0tC1SPNpLZnnkG6qHxpMLEDAsfpL9DvxjD/N+7t60n4qKGUod3Q3GI nOnWSWWlNGXT0GvGCYkJVekKPRBynx2RroIAQzhRf/mB+n1T0iz58RwS3HDCny+JqXvt xSKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=GL6ULwmZKVMA5OmtyuKJyT1/0Xil5I1+Y6Be+Xs99Ww=; b=SG9Y8xsPhIwTRGsZkDj1GgWQ22upCwl768l7M7osybgQLC9UgyVJobbxAXAFXMpc2h XnCoBiCSjmK2UnJ3rFHHuQRdhoyNcECPDC3v+fLelyoq2lCOFNhyUVGd5c3/eJn0CJK4 W2uQqiwg7CCVdciiYuqDMfJTWGMrhzdsLyzCXWZn55QoIOYpNIb3Xo2ak1Ku6v4FE2Os NW6ydnlr4bt5/VTsI+5/eIg3PIFCgqoYiPEWgSljKB5Ct17Vry4BggdA65NeDe3pAv2C RvdCTcFpk+GrN2/9B/bgUXbr8Rx5IoXsbUhr70mPLnYRh/PxOZBe/ZsbSKWu5IhfwCFl h6vw== X-Gm-Message-State: APjAAAVWCaigVssKUM0Ev7gt40PerZMHnqIaDZF5kNLROyHmXwNkIAPL /i59UdbBycTqrCvQy0mA4kjZanue X-Google-Smtp-Source: APXvYqwT5P5Xxr74zfavbC6RVuPOr/b41LMSTP63VGgfPgxwl+OP3vh2Li8SwYzO/ixMPT36SmoA2g== X-Received: by 2002:aed:23ca:: with SMTP id k10mr810267qtc.224.1556118892181; Wed, 24 Apr 2019 08:14:52 -0700 (PDT) Received: from stolee-gitdev.corp.microsoft.com ([2001:4898:8010:2:9efc:3578:ef3e:58d5]) by smtp.gmail.com with ESMTPSA id j129sm9671005qkd.51.2019.04.24.08.14.50 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 24 Apr 2019 08:14:51 -0700 (PDT) From: Derrick Stolee X-Google-Original-From: Derrick Stolee To: git@vger.kernel.org Cc: peff@peff.net, jrnieder@gmail.com, avarab@gmail.com, gitster@pobox.com, Derrick Stolee Subject: [PATCH v5 08/11] midx: implement midx_repack() Date: Wed, 24 Apr 2019 11:14:25 -0400 Message-Id: <20190424151428.170316-9-dstolee@microsoft.com> X-Mailer: git-send-email 2.21.0.1096.g1c91fdc207 In-Reply-To: <20190424151428.170316-1-dstolee@microsoft.com> References: <20190424151428.170316-1-dstolee@microsoft.com> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP To repack with a non-zero batch-size, first sort all pack-files by their modified time. Second, walk those pack-files from oldest to newest, compute their expected size, and add the packs to a list if they are smaller than the given batch-size. Stop when the total expected size is at least the batch size. If the batch size is zero, select all packs in the multi-pack-index. Finally, collect the objects from the multi-pack-index that are in the selected packs and send them to 'git pack-objects'. Write a new multi-pack-index that includes the new pack. Using a batch size of zero is very similar to a standard 'git repack' command, except that we do not delete the old packs and instead rely on the new multi-pack-index to prevent new processes from reading the old packs. This does not disrupt other Git processes that are currently reading the old packs based on the old multi-pack-index. While first designing a 'git multi-pack-index repack' operation, I started by collecting the batches based on the actual size of the objects instead of the size of the pack-files. This allows repacking a large pack-file that has very few referencd objects. However, this came at a significant cost of parsing pack-files instead of simply reading the multi-pack-index and getting the file information for the pack-files. The "expected size" version provides similar behavior, but could skip a pack-file if the average object size is much larger than the actual size of the referenced objects, or can create a large pack if the actual size of the referenced objects is larger than the expected size. Signed-off-by: Derrick Stolee --- midx.c | 150 +++++++++++++++++++++++++++++++++++- t/t5319-multi-pack-index.sh | 28 +++++++ 2 files changed, 177 insertions(+), 1 deletion(-) diff --git a/midx.c b/midx.c index 768a7dff73..01c6a05732 100644 --- a/midx.c +++ b/midx.c @@ -8,6 +8,7 @@ #include "sha1-lookup.h" #include "midx.h" #include "progress.h" +#include "run-command.h" #define MIDX_SIGNATURE 0x4d494458 /* "MIDX" */ #define MIDX_VERSION 1 @@ -1113,7 +1114,154 @@ int expire_midx_packs(const char *object_dir) return result; } -int midx_repack(const char *object_dir, size_t batch_size) +struct repack_info { + timestamp_t mtime; + uint32_t referenced_objects; + uint32_t pack_int_id; +}; + +static int compare_by_mtime(const void *a_, const void *b_) { + const struct repack_info *a, *b; + + a = (const struct repack_info *)a_; + b = (const struct repack_info *)b_; + + if (a->mtime < b->mtime) + return -1; + if (a->mtime > b->mtime) + return 1; + return 0; +} + +static int fill_included_packs_all(struct multi_pack_index *m, + unsigned char *include_pack) +{ + uint32_t i; + + for (i = 0; i < m->num_packs; i++) + include_pack[i] = 1; + + return m->num_packs < 2; +} + +static int fill_included_packs_batch(struct multi_pack_index *m, + unsigned char *include_pack, + size_t batch_size) +{ + uint32_t i, packs_to_repack; + size_t total_size; + struct repack_info *pack_info = xcalloc(m->num_packs, sizeof(struct repack_info)); + + for (i = 0; i < m->num_packs; i++) { + pack_info[i].pack_int_id = i; + + if (prepare_midx_pack(m, i)) + continue; + + pack_info[i].mtime = m->packs[i]->mtime; + } + + for (i = 0; batch_size && i < m->num_objects; i++) { + uint32_t pack_int_id = nth_midxed_pack_int_id(m, i); + pack_info[pack_int_id].referenced_objects++; + } + + QSORT(pack_info, m->num_packs, compare_by_mtime); + + total_size = 0; + packs_to_repack = 0; + for (i = 0; total_size < batch_size && i < m->num_packs; i++) { + int pack_int_id = pack_info[i].pack_int_id; + struct packed_git *p = m->packs[pack_int_id]; + size_t expected_size; + + if (!p) + continue; + if (open_pack_index(p) || !p->num_objects) + continue; + + expected_size = (size_t)(p->pack_size + * pack_info[i].referenced_objects); + expected_size /= p->num_objects; + + if (expected_size >= batch_size) + continue; + + packs_to_repack++; + total_size += expected_size; + include_pack[pack_int_id] = 1; + } + + free(pack_info); + + if (total_size < batch_size || packs_to_repack < 2) + return 1; + return 0; +} + +int midx_repack(const char *object_dir, size_t batch_size) +{ + int result = 0; + uint32_t i; + unsigned char *include_pack; + struct child_process cmd = CHILD_PROCESS_INIT; + struct strbuf base_name = STRBUF_INIT; + struct multi_pack_index *m = load_multi_pack_index(object_dir, 1); + + if (!m) + return 0; + + include_pack = xcalloc(m->num_packs, sizeof(unsigned char)); + + if (batch_size) { + if (fill_included_packs_batch(m, include_pack, batch_size)) + goto cleanup; + } else if (fill_included_packs_all(m, include_pack)) + goto cleanup; + + argv_array_push(&cmd.args, "pack-objects"); + + strbuf_addstr(&base_name, object_dir); + strbuf_addstr(&base_name, "/pack/pack"); + argv_array_push(&cmd.args, base_name.buf); + strbuf_release(&base_name); + + cmd.git_cmd = 1; + cmd.in = cmd.out = -1; + + if (start_command(&cmd)) { + error(_("could not start pack-objects")); + result = 1; + goto cleanup; + } + + for (i = 0; i < m->num_objects; i++) { + struct object_id oid; + uint32_t pack_int_id = nth_midxed_pack_int_id(m, i); + + if (!include_pack[pack_int_id]) + continue; + + nth_midxed_object_oid(&oid, m, i); + xwrite(cmd.in, oid_to_hex(&oid), the_hash_algo->hexsz); + xwrite(cmd.in, "\n", 1); + } + close(cmd.in); + + if (finish_command(&cmd)) { + error(_("could not finish pack-objects")); + result = 1; + goto cleanup; + } + + result = write_midx_internal(object_dir, m, NULL); + m = NULL; + +cleanup: + if (m) + close_midx(m); + free(include_pack); + return result; } diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index 26ae8b3f62..d6c1353514 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -435,4 +435,32 @@ test_expect_success 'repack with minimum size does not alter existing packs' ' ) ' +test_expect_success 'repack creates a new pack' ' + ( + cd dup && + ls .git/objects/pack/*idx >idx-list && + test_line_count = 5 idx-list && + THIRD_SMALLEST_SIZE=$(ls -l .git/objects/pack/*pack | awk "{print \$5;}" | sort -n | head -n 3 | tail -n 1) && + BATCH_SIZE=$(($THIRD_SMALLEST_SIZE + 1)) && + git multi-pack-index repack --batch-size=$BATCH_SIZE && + ls .git/objects/pack/*idx >idx-list && + test_line_count = 6 idx-list && + test-tool read-midx .git/objects | grep idx >midx-list && + test_line_count = 6 midx-list + ) +' + +test_expect_success 'expire removes repacked packs' ' + ( + cd dup && + ls -al .git/objects/pack/*pack && + ls -S .git/objects/pack/*pack | head -n 4 >expect && + git multi-pack-index expire && + ls -S .git/objects/pack/*pack >actual && + test_cmp expect actual && + test-tool read-midx .git/objects | grep idx >midx-list && + test_line_count = 4 midx-list + ) +' + test_done From patchwork Wed Apr 24 15:14:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 10915025 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2F22F1708 for ; Wed, 24 Apr 2019 15:14:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1F8E92074F for ; Wed, 24 Apr 2019 15:14:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 13F06287C0; Wed, 24 Apr 2019 15:14:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B36162074F for ; Wed, 24 Apr 2019 15:14:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730639AbfDXPOz (ORCPT ); Wed, 24 Apr 2019 11:14:55 -0400 Received: from mail-qt1-f195.google.com ([209.85.160.195]:34045 "EHLO mail-qt1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730578AbfDXPOy (ORCPT ); Wed, 24 Apr 2019 11:14:54 -0400 Received: by mail-qt1-f195.google.com with SMTP id j6so1441412qtq.1 for ; Wed, 24 Apr 2019 08:14:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=kq1FZu/0A6AKTssjPN3Fdz3QYTxlpCaOsGQ6vC7y6Y8=; b=j1Z6vrJsAe7E/xki63M+S8MW7CmY0GswfXilbpcEUJfTyeKpBhL4WVCHdIwtjxklGL ukTA05jPYab6nTcOtpOkaqtxMrSq3diDoQJRRYqBCE1KZEDREAnhYHQ52CNlDaIcBYQ2 O7LiTUpUmVMfmI70g8wyuzjaEEVW/vbDdQsazUHztQkI3nj1ZcuybjAmnjhgsOpzdj2l +QocIX3h3IZVUNcTPgAswDn40R7RoVKWAXKk75NRDDkc9VXr1LW4KsJWAU4n7d2XRWCD tiDYB3D17ktLrPZn5Cli+rwG3RbXC3xPhAvcrQapu5g10ziqh0OoO9efa87WgmGG3BI4 gcDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=kq1FZu/0A6AKTssjPN3Fdz3QYTxlpCaOsGQ6vC7y6Y8=; b=FKSgXvJcBt7sGCabxdwBYIk7Z9dlfGsKjsVJkcG/dUjW1RoHSmTjfVnHig8bnoxLWX Y0zeo+mV+ebEmvIpu9CJyIyR/IQ2O/cMF3A2kx7RwWPeRUHlxPqiY7iHALFzxqM0FRhO OBSB5foYOsQS3MP7dg0I41f6f7j12NyKlbvu7pPY7MnpXbJA4kZLQa8YwbAymfZj2/Ie w2wcHxc55I9lFY6cqgFlk5UyiMVWaLXkjLxgdsW6nqANxHpm4EziFNm/Bc3IMZaY3cTt MWq/gKG04jzmgF6XsXo6WWpX4Eg/ge3QwEsnbyz5Gs9dkvkA9z+fDwggE3ab2KXvlayK X8Nw== X-Gm-Message-State: APjAAAX8ge3k41tobR6SjBjnxhaucDCUxNGaYhNgglPEZoH0MM7mXFfw XbsGFD+0e7h5djKfZqcXToE48RxA X-Google-Smtp-Source: APXvYqy+UkQM2t1lqVaK55pqf4s62HmRz8tEFuhmom3FvcioYevclD2X7SJI/1FMm+xefx44DMrXDQ== X-Received: by 2002:a0c:c491:: with SMTP id u17mr6484688qvi.70.1556118893308; Wed, 24 Apr 2019 08:14:53 -0700 (PDT) Received: from stolee-gitdev.corp.microsoft.com ([2001:4898:8010:2:9efc:3578:ef3e:58d5]) by smtp.gmail.com with ESMTPSA id j129sm9671005qkd.51.2019.04.24.08.14.52 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 24 Apr 2019 08:14:52 -0700 (PDT) From: Derrick Stolee X-Google-Original-From: Derrick Stolee To: git@vger.kernel.org Cc: peff@peff.net, jrnieder@gmail.com, avarab@gmail.com, gitster@pobox.com, Derrick Stolee Subject: [PATCH v5 09/11] multi-pack-index: test expire while adding packs Date: Wed, 24 Apr 2019 11:14:26 -0400 Message-Id: <20190424151428.170316-10-dstolee@microsoft.com> X-Mailer: git-send-email 2.21.0.1096.g1c91fdc207 In-Reply-To: <20190424151428.170316-1-dstolee@microsoft.com> References: <20190424151428.170316-1-dstolee@microsoft.com> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP During development of the multi-pack-index expire subcommand, a version went out that improperly computed the pack order if a new pack was introduced while other packs were being removed. Part of the subtlety of the bug involved the new pack being placed before other packs that already existed in the multi-pack-index. Add a test to t5319-multi-pack-index.sh that catches this issue. The test adds new packs that cause another pack to be expired, and creates new packs that are lexicographically sorted before and after the existing packs. Signed-off-by: Derrick Stolee --- t/t5319-multi-pack-index.sh | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index d6c1353514..19b769eea0 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -463,4 +463,36 @@ test_expect_success 'expire removes repacked packs' ' ) ' +test_expect_success 'expire works when adding new packs' ' + ( + cd dup && + git pack-objects --revs .git/objects/pack/pack-combined <<-EOF && + refs/heads/A + ^refs/heads/B + EOF + git pack-objects --revs .git/objects/pack/pack-combined <<-EOF && + refs/heads/B + ^refs/heads/C + EOF + git pack-objects --revs .git/objects/pack/pack-combined <<-EOF && + refs/heads/C + ^refs/heads/D + EOF + git multi-pack-index write && + git pack-objects --revs .git/objects/pack/a-pack <<-EOF && + refs/heads/D + ^refs/heads/E + EOF + git multi-pack-index write && + git pack-objects --revs .git/objects/pack/z-pack <<-EOF && + refs/heads/E + EOF + git multi-pack-index expire && + ls .git/objects/pack/ | grep idx >expect && + test-tool read-midx .git/objects | grep idx >actual && + test_cmp expect actual && + git multi-pack-index verify + ) +' + test_done From patchwork Wed Apr 24 15:14:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 10915027 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ED0A5922 for ; Wed, 24 Apr 2019 15:14:58 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DC10C2074F for ; Wed, 24 Apr 2019 15:14:58 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D00A1287C0; Wed, 24 Apr 2019 15:14:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7C03F2074F for ; Wed, 24 Apr 2019 15:14:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730732AbfDXPO5 (ORCPT ); Wed, 24 Apr 2019 11:14:57 -0400 Received: from mail-qk1-f193.google.com ([209.85.222.193]:46335 "EHLO mail-qk1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730715AbfDXPOz (ORCPT ); Wed, 24 Apr 2019 11:14:55 -0400 Received: by mail-qk1-f193.google.com with SMTP id w73so8879913qkb.13 for ; Wed, 24 Apr 2019 08:14:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ictdyif0MDE7m6xyHh5Pe8kojmrVawufZ/3Rr7I3XpI=; b=Iq+3dLx8dzmFOwYfqF27iuDmf8qQW8KL0aBhT5xr7P9oSADBADiGVjsbzycwFNdg9/ 7ZnhMAtiG+XRECRpo0avsv7J2TftDfbk93Rkfe7auT0eZdRk7aKGOaq3vA0881vgbyzH 3HArFr9xx0celf0dgQt3nyXYp67mmie86GTmTi7GXdh/dNQ55pKC31EIVQgHCEtSstQx ZjjPu7/Y9dW3iZMlP7XsZ/xPcYYvkEXPnR1iNFx5y+ElT3bcstv+kr1UDKUiwdr3njaL f2u0+WZVHMXMcRf+8V78Feud06J30isPexed++bMzK25sbp32HWfAt05ab0l3YiaEq3A Escg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ictdyif0MDE7m6xyHh5Pe8kojmrVawufZ/3Rr7I3XpI=; b=FsIggNnoO0JfqST1jfBXxTbHGNcTMO8e4I/xFyKb5+BoVXU//SBaTvm1BdjdAcYjtd BtI1XlZkFVs5zE3AsCujL3kgn1kHqmJTJN96QDh/HPee3dSjj1vxAw6l6QAqwiIgjgVj 4wX0WwKjbunU7nRq4mljvHZBKTsIMKxBIK2fWrwTNfrxDpfnxrA7DX1QnufaKVr40iVb zL4/JZ4Y5pH45loyiJoVAEReCVBoeXkZF1UfHifx/QHxDmLyC2b/IX6XLuqv9cP55KHH r5rgYOtXKVkVzQOY36CN0atI8bDd+oAjhnB3qcZn7Es//X4JunAxkmEq00u1LNweAcYO UIWA== X-Gm-Message-State: APjAAAUDj0URFC1p57CrAi8MI9XVmIXEvVzKZ0gUSNOTESp4G4tmnc0k bg8ZCIWxVGqJ/aJElMeEyQ0qePdR X-Google-Smtp-Source: APXvYqyhrObuhn2WNBpVVnpcfv+OK9+533tSzrS0ZlNOLzHadb96kM3sPd61zKy5yu8QZnk7oC60NQ== X-Received: by 2002:ae9:e909:: with SMTP id x9mr24846519qkf.130.1556118894274; Wed, 24 Apr 2019 08:14:54 -0700 (PDT) Received: from stolee-gitdev.corp.microsoft.com ([2001:4898:8010:2:9efc:3578:ef3e:58d5]) by smtp.gmail.com with ESMTPSA id j129sm9671005qkd.51.2019.04.24.08.14.53 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 24 Apr 2019 08:14:53 -0700 (PDT) From: Derrick Stolee X-Google-Original-From: Derrick Stolee To: git@vger.kernel.org Cc: peff@peff.net, jrnieder@gmail.com, avarab@gmail.com, gitster@pobox.com, Derrick Stolee Subject: [PATCH v5 10/11] midx: add test that 'expire' respects .keep files Date: Wed, 24 Apr 2019 11:14:27 -0400 Message-Id: <20190424151428.170316-11-dstolee@microsoft.com> X-Mailer: git-send-email 2.21.0.1096.g1c91fdc207 In-Reply-To: <20190424151428.170316-1-dstolee@microsoft.com> References: <20190424151428.170316-1-dstolee@microsoft.com> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The 'git multi-pack-index expire' subcommand may delete packs that are not needed from the perspective of the multi-pack-index. If a pack has a .keep file, then we should not delete that pack. Add a test that ensures we preserve a pack that would otherwise be expired. First, create a new pack that contains every object in the repo, then add it to the multi-pack-index. Then create a .keep file for a pack starting with "a-pack" that was added in the previous test. Finally, expire and verify that the pack remains and the other packs were expired. Signed-off-by: Derrick Stolee --- t/t5319-multi-pack-index.sh | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index 19b769eea0..bcfa520401 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -495,4 +495,22 @@ test_expect_success 'expire works when adding new packs' ' ) ' +test_expect_success 'expire respects .keep files' ' + ( + cd dup && + git pack-objects --revs .git/objects/pack/pack-all <<-EOF && + refs/heads/A + EOF + git multi-pack-index write && + PACKA=$(ls .git/objects/pack/a-pack*\.pack | sed s/\.pack\$//) && + touch $PACKA.keep && + git multi-pack-index expire && + ls -S .git/objects/pack/a-pack* | grep $PACKA >a-pack-files && + test_line_count = 3 a-pack-files && + test-tool read-midx .git/objects | grep idx >midx-list && + test_line_count = 2 midx-list + ) +' + + test_done From patchwork Wed Apr 24 15:14:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 10915029 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C2E3814DB for ; Wed, 24 Apr 2019 15:14:59 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B1FEE22B27 for ; Wed, 24 Apr 2019 15:14:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A67C3287C0; Wed, 24 Apr 2019 15:14:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 513AC22B27 for ; Wed, 24 Apr 2019 15:14:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730757AbfDXPO6 (ORCPT ); Wed, 24 Apr 2019 11:14:58 -0400 Received: from mail-qk1-f195.google.com ([209.85.222.195]:33574 "EHLO mail-qk1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730578AbfDXPO4 (ORCPT ); Wed, 24 Apr 2019 11:14:56 -0400 Received: by mail-qk1-f195.google.com with SMTP id k189so11065384qkc.0 for ; Wed, 24 Apr 2019 08:14:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Hl2kohpjMFRdWldR0OVJ1OtbUQ6v2TVaz0BAtesidEQ=; b=jyUvPxkhZZoRtA7zyfDQIplrQEs4X38HtsjmFfJL8RJVfjRswwkFc9XEvkHvrgFozv XhdBKLyD78221FuXRsZXuBbNTssUMr3iFJ/9jPQNjdX6HQH7CIZpJLbkKpPVhdc69CSf CGRLuBKxvgvZEp7PLnXrbRIcWX28di4u+s4laJIOFBh3htQInVJ6a3yQztutpSObR286 m41gBHRedTV1Hy/HNGK/A6cZ7HZNssQD4JHUsFtU/GY0fV5LTH96sIVC6GRle3r77l4L zHfDgNLxnao+subxEaYkZLxNlxWAP2YqDgg2mUS15BSQHztsjspHJflslU38tPfVJ9t9 Q2hg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Hl2kohpjMFRdWldR0OVJ1OtbUQ6v2TVaz0BAtesidEQ=; b=MjSdNncPnimEsth5pZei0lW87dJraw1UQu0FFcf01L0wpDTIOSAhhFEAPfG3u+bZ4T W8P7anfTvaIjmM3KdE8Gq2s2rZv9gGdTLSpjhIT7QEqMVsMFqB1bGr1TRPgJYcTLksyi T8bnCP8T4xTvHfkblNDH2v3XF/tVISzUp4qBA5V0yFWEmxpXvz7pANFLdzF5vPEz1tns rw+KDChEsdDATZkrlJFQnU3EUe14P4Nm9RLRdsPO59tetNQpD/y+uaaH+81HYG1yM410 TWj1lHvHRxM3xzmW0XbNg4FPG7lFOOoeZmpqf5DJZQ7e38FW7Lx2Sp+8a9RJjFCXMq/9 DkKA== X-Gm-Message-State: APjAAAUZoOtrQ0W5c/3ViwfenD4dHaN3ViLK3VKPTtvP2U8NA5VA74jh K3RTlbpdOyy9NgnLbDNmbjhDc5Zk X-Google-Smtp-Source: APXvYqx//AMUXIhgZ+/TzJujsp5/2Q84dvpm4a1DSptIHmjhPlIM3KjaI8HbPJTIQ8kQ8G2p0lZzhw== X-Received: by 2002:ae9:e916:: with SMTP id x22mr25705303qkf.66.1556118895455; Wed, 24 Apr 2019 08:14:55 -0700 (PDT) Received: from stolee-gitdev.corp.microsoft.com ([2001:4898:8010:2:9efc:3578:ef3e:58d5]) by smtp.gmail.com with ESMTPSA id j129sm9671005qkd.51.2019.04.24.08.14.54 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 24 Apr 2019 08:14:54 -0700 (PDT) From: Derrick Stolee X-Google-Original-From: Derrick Stolee To: git@vger.kernel.org Cc: peff@peff.net, jrnieder@gmail.com, avarab@gmail.com, gitster@pobox.com, Derrick Stolee Subject: [PATCH v5 11/11] t5319-multi-pack-index.sh: test batch size zero Date: Wed, 24 Apr 2019 11:14:28 -0400 Message-Id: <20190424151428.170316-12-dstolee@microsoft.com> X-Mailer: git-send-email 2.21.0.1096.g1c91fdc207 In-Reply-To: <20190424151428.170316-1-dstolee@microsoft.com> References: <20190424151428.170316-1-dstolee@microsoft.com> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The 'git multi-pack-index repack' command can take a batch size of zero, which creates a new pack-file containing all objects in the multi-pack-index. The first 'repack' command will create one new pack-file, and an 'expire' command after that will delete the old pack-files, as they no longer contain any referenced objects in the multi-pack-index. We must remove the .keep file that was added in the previous test in order to expire that pack-file. Also test that a 'repack' will do nothing if there is only one pack-file. Signed-off-by: Derrick Stolee --- t/t5319-multi-pack-index.sh | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index bcfa520401..0f116b4b92 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -512,5 +512,24 @@ test_expect_success 'expire respects .keep files' ' ) ' +test_expect_success 'repack --batch-size=0 repacks everything' ' + ( + cd dup && + rm .git/objects/pack/*.keep && + ls .git/objects/pack/*idx >idx-list && + test_line_count = 2 idx-list && + git multi-pack-index repack --batch-size=0 && + ls .git/objects/pack/*idx >idx-list && + test_line_count = 3 idx-list && + test-tool read-midx .git/objects | grep idx >midx-list && + test_line_count = 3 midx-list && + git multi-pack-index expire && + ls -al .git/objects/pack/*idx >idx-list && + test_line_count = 1 idx-list && + git multi-pack-index repack --batch-size=0 && + ls -al .git/objects/pack/*idx >new-idx-list && + test_cmp idx-list new-idx-list + ) +' test_done