From patchwork Mon Jun 10 23:35:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee via GitGitGadget X-Patchwork-Id: 10985617 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0D0A0924 for ; Mon, 10 Jun 2019 23:35:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E40DC286A2 for ; Mon, 10 Jun 2019 23:35:28 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C685E28689; Mon, 10 Jun 2019 23:35:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 53C3D285CB for ; Mon, 10 Jun 2019 23:35:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2390568AbfFJXfY (ORCPT ); Mon, 10 Jun 2019 19:35:24 -0400 Received: from mail-ed1-f65.google.com ([209.85.208.65]:45413 "EHLO mail-ed1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2390523AbfFJXfY (ORCPT ); Mon, 10 Jun 2019 19:35:24 -0400 Received: by mail-ed1-f65.google.com with SMTP id a14so15095499edv.12 for ; Mon, 10 Jun 2019 16:35:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=sJMrQDICL9b/toaSjuLa++uCWddhhIkBGXD/UJovQFU=; b=G5kuW6JHEI6sEENCgdVoMcIOyEePmJ6KN/NNEsDvMS/wTgk1+o/OLvbNM2lqOaNCpU wpbLaunbImccBvExXfdjj38oqjar2fYh51KaJeVX0t1JXgJBn6CCrARo5FW8PrdVjeLn dkapSvSdNpKqsKzXFt1luBZQI3BtT666++uAs388+s8rLTGT9R2Wv4j4+B3QRuUWNel5 u3uKpvdKk1vwxptTgZkKUGRqbN0Xt4netvrQwAEw4COWXJ1FHWXz3uyqx6hMnmZcJfa2 7YTaBYK/bspPyAcDVwak3Ix//8o4zrV5CnDwqMMpODUhsNBXURCFEmnhLlZg3KL9vicF 9qbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=sJMrQDICL9b/toaSjuLa++uCWddhhIkBGXD/UJovQFU=; b=hmq4vziGeDvWhUPwuy68yJdIqD0znOWZ1KuCjItgnj7EC66sdz0B8AZtm62qk0GuQc /5uzZi52gkDNWK+cJP0uJZRNIrmgkpjqw5JZOeUaqQD5SvuJahbnJb/K96DtD44Vi7F0 Yv70CSyRPfN9S3vbHz6CzGw04yMw0NHMuGQxYj2oUkSg5atSo51lm8cIZ3e3dlH2HzsW yfvnkAzzqnASn08sVKpJeHPE2RbzeaF3ZnZAulAw9pzSljf4Gn1w2O9dUJSJY77pYRKW JXK/iZneN3zfX9+fuqKuA+UAyNJiHUhaaqWos/kzOOWSivqUOswFaJK7GIcjtCUP3rx3 d4Tg== X-Gm-Message-State: APjAAAVR4z+6WOVbsNYj9alvrcp6EH0UJ2jSxG6vZZdg8N5S+klU2vz8 mlOPZMsdJz/iQfxdGK/kQ/cfO9lq X-Google-Smtp-Source: APXvYqxO7e6fbY95F/D1DCi1X4LEx0AxiGVz6b3rc/lyLDGhmyDHruSwCpJmbRrHApbMJy+H4/Hulw== X-Received: by 2002:a50:b3fb:: with SMTP id t56mr25317954edd.303.1560209722464; Mon, 10 Jun 2019 16:35:22 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id z10sm3277747edl.35.2019.06.10.16.35.21 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 10 Jun 2019 16:35:22 -0700 (PDT) Date: Mon, 10 Jun 2019 16:35:22 -0700 (PDT) X-Google-Original-Date: Mon, 10 Jun 2019 23:35:10 GMT Message-Id: <3b424f7c2acafb400284089b218d2fb57f3192f2.1560209720.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Subject: [PATCH 01/11] repack: refactor pack deletion for future use Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sbeller@google.com, peff@peff.net, jrnieder@gmail.com, avarab@gmail.com, jonathantanmy@google.com, Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Derrick Stolee The repack builtin deletes redundant pack-files and their associated .idx, .promisor, .bitmap, and .keep files. We will want to re-use this logic in the future for other types of repack, so pull the logic into 'unlink_pack_path()' in packfile.c. The 'ignore_keep' parameter is enabled for the use in repack, but will be important for a future caller. Signed-off-by: Derrick Stolee --- builtin/repack.c | 14 ++------------ packfile.c | 28 ++++++++++++++++++++++++++++ packfile.h | 7 +++++++ 3 files changed, 37 insertions(+), 12 deletions(-) diff --git a/builtin/repack.c b/builtin/repack.c index 67f8978043..1f9e6fad1b 100644 --- a/builtin/repack.c +++ b/builtin/repack.c @@ -129,19 +129,9 @@ static void get_non_kept_pack_filenames(struct string_list *fname_list, static void remove_redundant_pack(const char *dir_name, const char *base_name) { - const char *exts[] = {".pack", ".idx", ".keep", ".bitmap", ".promisor"}; - int i; struct strbuf buf = STRBUF_INIT; - size_t plen; - - strbuf_addf(&buf, "%s/%s", dir_name, base_name); - plen = buf.len; - - for (i = 0; i < ARRAY_SIZE(exts); i++) { - strbuf_setlen(&buf, plen); - strbuf_addstr(&buf, exts[i]); - unlink(buf.buf); - } + strbuf_addf(&buf, "%s/%s.pack", dir_name, base_name); + unlink_pack_path(buf.buf, 1); strbuf_release(&buf); } diff --git a/packfile.c b/packfile.c index 060de420d1..683ce5674c 100644 --- a/packfile.c +++ b/packfile.c @@ -352,6 +352,34 @@ void close_all_packs(struct raw_object_store *o) } } +void unlink_pack_path(const char *pack_name, int force_delete) +{ + static const char *exts[] = {".pack", ".idx", ".keep", ".bitmap", ".promisor"}; + int i; + struct strbuf buf = STRBUF_INIT; + size_t plen; + + strbuf_addstr(&buf, pack_name); + strip_suffix_mem(buf.buf, &buf.len, ".pack"); + plen = buf.len; + + if (!force_delete) { + strbuf_addstr(&buf, ".keep"); + if (!access(buf.buf, F_OK)) { + strbuf_release(&buf); + return; + } + } + + for (i = 0; i < ARRAY_SIZE(exts); i++) { + strbuf_setlen(&buf, plen); + strbuf_addstr(&buf, exts[i]); + unlink(buf.buf); + } + + strbuf_release(&buf); +} + /* * The LRU pack is the one with the oldest MRU window, preferring packs * with no used windows, or the oldest mtime if it has no windows allocated. diff --git a/packfile.h b/packfile.h index 12baa6118a..09f5222113 100644 --- a/packfile.h +++ b/packfile.h @@ -94,6 +94,13 @@ void unuse_pack(struct pack_window **); void clear_delta_base_cache(void); struct packed_git *add_packed_git(const char *path, size_t path_len, int local); +/* + * Unlink the .pack and associated extension files. + * Does not unlink if 'force_delete' is false and the pack-file is + * marked as ".keep". + */ +extern void unlink_pack_path(const char *pack_name, int force_delete); + /* * Make sure that a pointer access into an mmap'd index file is within bounds, * and can provide at least 8 bytes of data. From patchwork Mon Jun 10 23:35:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Derrick Stolee via GitGitGadget X-Patchwork-Id: 10985619 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1F1FA18A6 for ; Mon, 10 Jun 2019 23:35:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0EF2F2845C for ; Mon, 10 Jun 2019 23:35:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F313128694; Mon, 10 Jun 2019 23:35:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7D12A28623 for ; Mon, 10 Jun 2019 23:35:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2390612AbfFJXf0 (ORCPT ); Mon, 10 Jun 2019 19:35:26 -0400 Received: from mail-ed1-f66.google.com ([209.85.208.66]:32881 "EHLO mail-ed1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2390524AbfFJXfY (ORCPT ); Mon, 10 Jun 2019 19:35:24 -0400 Received: by mail-ed1-f66.google.com with SMTP id h9so17007298edr.0 for ; Mon, 10 Jun 2019 16:35:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:mime-version :content-transfer-encoding:fcc:content-transfer-encoding:to:cc; bh=KFvWs3MtFPh9f4rs4LNngKoc4vcN++p1+neR0WjImg8=; b=Lbuhg5/0PFmm9mNAtCmTIxiS5DkTCQiiLa43xRoD2h1kGiXPhy6fgxdZjDxD5xFkQ8 D8DjU5n3aAGcRs781GN/aqzOtpvoewsFio1vOll9sVmXITkIAQ8Z8iQICj57gA9GKpsk yLvuSbFN5ULlL+s4t5dppqj6AlX0xIs2VwlZNRIZyftBxWHV2xPXiVDqVkOqDeSJ0Lbv KdhExin5FA1KhmLVL1qH/cdXE0TvcNu9VCfzEYLiUr50EcDbsOrfm8QKE9flMlTZq/hz t5LxNNcdeVgKRN5KfQ8IM3eXyi8ar+JUPf4r5NWvkgPkTmB1B0JKdEJv8Hrdlu3Da5o3 /51g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:mime-version:content-transfer-encoding:fcc :content-transfer-encoding:to:cc; bh=KFvWs3MtFPh9f4rs4LNngKoc4vcN++p1+neR0WjImg8=; b=FHF+Mu73VW+ci4rALmOvavygQsZsMUYL3aSByp9h5tdi9GnZudU3FAFleI7wAGcii3 TpY5VIeC6ZJVT3KThO6VujC4PkUIZNHussLU+6NHukM6fP5q9wu5YJmDxAyd3zgmTb+F 21BdbxIO8AR5QY6cdK7LMMX2A+D75SyDxKhcJwruYVA3TNIfm+dha5+0hZbFTqQ08hZC rN8lSX/FsMhYSgbbTMEMEvRHy2Sh/evI2Z66dQ3DRL8aPGXPa+XxxuE56W63Nt1n822+ P8xQXFg9sg94OtKBYWsvpThvrSivSbmsrlXlSu5VNDK28lk558pGiC9js2d01H/rFhoc OA6g== X-Gm-Message-State: APjAAAX7qPezW+zeK2ewuapvoab8NbxHkLNPNrxOANoPc7ZcMuzyw3Bd zNJB67F8g/81XF/JoYsQWIvRXMcL X-Google-Smtp-Source: APXvYqzHCsrRt0wDyZWUgyfPrd4t8xgTt77HqG5yvCq7e/m7VBGM/SLdg7dCRh4OSihZG9CVU9YVhg== X-Received: by 2002:a50:95ae:: with SMTP id w43mr48301492eda.115.1560209723163; Mon, 10 Jun 2019 16:35:23 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id a17sm3224948edt.63.2019.06.10.16.35.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 10 Jun 2019 16:35:22 -0700 (PDT) Date: Mon, 10 Jun 2019 16:35:22 -0700 (PDT) X-Google-Original-Date: Mon, 10 Jun 2019 23:35:11 GMT Message-Id: In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Subject: [PATCH 02/11] Docs: rearrange subcommands for multi-pack-index MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: sbeller@google.com, peff@peff.net, jrnieder@gmail.com, avarab@gmail.com, jonathantanmy@google.com, Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Derrick Stolee We will add new subcommands to the multi-pack-index, and that will make the documentation a bit messier. Clean up the 'verb' descriptions by renaming the concept to 'subcommand' and removing the reference to the object directory. Helped-by: Stefan Beller Helped-by: Szeder Gábor Signed-off-by: Derrick Stolee --- Documentation/git-multi-pack-index.txt | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt index f7778a2c85..1af406aca2 100644 --- a/Documentation/git-multi-pack-index.txt +++ b/Documentation/git-multi-pack-index.txt @@ -9,7 +9,7 @@ git-multi-pack-index - Write and verify multi-pack-indexes SYNOPSIS -------- [verse] -'git multi-pack-index' [--object-dir=] +'git multi-pack-index' [--object-dir=] DESCRIPTION ----------- @@ -23,13 +23,13 @@ OPTIONS `/packs/multi-pack-index` for the current MIDX file, and `/packs` for the pack-files to index. +The following subcommands are available: + write:: - When given as the verb, write a new MIDX file to - `/packs/multi-pack-index`. + Write a new MIDX file. verify:: - When given as the verb, verify the contents of the MIDX file - at `/packs/multi-pack-index`. + Verify the contents of the MIDX file. EXAMPLES From patchwork Mon Jun 10 23:35:23 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee via GitGitGadget X-Patchwork-Id: 10985623 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2C43D924 for ; Mon, 10 Jun 2019 23:35:31 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 18D912845C for ; Mon, 10 Jun 2019 23:35:31 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0979528623; Mon, 10 Jun 2019 23:35:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 725D92845C for ; Mon, 10 Jun 2019 23:35:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2390647AbfFJXf2 (ORCPT ); Mon, 10 Jun 2019 19:35:28 -0400 Received: from mail-ed1-f65.google.com ([209.85.208.65]:36428 "EHLO mail-ed1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2390542AbfFJXf0 (ORCPT ); Mon, 10 Jun 2019 19:35:26 -0400 Received: by mail-ed1-f65.google.com with SMTP id k21so13635550edq.3 for ; Mon, 10 Jun 2019 16:35:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=YPIY2XnjoE9ory6N/X2WJBLLO8KS5ZqTf6UTegR52Xc=; b=XulEK6Ukrd+01IuEKBhkPOTG+PSYi3ypQk1Rn8eCNCiLWAspGgiRFQeiCo3McxrZ0S NbiTpnJ50tARu8Vth49SqC3LckTokj14eqenrHzFnKIgY8x2ghYFF7+bX+Pt9rHMSkJk P6rId/fkEk1LH3JS6XqxJjsSSIrW2Cw1js0ckXHb0VhZw7xUYXV4lJegc1pJ9v205y4M hlkQE/oPJamW2MGBrDZsJqzALHuNjOHdTsM2LAHPHhfEOrE351OY8irAZs7wr+CW9B8d cwP/GMFAcfLdsxhFerH45Aakn/KVteM62cLO7kVATkEWyZBPnULlcpNrrxKR0+poP7sD dj5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=YPIY2XnjoE9ory6N/X2WJBLLO8KS5ZqTf6UTegR52Xc=; b=FzPo5v7t0OFdv37M2CbRQvjbOMYlOn7pIPIMEmVE/3XUfrpS5aCAcwyIVvYND6LnA0 Miub8MvEB9QBfFNDYBysXPZKOuO04Ew02AHJKFYy6Q4+nAsDEzSDKe7+klkOSli9jjn3 fXDrSOQFTQDzNKK6/JNCoTaY29kZs7gwgTKkxu2X8TuuA1Cx2DtZbJN15UkUsTWqtRbw OZX8A/VLqf8LtWxyaL95PFu3As/1fbQHa7vHHCgxE6niTzzUf4CFOVn9JkVFDtiYSMKm sEC7RgEWFhR7Iwj9dIhIfhQXf/tYM1HVDdOUtkvQFj/r9skp8xkq+mSZFl//+Pu9sHpp OWdw== X-Gm-Message-State: APjAAAX8EoxhALTzq1y5oPDGhMZWEg3nC6+I7wiN4mYaQk1CNd7vr0V3 aUfZ+J/eevKm6GdY7sEIBNJcQmeQ X-Google-Smtp-Source: APXvYqxSTqJpKASpx0fB+sVMkAOLaF+t7bSAOyfR9XSK+mZhYn5uzz3/8E1Kkv5SurWlCKpGYMrA1w== X-Received: by 2002:a05:6402:1701:: with SMTP id y1mr75031654edu.22.1560209723849; Mon, 10 Jun 2019 16:35:23 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m21sm3195132edv.83.2019.06.10.16.35.23 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 10 Jun 2019 16:35:23 -0700 (PDT) Date: Mon, 10 Jun 2019 16:35:23 -0700 (PDT) X-Google-Original-Date: Mon, 10 Jun 2019 23:35:12 GMT Message-Id: In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Subject: [PATCH 03/11] multi-pack-index: prepare for 'expire' subcommand Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sbeller@google.com, peff@peff.net, jrnieder@gmail.com, avarab@gmail.com, jonathantanmy@google.com, Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Derrick Stolee The multi-pack-index tracks objects in a collection of pack-files. Only one copy of each object is indexed, using the modified time of the pack-files to determine tie-breakers. It is possible to have a pack-file with no referenced objects because all objects have a duplicate in a newer pack-file. Introduce a new 'expire' subcommand to the multi-pack-index builtin. This subcommand will delete these unused pack-files and rewrite the multi-pack-index to no longer refer to those files. More details about the specifics will follow as the method is implemented. Add a test that verifies the 'expire' subcommand is correctly wired, but will still be valid when the verb is implemented. Specifically, create a set of packs that should all have referenced objects and should not be removed during an 'expire' operation. The packs are created carefully to ensure they have a specific order when sorted by size. This will be important in a later test. Signed-off-by: Derrick Stolee --- Documentation/git-multi-pack-index.txt | 5 +++ builtin/multi-pack-index.c | 4 ++- midx.c | 5 +++ midx.h | 1 + t/t5319-multi-pack-index.sh | 49 ++++++++++++++++++++++++++ 5 files changed, 63 insertions(+), 1 deletion(-) diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt index 1af406aca2..6186c4c936 100644 --- a/Documentation/git-multi-pack-index.txt +++ b/Documentation/git-multi-pack-index.txt @@ -31,6 +31,11 @@ write:: verify:: Verify the contents of the MIDX file. +expire:: + Delete the pack-files that are tracked by the MIDX file, but + have no objects referenced by the MIDX. Rewrite the MIDX file + afterward to remove all references to these pack-files. + EXAMPLES -------- diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c index 72dfd3dadc..ad10d40512 100644 --- a/builtin/multi-pack-index.c +++ b/builtin/multi-pack-index.c @@ -6,7 +6,7 @@ #include "trace2.h" static char const * const builtin_multi_pack_index_usage[] = { - N_("git multi-pack-index [--object-dir=] (write|verify)"), + N_("git multi-pack-index [--object-dir=] (write|verify|expire)"), NULL }; @@ -47,6 +47,8 @@ int cmd_multi_pack_index(int argc, const char **argv, return write_midx_file(opts.object_dir); if (!strcmp(argv[0], "verify")) return verify_midx_file(the_repository, opts.object_dir); + if (!strcmp(argv[0], "expire")) + return expire_midx_packs(the_repository, opts.object_dir); die(_("unrecognized verb: %s"), argv[0]); } diff --git a/midx.c b/midx.c index e7e1fe4d65..3b7da1a360 100644 --- a/midx.c +++ b/midx.c @@ -1140,3 +1140,8 @@ int verify_midx_file(struct repository *r, const char *object_dir) return verify_midx_error; } + +int expire_midx_packs(struct repository *r, const char *object_dir) +{ + return 0; +} diff --git a/midx.h b/midx.h index 3eb29731f2..505f1431b7 100644 --- a/midx.h +++ b/midx.h @@ -50,6 +50,7 @@ int prepare_multi_pack_index_one(struct repository *r, const char *object_dir, i int write_midx_file(const char *object_dir); void clear_midx_file(struct repository *r); int verify_midx_file(struct repository *r, const char *object_dir); +int expire_midx_packs(struct repository *r, const char *object_dir); void close_midx(struct multi_pack_index *m); diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index 1ebf19ec3c..1b2d32f475 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -363,4 +363,53 @@ test_expect_success 'verify incorrect 64-bit offset' ' "incorrect object offset" ' +test_expect_success 'setup expire tests' ' + mkdir dup && + ( + cd dup && + git init && + test-tool genrandom "data" 4096 >large_file.txt && + git update-index --add large_file.txt && + for i in $(test_seq 1 20) + do + test_commit $i + done && + git branch A HEAD && + git branch B HEAD~8 && + git branch C HEAD~13 && + git branch D HEAD~16 && + git branch E HEAD~18 && + git pack-objects --revs .git/objects/pack/pack-A <<-EOF && + refs/heads/A + ^refs/heads/B + EOF + git pack-objects --revs .git/objects/pack/pack-B <<-EOF && + refs/heads/B + ^refs/heads/C + EOF + git pack-objects --revs .git/objects/pack/pack-C <<-EOF && + refs/heads/C + ^refs/heads/D + EOF + git pack-objects --revs .git/objects/pack/pack-D <<-EOF && + refs/heads/D + ^refs/heads/E + EOF + git pack-objects --revs .git/objects/pack/pack-E <<-EOF && + refs/heads/E + EOF + git multi-pack-index write + ) +' + +test_expect_success 'expire does not remove any packs' ' + ( + cd dup && + ls .git/objects/pack >expect && + git multi-pack-index expire && + ls .git/objects/pack >actual && + test_cmp expect actual + ) +' + test_done From patchwork Mon Jun 10 23:35:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee via GitGitGadget X-Patchwork-Id: 10985633 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B51A2924 for ; Mon, 10 Jun 2019 23:35:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A60092845C for ; Mon, 10 Jun 2019 23:35:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9A50B28623; Mon, 10 Jun 2019 23:35:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 34EB82845C for ; Mon, 10 Jun 2019 23:35:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2390645AbfFJXf1 (ORCPT ); Mon, 10 Jun 2019 19:35:27 -0400 Received: from mail-ed1-f66.google.com ([209.85.208.66]:32883 "EHLO mail-ed1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2390500AbfFJXf0 (ORCPT ); Mon, 10 Jun 2019 19:35:26 -0400 Received: by mail-ed1-f66.google.com with SMTP id h9so17007368edr.0 for ; Mon, 10 Jun 2019 16:35:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=SggN8uz60g65Y4aU3BtykqZV2Pldn4U2tE9XQ7+fT8E=; b=GUZVWXcZrbxym5C4pXDQofEMbOrcrJVCEyRiejLhPXg6bD7Wr1bX2c/PBNtNqgJ54o A+x1sYY9f5eoJ+gC2yunlYtyvYjthAT+BQThEPwPDvvHdzvfxeaVjiczaBF0NuihQa2v 9IuF7zc7sx91+ZWQwhT4L3P5lyZmVcoz9waKj3np5nvroe5LGL0r4xVGyDZcdO656b2x hWdOglfXmRPTmjAXwtpqYaDHlEP6Euoe2eoyw5Np7fSQRuhqhV+QJTOSe0n+wyHgiQ5A +RKwpPqUpG5iDqEfugHAAFMIE/4IgPBdhjGj5GrL8vCWkXLpHCkNbAq/Ci9gsHpIe2Jm qjUw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=SggN8uz60g65Y4aU3BtykqZV2Pldn4U2tE9XQ7+fT8E=; b=FQYOVGGyxE00Hx7d1SbTz8hTI7hJ/pd0e7c3JoKphY2AxlQWFgHVlQXKrEoYzMAACh OhlzziNfInk+Py9zMrINTwG9e/tqAk9ZdLm+Hrr627NocJp+R+1FE0RpefYsJH/e4Ywn 8Eye6+6o+2EIJ/8OHzbt56IgoK+4EPJuk23WYy+gQR4oZfP1tiEgJQvwCMFhq+1TzyuQ JTk6y25+jPhvh/1CUxpQ7n6khe8mjVBDBGKMq18Kns/Yl3KZbgUW1V08wSwEkQDZ3HFj yaj7JTP4CXWvFkzE1RvCFYRTKx263mKJb/SW9p4MhUgAYO+1OPCsjvh/841GkfrDaL9A uYyg== X-Gm-Message-State: APjAAAWNYrvFX1lwL7VsXcszRGoSC7t351AVmXRFcl8HjpFPSS6Zf+Q6 cEGa8C+vPK9MJRJAe6g0leJJFVpx X-Google-Smtp-Source: APXvYqz+hYJfQXb24V6ZjpKpKeHDr4At4fB1UjIL07sNc5gTYO0QrQdX3FjqFgcKfcLy74kfQ0DTcQ== X-Received: by 2002:a50:b566:: with SMTP id z35mr42166948edd.129.1560209724509; Mon, 10 Jun 2019 16:35:24 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id y14sm492900ejj.84.2019.06.10.16.35.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 10 Jun 2019 16:35:24 -0700 (PDT) Date: Mon, 10 Jun 2019 16:35:24 -0700 (PDT) X-Google-Original-Date: Mon, 10 Jun 2019 23:35:13 GMT Message-Id: In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Subject: [PATCH 04/11] midx: simplify computation of pack name lengths Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sbeller@google.com, peff@peff.net, jrnieder@gmail.com, avarab@gmail.com, jonathantanmy@google.com, Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Derrick Stolee Before writing the multi-pack-index, we compute the length of the pack-index names concatenated together. This forms the data in the pack name chunk, and we precompute it to compute chunk offsets. The value is also modified to fit alignment needs. Previously, this computation was coupled with adding packs from the existing multi-pack-index and the remaining packs in the object dir not already covered by the multi-pack-index. In anticipation of this becoming more complicated with the 'expire' subcommand, simplify the computation by centralizing it to a single loop before writing the file. Signed-off-by: Derrick Stolee --- midx.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/midx.c b/midx.c index 3b7da1a360..62404620ad 100644 --- a/midx.c +++ b/midx.c @@ -433,7 +433,6 @@ struct pack_list { uint32_t nr; uint32_t alloc_list; uint32_t alloc_names; - size_t pack_name_concat_len; struct multi_pack_index *m; }; @@ -468,7 +467,6 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len, } packs->names[packs->nr] = xstrdup(file_name); - packs->pack_name_concat_len += strlen(file_name) + 1; packs->nr++; } } @@ -812,6 +810,7 @@ int write_midx_file(const char *object_dir) uint32_t nr_entries, num_large_offsets = 0; struct pack_midx_entry *entries = NULL; int large_offsets_needed = 0; + int pack_name_concat_len = 0; midx_name = get_midx_filename(object_dir); if (safe_create_leading_directories(midx_name)) { @@ -827,7 +826,6 @@ int write_midx_file(const char *object_dir) packs.alloc_names = packs.alloc_list; packs.list = NULL; packs.names = NULL; - packs.pack_name_concat_len = 0; ALLOC_ARRAY(packs.list, packs.alloc_list); ALLOC_ARRAY(packs.names, packs.alloc_names); @@ -838,7 +836,6 @@ int write_midx_file(const char *object_dir) packs.list[packs.nr] = NULL; packs.names[packs.nr] = xstrdup(packs.m->pack_names[i]); - packs.pack_name_concat_len += strlen(packs.names[packs.nr]) + 1; packs.nr++; } } @@ -848,10 +845,6 @@ int write_midx_file(const char *object_dir) if (packs.m && packs.nr == packs.m->num_packs) goto cleanup; - if (packs.pack_name_concat_len % MIDX_CHUNK_ALIGNMENT) - packs.pack_name_concat_len += MIDX_CHUNK_ALIGNMENT - - (packs.pack_name_concat_len % MIDX_CHUNK_ALIGNMENT); - ALLOC_ARRAY(pack_perm, packs.nr); sort_packs_by_name(packs.names, packs.nr, pack_perm); @@ -864,6 +857,13 @@ int write_midx_file(const char *object_dir) large_offsets_needed = 1; } + for (i = 0; i < packs.nr; i++) + pack_name_concat_len += strlen(packs.names[i]) + 1; + + if (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT) + pack_name_concat_len += MIDX_CHUNK_ALIGNMENT - + (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT); + hold_lock_file_for_update(&lk, midx_name, LOCK_DIE_ON_ERROR); f = hashfd(lk.tempfile->fd, lk.tempfile->filename.buf); FREE_AND_NULL(midx_name); @@ -881,7 +881,7 @@ int write_midx_file(const char *object_dir) cur_chunk++; chunk_ids[cur_chunk] = MIDX_CHUNKID_OIDFANOUT; - chunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + packs.pack_name_concat_len; + chunk_offsets[cur_chunk] = chunk_offsets[cur_chunk - 1] + pack_name_concat_len; cur_chunk++; chunk_ids[cur_chunk] = MIDX_CHUNKID_OIDLOOKUP; From patchwork Mon Jun 10 23:35:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee via GitGitGadget X-Patchwork-Id: 10985625 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E8E0F924 for ; Mon, 10 Jun 2019 23:35:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D9B2C285CB for ; Mon, 10 Jun 2019 23:35:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CDBB128623; Mon, 10 Jun 2019 23:35:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B8E432845C for ; Mon, 10 Jun 2019 23:35:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2390657AbfFJXfa (ORCPT ); Mon, 10 Jun 2019 19:35:30 -0400 Received: from mail-ed1-f42.google.com ([209.85.208.42]:38959 "EHLO mail-ed1-f42.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2390594AbfFJXf1 (ORCPT ); Mon, 10 Jun 2019 19:35:27 -0400 Received: by mail-ed1-f42.google.com with SMTP id m10so16979446edv.6 for ; Mon, 10 Jun 2019 16:35:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=mpfvp3LXyDFKA7YbepC/UoBn52fHXiFcGXN1OG04X44=; b=VllqpntLWwlx+TeEtBeHv8h9GbKYHwXiKBXcO2xQ6dwASimpSmH+iupoFpmFGMye0j abVFTbF3FCmIC8bU/k+e790wdkRB44yOeocLvTEj97nx5sBql0fM2oQU4QfK9gO7PfHF RGWFY4Sx+ctnmW5Y3SRtkibcjrNUW2wqPfxGWlIMzjTSpNdIlZOzk+vod0AC1lNSPeRY F5mNG3SIA4lXCuwQynlCCUGt+NccmP7JOFbwcEriXmr+qvue226swbJccvyB8M6IYLNs gwPWyoYg+ss2mJM7GMaF/X0LlqJRoepFAqyrqiPvHAXdwxDjmI4Qvd9PZ/HmcWS1j+zB usZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=mpfvp3LXyDFKA7YbepC/UoBn52fHXiFcGXN1OG04X44=; b=WG+vmN9Z59/v4TtPrcUoz/AlC5A8kHaI7NrnQZWDp8oX8j/W1KQgr9lBzvG9i46tQe BRX+FWzN3a5Z8oJ4L9nM2tsdUkQR/M4FzXfjorqnHqPiqjEDzLcMB1xwgqBxZFD5EqWA KV0MROy6vk5v8f3+Rr8WPYvCVMuPZ4lPYZ56n7vftNA1tJL0FOtYNh2Mgp+hYjGUkJDW geyfUo3+1M4/9oltfqpFRFxiK+gETEholxuwBqzOEF1yOlc041xCeuG2pjs/+/mZGTu0 DwM3AK2afFDA49QNe9vX/3qRXct6sjtQliB2TRZP6XcnAM8CwHE1+Vsc3a9/jx5mFNsH 2bbQ== X-Gm-Message-State: APjAAAUCenqla+ENVGhAojUQPrLACZaLD0SWJU/OJ+pmgIO0n/rEzlfB 0SmGnVMoSLaxkXa6GQ8g74EGlR6u X-Google-Smtp-Source: APXvYqwLjNVW3YgUnB0E/BrJPZVvz40pSCFdblSMR008PkQbxb/ZEwdGKrqPpzXFwDX23R0au/DCGw== X-Received: by 2002:a50:cac3:: with SMTP id f3mr75554272edi.91.1560209725156; Mon, 10 Jun 2019 16:35:25 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id z49sm1428670edz.61.2019.06.10.16.35.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 10 Jun 2019 16:35:24 -0700 (PDT) Date: Mon, 10 Jun 2019 16:35:24 -0700 (PDT) X-Google-Original-Date: Mon, 10 Jun 2019 23:35:14 GMT Message-Id: <989d49d0b284980fc1b46bd8c309448ff538fd78.1560209720.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Subject: [PATCH 05/11] midx: refactor permutation logic and pack sorting Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sbeller@google.com, peff@peff.net, jrnieder@gmail.com, avarab@gmail.com, jonathantanmy@google.com, Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Derrick Stolee In anticipation of the expire subcommand, refactor the way we sort the packfiles by name. This will greatly simplify our approach to dropping expired packs from the list. First, create 'struct pack_info' to replace 'struct pack_pair'. This struct contains the necessary information about a pack, including its name, a pointer to its packfile struct (if not already in the multi-pack-index), and the original pack-int-id. Second, track the pack information using an array of pack_info structs in the pack_list struct. This simplifies the logic around the multiple arrays we were tracking in that struct. Finally, update get_sorted_entries() to not permute the pack-int-id and instead supply the permutation to write_midx_object_offsets(). This requires sorting the packs after get_sorted_entries(). Signed-off-by: Derrick Stolee --- midx.c | 156 +++++++++++++++++++++++++-------------------------------- 1 file changed, 69 insertions(+), 87 deletions(-) diff --git a/midx.c b/midx.c index 62404620ad..6d4b84e243 100644 --- a/midx.c +++ b/midx.c @@ -427,12 +427,23 @@ static size_t write_midx_header(struct hashfile *f, return MIDX_HEADER_SIZE; } +struct pack_info { + uint32_t orig_pack_int_id; + char *pack_name; + struct packed_git *p; +}; + +static int pack_info_compare(const void *_a, const void *_b) +{ + struct pack_info *a = (struct pack_info *)_a; + struct pack_info *b = (struct pack_info *)_b; + return strcmp(a->pack_name, b->pack_name); +} + struct pack_list { - struct packed_git **list; - char **names; + struct pack_info *info; uint32_t nr; - uint32_t alloc_list; - uint32_t alloc_names; + uint32_t alloc; struct multi_pack_index *m; }; @@ -445,66 +456,32 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len, if (packs->m && midx_contains_pack(packs->m, file_name)) return; - ALLOC_GROW(packs->list, packs->nr + 1, packs->alloc_list); - ALLOC_GROW(packs->names, packs->nr + 1, packs->alloc_names); + ALLOC_GROW(packs->info, packs->nr + 1, packs->alloc); - packs->list[packs->nr] = add_packed_git(full_path, - full_path_len, - 0); + packs->info[packs->nr].p = add_packed_git(full_path, + full_path_len, + 0); - if (!packs->list[packs->nr]) { + if (!packs->info[packs->nr].p) { warning(_("failed to add packfile '%s'"), full_path); return; } - if (open_pack_index(packs->list[packs->nr])) { + if (open_pack_index(packs->info[packs->nr].p)) { warning(_("failed to open pack-index '%s'"), full_path); - close_pack(packs->list[packs->nr]); - FREE_AND_NULL(packs->list[packs->nr]); + close_pack(packs->info[packs->nr].p); + FREE_AND_NULL(packs->info[packs->nr].p); return; } - packs->names[packs->nr] = xstrdup(file_name); + packs->info[packs->nr].pack_name = xstrdup(file_name); + packs->info[packs->nr].orig_pack_int_id = packs->nr; packs->nr++; } } -struct pack_pair { - uint32_t pack_int_id; - char *pack_name; -}; - -static int pack_pair_compare(const void *_a, const void *_b) -{ - struct pack_pair *a = (struct pack_pair *)_a; - struct pack_pair *b = (struct pack_pair *)_b; - return strcmp(a->pack_name, b->pack_name); -} - -static void sort_packs_by_name(char **pack_names, uint32_t nr_packs, uint32_t *perm) -{ - uint32_t i; - struct pack_pair *pairs; - - ALLOC_ARRAY(pairs, nr_packs); - - for (i = 0; i < nr_packs; i++) { - pairs[i].pack_int_id = i; - pairs[i].pack_name = pack_names[i]; - } - - QSORT(pairs, nr_packs, pack_pair_compare); - - for (i = 0; i < nr_packs; i++) { - pack_names[i] = pairs[i].pack_name; - perm[pairs[i].pack_int_id] = i; - } - - free(pairs); -} - struct pack_midx_entry { struct object_id oid; uint32_t pack_int_id; @@ -530,7 +507,6 @@ static int midx_oid_compare(const void *_a, const void *_b) } static int nth_midxed_pack_midx_entry(struct multi_pack_index *m, - uint32_t *pack_perm, struct pack_midx_entry *e, uint32_t pos) { @@ -538,7 +514,7 @@ static int nth_midxed_pack_midx_entry(struct multi_pack_index *m, return 1; nth_midxed_object_oid(&e->oid, m, pos); - e->pack_int_id = pack_perm[nth_midxed_pack_int_id(m, pos)]; + e->pack_int_id = nth_midxed_pack_int_id(m, pos); e->offset = nth_midxed_offset(m, pos); /* consider objects in midx to be from "old" packs */ @@ -572,8 +548,7 @@ static void fill_pack_entry(uint32_t pack_int_id, * of a packfile containing the object). */ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m, - struct packed_git **p, - uint32_t *perm, + struct pack_info *info, uint32_t nr_packs, uint32_t *nr_objects) { @@ -584,7 +559,7 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m, uint32_t start_pack = m ? m->num_packs : 0; for (cur_pack = start_pack; cur_pack < nr_packs; cur_pack++) - total_objects += p[cur_pack]->num_objects; + total_objects += info[cur_pack].p->num_objects; /* * As we de-duplicate by fanout value, we expect the fanout @@ -609,7 +584,7 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m, for (cur_object = start; cur_object < end; cur_object++) { ALLOC_GROW(entries_by_fanout, nr_fanout + 1, alloc_fanout); - nth_midxed_pack_midx_entry(m, perm, + nth_midxed_pack_midx_entry(m, &entries_by_fanout[nr_fanout], cur_object); nr_fanout++; @@ -620,12 +595,12 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m, uint32_t start = 0, end; if (cur_fanout) - start = get_pack_fanout(p[cur_pack], cur_fanout - 1); - end = get_pack_fanout(p[cur_pack], cur_fanout); + start = get_pack_fanout(info[cur_pack].p, cur_fanout - 1); + end = get_pack_fanout(info[cur_pack].p, cur_fanout); for (cur_object = start; cur_object < end; cur_object++) { ALLOC_GROW(entries_by_fanout, nr_fanout + 1, alloc_fanout); - fill_pack_entry(perm[cur_pack], p[cur_pack], cur_object, &entries_by_fanout[nr_fanout]); + fill_pack_entry(cur_pack, info[cur_pack].p, cur_object, &entries_by_fanout[nr_fanout]); nr_fanout++; } } @@ -654,7 +629,7 @@ static struct pack_midx_entry *get_sorted_entries(struct multi_pack_index *m, } static size_t write_midx_pack_names(struct hashfile *f, - char **pack_names, + struct pack_info *info, uint32_t num_packs) { uint32_t i; @@ -662,14 +637,14 @@ static size_t write_midx_pack_names(struct hashfile *f, size_t written = 0; for (i = 0; i < num_packs; i++) { - size_t writelen = strlen(pack_names[i]) + 1; + size_t writelen = strlen(info[i].pack_name) + 1; - if (i && strcmp(pack_names[i], pack_names[i - 1]) <= 0) + if (i && strcmp(info[i].pack_name, info[i - 1].pack_name) <= 0) BUG("incorrect pack-file order: %s before %s", - pack_names[i - 1], - pack_names[i]); + info[i - 1].pack_name, + info[i].pack_name); - hashwrite(f, pack_names[i], writelen); + hashwrite(f, info[i].pack_name, writelen); written += writelen; } @@ -740,6 +715,7 @@ static size_t write_midx_oid_lookup(struct hashfile *f, unsigned char hash_len, } static size_t write_midx_object_offsets(struct hashfile *f, int large_offset_needed, + uint32_t *perm, struct pack_midx_entry *objects, uint32_t nr_objects) { struct pack_midx_entry *list = objects; @@ -749,7 +725,7 @@ static size_t write_midx_object_offsets(struct hashfile *f, int large_offset_nee for (i = 0; i < nr_objects; i++) { struct pack_midx_entry *obj = list++; - hashwrite_be32(f, obj->pack_int_id); + hashwrite_be32(f, perm[obj->pack_int_id]); if (large_offset_needed && obj->offset >> 31) hashwrite_be32(f, MIDX_LARGE_OFFSET_NEEDED | nr_large_offset++); @@ -822,20 +798,17 @@ int write_midx_file(const char *object_dir) packs.m = load_multi_pack_index(object_dir, 1); packs.nr = 0; - packs.alloc_list = packs.m ? packs.m->num_packs : 16; - packs.alloc_names = packs.alloc_list; - packs.list = NULL; - packs.names = NULL; - ALLOC_ARRAY(packs.list, packs.alloc_list); - ALLOC_ARRAY(packs.names, packs.alloc_names); + packs.alloc = packs.m ? packs.m->num_packs : 16; + packs.info = NULL; + ALLOC_ARRAY(packs.info, packs.alloc); if (packs.m) { for (i = 0; i < packs.m->num_packs; i++) { - ALLOC_GROW(packs.list, packs.nr + 1, packs.alloc_list); - ALLOC_GROW(packs.names, packs.nr + 1, packs.alloc_names); + ALLOC_GROW(packs.info, packs.nr + 1, packs.alloc); - packs.list[packs.nr] = NULL; - packs.names[packs.nr] = xstrdup(packs.m->pack_names[i]); + packs.info[packs.nr].orig_pack_int_id = i; + packs.info[packs.nr].pack_name = xstrdup(packs.m->pack_names[i]); + packs.info[packs.nr].p = NULL; packs.nr++; } } @@ -845,10 +818,7 @@ int write_midx_file(const char *object_dir) if (packs.m && packs.nr == packs.m->num_packs) goto cleanup; - ALLOC_ARRAY(pack_perm, packs.nr); - sort_packs_by_name(packs.names, packs.nr, pack_perm); - - entries = get_sorted_entries(packs.m, packs.list, pack_perm, packs.nr, &nr_entries); + entries = get_sorted_entries(packs.m, packs.info, packs.nr, &nr_entries); for (i = 0; i < nr_entries; i++) { if (entries[i].offset > 0x7fffffff) @@ -857,8 +827,21 @@ int write_midx_file(const char *object_dir) large_offsets_needed = 1; } + QSORT(packs.info, packs.nr, pack_info_compare); + + /* + * pack_perm stores a permutation between pack-int-ids from the + * previous multi-pack-index to the new one we are writing: + * + * pack_perm[old_id] = new_id + */ + ALLOC_ARRAY(pack_perm, packs.nr); + for (i = 0; i < packs.nr; i++) { + pack_perm[packs.info[i].orig_pack_int_id] = i; + } + for (i = 0; i < packs.nr; i++) - pack_name_concat_len += strlen(packs.names[i]) + 1; + pack_name_concat_len += strlen(packs.info[i].pack_name) + 1; if (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT) pack_name_concat_len += MIDX_CHUNK_ALIGNMENT - @@ -929,7 +912,7 @@ int write_midx_file(const char *object_dir) switch (chunk_ids[i]) { case MIDX_CHUNKID_PACKNAMES: - written += write_midx_pack_names(f, packs.names, packs.nr); + written += write_midx_pack_names(f, packs.info, packs.nr); break; case MIDX_CHUNKID_OIDFANOUT: @@ -941,7 +924,7 @@ int write_midx_file(const char *object_dir) break; case MIDX_CHUNKID_OBJECTOFFSETS: - written += write_midx_object_offsets(f, large_offsets_needed, entries, nr_entries); + written += write_midx_object_offsets(f, large_offsets_needed, pack_perm, entries, nr_entries); break; case MIDX_CHUNKID_LARGEOFFSETS: @@ -964,15 +947,14 @@ int write_midx_file(const char *object_dir) cleanup: for (i = 0; i < packs.nr; i++) { - if (packs.list[i]) { - close_pack(packs.list[i]); - free(packs.list[i]); + if (packs.info[i].p) { + close_pack(packs.info[i].p); + free(packs.info[i].p); } - free(packs.names[i]); + free(packs.info[i].pack_name); } - free(packs.list); - free(packs.names); + free(packs.info); free(entries); free(pack_perm); free(midx_name); From patchwork Mon Jun 10 23:35:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee via GitGitGadget X-Patchwork-Id: 10985639 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 35B16924 for ; Mon, 10 Jun 2019 23:35:40 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 24DC12845C for ; Mon, 10 Jun 2019 23:35:40 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1900628623; Mon, 10 Jun 2019 23:35:40 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 27B1F2845C for ; Mon, 10 Jun 2019 23:35:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2390687AbfFJXfg (ORCPT ); Mon, 10 Jun 2019 19:35:36 -0400 Received: from mail-ed1-f67.google.com ([209.85.208.67]:41800 "EHLO mail-ed1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2390524AbfFJXf3 (ORCPT ); Mon, 10 Jun 2019 19:35:29 -0400 Received: by mail-ed1-f67.google.com with SMTP id p15so16951165eds.8 for ; Mon, 10 Jun 2019 16:35:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Fuav/+nheF5SqDD8thcZ8OGP+BR7lbC5ETmzvGSjWYA=; b=al8WA9Cy4x1KK9Xh89McxwIElefU82S4T7uGy31nfuJchLfFxUU3DIlE4EW5mFHroK 9z9BkrtHsLVAwIckT+pvBmcC9cpJ4Oun2ewlFuNzSVF39K9/7ADKDLVOw5AGVrXhdz1Z 0LTZd52qrPrqwDutGz+6nKO5LRDmJLByYwYm+fbp0yyU+O0jx4nd6naEZ4lYeI0URuMH 5cRlvAKGCGuU2vbrnlkCJ2IqDD7K6bMs+Nt7gOP8M1HrmQ7VIjh+vEnikcdzuGcz2zQB +b0+qyxAqTLh+D5G8fZL14WQj+hqv4Qfe3EyjGYsbI2GDGkkxKHR+KGg9FU+e7jDFPsE g6ZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Fuav/+nheF5SqDD8thcZ8OGP+BR7lbC5ETmzvGSjWYA=; b=ltZrxpB2RGp7VUlskI1yl0mn+7c79OEJ1Dbd1nakvCESqjvsQnXavcA9PuZmfKoNBp bH7GBPiOw+E8O01UfShLRO0co031bkarN9ZE/Xa89px0unAe7AbG3EN9nKI4vtOm9FNj ItVYs+9S8cTYdjG/TR1MZCoq2PhQUWFVnYVC3JX341G/enq2od524zk0rWq2BvWeuQeY NRpSuK1CdNybkE/v0dJ6lgz8I6Suok4FzxVRo0+Ziu3K1zobSKGq/8FBrdR6LlpaTcQi HWOAG1u2pnQ1odiOcbtn1muqVWZ6Sa214bU/WMvdNZWc7RcYDAsUSPAb7DPY5nYQ/Pxz HMUQ== X-Gm-Message-State: APjAAAU9Z3Shz6k9Mi5zYuhuwgawDHegYYqzJkbFOIM2MqSDdX5921Jp v0FldYDEDjW9gHALPPtvAwVwRdi1 X-Google-Smtp-Source: APXvYqwv+UmP58z6ka+pXHKNwqZ0Xx2r0XbuyHRNNThQB5eJWfXbLTbUfMBGt1A4BImEEBUny4QxeA== X-Received: by 2002:aa7:cf0c:: with SMTP id a12mr4610765edy.146.1560209726091; Mon, 10 Jun 2019 16:35:26 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l45sm2474613edc.93.2019.06.10.16.35.25 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 10 Jun 2019 16:35:25 -0700 (PDT) Date: Mon, 10 Jun 2019 16:35:25 -0700 (PDT) X-Google-Original-Date: Mon, 10 Jun 2019 23:35:15 GMT Message-Id: <8213541052a7d040aad4352ee981c0de01cc6516.1560209720.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Subject: [PATCH 06/11] multi-pack-index: implement 'expire' subcommand Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sbeller@google.com, peff@peff.net, jrnieder@gmail.com, avarab@gmail.com, jonathantanmy@google.com, Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Derrick Stolee The 'git multi-pack-index expire' subcommand looks at the existing mult-pack-index, counts the number of objects referenced in each pack-file, deletes the pack-fils with no referenced objects, and rewrites the multi-pack-index to no longer reference those packs. Refactor the write_midx_file() method to call write_midx_internal() which now takes an existing 'struct multi_pack_index' and a list of pack-files to drop (as specified by the names of their pack- indexes). As we write the new multi-pack-index, we drop those file names from the list of known pack-files. The expire_midx_packs() method removes the unreferenced pack-files after carefully closing the packs to avoid open handles. Test that a new pack-file that covers the contents of two other pack-files leads to those pack-files being deleted during the expire subcommand. Be sure to read the multi-pack-index to ensure it no longer references those packs. Signed-off-by: Derrick Stolee --- midx.c | 119 +++++++++++++++++++++++++++++++++--- t/t5319-multi-pack-index.sh | 20 ++++++ 2 files changed, 129 insertions(+), 10 deletions(-) diff --git a/midx.c b/midx.c index 6d4b84e243..9b0b4c1520 100644 --- a/midx.c +++ b/midx.c @@ -34,6 +34,8 @@ #define MIDX_CHUNK_LARGE_OFFSET_WIDTH (sizeof(uint64_t)) #define MIDX_LARGE_OFFSET_NEEDED 0x80000000 +#define PACK_EXPIRED UINT_MAX + static char *get_midx_filename(const char *object_dir) { return xstrfmt("%s/pack/multi-pack-index", object_dir); @@ -431,6 +433,7 @@ struct pack_info { uint32_t orig_pack_int_id; char *pack_name; struct packed_git *p; + unsigned expired : 1; }; static int pack_info_compare(const void *_a, const void *_b) @@ -478,6 +481,7 @@ static void add_pack_to_midx(const char *full_path, size_t full_path_len, packs->info[packs->nr].pack_name = xstrdup(file_name); packs->info[packs->nr].orig_pack_int_id = packs->nr; + packs->info[packs->nr].expired = 0; packs->nr++; } } @@ -637,13 +641,17 @@ static size_t write_midx_pack_names(struct hashfile *f, size_t written = 0; for (i = 0; i < num_packs; i++) { - size_t writelen = strlen(info[i].pack_name) + 1; + size_t writelen; + + if (info[i].expired) + continue; if (i && strcmp(info[i].pack_name, info[i - 1].pack_name) <= 0) BUG("incorrect pack-file order: %s before %s", info[i - 1].pack_name, info[i].pack_name); + writelen = strlen(info[i].pack_name) + 1; hashwrite(f, info[i].pack_name, writelen); written += writelen; } @@ -725,6 +733,11 @@ static size_t write_midx_object_offsets(struct hashfile *f, int large_offset_nee for (i = 0; i < nr_objects; i++) { struct pack_midx_entry *obj = list++; + if (perm[obj->pack_int_id] == PACK_EXPIRED) + BUG("object %s is in an expired pack with int-id %d", + oid_to_hex(&obj->oid), + obj->pack_int_id); + hashwrite_be32(f, perm[obj->pack_int_id]); if (large_offset_needed && obj->offset >> 31) @@ -771,7 +784,8 @@ static size_t write_midx_large_offsets(struct hashfile *f, uint32_t nr_large_off return written; } -int write_midx_file(const char *object_dir) +static int write_midx_internal(const char *object_dir, struct multi_pack_index *m, + struct string_list *packs_to_drop) { unsigned char cur_chunk, num_chunks = 0; char *midx_name; @@ -787,6 +801,8 @@ int write_midx_file(const char *object_dir) struct pack_midx_entry *entries = NULL; int large_offsets_needed = 0; int pack_name_concat_len = 0; + int dropped_packs = 0; + int result = 0; midx_name = get_midx_filename(object_dir); if (safe_create_leading_directories(midx_name)) { @@ -795,7 +811,10 @@ int write_midx_file(const char *object_dir) midx_name); } - packs.m = load_multi_pack_index(object_dir, 1); + if (m) + packs.m = m; + else + packs.m = load_multi_pack_index(object_dir, 1); packs.nr = 0; packs.alloc = packs.m ? packs.m->num_packs : 16; @@ -809,13 +828,14 @@ int write_midx_file(const char *object_dir) packs.info[packs.nr].orig_pack_int_id = i; packs.info[packs.nr].pack_name = xstrdup(packs.m->pack_names[i]); packs.info[packs.nr].p = NULL; + packs.info[packs.nr].expired = 0; packs.nr++; } } for_each_file_in_pack_dir(object_dir, add_pack_to_midx, &packs); - if (packs.m && packs.nr == packs.m->num_packs) + if (packs.m && packs.nr == packs.m->num_packs && !packs_to_drop) goto cleanup; entries = get_sorted_entries(packs.m, packs.info, packs.nr, &nr_entries); @@ -829,6 +849,34 @@ int write_midx_file(const char *object_dir) QSORT(packs.info, packs.nr, pack_info_compare); + if (packs_to_drop && packs_to_drop->nr) { + int drop_index = 0; + int missing_drops = 0; + + for (i = 0; i < packs.nr && drop_index < packs_to_drop->nr; i++) { + int cmp = strcmp(packs.info[i].pack_name, + packs_to_drop->items[drop_index].string); + + if (!cmp) { + drop_index++; + packs.info[i].expired = 1; + } else if (cmp > 0) { + error(_("did not see pack-file %s to drop"), + packs_to_drop->items[drop_index].string); + drop_index++; + missing_drops++; + i--; + } else { + packs.info[i].expired = 0; + } + } + + if (missing_drops) { + result = 1; + goto cleanup; + } + } + /* * pack_perm stores a permutation between pack-int-ids from the * previous multi-pack-index to the new one we are writing: @@ -837,11 +885,18 @@ int write_midx_file(const char *object_dir) */ ALLOC_ARRAY(pack_perm, packs.nr); for (i = 0; i < packs.nr; i++) { - pack_perm[packs.info[i].orig_pack_int_id] = i; + if (packs.info[i].expired) { + dropped_packs++; + pack_perm[packs.info[i].orig_pack_int_id] = PACK_EXPIRED; + } else { + pack_perm[packs.info[i].orig_pack_int_id] = i - dropped_packs; + } } - for (i = 0; i < packs.nr; i++) - pack_name_concat_len += strlen(packs.info[i].pack_name) + 1; + for (i = 0; i < packs.nr; i++) { + if (!packs.info[i].expired) + pack_name_concat_len += strlen(packs.info[i].pack_name) + 1; + } if (pack_name_concat_len % MIDX_CHUNK_ALIGNMENT) pack_name_concat_len += MIDX_CHUNK_ALIGNMENT - @@ -857,7 +912,7 @@ int write_midx_file(const char *object_dir) cur_chunk = 0; num_chunks = large_offsets_needed ? 5 : 4; - written = write_midx_header(f, num_chunks, packs.nr); + written = write_midx_header(f, num_chunks, packs.nr - dropped_packs); chunk_ids[cur_chunk] = MIDX_CHUNKID_PACKNAMES; chunk_offsets[cur_chunk] = written + (num_chunks + 1) * MIDX_CHUNKLOOKUP_WIDTH; @@ -958,7 +1013,12 @@ int write_midx_file(const char *object_dir) free(entries); free(pack_perm); free(midx_name); - return 0; + return result; +} + +int write_midx_file(const char *object_dir) +{ + return write_midx_internal(object_dir, NULL, NULL); } void clear_midx_file(struct repository *r) @@ -1125,5 +1185,44 @@ int verify_midx_file(struct repository *r, const char *object_dir) int expire_midx_packs(struct repository *r, const char *object_dir) { - return 0; + uint32_t i, *count, result = 0; + struct string_list packs_to_drop = STRING_LIST_INIT_DUP; + struct multi_pack_index *m = load_multi_pack_index(object_dir, 1); + + if (!m) + return 0; + + count = xcalloc(m->num_packs, sizeof(uint32_t)); + for (i = 0; i < m->num_objects; i++) { + int pack_int_id = nth_midxed_pack_int_id(m, i); + count[pack_int_id]++; + } + + for (i = 0; i < m->num_packs; i++) { + char *pack_name; + + if (count[i]) + continue; + + if (prepare_midx_pack(r, m, i)) + continue; + + if (m->packs[i]->pack_keep) + continue; + + pack_name = xstrdup(m->packs[i]->pack_name); + close_pack(m->packs[i]); + + string_list_insert(&packs_to_drop, m->pack_names[i]); + unlink_pack_path(pack_name, 0); + free(pack_name); + } + + free(count); + + if (packs_to_drop.nr) + result = write_midx_internal(object_dir, m, &packs_to_drop); + + string_list_clear(&packs_to_drop, 0); + return result; } diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index 1b2d32f475..12570fe7ac 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -412,4 +412,24 @@ test_expect_success 'expire does not remove any packs' ' ) ' +test_expect_success 'expire removes unreferenced packs' ' + ( + cd dup && + git pack-objects --revs .git/objects/pack/pack-combined <<-EOF && + refs/heads/A + ^refs/heads/C + EOF + git multi-pack-index write && + ls .git/objects/pack | grep -v -e pack-[AB] >expect && + git multi-pack-index expire && + ls .git/objects/pack >actual && + test_cmp expect actual && + ls .git/objects/pack/ | grep idx >expect-idx && + test-tool read-midx .git/objects | grep idx >actual-midx && + test_cmp expect-idx actual-midx && + git multi-pack-index verify && + git fsck + ) +' + test_done From patchwork Mon Jun 10 23:35:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee via GitGitGadget X-Patchwork-Id: 10985635 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B240114BB for ; Mon, 10 Jun 2019 23:35:36 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A25B82845C for ; Mon, 10 Jun 2019 23:35:36 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 96AC928623; Mon, 10 Jun 2019 23:35:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E5E0C2845C for ; Mon, 10 Jun 2019 23:35:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2390680AbfFJXff (ORCPT ); Mon, 10 Jun 2019 19:35:35 -0400 Received: from mail-ed1-f65.google.com ([209.85.208.65]:44889 "EHLO mail-ed1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2390635AbfFJXf3 (ORCPT ); Mon, 10 Jun 2019 19:35:29 -0400 Received: by mail-ed1-f65.google.com with SMTP id k8so16942254edr.11 for ; Mon, 10 Jun 2019 16:35:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=XmZa6hJ/DAia1oLOApgobp7hn1ekOYDrB9JZoCOz58A=; b=KXWsjmH8rgAct7IpcIaTcum38eZBQPtN52VNLCVvCwgazlxjkJksaxmQOQ1GaGNefn eCl3OuX9INz4drbiGezcuRbxbdEsyo6HDSFg4qS5LlP0/2bMDfoIs9au3WihtfkDr1ak Tna3UVkWsqrcwgPTO7/Orm4UH8XB+RjY6aAefX7AfDE4y0tvaFQtZTZ0S+VqoBWfGDPI A8inGJahK46QbrnGWYvxLdVgGjCtOEuLc7kNGi/5ghzg7kqP7AeZaiyeEmUFXtN/q9Nm 6j00JY/kuIr3eJR5QQC50x5ru2aHtjPUwBsze/OmmUW5HjtxurxM0aARDRCzjl2Rs+iZ 5r6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=XmZa6hJ/DAia1oLOApgobp7hn1ekOYDrB9JZoCOz58A=; b=EidB61m8NZfOukW2ps5z8Q1SEnG49ko43hzmwDLo3N1O0OaG9Dz+N+qLZtbStCi3Qn erqaD8jVY5r6SgNDbX17ghe/3MIIJzHaHvufUwzvUM6U8UPdHcR2/V51blvpt1/Xme/c tvJVVbuSAFvgf2qAYUBfHHacWIOq6YDRhUWh0WEIBjDW9M1m67rnU8hJKrlFCrgYeIhh XZscZZnLpMIMBLmBCEISeeOdgjYMlzAA3CTrg3P1dMdSu4k1oX58Fk6z6F5cryoEvqet CL1ZcqwV4s9ww+2Evcl+Pyhw7uMoDIJNQkhbYHbn0BF9jGibYUyDXYX4pHH+6JhGZza4 xyNQ== X-Gm-Message-State: APjAAAVsrA0nFEabxlqtr9K6MvZBrTDZlQH9vGaI4oluZNbMrLGqEAUo dOeipT4ccNiPDDqZkxrjQ+3DPeSy X-Google-Smtp-Source: APXvYqyd/GbCqgMi8MNFRFhGqic7HYZsWirAn0CVkyec/+CuMtbKKIIdINY4ym0qMEKeOKJMIGpByw== X-Received: by 2002:a17:906:3102:: with SMTP id 2mr62792294ejx.304.1560209726811; Mon, 10 Jun 2019 16:35:26 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id j19sm3200349edr.69.2019.06.10.16.35.26 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 10 Jun 2019 16:35:26 -0700 (PDT) Date: Mon, 10 Jun 2019 16:35:26 -0700 (PDT) X-Google-Original-Date: Mon, 10 Jun 2019 23:35:16 GMT Message-Id: <1776e36f1952cb6ed7fea73e90167134285c331b.1560209720.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Subject: [PATCH 07/11] multi-pack-index: prepare 'repack' subcommand Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sbeller@google.com, peff@peff.net, jrnieder@gmail.com, avarab@gmail.com, jonathantanmy@google.com, Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Derrick Stolee In an environment where the multi-pack-index is useful, it is due to many pack-files and an inability to repack the object store into a single pack-file. However, it is likely that many of these pack-files are rather small, and could be repacked into a slightly larger pack-file without too much effort. It may also be important to ensure the object store is highly available and the repack operation does not interrupt concurrent git commands. Introduce a 'repack' subcommand to 'git multi-pack-index' that takes a '--batch-size' option. The subcommand will inspect the multi-pack-index for referenced pack-files whose size is smaller than the batch size, until collecting a list of pack-files whose sizes sum to larger than the batch size. Then, a new pack-file will be created containing the objects from those pack-files that are referenced by the multi-pack-index. The resulting pack is likely to actually be smaller than the batch size due to compression and the fact that there may be objects in the pack- files that have duplicate copies in other pack-files. The current change introduces the command-line arguments, and we add a test that ensures we parse these options properly. Since we specify a small batch size, we will guarantee that future implementations do not change the list of pack-files. In addition, we hard-code the modified times of the packs in the pack directory to ensure the list of packs sorted by modified time matches the order if sorted by size (ascending). This will be important in a future test. Signed-off-by: Derrick Stolee --- Documentation/git-multi-pack-index.txt | 17 +++++++++++++++++ builtin/multi-pack-index.c | 12 ++++++++++-- midx.c | 5 +++++ midx.h | 1 + t/t5319-multi-pack-index.sh | 20 +++++++++++++++++++- 5 files changed, 52 insertions(+), 3 deletions(-) diff --git a/Documentation/git-multi-pack-index.txt b/Documentation/git-multi-pack-index.txt index 6186c4c936..233b2b7862 100644 --- a/Documentation/git-multi-pack-index.txt +++ b/Documentation/git-multi-pack-index.txt @@ -36,6 +36,23 @@ expire:: have no objects referenced by the MIDX. Rewrite the MIDX file afterward to remove all references to these pack-files. +repack:: + Create a new pack-file containing objects in small pack-files + referenced by the multi-pack-index. If the size given by the + `--batch-size=` argument is zero, then create a pack + containing all objects referenced by the multi-pack-index. For + a non-zero batch size, Select the pack-files by examining packs + from oldest-to-newest, computing the "expected size" by counting + the number of objects in the pack referenced by the + multi-pack-index, then divide by the total number of objects in + the pack and multiply by the pack size. We select packs with + expected size below the batch size until the set of packs have + total expected size at least the batch size. If the total size + does not reach the batch size, then do nothing. If a new pack- + file is created, rewrite the multi-pack-index to reference the + new pack-file. A later run of 'git multi-pack-index expire' will + delete the pack-files that were part of this batch. + EXAMPLES -------- diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c index ad10d40512..b1ea1a6aa1 100644 --- a/builtin/multi-pack-index.c +++ b/builtin/multi-pack-index.c @@ -6,12 +6,13 @@ #include "trace2.h" static char const * const builtin_multi_pack_index_usage[] = { - N_("git multi-pack-index [--object-dir=] (write|verify|expire)"), + N_("git multi-pack-index [--object-dir=] (write|verify|expire|repack --batch-size=)"), NULL }; static struct opts_multi_pack_index { const char *object_dir; + unsigned long batch_size; } opts; int cmd_multi_pack_index(int argc, const char **argv, @@ -20,6 +21,8 @@ int cmd_multi_pack_index(int argc, const char **argv, static struct option builtin_multi_pack_index_options[] = { OPT_FILENAME(0, "object-dir", &opts.object_dir, N_("object directory containing set of packfile and pack-index pairs")), + OPT_MAGNITUDE(0, "batch-size", &opts.batch_size, + N_("during repack, collect pack-files of smaller size into a batch that is larger than this size")), OPT_END(), }; @@ -43,6 +46,11 @@ int cmd_multi_pack_index(int argc, const char **argv, trace2_cmd_mode(argv[0]); + if (!strcmp(argv[0], "repack")) + return midx_repack(the_repository, opts.object_dir, (size_t)opts.batch_size); + if (opts.batch_size) + die(_("--batch-size option is only for 'repack' subcommand")); + if (!strcmp(argv[0], "write")) return write_midx_file(opts.object_dir); if (!strcmp(argv[0], "verify")) @@ -50,5 +58,5 @@ int cmd_multi_pack_index(int argc, const char **argv, if (!strcmp(argv[0], "expire")) return expire_midx_packs(the_repository, opts.object_dir); - die(_("unrecognized verb: %s"), argv[0]); + die(_("unrecognized subcommand: %s"), argv[0]); } diff --git a/midx.c b/midx.c index 9b0b4c1520..fbed8a8adb 100644 --- a/midx.c +++ b/midx.c @@ -1226,3 +1226,8 @@ int expire_midx_packs(struct repository *r, const char *object_dir) string_list_clear(&packs_to_drop, 0); return result; } + +int midx_repack(struct repository *r, const char *object_dir, size_t batch_size) +{ + return 0; +} diff --git a/midx.h b/midx.h index 505f1431b7..f0ae656b5d 100644 --- a/midx.h +++ b/midx.h @@ -51,6 +51,7 @@ int write_midx_file(const char *object_dir); void clear_midx_file(struct repository *r); int verify_midx_file(struct repository *r, const char *object_dir); int expire_midx_packs(struct repository *r, const char *object_dir); +int midx_repack(struct repository *r, const char *object_dir, size_t batch_size); void close_midx(struct multi_pack_index *m); diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index 12570fe7ac..133d5b7068 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -398,7 +398,8 @@ test_expect_success 'setup expire tests' ' git pack-objects --revs .git/objects/pack/pack-E <<-EOF && refs/heads/E EOF - git multi-pack-index write + git multi-pack-index write && + cp -r .git/objects/pack .git/objects/pack-backup ) ' @@ -432,4 +433,21 @@ test_expect_success 'expire removes unreferenced packs' ' ) ' +test_expect_success 'repack with minimum size does not alter existing packs' ' + ( + cd dup && + rm -rf .git/objects/pack && + mv .git/objects/pack-backup .git/objects/pack && + touch -m -t 201901010000 .git/objects/pack/pack-D* && + touch -m -t 201901010001 .git/objects/pack/pack-C* && + touch -m -t 201901010002 .git/objects/pack/pack-B* && + touch -m -t 201901010003 .git/objects/pack/pack-A* && + ls .git/objects/pack >expect && + MINSIZE=$(ls -l .git/objects/pack/*pack | awk "{print \$5;}" | sort -n | head -n 1) && + git multi-pack-index repack --batch-size=$MINSIZE && + ls .git/objects/pack >actual && + test_cmp expect actual + ) +' + test_done From patchwork Mon Jun 10 23:35:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee via GitGitGadget X-Patchwork-Id: 10985631 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E2079924 for ; Mon, 10 Jun 2019 23:35:34 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D0C112845C for ; Mon, 10 Jun 2019 23:35:34 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C4D6228623; Mon, 10 Jun 2019 23:35:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 245E62845C for ; Mon, 10 Jun 2019 23:35:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2390664AbfFJXfb (ORCPT ); Mon, 10 Jun 2019 19:35:31 -0400 Received: from mail-ed1-f66.google.com ([209.85.208.66]:37895 "EHLO mail-ed1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2390542AbfFJXf3 (ORCPT ); Mon, 10 Jun 2019 19:35:29 -0400 Received: by mail-ed1-f66.google.com with SMTP id g13so16973020edu.5 for ; Mon, 10 Jun 2019 16:35:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=ycyscfLXOrkbmY2w0Hk1HLxjyvVhXJ4MRJmYgcdjxIM=; b=ayRRZNAG/fdDcwjdq3emE2O75SdpxUuXEU5Qhjo7mUuQCdCd592GDTm1MJALP2eJGV GtRcZv3Dg7xQC1K8cFPjJSUqw3amAdIyhO5LUKlupBtUbiME7va/TFObaLV7RuVH9PtD 4ogfCE9P0s8v6bGX93HF0JsGu/8+OszemaVGxaMtHUXOvYz/TpA2u5VouQqHZYp6xq+C qkxm4PWZVT4AB5DOhWp3rnuL/evPRy4UUJbSSzx3Ffk1RVzXNBWf16MnpPraEoFu+U9i HPB5UmtKB2QOEwmgzV+9/sNQbGE8JFjq1CJRVU9hLGFKL+EUQWxQxpe+2VTc18eGYi35 ZfvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=ycyscfLXOrkbmY2w0Hk1HLxjyvVhXJ4MRJmYgcdjxIM=; b=s9oO8XIGIR94+4rLy1O2/MB9ugb5l79p6NY3w35R3qeU8y3fQknrdPSJ8fOIt2XAV6 FY5ugjVFSz7abq0d3dXhdcjOODn2G6+kMwir4428ppXTzVpbpYG/jxYsfeRQ+DQG/UvC fdAwFwEdgFO/L6jVYA5rC1siV5JYb2ymprEVGhgc7fs5QPSa01xUZex70b0cRWfX9Qob z+A4AcBRqXPehDJVDl3G8W5Mjy+JiCGwHusEtXbRqvWYqAMtnjs28KIN5gvCEluU+YBw nF+Y7szFOoLS7vxyKen5LjQNLdQ2uU7vRHz9M0QlOx4MxXh0BglBU5ChLp1gPT2JMXFe KPPQ== X-Gm-Message-State: APjAAAWV0ylKUIm9GjwEplsXc7P9QJHzv9pb2kncLzD31pLGiQyox1KE EYHuBC37kHhW5FxqBOcW9J5YZH2e X-Google-Smtp-Source: APXvYqznYiIWRqbPw/+ITJvRg5zeH925bMg928+Neqt7yd01qW2Eq7Zz8rP7YYRuzhTJ+zc4E71zWg== X-Received: by 2002:a05:6402:1557:: with SMTP id p23mr1051125edx.207.1560209727557; Mon, 10 Jun 2019 16:35:27 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id j10sm1991102ejk.49.2019.06.10.16.35.26 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 10 Jun 2019 16:35:27 -0700 (PDT) Date: Mon, 10 Jun 2019 16:35:27 -0700 (PDT) X-Google-Original-Date: Mon, 10 Jun 2019 23:35:17 GMT Message-Id: In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Subject: [PATCH 08/11] midx: implement midx_repack() Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sbeller@google.com, peff@peff.net, jrnieder@gmail.com, avarab@gmail.com, jonathantanmy@google.com, Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Derrick Stolee To repack with a non-zero batch-size, first sort all pack-files by their modified time. Second, walk those pack-files from oldest to newest, compute their expected size, and add the packs to a list if they are smaller than the given batch-size. Stop when the total expected size is at least the batch size. If the batch size is zero, select all packs in the multi-pack-index. Finally, collect the objects from the multi-pack-index that are in the selected packs and send them to 'git pack-objects'. Write a new multi-pack-index that includes the new pack. Using a batch size of zero is very similar to a standard 'git repack' command, except that we do not delete the old packs and instead rely on the new multi-pack-index to prevent new processes from reading the old packs. This does not disrupt other Git processes that are currently reading the old packs based on the old multi-pack-index. While first designing a 'git multi-pack-index repack' operation, I started by collecting the batches based on the actual size of the objects instead of the size of the pack-files. This allows repacking a large pack-file that has very few referencd objects. However, this came at a significant cost of parsing pack-files instead of simply reading the multi-pack-index and getting the file information for the pack-files. The "expected size" version provides similar behavior, but could skip a pack-file if the average object size is much larger than the actual size of the referenced objects, or can create a large pack if the actual size of the referenced objects is larger than the expected size. Signed-off-by: Derrick Stolee --- midx.c | 151 +++++++++++++++++++++++++++++++++++- t/t5319-multi-pack-index.sh | 28 +++++++ 2 files changed, 178 insertions(+), 1 deletion(-) diff --git a/midx.c b/midx.c index fbed8a8adb..d649644420 100644 --- a/midx.c +++ b/midx.c @@ -9,6 +9,7 @@ #include "midx.h" #include "progress.h" #include "trace2.h" +#include "run-command.h" #define MIDX_SIGNATURE 0x4d494458 /* "MIDX" */ #define MIDX_VERSION 1 @@ -1227,7 +1228,155 @@ int expire_midx_packs(struct repository *r, const char *object_dir) return result; } -int midx_repack(struct repository *r, const char *object_dir, size_t batch_size) +struct repack_info { + timestamp_t mtime; + uint32_t referenced_objects; + uint32_t pack_int_id; +}; + +static int compare_by_mtime(const void *a_, const void *b_) { + const struct repack_info *a, *b; + + a = (const struct repack_info *)a_; + b = (const struct repack_info *)b_; + + if (a->mtime < b->mtime) + return -1; + if (a->mtime > b->mtime) + return 1; + return 0; +} + +static int fill_included_packs_all(struct multi_pack_index *m, + unsigned char *include_pack) +{ + uint32_t i; + + for (i = 0; i < m->num_packs; i++) + include_pack[i] = 1; + + return m->num_packs < 2; +} + +static int fill_included_packs_batch(struct repository *r, + struct multi_pack_index *m, + unsigned char *include_pack, + size_t batch_size) +{ + uint32_t i, packs_to_repack; + size_t total_size; + struct repack_info *pack_info = xcalloc(m->num_packs, sizeof(struct repack_info)); + + for (i = 0; i < m->num_packs; i++) { + pack_info[i].pack_int_id = i; + + if (prepare_midx_pack(r, m, i)) + continue; + + pack_info[i].mtime = m->packs[i]->mtime; + } + + for (i = 0; batch_size && i < m->num_objects; i++) { + uint32_t pack_int_id = nth_midxed_pack_int_id(m, i); + pack_info[pack_int_id].referenced_objects++; + } + + QSORT(pack_info, m->num_packs, compare_by_mtime); + + total_size = 0; + packs_to_repack = 0; + for (i = 0; total_size < batch_size && i < m->num_packs; i++) { + int pack_int_id = pack_info[i].pack_int_id; + struct packed_git *p = m->packs[pack_int_id]; + size_t expected_size; + + if (!p) + continue; + if (open_pack_index(p) || !p->num_objects) + continue; + + expected_size = (size_t)(p->pack_size + * pack_info[i].referenced_objects); + expected_size /= p->num_objects; + + if (expected_size >= batch_size) + continue; + + packs_to_repack++; + total_size += expected_size; + include_pack[pack_int_id] = 1; + } + + free(pack_info); + + if (total_size < batch_size || packs_to_repack < 2) + return 1; + return 0; } + +int midx_repack(struct repository *r, const char *object_dir, size_t batch_size) +{ + int result = 0; + uint32_t i; + unsigned char *include_pack; + struct child_process cmd = CHILD_PROCESS_INIT; + struct strbuf base_name = STRBUF_INIT; + struct multi_pack_index *m = load_multi_pack_index(object_dir, 1); + + if (!m) + return 0; + + include_pack = xcalloc(m->num_packs, sizeof(unsigned char)); + + if (batch_size) { + if (fill_included_packs_batch(r, m, include_pack, batch_size)) + goto cleanup; + } else if (fill_included_packs_all(m, include_pack)) + goto cleanup; + + argv_array_push(&cmd.args, "pack-objects"); + + strbuf_addstr(&base_name, object_dir); + strbuf_addstr(&base_name, "/pack/pack"); + argv_array_push(&cmd.args, base_name.buf); + strbuf_release(&base_name); + + cmd.git_cmd = 1; + cmd.in = cmd.out = -1; + + if (start_command(&cmd)) { + error(_("could not start pack-objects")); + result = 1; + goto cleanup; + } + + for (i = 0; i < m->num_objects; i++) { + struct object_id oid; + uint32_t pack_int_id = nth_midxed_pack_int_id(m, i); + + if (!include_pack[pack_int_id]) + continue; + + nth_midxed_object_oid(&oid, m, i); + xwrite(cmd.in, oid_to_hex(&oid), the_hash_algo->hexsz); + xwrite(cmd.in, "\n", 1); + } + close(cmd.in); + + if (finish_command(&cmd)) { + error(_("could not finish pack-objects")); + result = 1; + goto cleanup; + } + + result = write_midx_internal(object_dir, m, NULL); + m = NULL; + +cleanup: + if (m) + close_midx(m); + free(include_pack); + return result; +} diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index 133d5b7068..6e47e5d0b2 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -450,4 +450,32 @@ test_expect_success 'repack with minimum size does not alter existing packs' ' ) ' +test_expect_success 'repack creates a new pack' ' + ( + cd dup && + ls .git/objects/pack/*idx >idx-list && + test_line_count = 5 idx-list && + THIRD_SMALLEST_SIZE=$(ls -l .git/objects/pack/*pack | awk "{print \$5;}" | sort -n | head -n 3 | tail -n 1) && + BATCH_SIZE=$(($THIRD_SMALLEST_SIZE + 1)) && + git multi-pack-index repack --batch-size=$BATCH_SIZE && + ls .git/objects/pack/*idx >idx-list && + test_line_count = 6 idx-list && + test-tool read-midx .git/objects | grep idx >midx-list && + test_line_count = 6 midx-list + ) +' + +test_expect_success 'expire removes repacked packs' ' + ( + cd dup && + ls -al .git/objects/pack/*pack && + ls -S .git/objects/pack/*pack | head -n 4 >expect && + git multi-pack-index expire && + ls -S .git/objects/pack/*pack >actual && + test_cmp expect actual && + test-tool read-midx .git/objects | grep idx >midx-list && + test_line_count = 4 midx-list + ) +' + test_done From patchwork Mon Jun 10 23:35:27 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee via GitGitGadget X-Patchwork-Id: 10985637 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 07C2714BB for ; Mon, 10 Jun 2019 23:35:38 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EC1C72845C for ; Mon, 10 Jun 2019 23:35:37 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E09E728623; Mon, 10 Jun 2019 23:35:37 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8646E2845C for ; Mon, 10 Jun 2019 23:35:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2390685AbfFJXfg (ORCPT ); Mon, 10 Jun 2019 19:35:36 -0400 Received: from mail-ed1-f68.google.com ([209.85.208.68]:34075 "EHLO mail-ed1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2390650AbfFJXf3 (ORCPT ); Mon, 10 Jun 2019 19:35:29 -0400 Received: by mail-ed1-f68.google.com with SMTP id c26so16994953edt.1 for ; Mon, 10 Jun 2019 16:35:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=KnA30+e3AAJbIXMVx/anXWrN4GRIvEfiqITdtg6IWO4=; b=GFbFdbWRLde+6+UUxxE4feUQRREiW+lTA5HZTLBHgZM0IDUFb3kVl9zhl85rkHeaI/ fgXDnTUY5IdHGr6LrDfvQ9yFeD8MYk2fOcZ2e/oXKL7FpfPnhQpPb1e/+t/Oc/O5btzv phZcC3aVVceatTCidZWbBMiO3asfS/k17b4bqkGcmOjtUruDzpUhewGIoiwUVStfINY/ ZuDMyEmCRh/9in0QDo/ZBUlKPr/VOf7SxeaJW7WjHM4eQT2bt2zMLX7jFS2+VCtYzYxL +ZZCZcf1gak1F7yLu1zkJAkjYQe9/Zr72dcYcD3NB69I0Eo6KkcfgAmWesxV9TbMLVEG SZVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=KnA30+e3AAJbIXMVx/anXWrN4GRIvEfiqITdtg6IWO4=; b=gDs3/P//ApCMpO5afbnhjAYqJDzmcHUIOd78ORHdrQJ4YogQ+KH0CHsPd/hqhDVXuj JKaK48hNWaXfuzVazErN+mnt5LPlhXhgnmdwCatRlrK7f/BsGWwMdP+qmibPzoJiHhp3 7RR8JuBotVRuzq+FLirsRmgp6ZxUVa4ElTPEJCcb3l3X6vc4NeQaRaJzIJes0shh4YdY lEm8HwPYePl/OVt79aS6ZDmgvc/6CZt+Y+nequZQU3lxX57CUDt0NR+8pGL+aDGJFa6W HbD+btAdMdo4gGpvz8hVVAA0GVdq/KZBaWZeKGUmv6cNBKGmApkzNLfhxa/gylOVH/Fn LZ+A== X-Gm-Message-State: APjAAAW0XAIAm350XilFqXJp7l+s1xE4tP3CY48Kidu5d7cerGC4HGGp cPPYcFyppk4fXthlTdC2UcnE1uhu X-Google-Smtp-Source: APXvYqy5X468jnX1nhe+BUqGKdPOarfUBOUBDvOgyTrVmQUDBF8LCcXPaOd30/CM7APKxQ0DuSuVZA== X-Received: by 2002:a17:906:79d6:: with SMTP id m22mr50522033ejo.187.1560209728222; Mon, 10 Jun 2019 16:35:28 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n5sm3152165edt.65.2019.06.10.16.35.27 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 10 Jun 2019 16:35:27 -0700 (PDT) Date: Mon, 10 Jun 2019 16:35:27 -0700 (PDT) X-Google-Original-Date: Mon, 10 Jun 2019 23:35:18 GMT Message-Id: <80c7d2e5810cf851af11154dbe1e5abaa852a224.1560209720.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Subject: [PATCH 09/11] multi-pack-index: test expire while adding packs Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sbeller@google.com, peff@peff.net, jrnieder@gmail.com, avarab@gmail.com, jonathantanmy@google.com, Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Derrick Stolee During development of the multi-pack-index expire subcommand, a version went out that improperly computed the pack order if a new pack was introduced while other packs were being removed. Part of the subtlety of the bug involved the new pack being placed before other packs that already existed in the multi-pack-index. Add a test to t5319-multi-pack-index.sh that catches this issue. The test adds new packs that cause another pack to be expired, and creates new packs that are lexicographically sorted before and after the existing packs. Signed-off-by: Derrick Stolee --- t/t5319-multi-pack-index.sh | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index 6e47e5d0b2..8e04ce2821 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -478,4 +478,36 @@ test_expect_success 'expire removes repacked packs' ' ) ' +test_expect_success 'expire works when adding new packs' ' + ( + cd dup && + git pack-objects --revs .git/objects/pack/pack-combined <<-EOF && + refs/heads/A + ^refs/heads/B + EOF + git pack-objects --revs .git/objects/pack/pack-combined <<-EOF && + refs/heads/B + ^refs/heads/C + EOF + git pack-objects --revs .git/objects/pack/pack-combined <<-EOF && + refs/heads/C + ^refs/heads/D + EOF + git multi-pack-index write && + git pack-objects --revs .git/objects/pack/a-pack <<-EOF && + refs/heads/D + ^refs/heads/E + EOF + git multi-pack-index write && + git pack-objects --revs .git/objects/pack/z-pack <<-EOF && + refs/heads/E + EOF + git multi-pack-index expire && + ls .git/objects/pack/ | grep idx >expect && + test-tool read-midx .git/objects | grep idx >actual && + test_cmp expect actual && + git multi-pack-index verify + ) +' + test_done From patchwork Mon Jun 10 23:35:28 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee via GitGitGadget X-Patchwork-Id: 10985627 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4BA7E14BB for ; Mon, 10 Jun 2019 23:35:33 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3D8412845C for ; Mon, 10 Jun 2019 23:35:33 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3148D28689; Mon, 10 Jun 2019 23:35:33 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CFBB72845C for ; Mon, 10 Jun 2019 23:35:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2390670AbfFJXfb (ORCPT ); Mon, 10 Jun 2019 19:35:31 -0400 Received: from mail-ed1-f54.google.com ([209.85.208.54]:43930 "EHLO mail-ed1-f54.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2390655AbfFJXfa (ORCPT ); Mon, 10 Jun 2019 19:35:30 -0400 Received: by mail-ed1-f54.google.com with SMTP id w33so16961605edb.10 for ; Mon, 10 Jun 2019 16:35:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=U8L2yvKWeo5aSk9NjQTKZ/6SYIVvl2Xzu82SuSIL7jE=; b=Y4g32xDqKYJTK1+b7gmYcyfzwC93Y992evgMnyhZVrk92Ad8IuWA/XDhrlPqi1ikge 5lbEVgPoRwAy7yVrfO3T8m4SYxSfdrPiAOW6zxOP5vHXfnIZbqddJoNSRFqa97fQhyLr AybvgPJ/JGuLVPx2CGO8Whr4jMkd9olmnfz/epwRndIm6I6HWtefpN/kqZ+TsNTjc7HY ZB8uEa5UZcd7ezfRSkomD4kLjVz72wgySu6nxNmQK66XMsuwnpeSWasP0E3wNTaRIUFS v90CPK8fEsc9d9EdhbxO/zga8+UrJ5uYYAqct5qJgAjsfXy6hPeBW+KItEN9A7QzUCiw 9JJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=U8L2yvKWeo5aSk9NjQTKZ/6SYIVvl2Xzu82SuSIL7jE=; b=RwbiQh+bPVaolF64LkzRFfTIwHCqRs6loehblorlnjWDLTUcYMF6Ax2pZ8qApxtE0B QgPO7ADXgfgRBGezHHP61er/TWB21c9LNaueepqvF8zlIWbQ8hWVBN5XQzJ522xwwP7c Xi1EVveVz8Q9IVEXQBNEF8d2P59KE2tR/ePoydyyvOhULlt4twyZpcSbaNhYdqXjpCqo 5N+oumAfTPlx67VNDUbpbxMXX0ra3CAfF3WhnJk/M0g9AbLxrsh5R5htk1JfLiU1+VzD P9Aw7syYqglO6Do40CTIZS8qzdyS4Jh3mffkyU3o+h2MyfaAmz503cUA43JxswbVGCOm ceOQ== X-Gm-Message-State: APjAAAVFZJrIoMx18/u2CsUf2rulJD+iG8HwAQA9SH9+Aqdh/7UCB0E+ 7unxDFt5nqQzidmrlOiD8BvI/nAD X-Google-Smtp-Source: APXvYqxyU2xrFWznv7bcV0MNakgJotgoqrSVqWdhteYk4NwJU5OooZ7O1IliYerzZNlRa2gQgup7OQ== X-Received: by 2002:a17:906:604e:: with SMTP id p14mr17270806ejj.192.1560209728962; Mon, 10 Jun 2019 16:35:28 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m3sm3184487edi.33.2019.06.10.16.35.28 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 10 Jun 2019 16:35:28 -0700 (PDT) Date: Mon, 10 Jun 2019 16:35:28 -0700 (PDT) X-Google-Original-Date: Mon, 10 Jun 2019 23:35:19 GMT Message-Id: <8e243939efc9131dd4f765e7931e4ad55f7e6b29.1560209720.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Subject: [PATCH 10/11] midx: add test that 'expire' respects .keep files Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sbeller@google.com, peff@peff.net, jrnieder@gmail.com, avarab@gmail.com, jonathantanmy@google.com, Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Derrick Stolee The 'git multi-pack-index expire' subcommand may delete packs that are not needed from the perspective of the multi-pack-index. If a pack has a .keep file, then we should not delete that pack. Add a test that ensures we preserve a pack that would otherwise be expired. First, create a new pack that contains every object in the repo, then add it to the multi-pack-index. Then create a .keep file for a pack starting with "a-pack" that was added in the previous test. Finally, expire and verify that the pack remains and the other packs were expired. Signed-off-by: Derrick Stolee --- t/t5319-multi-pack-index.sh | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index 8e04ce2821..c288901401 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -510,4 +510,22 @@ test_expect_success 'expire works when adding new packs' ' ) ' +test_expect_success 'expire respects .keep files' ' + ( + cd dup && + git pack-objects --revs .git/objects/pack/pack-all <<-EOF && + refs/heads/A + EOF + git multi-pack-index write && + PACKA=$(ls .git/objects/pack/a-pack*\.pack | sed s/\.pack\$//) && + touch $PACKA.keep && + git multi-pack-index expire && + ls -S .git/objects/pack/a-pack* | grep $PACKA >a-pack-files && + test_line_count = 3 a-pack-files && + test-tool read-midx .git/objects | grep idx >midx-list && + test_line_count = 2 midx-list + ) +' + + test_done From patchwork Mon Jun 10 23:35:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee via GitGitGadget X-Patchwork-Id: 10985629 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 84EC318A6 for ; Mon, 10 Jun 2019 23:35:33 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 74B492845C for ; Mon, 10 Jun 2019 23:35:33 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 694DC28623; Mon, 10 Jun 2019 23:35:33 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 179A6285CB for ; Mon, 10 Jun 2019 23:35:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2390674AbfFJXfc (ORCPT ); Mon, 10 Jun 2019 19:35:32 -0400 Received: from mail-ed1-f68.google.com ([209.85.208.68]:32903 "EHLO mail-ed1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2390662AbfFJXfb (ORCPT ); Mon, 10 Jun 2019 19:35:31 -0400 Received: by mail-ed1-f68.google.com with SMTP id h9so17007711edr.0 for ; Mon, 10 Jun 2019 16:35:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:message-id:in-reply-to:references:from:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=W5w8rS4sCM3WmqZuB0Oa69IMUfsWAR/eAtBqwbM0CqM=; b=SkC3Q4nNbtCW3gKcemc2Cr6FzRld74y/VyGz6pwW3FVdMCWkLYfaD4gZ8YpCugtYHW qRvR9trTvaTAcs6PiC01n5q/3l3sLyZtab5rWTx4UxscFWH4S+VMphk/H6d2zDP7Kpt8 5w3AiYa1q6+nnYji3u2v5PbdGcRE7oRYzXDogBgKh1+aUbemSgjeQzTaUOIwcbR7Moe5 shLxoNQk6weHuW3PJWP3AgNqb8hj+0imY1NkYUvZcrddtRB+/rkUVY4JFSyexCnRfmRW OpvSm7oBfgONZc4Zv8IbnhcpXEzkRlXl+gIQsH3SjJv6Wu87xolXXpPz3gH0LnH/pKHz 4//g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:in-reply-to:references:from :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=W5w8rS4sCM3WmqZuB0Oa69IMUfsWAR/eAtBqwbM0CqM=; b=AbSzkGjTRplCBOFp0FKMCFmNT1MwRBJ3RuNZiAkPzNDgTUtMKz4vQFxpI6et55q54B mZp6elTWJEpbezT4nTCHReYhciginHqpZzIPmXk3TRGKhP2HLejUgsQmWIyJxmaj0C8e mf47MuZGC430bCORfqcmgn8CELLQ7tKrhskQfYPcEQUt1Ig95YZUWYgZz9qVwdNMruI/ 228A4aivQ4IBUA2I+6w+/jUYP+A8m+NSsW1Hd1v98Mcnj0u/qJy638G0XoYRV83XKFpd 6ycvv7SuXOys36+U1Bbrzxcgu6BPKpIa2a/OP00vtN7bq0ij3xdL7VM7PHUxsXTZtdfA MV/Q== X-Gm-Message-State: APjAAAXPxQOmFnzJplT7CcxyIp8ERKq5i7YYsLt8rJvlbSS5QrE6RygC tTX1TD5hpFLxj3GxzL3vBg3Wugwd X-Google-Smtp-Source: APXvYqyt1u63B4dBRomx3qy0S3pRPaiYtjH3kKX+VuSeg9ezFgnhtjbHH65yqpzauib4bDWPysq35A== X-Received: by 2002:a17:906:1e8a:: with SMTP id e10mr23422759ejj.261.1560209729659; Mon, 10 Jun 2019 16:35:29 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id j10sm1991113ejk.49.2019.06.10.16.35.29 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 10 Jun 2019 16:35:29 -0700 (PDT) Date: Mon, 10 Jun 2019 16:35:29 -0700 (PDT) X-Google-Original-Date: Mon, 10 Jun 2019 23:35:20 GMT Message-Id: <3ed388f0a8295c504da5d0a39887f246ec532aaa.1560209720.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Subject: [PATCH 11/11] t5319-multi-pack-index.sh: test batch size zero Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sbeller@google.com, peff@peff.net, jrnieder@gmail.com, avarab@gmail.com, jonathantanmy@google.com, Junio C Hamano , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Derrick Stolee The 'git multi-pack-index repack' command can take a batch size of zero, which creates a new pack-file containing all objects in the multi-pack-index. The first 'repack' command will create one new pack-file, and an 'expire' command after that will delete the old pack-files, as they no longer contain any referenced objects in the multi-pack-index. We must remove the .keep file that was added in the previous test in order to expire that pack-file. Also test that a 'repack' will do nothing if there is only one pack-file. Signed-off-by: Derrick Stolee --- t/t5319-multi-pack-index.sh | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh index c288901401..79bfaeafa9 100755 --- a/t/t5319-multi-pack-index.sh +++ b/t/t5319-multi-pack-index.sh @@ -527,5 +527,24 @@ test_expect_success 'expire respects .keep files' ' ) ' +test_expect_success 'repack --batch-size=0 repacks everything' ' + ( + cd dup && + rm .git/objects/pack/*.keep && + ls .git/objects/pack/*idx >idx-list && + test_line_count = 2 idx-list && + git multi-pack-index repack --batch-size=0 && + ls .git/objects/pack/*idx >idx-list && + test_line_count = 3 idx-list && + test-tool read-midx .git/objects | grep idx >midx-list && + test_line_count = 3 midx-list && + git multi-pack-index expire && + ls -al .git/objects/pack/*idx >idx-list && + test_line_count = 1 idx-list && + git multi-pack-index repack --batch-size=0 && + ls -al .git/objects/pack/*idx >new-idx-list && + test_cmp idx-list new-idx-list + ) +' test_done