[v3,0/3] midx: apply gitconfig to midx repack

Message

Philippe Blain via GitGitGadget May 9, 2020, 2:24 p.m. UTC

Midx repack has largely been used in Microsoft Scalar on the client side to
optimize the repository multiple packs state. However when I tried to apply
this onto the server-side, I realized that there are certain features that
were lacking compare to git repack. Most of these features are highly
desirable on the server-side to create the most optimized pack possible.

One of the example is delta_base_offset, comparing an midx repack
with/without delta_base_offset, we can observe significant size differences.

> du objects/pack/*pack
14536   objects/pack/pack-08a017b424534c88191addda1aa5dd6f24bf7a29.pack
9435280 objects/pack/pack-8829c53ad1dca02e7311f8e5b404962ab242e8f1.pack

Latest 2.26.2 (without delta_base_offset)
> git multi-pack-index write
> git multi-pack-index repack
> git multi-pack-index expire
> du objects/pack/*pack
9446096 objects/pack/pack-366c75e2c2f987b9836d3bf0bf5e4a54b6975036.pack

With delta_base_offset
> git version
git version 2.26.2.672.g232c24e857.dirty
> git multi-pack-index write
> git multi-pack-index repack
> git multi-pack-index expire
> du objects/pack/*pack
9152512 objects/pack/pack-3bc8c1ec496ab95d26875f8367ff6807081e9e7d.pack

Note that repack.writeBitmaps configuration is ignored, as the pack bitmap
facility is useful only with a single packfile.

Derrick Stolee's following patch will address repack.packKeptObjects 
support.

Derrick Stolee (1):
  multi-pack-index: respect repack.packKeptObjects=false

Son Luong Ngoc (2):
  midx: teach "git multi-pack-index repack" honor "git repack"
    configurations
  Ensured t5319 follows arith expansion guideline

 Documentation/git-multi-pack-index.txt |  3 ++
 midx.c                                 | 36 ++++++++++++---
 t/t5319-multi-pack-index.sh            | 62 ++++++++++++++++++--------
 3 files changed, 78 insertions(+), 23 deletions(-)


base-commit: b994622632154fc3b17fb40a38819ad954a5fb88
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-626%2Fsluongng%2Fsluongngoc%2Fmidx-config-v3
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-626/sluongng/sluongngoc/midx-config-v3
Pull-Request: https://github.com/gitgitgadget/git/pull/626

Range-diff vs v2:

 1:  21c648cc486 ! 1:  a925307d4c5 midx: apply gitconfig to midx repack
     @@ Metadata
      Author: Son Luong Ngoc <[e-mail]>
      
       ## Commit message ##
     -    midx: apply gitconfig to midx repack
     +    midx: teach "git multi-pack-index repack" honor "git repack" configurations
      
     -    Multi-Pack-Index repack is an incremental, repack solutions
     -    that allows user to consolidate multiple packfiles in a non-disruptive
     -    way. However the new packfile could be created without some of the
     -    capabilities of a packfile that is created by calling `git repack`.
     +    Previously, when the "repack" subcommand of "git multi-pack-index" command
     +    creates new packfile(s), it does not call the "git repack" command but
     +    instead directly calls the "git pack-objects" command, and the
     +    configuration variables meant for the "git repack" command, like
     +    "repack.usedaeltabaseoffset", are ignored.
      
     -    This is because with `git repack`, there are configuration that would
     -    enable different flags to be passed down to `git pack-objects` plumbing.
     +    This patch ensured "git multi-pack-index" checks the configuration
     +    variables used by "git repack" and passes the corresponding options to
     +    the underlying "git pack-objects" command.
      
     -    In this patch, I applies those flags into `git multi-pack-index repack`
     -    so that it respect the `repack.*` config series.
     -
     -    Note:
     -    - `repack.packKeptObjects` will be addressed by Derrick Stolee in
     -    the following patch
     -    - `repack.writeBitmaps` when `--batch-size=0` was NOT adopted here as it
     -    requires `--all` to be passed onto `git pack-objects`, which is very
     -    slow. I think it would be nice to have this in a future patch.
     +    Note that `repack.writeBitmaps` configuration is ignored, as the
     +    pack bitmap facility is useful only with a single packfile.
      
          Signed-off-by: Son Luong Ngoc <[e-mail]>
      
     @@ midx.c: int midx_repack(struct repository *r, const char *object_dir, size_t bat
       	struct strbuf base_name = STRBUF_INIT;
       	struct multi_pack_index *m = load_multi_pack_index(object_dir, 1);
      +	int delta_base_offset = 1;
     -+	int use_delta_islands;
     ++	int use_delta_islands = 0;
       
       	if (!m)
       		return 0;
 2:  3d7b334f5c6 = 2:  988697dd512 multi-pack-index: respect repack.packKeptObjects=false
 -:  ----------- > 3:  efeb3d7d132 Ensured t5319 follows arith expansion guideline