[v15,0/6] unpack-objects: support streaming blobs to disk

Message

Han Xin June 11, 2022, 2:44 a.m. UTC

This series makes "unpack-objects" capable of streaming large objects
to disk.

As 6/6 shows streaming e.g. a 100MB blob now uses ~5MB of memory
instead of ~105MB. This streaming method is slower if you've got
memory to handle the blobs in-core, but if you don't it allows you to
unpack objects at all, as you might otherwise OOM.

Changes since v14:

* Remove "object-file.c: do fsync() and close() before post-write die()"
  as it's not necessary anymore. It was first introduced in v10 and was
  no longer in the utility function end_loose_object_common() since v12.
  We can see the discussion[1].

* Minor grammar/comment etc. fixes throughout.

1. https://lore.kernel.org/git/[e-mail]/

Han Xin (4):
  unpack-objects: low memory footprint for get_data() in dry_run mode
  object-file.c: refactor write_loose_object() to several steps
  object-file.c: add "stream_loose_object()" to handle large object
  unpack-objects: use stream_loose_object() to unpack large objects

Ævar Arnfjörð Bjarmason (2):
  object-file.c: factor out deflate part of write_loose_object()
  core doc: modernize core.bigFileThreshold documentation

 Documentation/config/core.txt   |  33 +++--
 builtin/unpack-objects.c        | 106 +++++++++++++--
 object-file.c                   | 233 ++++++++++++++++++++++++++++----
 object-store.h                  |   8 ++
 t/t5351-unpack-large-objects.sh |  76 +++++++++++
 5 files changed, 405 insertions(+), 51 deletions(-)
 create mode 100755 t/t5351-unpack-large-objects.sh

Range-diff against v14:
1:  bf600a2fa8 ! 1:  9a776f717d unpack-objects: low memory footprint for get_data() in dry_run mode
    @@ Commit message
         d9545c7f465 (fast-import: implement unpack limit, 2016-04-25).
     
         Suggested-by: Jiang Xin <[e-mail]>
    -    Signed-off-by: Han Xin <[e-mail]>
    +    Signed-off-by: Han Xin <[e-mail]>
         Signed-off-by: Ævar Arnfjörð Bjarmason <[e-mail]>
     
      ## builtin/unpack-objects.c ##
2:  a327f484f7 < -:  ---------- object-file.c: do fsync() and close() before post-write die()
3:  9bc8002282 ! 2:  a1e090d338 object-file.c: refactor write_loose_object() to several steps
    @@ Commit message
     
         Helped-by: Ævar Arnfjörð Bjarmason <[e-mail]>
         Helped-by: Jiang Xin <[e-mail]>
    -    Signed-off-by: Han Xin <[e-mail]>
    +    Signed-off-by: Han Xin <[e-mail]>
         Signed-off-by: Ævar Arnfjörð Bjarmason <[e-mail]>
     
      ## object-file.c ##
    @@ object-file.c: static int write_loose_object(const struct object_id *oid, char *
     -	ret = git_deflate_end_gently(&stream);
     +	ret = end_loose_object_common(&c, &stream, &parano_oid);
      	if (ret != Z_OK)
    --		die(_("deflateEnd on object %s failed (%d)"), oid_to_hex(oid),
    --		    ret);
    + 		die(_("deflateEnd on object %s failed (%d)"), oid_to_hex(oid),
    + 		    ret);
     -	the_hash_algo->final_oid_fn(&parano_oid, &c);
    -+		die(_("deflateEnd on object %s failed (%d)"), oid_to_hex(oid), ret);
    - 	close_loose_object(fd, tmp_file.buf);
    - 
      	if (!oideq(oid, &parano_oid))
    + 		die(_("confused by unstable object source data for %s"),
    + 		    oid_to_hex(oid));
4:  7c73815f18 = 3:  0ddf912d47 object-file.c: factor out deflate part of write_loose_object()
5:  28a9588f9c ! 4:  f9e51d3c68 object-file.c: add "stream_loose_object()" to handle large object
    @@ Commit message
         Helped-by: René Scharfe <[e-mail]>
         Helped-by: Ævar Arnfjörð Bjarmason <[e-mail]>
         Helped-by: Jiang Xin <[e-mail]>
    -    Signed-off-by: Han Xin <[e-mail]>
    +    Signed-off-by: Han Xin <[e-mail]>
         Signed-off-by: Ævar Arnfjörð Bjarmason <[e-mail]>
     
      ## object-file.c ##
6:  dea5c4172b ! 5:  61ae1c1632 core doc: modernize core.bigFileThreshold documentation
    @@ Documentation/config/core.txt: You probably do not need to adjust this value.
     -Common unit suffixes of 'k', 'm', or 'g' are supported.
     +* Stored deflated in packfiles, without attempting delta compression.
     ++
    -+The default limit is primarily set with this use-case in mind. With it
    ++The default limit is primarily set with this use-case in mind. With it,
     +most projects will have their source code and other text files delta
     +compressed, but not larger binary media files.
     ++
     +Storing large files without delta compression avoids excessive memory
     +usage, at the slight expense of increased disk usage.
     ++
    -+* Will be treated as if though they were labeled "binary" (see
    ++* Will be treated as if they were labeled "binary" (see
     +  linkgit:gitattributes[5]). e.g. linkgit:git-log[1] and
    -+  linkgit:git-diff[1] will not diffs for files above this limit.
    ++  linkgit:git-diff[1] will not compute diffs for files above this limit.
     ++
    -+* Will be generally be streamed when written, which avoids excessive
    ++* Will generally be streamed when written, which avoids excessive
     +memory usage, at the cost of some fixed overhead. Commands that make
     +use of this include linkgit:git-archive[1],
     +linkgit:git-fast-import[1], linkgit:git-index-pack[1] and
7:  d236230a4c ! 6:  5a4782d746 unpack-objects: use stream_loose_object() to unpack large objects
    @@ Commit message
         Helped-by: Ævar Arnfjörð Bjarmason <[e-mail]>
         Helped-by: Derrick Stolee <[e-mail]>
         Helped-by: Jiang Xin <[e-mail]>
    -    Signed-off-by: Han Xin <[e-mail]>
    +    Signed-off-by: Han Xin <[e-mail]>
         Signed-off-by: Ævar Arnfjörð Bjarmason <[e-mail]>
     
      ## Documentation/config/core.txt ##
     @@ Documentation/config/core.txt: usage, at the slight expense of increased disk usage.
    - * Will be generally be streamed when written, which avoids excessive
    + * Will generally be streamed when written, which avoids excessive
      memory usage, at the cost of some fixed overhead. Commands that make
      use of this include linkgit:git-archive[1],
     -linkgit:git-fast-import[1], linkgit:git-index-pack[1] and