[v4,0/5] rev-list: introduce NUL-delimited output mode

Message

Justin Tobler March 19, 2025, 6:34 p.m. UTC

When walking objects, git-rev-list(1) prints each object entry on a
separate line in the form:

        <oid> LF

Some options, such as `--objects`, may print additional information
about the object on the same line:

        <oid> SP [<path>] LF

In this mode, if the object path contains a newline it is truncated at
the newline.

The `--boundary` option also modifies output by prefixing boundary
objects with `-`:

        -<oid> LF

When the `--missing={print,print-info}` option is provided, information
about any missing objects encountered during the object walk are also
printed in the form:

        ?<oid> [SP <token>=<value>]... LF

where values containing LF or SP are printed in a token specific fashion
so that the resulting encoded value does not contain either of these two
problematic bytes. For example, missing object paths are quoted in the C
style when they contain LF or SP.

To make machine parsing easier, this series introduces a NUL-delimited
output mode for git-rev-list(1) via a `-z` option. In this mode, the
output format for object records is unified such that each object and
its accompanying metadata is formatted without relying on object
metadata order. This format follows the existing `<token>=<value>` used
by the `--missing` option to represent object metadata in the form:

        <oid> NUL [<token>=<value> NUL]...

        # Examples
        <oid> LF                       -> <oid> NUL
        <oid> SP <path> LF             -> <oid> NUL path=<path> NUL
        -<oid> LF                      -> <oid> NUL boundary=yes NUL
        ?<oid> [SP <token>=<value>]... -> <oid> NUL missing=yes NUL [<token>=<value> NUL]...

Note that token value info is printed as-is without any special encoding
or truncation. Prefixes such as '-' and '?' are dropped in favor using a
token/value pair to signal the same information.

For now this series only adds support for use with the `--objects`,
`--boundary` and `--missing` output options. Usage of `-z` with other
options is rejected, so it can potentially be added in the future.

This series is structured as follows:

        - Patches 1 and 2 do some minor preparatory refactors.

        - Patch 3 adds the `-z` option to git-rev-list(1) to print
          objects in a NUL-delimited fashion.

        - Patch 4 teaches the `--boundary` option how to print info in a
          NUL-delimited fashino using the unified output format.

        - Patch 5 teaches the `--missing` option how to print info in a
          NUL-delimited fashion using the unified output format.

Changes since V3:

        - The -z option now only makes output NUL-delimited. Input
          parsed on stdin via the `--stdin` option remains unchanged.
          This is done to remain more consistent with other log family
          commands. Support for more explicit options to control
          NUL-delimited input/ouput behavior may be added in a future
          series via `--NUL-delimited-{input,output}` options.

        - Changed some variable names in tests that were a little
          confusing.

Changes since V2:

        - In patch 4, the documentation for the -z option now points out
          the `--stdin` behavior change earlier.

        - Minor code style and documentation changes in patch 6.

Changes since V1:

        - Use unified output format with `<token>=<value>` pairs for
          all object metadata.

        - Add support for the `--boundary` option in NUL-delimited mode.

        - Add support for NUL-delimited stdin argument parsing in
          NUL-delimited mode.

        - Instead of using two NUL bytes to delimit between object
          records, a single NUL byte is used. Now that object metadata
          is always in the form `<token>=<value>`, we know a new object
          record starts when there is an OID entry which will not
          contain '='.

Thanks for taking a look,
-Justin

Justin Tobler (5):
  rev-list: inline `show_object_with_name()` in `show_object()`
  rev-list: refactor early option parsing
  rev-list: support delimiting objects with NUL bytes
  rev-list: support NUL-delimited --boundary option
  rev-list: support NUL-delimited --missing option

 Documentation/rev-list-options.adoc | 24 ++++++++
 builtin/rev-list.c                  | 93 +++++++++++++++++++++--------
 revision.c                          |  8 ---
 revision.h                          |  2 -
 t/t6000-rev-list-misc.sh            | 51 ++++++++++++++++
 t/t6022-rev-list-missing.sh         | 31 ++++++++++
 6 files changed, 175 insertions(+), 34 deletions(-)

Range-diff against v3:
1:  d2eded3ac7 = 1:  d2eded3ac7 rev-list: inline `show_object_with_name()` in `show_object()`
2:  03cd08c859 = 2:  03cd08c859 rev-list: refactor early option parsing
3:  803a49933a < -:  ---------- revision: support NUL-delimited --stdin mode
4:  8eb7669089 ! 3:  f6ee01571d rev-list: support delimiting objects with NUL bytes
    @@ Commit message
         from `--objects` is appended to the record as a token/value pair
         `path=<path>` as-is without any truncation.
     
    -    In this mode, revision and pathspec arguments provided on stdin with the
    -    `--stdin` option are also separated by a NUL byte instead of being
    -    newline delimited.
    -
    -    For now, the `--objects` and `--stdin` flag are the only options that
    -    can be used in combination with `-z`. In a subsequent commit,
    -    NUL-delimited support for other options is added. Other options that do
    -    not make sense with be used in combination with `-z` are rejected.
    +    For now, the `--objects` flag is the only options that can be used in
    +    combination with `-z`. In a subsequent commit, NUL-delimited support for
    +    other options is added. Other options that do not make sense when used
    +    in combination with `-z` are rejected.
     
         Signed-off-by: Justin Tobler <[e-mail]>
     
    @@ Documentation/rev-list-options.adoc: ifdef::git-rev-list[]
     +
     +-z::
     +	Instead of being newline-delimited, each outputted object and its
    -+	accompanying metadata is delimited using NUL bytes. In this mode, when
    -+	the `--stdin` option is provided, revision and pathspec arguments on
    -+	stdin are also delimited using a NUL byte. Output is printed in the
    -+	following form:
    ++	accompanying metadata is delimited using NUL bytes. Output is printed
    ++	in the following form:
     ++
     +-----------------------------------------------------------------------
     +<OID> NUL [<token>=<value> NUL]...
    @@ builtin/rev-list.c: int cmd_rev_list(int argc,
      		} else if (skip_prefix(arg, "--missing=", &arg)) {
      			parse_missing_action_value(arg);
     +		} else if (!strcmp(arg, "-z")) {
    -+			s_r_opt.nul_delim_stdin = 1;
     +			line_term = '\0';
     +			info_term = '\0';
      		}
    @@ t/t6000-rev-list-misc.sh: test_expect_success 'rev-list --unpacked' '
     +	test_commit -C repo 1 &&
     +	test_commit -C repo 2 &&
     +
    -+	oid1=$(git -C repo rev-parse HEAD) &&
    -+	oid2=$(git -C repo rev-parse HEAD~) &&
    ++	oid1=$(git -C repo rev-parse HEAD~) &&
    ++	oid2=$(git -C repo rev-parse HEAD) &&
     +
    -+	printf "%s\0%s\0" "$oid1" "$oid2" >expect &&
    ++	printf "%s\0%s\0" "$oid2" "$oid1" >expect &&
     +	git -C repo rev-list -z HEAD >actual &&
     +
     +	test_cmp expect actual
    @@ t/t6000-rev-list-misc.sh: test_expect_success 'rev-list --unpacked' '
     +'
     +
      test_done
    -
    - ## t/t6017-rev-list-stdin.sh ##
    -@@ t/t6017-rev-list-stdin.sh: test_expect_success '--not via stdin does not influence revisions from command l
    - 	test_cmp expect actual
    - '
    - 
    -+test_expect_success 'NUL-delimited stdin' '
    -+	printf "%s\0%s\0%s\0" "HEAD" "--" "file-1" > input &&
    -+
    -+	git rev-list -z --objects HEAD -- file-1 >expect &&
    -+	git rev-list -z --objects --stdin <input >actual &&
    -+
    -+	test_cmp expect actual
    -+'
    -+
    - test_done
5:  591a2c7dac ! 4:  ccf6bd8d35 rev-list: support NUL-delimited --boundary option
    @@ t/t6000-rev-list-misc.sh: test_expect_success 'rev-list -z --objects' '
     +	test_commit -C repo 1 &&
     +	test_commit -C repo 2 &&
     +
    -+	oid1=$(git -C repo rev-parse HEAD) &&
    -+	oid2=$(git -C repo rev-parse HEAD~) &&
    ++	oid1=$(git -C repo rev-parse HEAD~) &&
    ++	oid2=$(git -C repo rev-parse HEAD) &&
     +
    -+	printf "%s\0%s\0boundary=yes\0" "$oid1" "$oid2" >expect &&
    ++	printf "%s\0%s\0boundary=yes\0" "$oid2" "$oid1" >expect &&
     +	git -C repo rev-list -z --boundary HEAD~.. >actual &&
     +
     +	test_cmp expect actual
6:  669b3b5d9f = 5:  b1bd245155 rev-list: support NUL-delimited --missing option

base-commit: 87a0bdbf0f72b7561f3cd50636eee33dcb7dbcc3