[v2,00/15] bloom: changed-path Bloom filters v2

Message

Jonathan Tan Aug. 30, 2023, 4:43 p.m. UTC

Here's an updated patch set containing updates to the tests, making them
run on non-root commits to ensure that we exercise the Bloom filters.
Thanks to SZEDER Gábor for spotting this.

Jonathan Tan (4):
  gitformat-commit-graph: describe version 2 of BDAT
  t4216: test changed path filters with high bit paths
  repo-settings: introduce commitgraph.changedPathsVersion
  commit-graph: new filter ver. that fixes murmur3

Taylor Blau (11):
  t/helper/test-read-graph.c: extract `dump_graph_info()`
  bloom.h: make `load_bloom_filter_from_graph()` public
  t/helper/test-read-graph: implement `bloom-filters` mode
  bloom: annotate filters with hash version
  bloom: prepare to discard incompatible Bloom filters
  t/t4216-log-bloom.sh: harden `test_bloom_filters_not_used()`
  commit-graph.c: unconditionally load Bloom filters
  commit-graph: drop unnecessary `graph_read_bloom_data_context`
  object.h: fix mis-aligned flag bits table
  commit-graph: reuse existing Bloom filters where possible
  bloom: introduce `deinit_bloom_filters()`

 Documentation/config/commitgraph.txt     |  26 ++-
 Documentation/gitformat-commit-graph.txt |   9 +-
 bloom.c                                  | 208 +++++++++++++++++++++--
 bloom.h                                  |  38 ++++-
 commit-graph.c                           |  36 +++-
 object.h                                 |   3 +-
 oss-fuzz/fuzz-commit-graph.c             |   2 +-
 repo-settings.c                          |   6 +-
 repository.h                             |   2 +-
 t/helper/test-bloom.c                    |   9 +-
 t/helper/test-read-graph.c               |  65 +++++--
 t/t0095-bloom.sh                         |   8 +
 t/t4216-log-bloom.sh                     | 202 +++++++++++++++++++++-
 13 files changed, 564 insertions(+), 50 deletions(-)

Range-diff against v1:
 1:  dcfc987741 =  1:  dcfc987741 gitformat-commit-graph: describe version 2 of BDAT
 2:  0c56f2a9e9 =  2:  0c56f2a9e9 t/helper/test-read-graph.c: extract `dump_graph_info()`
 3:  8405b845e5 =  3:  8405b845e5 bloom.h: make `load_bloom_filter_from_graph()` public
 4:  3a25f90c15 =  4:  3a25f90c15 t/helper/test-read-graph: implement `bloom-filters` mode
 5:  e300d338e1 !  5:  7858010665 t4216: test changed path filters with high bit paths
    @@ t/t4216-log-bloom.sh: test_expect_success 'Bloom generation backfills empty comm
     +	true
     +'
     +
    ++test_expect_success 'setup make another commit' '
    ++	# "git log" does not use Bloom filters for root commits - see how, in
    ++	# revision.c, rev_compare_tree() (the only code path that eventually calls
    ++	# get_bloom_filter()) is only called by try_to_simplify_commit() when the commit
    ++	# has one parent. Therefore, make another commit so that we perform the tests on
    ++	# a non-root commit.
    ++	test_commit -C highbit1 anotherc1 "another$CENT"
    ++'
    ++
     +test_expect_success 'version 1 changed-path used when version 1 requested' '
     +	(
     +		cd highbit1 &&
    -+		test_bloom_filters_used "-- $CENT"
    ++		test_bloom_filters_used "-- another$CENT"
     +	)
     +'
     +
 6:  6bc665e1d3 =  6:  94ad289dbb repo-settings: introduce commitgraph.changedPathsVersion
 7:  7ef3b2bbbd !  7:  44d3163125 commit-graph: new filter ver. that fixes murmur3
    @@ t/t4216-log-bloom.sh: test_expect_success 'version 1 changed-path used when vers
     +	(
     +		cd highbit1 &&
     +		git config --add commitgraph.changedPathsVersion 2 &&
    -+		test_bloom_filters_not_used "-- $CENT"
    ++		test_bloom_filters_not_used "-- another$CENT"
     +	)
     +'
     +
    @@ t/t4216-log-bloom.sh: test_expect_success 'version 1 changed-path used when vers
     +	(
     +		cd highbit1 &&
     +		git config --add commitgraph.changedPathsVersion -1 &&
    -+		test_bloom_filters_used "-- $CENT"
    ++		test_bloom_filters_used "-- another$CENT"
     +	)
     +'
     +
    @@ t/t4216-log-bloom.sh: test_expect_success 'version 1 changed-path used when vers
     +	)
     +'
     +
    ++test_expect_success 'setup make another commit' '
    ++	# "git log" does not use Bloom filters for root commits - see how, in
    ++	# revision.c, rev_compare_tree() (the only code path that eventually calls
    ++	# get_bloom_filter()) is only called by try_to_simplify_commit() when the commit
    ++	# has one parent. Therefore, make another commit so that we perform the tests on
    ++	# a non-root commit.
    ++	test_commit -C highbit2 anotherc2 "another$CENT"
    ++'
    ++
     +test_expect_success 'version 2 changed-path used when version 2 requested' '
     +	(
     +		cd highbit2 &&
    -+		test_bloom_filters_used "-- $CENT"
    ++		test_bloom_filters_used "-- another$CENT"
     +	)
     +'
     +
    @@ t/t4216-log-bloom.sh: test_expect_success 'version 1 changed-path used when vers
     +	(
     +		cd highbit2 &&
     +		git config --add commitgraph.changedPathsVersion 1 &&
    -+		test_bloom_filters_not_used "-- $CENT"
    ++		test_bloom_filters_not_used "-- another$CENT"
     +	)
     +'
     +
    @@ t/t4216-log-bloom.sh: test_expect_success 'version 1 changed-path used when vers
     +	(
     +		cd highbit2 &&
     +		git config --add commitgraph.changedPathsVersion -1 &&
    -+		test_bloom_filters_used "-- $CENT"
    ++		test_bloom_filters_used "-- another$CENT"
     +	)
     +'
     +
 8:  302caee39d =  8:  a5bf23a7d3 bloom: annotate filters with hash version
 9:  d2b0726266 =  9:  3de6cd8460 bloom: prepare to discard incompatible Bloom filters
10:  9a9992220f = 10:  ef04389a0e t/t4216-log-bloom.sh: harden `test_bloom_filters_not_used()`
11:  607945ab05 = 11:  e0c1c1ccec commit-graph.c: unconditionally load Bloom filters
12:  e397d83895 = 12:  4d57f51854 commit-graph: drop unnecessary `graph_read_bloom_data_context`
13:  bd3ad6b6c0 = 13:  a3b4e7ef59 object.h: fix mis-aligned flag bits table
14:  2996f0fdb6 = 14:  05357f9533 commit-graph: reuse existing Bloom filters where possible
15:  a3b5b22db0 = 15:  58a1d90e6d bloom: introduce `deinit_bloom_filters()`

Comments

Junio C Hamano Aug. 30, 2023, 7:38 p.m. UTC | #1

Jonathan Tan <[e-mail]> writes:

> Here's an updated patch set containing updates to the tests, making them
> run on non-root commits to ensure that we exercise the Bloom filters.

Thanks for a quick turnaround.

> Thanks to SZEDER Gábor for spotting this.

With this everybody involved are happy, I presume.  Thank you,
everybody, who's worked on polishing and assembling these patches.