[0/8] refs/reftable: reuse iterators when reading refs

Message

Patrick Steinhardt Nov. 4, 2024, 3:11 p.m. UTC

Hi,

this patch series refactors the reftable backend to reuse reftable
iterators when reading random references. This removes the overhead of
having to recreate the iterator on every read and thus leads to better
performance and less allocation churn. It also gives us the ability to
further optimize reads by optimizing re-seeking iterators in the future.

Overall this leads to a 7% speedup when creating many refs in a
transaction, which performs many random reads. But this change also
positively impacts other usecases.

Thanks!

Patrick

Patrick Steinhardt (8):
  refs/reftable: encapsulate reftable stack
  refs/reftable: handle reloading stacks in the reftable backend
  refs/reftable: read references via `struct reftable_backend`
  refs/reftable: refactor reading symbolic refs to use reftable backend
  refs/reftable: refactor reflog expiry to use reftable backend
  reftable/stack: add mechanism to notify callers on reload
  reftable/merged: drain priority queue on reseek
  refs/reftable: reuse iterators when reading refs

 refs/reftable-backend.c          | 356 ++++++++++++++++++-------------
 reftable/merged.c                |   2 +
 reftable/reftable-stack.h        |   3 +
 reftable/reftable-writer.h       |   9 +
 reftable/stack.c                 |   9 +
 t/unit-tests/t-reftable-merged.c |  73 +++++++
 6 files changed, 308 insertions(+), 144 deletions(-)

Comments

Junio C Hamano Nov. 5, 2024, 4:49 a.m. UTC | #1

Patrick Steinhardt <[e-mail]> writes:

> this patch series refactors the reftable backend to reuse reftable
> iterators when reading random references. This removes the overhead of
> having to recreate the iterator on every read and thus leads to better
> performance and less allocation churn. It also gives us the ability to
> further optimize reads by optimizing re-seeking iterators in the future.
>
> Overall this leads to a 7% speedup when creating many refs in a
> transaction, which performs many random reads. But this change also
> positively impacts other usecases.

When merged to 'seen', this seems to break a handful of tests.  I am
reasonably sure that it is interaction with other topics, as these
tests passes in this topic itself without other topics merged.

Thanks.


Test Summary Report
-------------------
t0611-reftable-httpd.sh             (Wstat: 256 (exited 1) Tests: 1 Failed: 1)
  Failed test:  1
  Non-zero exit status: 1
t0613-reftable-write-options.sh     (Wstat: 256 (exited 1) Tests: 11 Failed: 9)
  Failed tests:  1-3, 5-8, 10-11
  Non-zero exit status: 1
t7424-submodule-mixed-ref-formats.sh (Wstat: 256 (exited 1) Tests: 7 Failed: 6)
  Failed tests:  2-7
  Non-zero exit status: 1
t1460-refs-migrate.sh               (Wstat: 256 (exited 1) Tests: 30 Failed: 16)
  Failed tests:  9-15, 21-27, 29-30
  Non-zero exit status: 1
t0610-reftable-basics.sh            (Wstat: 256 (exited 1) Tests: 89 Failed: 53)
  Failed tests:  8, 10, 12, 14, 16, 18, 20, 22, 24-27, 38-42
                44, 47-52, 55-56, 58, 62-71, 73, 75-89
  Non-zero exit status: 1