[v3,0/3] reftable: graceful concurrent writes

Message

Patrick Steinhardt Sept. 18, 2024, 9:59 a.m. UTC

Hi,

this is the third version of my patch series that implements support for
graceful concurrent writes with the reftable backend.

There is only a single change compared to v2, namely that we handle `0`
and `-1` for the lock timeout config. With `0` we fail immediately, with
`-1` we lock indefinitely. This matches semantics of loose and packed
ref locking.

Thanks!

Patrick

Patrick Steinhardt (3):
  refs/reftable: introduce "reftable.lockTimeout"
  reftable/stack: allow locking of outdated stacks
  refs/reftable: reload locked stack when preparing transaction

 Documentation/config/reftable.txt |  8 ++++
 refs/reftable-backend.c           | 13 +++++-
 reftable/reftable-stack.h         | 13 +++++-
 reftable/reftable-writer.h        | 11 +++++
 reftable/stack.c                  | 38 ++++++++++++------
 t/t0610-reftable-basics.sh        | 58 ++++++++++++++++++++++++++
 t/unit-tests/t-reftable-stack.c   | 67 ++++++++++++++++++++++++++++++-
 7 files changed, 189 insertions(+), 19 deletions(-)

Range-diff against v2:
1:  700a35df125 ! 1:  77cffd3b1eb refs/reftable: introduce "reftable.lockTimeout"
    @@ Documentation/config/reftable.txt: reftable.geometricFactor::
     +	Whenever the reftable backend appends a new table to the stack, it has
     +	to lock the central "tables.list" file before updating it. This config
     +	controls how long the process will wait to acquire the lock in case
    -+	another process has already acquired it. Default is 100 (i.e., retry
    -+	for 100ms).
    ++	another process has already acquired it. Value 0 means not to retry at
    ++	all; -1 means to try indefinitely. Default is 100 (i.e., retry for
    ++	100ms).
     
      ## refs/reftable-backend.c ##
     @@ refs/reftable-backend.c: static int reftable_be_config(const char *var, const char *value,
    @@ refs/reftable-backend.c: static int reftable_be_config(const char *var, const ch
      			die("reftable geometric factor cannot exceed %u", (unsigned)UINT8_MAX);
      		opts->auto_compaction_factor = factor;
     +	} else if (!strcmp(var, "reftable.locktimeout")) {
    -+		unsigned long lock_timeout = git_config_ulong(var, value, ctx->kvi);
    ++		int64_t lock_timeout = git_config_int64(var, value, ctx->kvi);
    ++		if (lock_timeout > LONG_MAX)
    ++			die("reftable lock timeout cannot exceed %"PRIdMAX, (intmax_t)LONG_MAX);
    ++		if (lock_timeout < 0 && lock_timeout != -1)
    ++			die("reftable lock timeout does not support negative values other than -1");
     +		opts->lock_timeout_ms = lock_timeout;
      	}
      
    @@ reftable/reftable-writer.h: struct reftable_write_options {
     +	 * Note that this does not apply to locking individual tables, as these
     +	 * should only ever be locked when already holding the "tables.list"
     +	 * lock.
    ++	 *
    ++	 * Passing 0 will fail immediately when the file is locked, passing a
    ++	 * negative value will cause us to block indefinitely.
     +	 */
    -+	unsigned lock_timeout_ms;
    ++	long lock_timeout_ms;
      };
      
      /* reftable_block_stats holds statistics for a single block type */
2:  f4be0966e17 = 2:  6130565498e reftable/stack: allow locking of outdated stacks
3:  111b497ef17 = 3:  25d4e513a36 refs/reftable: reload locked stack when preparing transaction

Comments

James Liu Sept. 18, 2024, 11:23 p.m. UTC | #1

On Wed Sep 18, 2024 at 7:59 PM AEST, Patrick Steinhardt wrote:
> Hi,
>
> this is the third version of my patch series that implements support for
> graceful concurrent writes with the reftable backend.
>
> There is only a single change compared to v2, namely that we handle `0`
> and `-1` for the lock timeout config. With `0` we fail immediately, with
> `-1` we lock indefinitely. This matches semantics of loose and packed
> ref locking.

I guess you meant "we retry indefinitely" here :D

>
> Thanks!
>
> Patrick

Thanks Patrick, no more comments from me!

Cheers,
James

Patrick Steinhardt Sept. 24, 2024, 5:33 a.m. UTC | #2

On Thu, Sep 19, 2024 at 09:23:21AM +1000, James Liu wrote:
> On Wed Sep 18, 2024 at 7:59 PM AEST, Patrick Steinhardt wrote:
> > Hi,
> >
> > this is the third version of my patch series that implements support for
> > graceful concurrent writes with the reftable backend.
> >
> > There is only a single change compared to v2, namely that we handle `0`
> > and `-1` for the lock timeout config. With `0` we fail immediately, with
> > `-1` we lock indefinitely. This matches semantics of loose and packed
> > ref locking.
> 
> I guess you meant "we retry indefinitely" here :D

Oops, yes :)

> >
> > Thanks!
> >
> > Patrick
> 
> Thanks Patrick, no more comments from me!

Thanks for your review!

Patrick