| Message ID | 20220527081915.2024853-2-hch@lst.de (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | [01/10] btrfs: add a helpers for read repair testing | expand |
On Fri, May 27, 2022 at 10:19:06AM +0200, Christoph Hellwig wrote: > Add a few helpers to consolidate code for btrfs read repair testing: > > - _btrfs_get_first_logical() gets the btrfs logical address for the > first extent in a file > - _btrfs_get_device_path and _btrfs_get_physical use the > btrfs-map-logical tool to find the device path and physical address > for btrfs logical address for a specific mirror > - _btrfs_direct_read_on_mirror and _btrfs_buffered_read_on_mirror > read the data from a specific mirror > > These will be used to consolidate the read repair tests and avoid > duplication for new tests. > > Signed-off-by: Christoph Hellwig <hch@lst.de> > Reviewed-by: Qu Wenruo <wqu@suse.com> > --- > common/btrfs | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++ > common/config | 1 + > 2 files changed, 76 insertions(+) > > diff --git a/common/btrfs b/common/btrfs > index ac597ca4..b69feeee 100644 > --- a/common/btrfs > +++ b/common/btrfs > @@ -505,3 +505,78 @@ _btrfs_metadump() > $BTRFS_IMAGE_PROG "$device" "$dumpfile" > [ -n "$DUMP_COMPRESSOR" ] && $DUMP_COMPRESSOR -f "$dumpfile" &> /dev/null > } > + > +# Return the btrfs logical address for the first block in a file > +_btrfs_get_first_logical() > +{ > + local file=$1 > + _require_command "$FILEFRAG_PROG" filefrag > + > + ${FILEFRAG_PROG} -v $SCRATCH_MNT/foobar >> $seqres.full ^^ $file You can send a single fixed patch for this one Thanks, Zorro > + ${FILEFRAG_PROG} -v $file | _filter_filefrag | cut -d '#' -f 1 > +} > + > +# Find the device path for a btrfs logical offset > +_btrfs_get_device_path() > +{ > + local logical=$1 > + local stripe=$2 > + > + _require_command "$BTRFS_MAP_LOGICAL_PROG" btrfs-map-logical > + > + $BTRFS_MAP_LOGICAL_PROG -l $logical $SCRATCH_DEV | \ > + $AWK_PROG "(\$1 ~ /mirror/ && \$2 ~ /$stripe/) { print \$8 }" > +} > + > + > +# Find the device physical sector for a btrfs logical offset > +_btrfs_get_physical() > +{ > + local logical=$1 > + local stripe=$2 > + > + _require_command "$BTRFS_MAP_LOGICAL_PROG" btrfs-map-logical > + > + $BTRFS_MAP_LOGICAL_PROG -b -l $logical $SCRATCH_DEV >> $seqres.full 2>&1 > + $BTRFS_MAP_LOGICAL_PROG -l $logical $SCRATCH_DEV | \ > + $AWK_PROG "(\$1 ~ /mirror/ && \$2 ~ /$stripe/) { print \$6 }" > +} > + > +# Read from a specific stripe to test read recovery that corrupted a specific > +# stripe. Btrfs uses the PID to select the mirror, so keep reading until the > +# xfs_io process that performed the read was executed with a PID that ends up > +# on the intended mirror. > +_btrfs_direct_read_on_mirror() > +{ > + local mirror=$1 > + local nr_mirrors=$2 > + local file=$3 > + local offset=$4 > + local size=$5 > + > + while [[ -z $( (( BASHPID % nr_mirrors == mirror )) && > + exec $XFS_IO_PROG -d \ > + -c "pread -b $size $offset $size" $file) ]]; do > + : > + done > +} > + > +# Read from a specific stripe to test read recovery that corrupted a specific > +# stripe. Btrfs uses the PID to select the mirror, so keep reading until the > +# xfs_io process that performed the read was executed with a PID that ends up > +# on the intended mirror. > +_btrfs_buffered_read_on_mirror() > +{ > + local mirror=$1 > + local nr_mirrors=$2 > + local file=$3 > + local offset=$4 > + local size=$5 > + > + echo 3 > /proc/sys/vm/drop_caches > + while [[ -z $( (( BASHPID % nr_mirrors == mirror )) && > + exec $XFS_IO_PROG \ > + -c "pread -b $size $offset $size" $file) ]]; do > + : > + done > +} > diff --git a/common/config b/common/config > index c6428f90..df20afc1 100644 > --- a/common/config > +++ b/common/config > @@ -228,6 +228,7 @@ export E2IMAGE_PROG="$(type -P e2image)" > export BLKZONE_PROG="$(type -P blkzone)" > export GZIP_PROG="$(type -P gzip)" > export BTRFS_IMAGE_PROG="$(type -P btrfs-image)" > +export BTRFS_MAP_LOGICAL_PROG=$(type -P btrfs-map-logical) > > # use 'udevadm settle' or 'udevsettle' to wait for lv to be settled. > # newer systems have udevadm command but older systems like RHEL5 don't. > -- > 2.30.2 >
On Fri, May 27, 2022 at 10:54:45PM +0800, Zorro Lang wrote:
> You can send a single fixed patch for this one
I could do that, but I also need to fix up the comments in 266 and 267
to say 64k instead of 4k.
On Fri, May 27, 2022 at 05:03:06PM +0200, Christoph Hellwig wrote: > On Fri, May 27, 2022 at 10:54:45PM +0800, Zorro Lang wrote: > > You can send a single fixed patch for this one > > I could do that, but I also need to fix up the comments in 266 and 267 > to say 64k instead of 4k. Hi Christoph, I can help to do this change when I merge this patchset, if you don't have more changes need to do. Actually I'm trying to wait this patchset for this weekend fstests release, due to it's almost done. I need to do a regression test today before pushing these patches on Sunday (if no exception:) Thanks, Zorro >
On Sat, May 28, 2022 at 11:34:14AM +0800, Zorro Lang wrote: > I can help to do this change when I merge this patchset, if you don't have > more changes need to do. Actually I'm trying to wait this patchset for this > weekend fstests release, due to it's almost done. I need to do a regression > test today before pushing these patches on Sunday (if no exception:) Ok. Here is the updated version of this first patch: --- From 3de26208af33e9aa83deac77a9d34dec3ccb67c9 Mon Sep 17 00:00:00 2001 From: Christoph Hellwig <hch@lst.de> Date: Tue, 24 May 2022 09:06:38 +0200 Subject: btrfs: add a helpers for read repair testing Add a few helpers to consolidate code for btrfs read repair testing: - _btrfs_get_first_logical() gets the btrfs logical address for the first extent in a file - _btrfs_get_device_path and _btrfs_get_physical use the btrfs-map-logical tool to find the device path and physical address for btrfs logical address for a specific mirror - _btrfs_direct_read_on_mirror and _btrfs_buffered_read_on_mirror read the data from a specific mirror These will be used to consolidate the read repair tests and avoid duplication for new tests. Signed-off-by: Christoph Hellwig <hch@lst.de> Reviewed-by: Qu Wenruo <wqu@suse.com> --- common/btrfs | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++ common/config | 1 + 2 files changed, 76 insertions(+) diff --git a/common/btrfs b/common/btrfs index ac597ca4..c7058918 100644 --- a/common/btrfs +++ b/common/btrfs @@ -505,3 +505,78 @@ _btrfs_metadump() $BTRFS_IMAGE_PROG "$device" "$dumpfile" [ -n "$DUMP_COMPRESSOR" ] && $DUMP_COMPRESSOR -f "$dumpfile" &> /dev/null } + +# Return the btrfs logical address for the first block in a file +_btrfs_get_first_logical() +{ + local file=$1 + _require_command "$FILEFRAG_PROG" filefrag + + ${FILEFRAG_PROG} -v $file >> $seqres.full + ${FILEFRAG_PROG} -v $file | _filter_filefrag | cut -d '#' -f 1 +} + +# Find the device path for a btrfs logical offset +_btrfs_get_device_path() +{ + local logical=$1 + local stripe=$2 + + _require_command "$BTRFS_MAP_LOGICAL_PROG" btrfs-map-logical + + $BTRFS_MAP_LOGICAL_PROG -l $logical $SCRATCH_DEV | \ + $AWK_PROG "(\$1 ~ /mirror/ && \$2 ~ /$stripe/) { print \$8 }" +} + + +# Find the device physical sector for a btrfs logical offset +_btrfs_get_physical() +{ + local logical=$1 + local stripe=$2 + + _require_command "$BTRFS_MAP_LOGICAL_PROG" btrfs-map-logical + + $BTRFS_MAP_LOGICAL_PROG -b -l $logical $SCRATCH_DEV >> $seqres.full 2>&1 + $BTRFS_MAP_LOGICAL_PROG -l $logical $SCRATCH_DEV | \ + $AWK_PROG "(\$1 ~ /mirror/ && \$2 ~ /$stripe/) { print \$6 }" +} + +# Read from a specific stripe to test read recovery that corrupted a specific +# stripe. Btrfs uses the PID to select the mirror, so keep reading until the +# xfs_io process that performed the read was executed with a PID that ends up +# on the intended mirror. +_btrfs_direct_read_on_mirror() +{ + local mirror=$1 + local nr_mirrors=$2 + local file=$3 + local offset=$4 + local size=$5 + + while [[ -z $( (( BASHPID % nr_mirrors == mirror )) && + exec $XFS_IO_PROG -d \ + -c "pread -b $size $offset $size" $file) ]]; do + : + done +} + +# Read from a specific stripe to test read recovery that corrupted a specific +# stripe. Btrfs uses the PID to select the mirror, so keep reading until the +# xfs_io process that performed the read was executed with a PID that ends up +# on the intended mirror. +_btrfs_buffered_read_on_mirror() +{ + local mirror=$1 + local nr_mirrors=$2 + local file=$3 + local offset=$4 + local size=$5 + + echo 3 > /proc/sys/vm/drop_caches + while [[ -z $( (( BASHPID % nr_mirrors == mirror )) && + exec $XFS_IO_PROG \ + -c "pread -b $size $offset $size" $file) ]]; do + : + done +} diff --git a/common/config b/common/config index c6428f90..df20afc1 100644 --- a/common/config +++ b/common/config @@ -228,6 +228,7 @@ export E2IMAGE_PROG="$(type -P e2image)" export BLKZONE_PROG="$(type -P blkzone)" export GZIP_PROG="$(type -P gzip)" export BTRFS_IMAGE_PROG="$(type -P btrfs-image)" +export BTRFS_MAP_LOGICAL_PROG=$(type -P btrfs-map-logical) # use 'udevadm settle' or 'udevsettle' to wait for lv to be settled. # newer systems have udevadm command but older systems like RHEL5 don't.
On 5/27/22 13:49, Christoph Hellwig wrote: > Add a few helpers to consolidate code for btrfs read repair testing: > > - _btrfs_get_first_logical() gets the btrfs logical address for the > first extent in a file > - _btrfs_get_device_path and _btrfs_get_physical use the > btrfs-map-logical tool to find the device path and physical address > for btrfs logical address for a specific mirror > - _btrfs_direct_read_on_mirror and _btrfs_buffered_read_on_mirror > read the data from a specific mirror > > These will be used to consolidate the read repair tests and avoid > duplication for new tests. > > Signed-off-by: Christoph Hellwig <hch@lst.de> > Reviewed-by: Qu Wenruo <wqu@suse.com> > --- > common/btrfs | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++ > common/config | 1 + > 2 files changed, 76 insertions(+) > > diff --git a/common/btrfs b/common/btrfs > index ac597ca4..b69feeee 100644 > --- a/common/btrfs > +++ b/common/btrfs > @@ -505,3 +505,78 @@ _btrfs_metadump() > $BTRFS_IMAGE_PROG "$device" "$dumpfile" > [ -n "$DUMP_COMPRESSOR" ] && $DUMP_COMPRESSOR -f "$dumpfile" &> /dev/null > } > + > +# Return the btrfs logical address for the first block in a file > +_btrfs_get_first_logical() > +{ > + local file=$1 > + _require_command "$FILEFRAG_PROG" filefrag > + > + ${FILEFRAG_PROG} -v $SCRATCH_MNT/foobar >> $seqres.full > + ${FILEFRAG_PROG} -v $file | _filter_filefrag | cut -d '#' -f 1 > +} > + > +# Find the device path for a btrfs logical offset > +_btrfs_get_device_path() > +{ > + local logical=$1 > + local stripe=$2 > + > + _require_command "$BTRFS_MAP_LOGICAL_PROG" btrfs-map-logical > + > + $BTRFS_MAP_LOGICAL_PROG -l $logical $SCRATCH_DEV | \ > + $AWK_PROG "(\$1 ~ /mirror/ && \$2 ~ /$stripe/) { print \$8 }" > +} > + > + > +# Find the device physical sector for a btrfs logical offset > +_btrfs_get_physical() > +{ > + local logical=$1 > + local stripe=$2 > + > + _require_command "$BTRFS_MAP_LOGICAL_PROG" btrfs-map-logical > + > + $BTRFS_MAP_LOGICAL_PROG -b -l $logical $SCRATCH_DEV >> $seqres.full 2>&1 > + $BTRFS_MAP_LOGICAL_PROG -l $logical $SCRATCH_DEV | \ > + $AWK_PROG "(\$1 ~ /mirror/ && \$2 ~ /$stripe/) { print \$6 }" > +} > + > +# Read from a specific stripe to test read recovery that corrupted a specific > +# stripe. Btrfs uses the PID to select the mirror, so keep reading until the > +# xfs_io process that performed the read was executed with a PID that ends up > +# on the intended mirror. > +_btrfs_direct_read_on_mirror() > +{ > + local mirror=$1 > + local nr_mirrors=$2 > + local file=$3 > + local offset=$4 > + local size=$5 > + > + while [[ -z $( (( BASHPID % nr_mirrors == mirror )) && > + exec $XFS_IO_PROG -d \ > + -c "pread -b $size $offset $size" $file) ]]; do > + : > + done > +} > + > +# Read from a specific stripe to test read recovery that corrupted a specific > +# stripe. Btrfs uses the PID to select the mirror, so keep reading until the > +# xfs_io process that performed the read was executed with a PID that ends up > +# on the intended mirror. > +_btrfs_buffered_read_on_mirror() > +{ > + local mirror=$1 > + local nr_mirrors=$2 > + local file=$3 > + local offset=$4 > + local size=$5 > + > + echo 3 > /proc/sys/vm/drop_caches > + while [[ -z $( (( BASHPID % nr_mirrors == mirror )) && > + exec $XFS_IO_PROG \ > + -c "pread -b $size $offset $size" $file) ]]; do I am confused if it should be BASHPID or PID? Next, it is ok if the xfs_io_prog fails and returns != 0. (Part of the test). But then we will continue in the while loop. No? Sorry, I am sceptical about this. Could you please clarify how this works? Thanks, Anand > + : > + done > +} > diff --git a/common/config b/common/config > index c6428f90..df20afc1 100644 > --- a/common/config > +++ b/common/config > @@ -228,6 +228,7 @@ export E2IMAGE_PROG="$(type -P e2image)" > export BLKZONE_PROG="$(type -P blkzone)" > export GZIP_PROG="$(type -P gzip)" > export BTRFS_IMAGE_PROG="$(type -P btrfs-image)" > +export BTRFS_MAP_LOGICAL_PROG=$(type -P btrfs-map-logical) > > # use 'udevadm settle' or 'udevsettle' to wait for lv to be settled. > # newer systems have udevadm command but older systems like RHEL5 don't.
On 2022/5/30 08:23, Anand Jain wrote: > On 5/27/22 13:49, Christoph Hellwig wrote: >> Add a few helpers to consolidate code for btrfs read repair testing: >> >> - _btrfs_get_first_logical() gets the btrfs logical address for the >> first extent in a file >> - _btrfs_get_device_path and _btrfs_get_physical use the >> btrfs-map-logical tool to find the device path and physical address >> for btrfs logical address for a specific mirror >> - _btrfs_direct_read_on_mirror and _btrfs_buffered_read_on_mirror >> read the data from a specific mirror >> >> These will be used to consolidate the read repair tests and avoid >> duplication for new tests. >> >> Signed-off-by: Christoph Hellwig <hch@lst.de> >> Reviewed-by: Qu Wenruo <wqu@suse.com> >> --- >> common/btrfs | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++ >> common/config | 1 + >> 2 files changed, 76 insertions(+) >> >> diff --git a/common/btrfs b/common/btrfs >> index ac597ca4..b69feeee 100644 >> --- a/common/btrfs >> +++ b/common/btrfs >> @@ -505,3 +505,78 @@ _btrfs_metadump() >> $BTRFS_IMAGE_PROG "$device" "$dumpfile" >> [ -n "$DUMP_COMPRESSOR" ] && $DUMP_COMPRESSOR -f "$dumpfile" &> >> /dev/null >> } >> + >> +# Return the btrfs logical address for the first block in a file >> +_btrfs_get_first_logical() >> +{ >> + local file=$1 >> + _require_command "$FILEFRAG_PROG" filefrag >> + >> + ${FILEFRAG_PROG} -v $SCRATCH_MNT/foobar >> $seqres.full >> + ${FILEFRAG_PROG} -v $file | _filter_filefrag | cut -d '#' -f 1 >> +} >> + >> +# Find the device path for a btrfs logical offset >> +_btrfs_get_device_path() >> +{ >> + local logical=$1 >> + local stripe=$2 >> + >> + _require_command "$BTRFS_MAP_LOGICAL_PROG" btrfs-map-logical >> + >> + $BTRFS_MAP_LOGICAL_PROG -l $logical $SCRATCH_DEV | \ >> + $AWK_PROG "(\$1 ~ /mirror/ && \$2 ~ /$stripe/) { print \$8 }" >> +} >> + >> + >> +# Find the device physical sector for a btrfs logical offset >> +_btrfs_get_physical() >> +{ >> + local logical=$1 >> + local stripe=$2 >> + >> + _require_command "$BTRFS_MAP_LOGICAL_PROG" btrfs-map-logical >> + >> + $BTRFS_MAP_LOGICAL_PROG -b -l $logical $SCRATCH_DEV >> >> $seqres.full 2>&1 >> + $BTRFS_MAP_LOGICAL_PROG -l $logical $SCRATCH_DEV | \ >> + $AWK_PROG "(\$1 ~ /mirror/ && \$2 ~ /$stripe/) { print \$6 }" >> +} >> + >> +# Read from a specific stripe to test read recovery that corrupted a >> specific >> +# stripe. Btrfs uses the PID to select the mirror, so keep reading >> until the >> +# xfs_io process that performed the read was executed with a PID that >> ends up >> +# on the intended mirror. >> +_btrfs_direct_read_on_mirror() >> +{ >> + local mirror=$1 >> + local nr_mirrors=$2 >> + local file=$3 >> + local offset=$4 >> + local size=$5 >> + >> + while [[ -z $( (( BASHPID % nr_mirrors == mirror )) && >> + exec $XFS_IO_PROG -d \ >> + -c "pread -b $size $offset $size" $file) ]]; do >> + : >> + done >> +} >> + >> +# Read from a specific stripe to test read recovery that corrupted a >> specific >> +# stripe. Btrfs uses the PID to select the mirror, so keep reading >> until the >> +# xfs_io process that performed the read was executed with a PID that >> ends up >> +# on the intended mirror. >> +_btrfs_buffered_read_on_mirror() >> +{ >> + local mirror=$1 >> + local nr_mirrors=$2 >> + local file=$3 >> + local offset=$4 >> + local size=$5 >> + >> + echo 3 > /proc/sys/vm/drop_caches > > >> + while [[ -z $( (( BASHPID % nr_mirrors == mirror )) && >> + exec $XFS_IO_PROG \ >> + -c "pread -b $size $offset $size" $file) ]]; do > > I am confused if it should be BASHPID or PID? > > Next, it is ok if the xfs_io_prog fails and returns != 0. > (Part of the test). > But then we will continue in the while loop. No? The loop goes like this: If "BASHPID % nr_mirrors" isn't mirror, then the whole condition inside the $() part is already 0, no need to exec the xfs_io command. Then -z 0 return true, we continue the while loop (aka, doing nothing). Only when (BASHPID % nr_mirrors == mirror) is true, we will try to exec c XFS_IO_PROGS in the same PID, and normally we will exit. Some of your concern are valid. Like if some code bug makes the pread failed, then we will continue the loop, possibly lead to an infinite loop. The pid-check-then-exec idea is pretty good, it allows us to only initiate the read, without doing some unnecessary read then check the pid. But putting the whole exec into the pid check is indeed hacky and can lead to problems. Can we make it less hacky? Thanks, Qu Thanks, Qu > > Sorry, I am sceptical about this. Could you please clarify > how this works? > > > Thanks, Anand > > > >> + : >> + done > > > > >> +} >> diff --git a/common/config b/common/config >> index c6428f90..df20afc1 100644 >> --- a/common/config >> +++ b/common/config >> @@ -228,6 +228,7 @@ export E2IMAGE_PROG="$(type -P e2image)" >> export BLKZONE_PROG="$(type -P blkzone)" >> export GZIP_PROG="$(type -P gzip)" >> export BTRFS_IMAGE_PROG="$(type -P btrfs-image)" >> +export BTRFS_MAP_LOGICAL_PROG=$(type -P btrfs-map-logical) >> # use 'udevadm settle' or 'udevsettle' to wait for lv to be settled. >> # newer systems have udevadm command but older systems like RHEL5 >> don't. >
On Mon, May 30, 2022 at 09:20:27AM +0800, Qu Wenruo wrote: > > > On 2022/5/30 08:23, Anand Jain wrote: > > On 5/27/22 13:49, Christoph Hellwig wrote: > > > Add a few helpers to consolidate code for btrfs read repair testing: > > > > > > - _btrfs_get_first_logical() gets the btrfs logical address for the > > > first extent in a file > > > - _btrfs_get_device_path and _btrfs_get_physical use the > > > btrfs-map-logical tool to find the device path and physical address > > > for btrfs logical address for a specific mirror > > > - _btrfs_direct_read_on_mirror and _btrfs_buffered_read_on_mirror > > > read the data from a specific mirror > > > > > > These will be used to consolidate the read repair tests and avoid > > > duplication for new tests. > > > > > > Signed-off-by: Christoph Hellwig <hch@lst.de> > > > Reviewed-by: Qu Wenruo <wqu@suse.com> > > > --- > > > common/btrfs | 75 +++++++++++++++++++++++++++++++++++++++++++++++++++ > > > common/config | 1 + > > > 2 files changed, 76 insertions(+) > > > > > > diff --git a/common/btrfs b/common/btrfs > > > index ac597ca4..b69feeee 100644 > > > --- a/common/btrfs > > > +++ b/common/btrfs > > > @@ -505,3 +505,78 @@ _btrfs_metadump() > > > $BTRFS_IMAGE_PROG "$device" "$dumpfile" > > > [ -n "$DUMP_COMPRESSOR" ] && $DUMP_COMPRESSOR -f "$dumpfile" &> > > > /dev/null > > > } > > > + > > > +# Return the btrfs logical address for the first block in a file > > > +_btrfs_get_first_logical() > > > +{ > > > + local file=$1 > > > + _require_command "$FILEFRAG_PROG" filefrag > > > + > > > + ${FILEFRAG_PROG} -v $SCRATCH_MNT/foobar >> $seqres.full > > > + ${FILEFRAG_PROG} -v $file | _filter_filefrag | cut -d '#' -f 1 > > > +} > > > + > > > +# Find the device path for a btrfs logical offset > > > +_btrfs_get_device_path() > > > +{ > > > + local logical=$1 > > > + local stripe=$2 > > > + > > > + _require_command "$BTRFS_MAP_LOGICAL_PROG" btrfs-map-logical > > > + > > > + $BTRFS_MAP_LOGICAL_PROG -l $logical $SCRATCH_DEV | \ > > > + $AWK_PROG "(\$1 ~ /mirror/ && \$2 ~ /$stripe/) { print \$8 }" > > > +} > > > + > > > + > > > +# Find the device physical sector for a btrfs logical offset > > > +_btrfs_get_physical() > > > +{ > > > + local logical=$1 > > > + local stripe=$2 > > > + > > > + _require_command "$BTRFS_MAP_LOGICAL_PROG" btrfs-map-logical > > > + > > > + $BTRFS_MAP_LOGICAL_PROG -b -l $logical $SCRATCH_DEV >> > > > $seqres.full 2>&1 > > > + $BTRFS_MAP_LOGICAL_PROG -l $logical $SCRATCH_DEV | \ > > > + $AWK_PROG "(\$1 ~ /mirror/ && \$2 ~ /$stripe/) { print \$6 }" > > > +} > > > + > > > +# Read from a specific stripe to test read recovery that corrupted a > > > specific > > > +# stripe. Btrfs uses the PID to select the mirror, so keep reading > > > until the > > > +# xfs_io process that performed the read was executed with a PID that > > > ends up > > > +# on the intended mirror. > > > +_btrfs_direct_read_on_mirror() > > > +{ > > > + local mirror=$1 > > > + local nr_mirrors=$2 > > > + local file=$3 > > > + local offset=$4 > > > + local size=$5 > > > + > > > + while [[ -z $( (( BASHPID % nr_mirrors == mirror )) && > > > + exec $XFS_IO_PROG -d \ > > > + -c "pread -b $size $offset $size" $file) ]]; do > > > + : > > > + done > > > +} > > > + > > > +# Read from a specific stripe to test read recovery that corrupted a > > > specific > > > +# stripe. Btrfs uses the PID to select the mirror, so keep reading > > > until the > > > +# xfs_io process that performed the read was executed with a PID that > > > ends up > > > +# on the intended mirror. > > > +_btrfs_buffered_read_on_mirror() > > > +{ > > > + local mirror=$1 > > > + local nr_mirrors=$2 > > > + local file=$3 > > > + local offset=$4 > > > + local size=$5 > > > + > > > + echo 3 > /proc/sys/vm/drop_caches > > > > > > > + while [[ -z $( (( BASHPID % nr_mirrors == mirror )) && > > > + exec $XFS_IO_PROG \ > > > + -c "pread -b $size $offset $size" $file) ]]; do > > > > I am confused if it should be BASHPID or PID? > > > > Next, it is ok if the xfs_io_prog fails and returns != 0. > > (Part of the test). > > But then we will continue in the while loop. No? > > The loop goes like this: > > If "BASHPID % nr_mirrors" isn't mirror, then the whole condition inside > the $() part is already 0, no need to exec the xfs_io command. > > Then -z 0 return true, we continue the while loop (aka, doing nothing). > > Only when (BASHPID % nr_mirrors == mirror) is true, we will try to exec > c XFS_IO_PROGS in the same PID, and normally we will exit. > > > Some of your concern are valid. > Like if some code bug makes the pread failed, then we will continue the > loop, possibly lead to an infinite loop. > > The pid-check-then-exec idea is pretty good, it allows us to only > initiate the read, without doing some unnecessary read then check the pid. > > But putting the whole exec into the pid check is indeed hacky and can > lead to problems. > > Can we make it less hacky? Hi Anand and Wenruo, This patchset has been merged into fstests release v2022.05.29 last weekend. If you feel something isn't good enough, feel free to send new patches to fix/improve it. Thanks, Zorro > > Thanks, > Qu > > Thanks, > Qu > > > > > Sorry, I am sceptical about this. Could you please clarify > > how this works? > > > > > > Thanks, Anand > > > > > > > > > + : > > > + done > > > > > > > > > > > +} > > > diff --git a/common/config b/common/config > > > index c6428f90..df20afc1 100644 > > > --- a/common/config > > > +++ b/common/config > > > @@ -228,6 +228,7 @@ export E2IMAGE_PROG="$(type -P e2image)" > > > export BLKZONE_PROG="$(type -P blkzone)" > > > export GZIP_PROG="$(type -P gzip)" > > > export BTRFS_IMAGE_PROG="$(type -P btrfs-image)" > > > +export BTRFS_MAP_LOGICAL_PROG=$(type -P btrfs-map-logical) > > > # use 'udevadm settle' or 'udevsettle' to wait for lv to be settled. > > > # newer systems have udevadm command but older systems like RHEL5 > > > don't. > > >
On Mon, May 30, 2022 at 12:36:32PM +0800, Zorro Lang wrote: > This patchset has been merged into fstests release v2022.05.29 last weekend. > If you feel something isn't good enough, feel free to send new patches to > fix/improve it. Also this exact same code existed in two tests before and was just factored into a common helper.
diff --git a/common/btrfs b/common/btrfs index ac597ca4..b69feeee 100644 --- a/common/btrfs +++ b/common/btrfs @@ -505,3 +505,78 @@ _btrfs_metadump() $BTRFS_IMAGE_PROG "$device" "$dumpfile" [ -n "$DUMP_COMPRESSOR" ] && $DUMP_COMPRESSOR -f "$dumpfile" &> /dev/null } + +# Return the btrfs logical address for the first block in a file +_btrfs_get_first_logical() +{ + local file=$1 + _require_command "$FILEFRAG_PROG" filefrag + + ${FILEFRAG_PROG} -v $SCRATCH_MNT/foobar >> $seqres.full + ${FILEFRAG_PROG} -v $file | _filter_filefrag | cut -d '#' -f 1 +} + +# Find the device path for a btrfs logical offset +_btrfs_get_device_path() +{ + local logical=$1 + local stripe=$2 + + _require_command "$BTRFS_MAP_LOGICAL_PROG" btrfs-map-logical + + $BTRFS_MAP_LOGICAL_PROG -l $logical $SCRATCH_DEV | \ + $AWK_PROG "(\$1 ~ /mirror/ && \$2 ~ /$stripe/) { print \$8 }" +} + + +# Find the device physical sector for a btrfs logical offset +_btrfs_get_physical() +{ + local logical=$1 + local stripe=$2 + + _require_command "$BTRFS_MAP_LOGICAL_PROG" btrfs-map-logical + + $BTRFS_MAP_LOGICAL_PROG -b -l $logical $SCRATCH_DEV >> $seqres.full 2>&1 + $BTRFS_MAP_LOGICAL_PROG -l $logical $SCRATCH_DEV | \ + $AWK_PROG "(\$1 ~ /mirror/ && \$2 ~ /$stripe/) { print \$6 }" +} + +# Read from a specific stripe to test read recovery that corrupted a specific +# stripe. Btrfs uses the PID to select the mirror, so keep reading until the +# xfs_io process that performed the read was executed with a PID that ends up +# on the intended mirror. +_btrfs_direct_read_on_mirror() +{ + local mirror=$1 + local nr_mirrors=$2 + local file=$3 + local offset=$4 + local size=$5 + + while [[ -z $( (( BASHPID % nr_mirrors == mirror )) && + exec $XFS_IO_PROG -d \ + -c "pread -b $size $offset $size" $file) ]]; do + : + done +} + +# Read from a specific stripe to test read recovery that corrupted a specific +# stripe. Btrfs uses the PID to select the mirror, so keep reading until the +# xfs_io process that performed the read was executed with a PID that ends up +# on the intended mirror. +_btrfs_buffered_read_on_mirror() +{ + local mirror=$1 + local nr_mirrors=$2 + local file=$3 + local offset=$4 + local size=$5 + + echo 3 > /proc/sys/vm/drop_caches + while [[ -z $( (( BASHPID % nr_mirrors == mirror )) && + exec $XFS_IO_PROG \ + -c "pread -b $size $offset $size" $file) ]]; do + : + done +} diff --git a/common/config b/common/config index c6428f90..df20afc1 100644 --- a/common/config +++ b/common/config @@ -228,6 +228,7 @@ export E2IMAGE_PROG="$(type -P e2image)" export BLKZONE_PROG="$(type -P blkzone)" export GZIP_PROG="$(type -P gzip)" export BTRFS_IMAGE_PROG="$(type -P btrfs-image)" +export BTRFS_MAP_LOGICAL_PROG=$(type -P btrfs-map-logical) # use 'udevadm settle' or 'udevsettle' to wait for lv to be settled. # newer systems have udevadm command but older systems like RHEL5 don't.