[v7,0/4] btrfs-progs: image: new data dump feature

Message

Qu Wenruo Aug. 24, 2021, 7:41 a.m. UTC

This patchset includes the following features:

- Introduce data dump feature to dump the whole fs.
  This will introduce a new magic number to prevent old btrfs-image to
  hit failure as the item size limit is enlarged.
  Patch 1 and 2.

- Reduce memory usage for compressed data dump restore
  This is mostly due to the fact that we have much larger
  max_pending_size introduced by data dump(256K -> 256M).
  Using 4 * max_pending_size for each decompress thread as buffer is way
  too expensive now.
  Use proper inflate() to replace uncompress() calls.
  Patch 3

- A fix for small dev extent size mismatch with superblock
  This no longer affects single device dump restore, thus it's only
  for multi-device dump restore.
  Patch 4

Changelog:
v2:
- New small fixes:
  * Fix a confusing error message due to unpopulated errno
  * Output error message for chunk tree build error
  
- Fix a regression of previous version
  Patch "btrfs-progs: image: Rework how we search chunk tree blocks"
  deleted a "ret = 0" line which could cause false early exit.

- Reduce memory usage for data dump

v2.1:
- Rebased to devel branch
  Removing 4 already merged patches from the patchset.

- Re-order the patchset
  Put small and independent patches at the top of queue, and put the
  data dump related feature at the end.

- Fix -Wmaybe-uninitialized warnings
  Strangely, D=1 won't trigger these warnings thus they sneak into v2
  without being detected.

- Fix FROM: line
  Reverted to old smtp setup. The new setup will override FROM: line,
  messing up the name of author.

v3:
- Fix a wrong option in error string
- Fix a bug that we always dump data extents

v4:
- Rebased to latest devel branch
- Add a new small fix to kill the tiny dev extent size mismatch.

v5:
- Rebased to latest devel branch
- Checkpatch fixes

v6:
- Rebased to latest devel branch
  No conflicts at all.

v7:
- Rebased to latest devel branch
  No conflicts at all.

- Hide the data dump feature behind the experimental features
  Now to make "btrfs-image -d" work, one must enable experimental
  features, or "btrfs-image -d" will only output an error message.

  And if experimental features are not enabled, the new header format
  will also be disabled, so that btrfs-image can't detect the new format
  either.

  Although the "-d" option is still enabled for the non-experimental
  build.

  This only affects the 2nd patch, the remaining patches are not
  touched.

Qu Wenruo (4):
  btrfs-progs: image: introduce framework for more dump versions
  btrfs-progs: image: introduce -d option to dump data
  btrfs-progs: image: reduce memory requirement for decompression
  btrfs-progs: image: fix restored image size misalignment

 image/main.c     | 358 ++++++++++++++++++++++++++++++++++-------------
 image/metadump.h |  12 +-
 2 files changed, 273 insertions(+), 97 deletions(-)

Comments

David Sterba Aug. 24, 2021, 4:08 p.m. UTC | #1

On Tue, Aug 24, 2021 at 03:41:04PM +0800, Qu Wenruo wrote:
> v7:
> - Rebased to latest devel branch
>   No conflicts at all.
> 
> - Hide the data dump feature behind the experimental features
>   Now to make "btrfs-image -d" work, one must enable experimental
>   features, or "btrfs-image -d" will only output an error message.
> 
>   And if experimental features are not enabled, the new header format
>   will also be disabled, so that btrfs-image can't detect the new format
>   either.
> 
>   Although the "-d" option is still enabled for the non-experimental
>   build.

Thanks.  I've created issue #394 for tracking, with label 'experimental'
so we know what's in progress. If there are separate tasks related to
that then they could be tracked in other issues and referenced from that
one.