[v2,00/30] btrfs-progs: extent tree v2 support, global roots

Message

Josef Bacik Nov. 10, 2021, 8:14 p.m. UTC

v1->v2:
- These depend on the v3 of the prep patches (it's marked as v2 because I'm
  stupid, but the second v2 posting I sent.)
- I've moved the global root rb tree patches into this series to differentiate
  them from the actual fixes in the prep series.

--- Original email ---
Hello,

These patches are the first chunk of the extent tree v2 format changes.  This
includes the separate block group root which will hold all of the block group
items.  This also includes the global root support, which is the work to allow
us to have multiple extent, csum, and free space trees in the same file system.

The goal of these two changes are straightforward.  For the block group root, on
very large file systems the block group items are very widely separated, which
means it takes a very long time to mount the file system on large, slow disks.
Putting the block group items in their own root will allow us to densely
populate the tree and dramatically increase mount times in these cases.

The global roots change is motivated by lock contention on the root nodes of
these global roots.  I've had to make many changes to how we run delayed refs to
speed up things like the transaction commit because of all the delayed refs
going into one tree and contending on the root node of the extent tree.  In the
same token you can have heavy lock contention on the csum roots when writing to
many files.  Allowing for multiple roots will let us spread the lock contention
load around.

I have disabled a few key features, namely balance and qgroups.  There will be
more to come as I make more and more invasive changes, and then they will slowly
be re-enabled as the work is added.  These are disabled to avoid a bunch of work
that would be thrown away by future changes.

These patches have passed xfstests without panicing, but clearly failing a lot
of tests because of the disabled features.  I've also run it through fsperf to
validate that there are no major performance regressions.

WARNING: there are many more format changes planned, this is just the first
batch.  If you want to test then please feel free, but know that the format is
still in flux.  Thanks,

Josef

Josef Bacik (30):
  btrfs-progs: stop accessing ->extent_root directly
  btrfs-progs: stop accessing ->free_space_root directly
  btrfs-progs: track csum, extent, and free space trees in a rb tree
  btrfs-progs: check: make reinit work per found root item
  btrfs-progs: check: check the global roots for uptodate root nodes
  btrfs-progs: check: check all of the csum roots
  btrfs-progs: check: fill csum root from all extent roots
  btrfs-progs: common: search all extent roots for marking used space
  btrfs-progs: common: allow users to select extent-tree-v2 option
  btrfs-progs: add definitions for the block group tree
  btrfs-progs: add on disk pointers to global tree ids
  btrfs-progs: add support for loading the block group root
  btrfs-progs: add print support for the block group tree
  btrfs-progs: mkfs: use the btrfs_block_group_root helper
  btrfs-progs: check-lowmem: use the btrfs_block_group_root helper
  btrfs-progs: handle no bg item in extent tree for free space tree
  btrfs-progs: mkfs: add support for the block group tree
  btrfs-progs: check: add block group tree support
  btrfs-progs: qgroup-verify: scan extents based on block groups
  btrfs-progs: check: make free space tree validation extent tree v2
    aware
  btrfs-progs: check: add helper to reinit the root based on a key
  btrfs-progs: check: handle the block group tree properly
  btrfs-progs: load the number of global roots into the fs_info
  btrfs-progs: handle the per-block group global root id
  btrfs-progs: add a btrfs_delete_and_free_root helper
  btrfs-progs: make btrfs_clear_free_space_tree extent tree v2 aware
  btrfs-progs: make btrfs_create_tree take a key for the root key
  btrfs-progs: mkfs: set chunk_item_objectid properly for extent tree v2
  btrfs-progs: mkfs: create the global root's
  btrfs-progs: check: don't do the root item check for extent tree v2

 btrfs-corrupt-block.c           |  13 +-
 btrfs-map-logical.c             |   9 +-
 btrfstune.c                     |   2 +-
 check/main.c                    | 429 ++++++++++++++++---------
 check/mode-common.c             |   8 +-
 check/mode-lowmem.c             |  76 +++--
 check/qgroup-verify.c           |  34 +-
 cmds/inspect-dump-tree.c        |  37 ++-
 cmds/rescue-chunk-recover.c     |  18 +-
 common/fsfeatures.c             |  11 +
 common/repair.c                 | 146 ++++++---
 common/repair.h                 |   2 +
 convert/main.c                  |   4 +-
 image/main.c                    |   2 +-
 kernel-shared/backref.c         |  10 +-
 kernel-shared/ctree.h           |  70 +++-
 kernel-shared/disk-io.c         | 553 ++++++++++++++++++++++++++------
 kernel-shared/disk-io.h         |  19 +-
 kernel-shared/extent-tree.c     |  66 ++--
 kernel-shared/free-space-tree.c | 124 ++++---
 kernel-shared/print-tree.c      |  26 +-
 kernel-shared/transaction.c     |   2 +
 kernel-shared/volumes.c         |   3 +-
 kernel-shared/zoned.c           |   2 +-
 libbtrfsutil/btrfs_tree.h       |   3 +
 mkfs/common.c                   |  93 ++++--
 mkfs/common.h                   |  12 +
 mkfs/main.c                     | 122 ++++++-
 28 files changed, 1411 insertions(+), 485 deletions(-)