mbox series

[GIT,PULL,13/23] xfs_scrub: use free space histograms to reduce fstrim runtime

Message ID 172230459074.1455085.705786233665014114.stg-ugh@frogsfrogsfrogs (mailing list archive)
State Accepted, archived
Headers show
Series [GIT,PULL,01/23] libxfs: fixes for 6.9 | expand

Pull-request

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/scrub-fstrim-minlen-freesp-histogram-6.10_2024-07-29

Message

Darrick J. Wong July 30, 2024, 2:43 a.m. UTC 
Hi Carlos,

Please pull this branch with changes for xfsprogs for 6.10-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

The following changes since commit 5ccdd24dc9987b50333332b7381ff1a305e67ef7:

xfs_scrub: improve responsiveness while trimming the filesystem (2024-07-29 17:01:09 -0700)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/scrub-fstrim-minlen-freesp-histogram-6.10_2024-07-29

for you to fetch changes up to 34bed605490f936c3ead49e2e1cad78505260461:

xfs_scrub: tune fstrim minlen parameter based on free space histograms (2024-07-29 17:01:09 -0700)

----------------------------------------------------------------
xfs_scrub: use free space histograms to reduce fstrim runtime [v30.9 13/28]

This patchset dramatically reduces the runtime of the FITRIM calls made
during phase 8 of xfs_scrub.  It turns out that phase 8 can really get
bogged down if the free space contains a large number of very small
extents.  In these cases, the runtime can increase by an order of
magnitude to free less than 1%% of the free space.  This is not worth the
time, since we're spending a lot of time to do very little work.  The
FITRIM ioctl allows us to specify a minimum extent length, so we can use
statistical methods to compute a minlen parameter.

It turns out xfs_db/spaceman already have the code needed to create
histograms of free space extent lengths.  We add the ability to compute
a CDF of the extent lengths, which make it easy to pick a minimum length
corresponding to 99%% of the free space.  In most cases, this results in
dramatic reductions in phase 8 runtime.  Hence, move the histogram code
to libfrog, and wire up xfs_scrub, since phase 7 already walks the
fsmap.

We also add a new -o suboption to xfs_scrub so that people who /do/ want
to examine every free extent can do so.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Darrick J. Wong (7):
libfrog: hoist free space histogram code
libfrog: print wider columns for free space histogram
libfrog: print cdf of free space buckets
xfs_scrub: don't close stdout when closing the progress bar
xfs_scrub: remove pointless spacemap.c arguments
xfs_scrub: collect free space histograms during phase 7
xfs_scrub: tune fstrim minlen parameter based on free space histograms

db/freesp.c          |  89 +++++------------
libfrog/Makefile     |   2 +
libfrog/histogram.c  | 270 +++++++++++++++++++++++++++++++++++++++++++++++++++
libfrog/histogram.h  |  77 +++++++++++++++
man/man8/xfs_scrub.8 |  16 +++
scrub/phase7.c       |  47 ++++++++-
scrub/phase8.c       |  91 ++++++++++++++++-
scrub/spacemap.c     |  11 +--
scrub/vfs.c          |   4 +-
scrub/vfs.h          |   2 +-
scrub/xfs_scrub.c    |  45 ++++++++-
scrub/xfs_scrub.h    |  16 +++
spaceman/freesp.c    |  99 +++++++------------
13 files changed, 619 insertions(+), 150 deletions(-)
create mode 100644 libfrog/histogram.c
create mode 100644 libfrog/histogram.h