| Message ID | cover.1687296675.git.osandov@osandov.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | xfs: CPU usage optimizations for realtime allocator | expand |
On Tue, Jun 20, 2023 at 02:32:10PM -0700, Omar Sandoval wrote: > Hello, > > Our distributed storage system uses XFS's realtime device support as a > way to split an XFS filesystem between an SSD and an HDD -- we configure > the HDD as the realtime device so that metadata goes on the SSD and data > goes on the HDD. > > We've been running this in production for a few years now, so we have > some fairly fragmented filesystems. This has exposed various CPU > inefficiencies in the realtime allocator. These became even worse when > we experimented with using XFS_XFLAG_EXTSIZE to force files to be > allocated contiguously. > > This series adds several optimizations that don't change the realtime > allocator's decisions, but make them happen more efficiently, mainly by > avoiding redundant work. We've tested these in production and measured > ~10% lower CPU utilization. Furthermore, it made it possible to use > XFS_XFLAG_EXTSIZE to force contiguous allocations -- without these > patches, our most fragmented systems would become unresponsive due to > high CPU usage in the realtime allocator, but with them, CPU utilization > is actually ~4-6% lower than before, and disk I/O utilization is 15-20% > lower. > > Patches 2 and 3 are preparations for later optimizations; the remaining > patches are the optimizations themselves. > > This is based on Linus' tree as of today (commit > 692b7dc87ca6d55ab254f8259e6f970171dc9d01). > > Thanks! > > Omar Sandoval (6): > xfs: cache last bitmap block in realtime allocator > xfs: invert the realtime summary cache > xfs: return maximum free size from xfs_rtany_summary() > xfs: limit maxlen based on available space in > xfs_rtallocate_extent_near() > xfs: don't try redundant allocations in xfs_rtallocate_extent_near() > xfs: don't look for end of extent further than necessary in > xfs_rtallocate_extent_near() > > fs/xfs/libxfs/xfs_rtbitmap.c | 173 ++++++++++++++-------------- > fs/xfs/xfs_mount.h | 6 +- > fs/xfs/xfs_rtalloc.c | 215 ++++++++++++++++------------------- > fs/xfs/xfs_rtalloc.h | 28 +++-- > 4 files changed, 207 insertions(+), 215 deletions(-) Gentle ping.
On Thu, Jul 06, 2023 at 02:39:08PM -0700, Omar Sandoval wrote: > On Tue, Jun 20, 2023 at 02:32:10PM -0700, Omar Sandoval wrote: > > Hello, > > > > Our distributed storage system uses XFS's realtime device support as a > > way to split an XFS filesystem between an SSD and an HDD -- we configure > > the HDD as the realtime device so that metadata goes on the SSD and data > > goes on the HDD. > > > > We've been running this in production for a few years now, so we have > > some fairly fragmented filesystems. This has exposed various CPU > > inefficiencies in the realtime allocator. These became even worse when > > we experimented with using XFS_XFLAG_EXTSIZE to force files to be > > allocated contiguously. > > > > This series adds several optimizations that don't change the realtime > > allocator's decisions, but make them happen more efficiently, mainly by > > avoiding redundant work. We've tested these in production and measured > > ~10% lower CPU utilization. Furthermore, it made it possible to use > > XFS_XFLAG_EXTSIZE to force contiguous allocations -- without these > > patches, our most fragmented systems would become unresponsive due to > > high CPU usage in the realtime allocator, but with them, CPU utilization > > is actually ~4-6% lower than before, and disk I/O utilization is 15-20% > > lower. > > > > Patches 2 and 3 are preparations for later optimizations; the remaining > > patches are the optimizations themselves. > > > > This is based on Linus' tree as of today (commit > > 692b7dc87ca6d55ab254f8259e6f970171dc9d01). > > > > Thanks! > > > > Omar Sandoval (6): > > xfs: cache last bitmap block in realtime allocator > > xfs: invert the realtime summary cache > > xfs: return maximum free size from xfs_rtany_summary() > > xfs: limit maxlen based on available space in > > xfs_rtallocate_extent_near() > > xfs: don't try redundant allocations in xfs_rtallocate_extent_near() > > xfs: don't look for end of extent further than necessary in > > xfs_rtallocate_extent_near() > > > > fs/xfs/libxfs/xfs_rtbitmap.c | 173 ++++++++++++++-------------- > > fs/xfs/xfs_mount.h | 6 +- > > fs/xfs/xfs_rtalloc.c | 215 ++++++++++++++++------------------- > > fs/xfs/xfs_rtalloc.h | 28 +++-- > > 4 files changed, 207 insertions(+), 215 deletions(-) > > Gentle ping. Sorry, I haven't had time to get to this yet. I've still got a couple more bug reports to work through before I can really start thinking about looking at anything else.. Cheers, Dave.
Hello, Our distributed storage system uses XFS's realtime device support as a way to split an XFS filesystem between an SSD and an HDD -- we configure the HDD as the realtime device so that metadata goes on the SSD and data goes on the HDD. We've been running this in production for a few years now, so we have some fairly fragmented filesystems. This has exposed various CPU inefficiencies in the realtime allocator. These became even worse when we experimented with using XFS_XFLAG_EXTSIZE to force files to be allocated contiguously. This series adds several optimizations that don't change the realtime allocator's decisions, but make them happen more efficiently, mainly by avoiding redundant work. We've tested these in production and measured ~10% lower CPU utilization. Furthermore, it made it possible to use XFS_XFLAG_EXTSIZE to force contiguous allocations -- without these patches, our most fragmented systems would become unresponsive due to high CPU usage in the realtime allocator, but with them, CPU utilization is actually ~4-6% lower than before, and disk I/O utilization is 15-20% lower. Patches 2 and 3 are preparations for later optimizations; the remaining patches are the optimizations themselves. This is based on Linus' tree as of today (commit 692b7dc87ca6d55ab254f8259e6f970171dc9d01). Thanks! Omar Sandoval (6): xfs: cache last bitmap block in realtime allocator xfs: invert the realtime summary cache xfs: return maximum free size from xfs_rtany_summary() xfs: limit maxlen based on available space in xfs_rtallocate_extent_near() xfs: don't try redundant allocations in xfs_rtallocate_extent_near() xfs: don't look for end of extent further than necessary in xfs_rtallocate_extent_near() fs/xfs/libxfs/xfs_rtbitmap.c | 173 ++++++++++++++-------------- fs/xfs/xfs_mount.h | 6 +- fs/xfs/xfs_rtalloc.c | 215 ++++++++++++++++------------------- fs/xfs/xfs_rtalloc.h | 28 +++-- 4 files changed, 207 insertions(+), 215 deletions(-)