[GIT,PULL,10/23] xfs_scrub: improve scheduling of repair items

Message ID	172230458756.1455085.11655778070760827192.stg-ugh@frogsfrogsfrogs (mailing list archive)
State	Accepted, archived
Headers	show Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 379B0144312 for <linux-xfs@vger.kernel.org>; Tue, 30 Jul 2024 02:42:55 +0000 (UTC) Date: Mon, 29 Jul 2024 19:42:54 -0700 Subject: [GIT PULL 10/23] xfs_scrub: improve scheduling of repair items From: "Darrick J. Wong" <djwong@kernel.org> To: cem@kernel.org, djwong@kernel.org Cc: hch@lst.de, linux-xfs@vger.kernel.org Message-ID: <172230458756.1455085.11655778070760827192.stg-ugh@frogsfrogsfrogs> Precedence: bulk MIME-Version: 1.0 In-Reply-To: <20240730013626.GF6352@frogsfrogsfrogs> References: <20240730013626.GF6352@frogsfrogsfrogs> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit
Series	[GIT,PULL,01/23] libxfs: fixes for 6.9 \| expand [GIT,PULL,01/23] libxfs: fixes for 6.9 [GIT,PULL,02/23] libxfs: sync with 6.10 [GIT,PULL,03/23] xfsprogs: atomic file updates [GIT,PULL,04/23] xfsprogs: set and validate dir/attr block owners [GIT,PULL,05/23] xfsprogs: inode-related repair fixes [GIT,PULL,06/23] xfs_scrub: fixes to the repair code [GIT,PULL,07/23] xfs_scrub: improve warnings about difficult repairs [GIT,PULL,08/23] xfs_scrub: track data dependencies for repairs [GIT,PULL,09/23] xfs_scrub: use scrub_item to track check progress [GIT,PULL,10/23] xfs_scrub: improve scheduling of repair items [GIT,PULL,11/23] xfs_scrub: detect deceptive filename extensions [GIT,PULL,12/23] xfs_scrub: move fstrim to a separate phase [GIT,PULL,13/23] xfs_scrub: use free space histograms to reduce fstrim runtime [GIT,PULL,14/23] xfs_scrub: tighten security of systemd services [GIT,PULL,15/23] xfs_scrub_all: automatic media scan service [GIT,PULL,16/23] xfs_scrub_all: improve systemd handling [GIT,PULL,17/23] xfsprogs: improve extended attribute validation [GIT,PULL,18/23] xfsprogs: Parent Pointers [GIT,PULL,19/23] xfsprogs: scrubbing for parent pointers [GIT,PULL,20/23] xfsprogs: offline repair for parent pointers [GIT,PULL,21/23] xfsprogs: detect and correct directory tree problems [GIT,PULL,22/23] xfs_scrub: vectorize kernel calls [GIT,PULL,23/23] xfs_repair: fixes for kernel 6.10

Message ID

172230458756.1455085.11655778070760827192.stg-ugh@frogsfrogsfrogs (mailing list archive)

State

Accepted, archived

Headers

Date: Mon, 29 Jul 2024 19:42:54 -0700
Subject: [GIT PULL 10/23] xfs_scrub: improve scheduling of repair items
From: "Darrick J. Wong" <djwong@kernel.org>
To: cem@kernel.org, djwong@kernel.org
Cc: hch@lst.de, linux-xfs@vger.kernel.org
Message-ID: 
 <172230458756.1455085.11655778070760827192.stg-ugh@frogsfrogsfrogs>
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <20240730013626.GF6352@frogsfrogsfrogs>
References: <20240730013626.GF6352@frogsfrogsfrogs>
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit

Series

[GIT,PULL,01/23] libxfs: fixes for 6.9 | expand

Message

Darrick J. Wong July 30, 2024, 2:42 a.m. UTC

Hi Carlos,

Please pull this branch with changes for xfsprogs for 6.10-rc1.

As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts.  Please let me know if you
encounter any problems.

The following changes since commit 8dd67c8eccbe0e6b8dd7975ba53f9ccaf532aa9c:

xfs_scrub: hoist scrub retry loop to scrub_item_check_file (2024-07-29 17:01:08 -0700)

are available in the Git repository at:

https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfsprogs-dev.git tags/scrub-repair-scheduling-6.10_2024-07-29

for you to fetch changes up to 474ff27d466c053f1cd51024e6b0c5a741a2d4bd:

xfs_scrub: try to repair space metadata before file metadata (2024-07-29 17:01:08 -0700)

----------------------------------------------------------------
xfs_scrub: improve scheduling of repair items [v30.9 10/28]

Currently, phase 4 of xfs_scrub uses per-AG repair item lists to
schedule repair work across a thread pool.  This scheme is suboptimal
when most of the repairs involve a single AG because all the work gets
dumped on a single pool thread.

Instead, we should create a thread pool with the same number of workers
as CPUs, and dispatch individual repair tickets as separate work items
to maximize parallelization.

However, we also need to ensure that repairs to space metadata and file
metadata are kept in separate queues because file repairs generally
depend on correctness of space metadata.

This has been running on the djcloud for months with no problems.  Enjoy!

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

----------------------------------------------------------------
Darrick J. Wong (4):
libfrog: enhance ptvar to support initializer functions
xfs_scrub: improve thread scheduling repair items during phase 4
xfs_scrub: recheck entire metadata objects after corruption repairs
xfs_scrub: try to repair space metadata before file metadata

libfrog/ptvar.c       |   9 +-
libfrog/ptvar.h       |   4 +-
scrub/counter.c       |   2 +-
scrub/descr.c         |   2 +-
scrub/phase1.c        |  15 +++-
scrub/phase2.c        |  23 ++++-
scrub/phase3.c        | 106 ++++++++++++++--------
scrub/phase4.c        | 244 +++++++++++++++++++++++++++++++++++++-------------
scrub/phase7.c        |   2 +-
scrub/read_verify.c   |   2 +-
scrub/repair.c        | 172 ++++++++++++++++++++++-------------
scrub/repair.h        |  37 ++++++--
scrub/scrub.c         |   5 +-
scrub/scrub.h         |  10 +++
scrub/scrub_private.h |   2 +
scrub/xfs_scrub.h     |   3 +-
16 files changed, 455 insertions(+), 183 deletions(-)

[GIT,PULL,10/23] xfs_scrub: improve scheduling of repair items

Pull-request

Message