From patchwork Tue Jul 30 01:14:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13746138 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 052048F40 for ; Tue, 30 Jul 2024 01:14:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722302081; cv=none; b=ufQdQzs3JsgH6hgl0bb30dIg6d+UMLtz2cYd7ATO78IEWWntmSHJ56SSP8F/wY8UHU+BckJOT3NHsY5LmnvtOrdRXAe4+tPsO2WQn1RV4nQHbbyO+9qn1Xug03U2rKZREGgEZeQn1hTYwcoo/FKpDrhvFTMGXQvajel8YWOcgUM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722302081; c=relaxed/simple; bh=F0Z0N7fa6PShlvZPhgqIpndE5jMiy2Dv/vMTW3HifBg=; h=Date:Subject:From:To:Cc:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=n7XQcmLrq12jXk5VxdHUCy6kRuHlD7gb+sLEdajguuGMAh1W6SfGzRLXxGeXVfGRKp7b7uE9wf7DppSLHIci+9mDbfse/inOSK/f0mDgYvITYHsypcc8b74YJFRM2jdjrDZsLNqbkCftBz2S1Ce+KLA183eMm1FKqejB0UrQEtw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=BZ/jz7LN; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="BZ/jz7LN" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C939BC32786; Tue, 30 Jul 2024 01:14:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1722302080; bh=F0Z0N7fa6PShlvZPhgqIpndE5jMiy2Dv/vMTW3HifBg=; h=Date:Subject:From:To:Cc:In-Reply-To:References:From; b=BZ/jz7LNQB/DO+CSD/w4pN/NbSSdW30PsoMdS9zbjuB3ZT35wf/aFKuxPsdQmq7bh YkvGZIXPGgCBlMgPjnp1R1y/2ZcKnt2jsNCq99bf0DIF/eheIS8nl+kus8D4TGu8lo zAsbC7ILN/SET0OmC002OC6utUNe0wpllLBEAPT7/c/LT9f2UK7DgncnATPa963c1s om9+Tz8+7cN1x3sQqvIWw/RqUYKJUEaanVHHQNzuXIVQSWSq1a0DgRXjgG86dZtey6 BV8ZOXiiMSlm0piJELqP0/njPUUpckQ87mJkZ40nJAZR/WsZQypT4JVUtc7gQSVT4+ PRUT6f1muXqPw== Date: Mon, 29 Jul 2024 18:14:40 -0700 Subject: [PATCH 1/6] xfs_scrub_all: only use the xfs_scrub@ systemd services in service mode From: "Darrick J. Wong" To: djwong@kernel.org, cem@kernel.org Cc: Christoph Hellwig , linux-xfs@vger.kernel.org Message-ID: <172229849261.1350165.10506383191812810534.stgit@frogsfrogsfrogs> In-Reply-To: <172229849240.1350165.13200329618269649031.stgit@frogsfrogsfrogs> References: <172229849240.1350165.13200329618269649031.stgit@frogsfrogsfrogs> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-xfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Darrick J. Wong Since the per-mount xfs_scrub@.service definition includes a bunch of resource usage constraints, we no longer want to use those services if xfs_scrub_all is being run directly by the sysadmin (aka not in service mode) on the presumption that sysadmins want answers as quickly as possible. Therefore, only try to call the systemd service from xfs_scrub_all if SERVICE_MODE is set in the environment. If reaching out to systemd fails and we're in service mode, we still want to run xfs_scrub directly. Split the makefile variables as necessary so that we only pass -b to xfs_scrub in service mode. Signed-off-by: Darrick J. Wong Reviewed-by: Christoph Hellwig --- scrub/Makefile | 5 ++++- scrub/xfs_scrub@.service.in | 2 +- scrub/xfs_scrub_all.in | 11 ++++++++--- 3 files changed, 13 insertions(+), 5 deletions(-) diff --git a/scrub/Makefile b/scrub/Makefile index 2a257e080..2050fe28f 100644 --- a/scrub/Makefile +++ b/scrub/Makefile @@ -15,7 +15,8 @@ LTCOMMAND = xfs_scrub INSTALL_SCRUB = install-scrub XFS_SCRUB_ALL_PROG = xfs_scrub_all XFS_SCRUB_FAIL_PROG = xfs_scrub_fail -XFS_SCRUB_ARGS = -b -n +XFS_SCRUB_ARGS = -n +XFS_SCRUB_SERVICE_ARGS = -b ifeq ($(HAVE_SYSTEMD),yes) INSTALL_SCRUB += install-systemd SYSTEMD_SERVICES=\ @@ -113,6 +114,7 @@ xfs_scrub_all: xfs_scrub_all.in $(builddefs) $(Q)$(SED) -e "s|@sbindir@|$(PKG_SBIN_DIR)|g" \ -e "s|@scrub_svcname@|$(scrub_svcname)|g" \ -e "s|@pkg_version@|$(PKG_VERSION)|g" \ + -e "s|@scrub_service_args@|$(XFS_SCRUB_SERVICE_ARGS)|g" \ -e "s|@scrub_args@|$(XFS_SCRUB_ARGS)|g" < $< > $@ $(Q)chmod a+x $@ @@ -132,6 +134,7 @@ install: $(INSTALL_SCRUB) %.service: %.service.in $(builddefs) @echo " [SED] $@" $(Q)$(SED) -e "s|@sbindir@|$(PKG_SBIN_DIR)|g" \ + -e "s|@scrub_service_args@|$(XFS_SCRUB_SERVICE_ARGS)|g" \ -e "s|@scrub_args@|$(XFS_SCRUB_ARGS)|g" \ -e "s|@pkg_libexec_dir@|$(PKG_LIBEXEC_DIR)|g" \ < $< > $@ diff --git a/scrub/xfs_scrub@.service.in b/scrub/xfs_scrub@.service.in index a8dd9052f..5fa5f3282 100644 --- a/scrub/xfs_scrub@.service.in +++ b/scrub/xfs_scrub@.service.in @@ -22,7 +22,7 @@ RequiresMountsFor=%f [Service] Type=oneshot Environment=SERVICE_MODE=1 -ExecStart=@sbindir@/xfs_scrub @scrub_args@ -M /tmp/scrub/ %f +ExecStart=@sbindir@/xfs_scrub @scrub_service_args@ @scrub_args@ -M /tmp/scrub/ %f SyslogIdentifier=%N # Run scrub with minimal CPU and IO priority so that nothing else will starve. diff --git a/scrub/xfs_scrub_all.in b/scrub/xfs_scrub_all.in index d0ab27fd3..f27251fa5 100644 --- a/scrub/xfs_scrub_all.in +++ b/scrub/xfs_scrub_all.in @@ -162,9 +162,10 @@ def run_scrub(mnt, cond, running_devs, mntdevs, killfuncs): if terminate: return - # Try it the systemd way + # Run per-mount systemd xfs_scrub service only if we ourselves + # are running as a systemd service. unitname = path_to_serviceunit(path) - if unitname is not None: + if unitname is not None and 'SERVICE_MODE' in os.environ: ret = systemctl_start(unitname, killfuncs) if ret == 0 or ret == 1: print("Scrubbing %s done, (err=%d)" % (mnt, ret)) @@ -175,8 +176,12 @@ def run_scrub(mnt, cond, running_devs, mntdevs, killfuncs): if terminate: return - # Invoke xfs_scrub manually + # Invoke xfs_scrub manually if we're running in the foreground. + # We also permit this if we're running as a cronjob where + # systemd services are unavailable. cmd = ['@sbindir@/xfs_scrub'] + if 'SERVICE_MODE' in os.environ: + cmd += '@scrub_service_args@'.split() cmd += '@scrub_args@'.split() cmd += [mnt] ret = run_killable(cmd, None, killfuncs) From patchwork Tue Jul 30 01:14:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13746139 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E2EBB8F40 for ; Tue, 30 Jul 2024 01:14:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722302097; cv=none; b=JEEzagG5cSl1Eyc1Avk8nnsOY8lH8EmnUwWKYAjzoPXHe+7keVtvcZfruRdFTTaT8N3c2ELSY3zXZg6qvmNqt70diR2bcBHkMFdPhH68Ecb47yqbc5J/oS57uoN1VGCFcsVYETv5mKi/Nl5x7/ytIau7DILMjO4qOS2xMFDPu7I= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722302097; c=relaxed/simple; bh=twE7QfeBFmCQsbI8pyGf5U8axmg/NA/KYQRh2fP0bCo=; h=Date:Subject:From:To:Cc:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=Mp/tq0bjuK3SW2e84Gf26oAIgWoLdQrl+S7OpzfSIVRyW7GjizBgvaQ7PU2BsgijpZ2GOGpGmnnQgD8k8SEtyN342zAVUhoe/09lcirvsGtfkW0UAu8P2TB30taxdU60DiksLA5ISI/ELl1iHCrthAHtHy0mC+0MlbdRh4/8J50= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=id7Q11a7; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="id7Q11a7" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 718C4C32786; Tue, 30 Jul 2024 01:14:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1722302096; bh=twE7QfeBFmCQsbI8pyGf5U8axmg/NA/KYQRh2fP0bCo=; h=Date:Subject:From:To:Cc:In-Reply-To:References:From; b=id7Q11a74IdqttL1dvI7MDNok2PHy2prclQp8NYI8wIZiG6S3cgQKtt25vnwIvzTG uFg4sEUGutJpVmMHLVNUEm9oiwOGn8MCqMwJFd0URLbZEGlCLwqp+x/9huMcUFDKbN YsTF1t2QtZS6+PhlzfehG50keDIAtZ4d1EvVCx7jks43PplnT93GiRe8BENGMmgJDd H8AbXSuOH0GEBeP+77uPhLSRJjq43GXTSoOL2fHD0A1vJ9bK0peT/ahRoVrsap6UBF VYs5/IwJ2Cteh2ZE8k/lDPnmKgtirCyDFXkHB0keDoMcUZljWdU9sjr7lG6Ee10SZX XHZPc1v/7PPsg== Date: Mon, 29 Jul 2024 18:14:55 -0700 Subject: [PATCH 2/6] xfs_scrub_all: remove journalctl background process From: "Darrick J. Wong" To: djwong@kernel.org, cem@kernel.org Cc: Christoph Hellwig , linux-xfs@vger.kernel.org Message-ID: <172229849276.1350165.9133033700785877949.stgit@frogsfrogsfrogs> In-Reply-To: <172229849240.1350165.13200329618269649031.stgit@frogsfrogsfrogs> References: <172229849240.1350165.13200329618269649031.stgit@frogsfrogsfrogs> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-xfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Darrick J. Wong Now that we only start systemd services if we're running in service mode, there's no need for the background journalctl process that only ran if we had started systemd services in non-service mode. Signed-off-by: Darrick J. Wong Reviewed-by: Christoph Hellwig --- scrub/xfs_scrub_all.in | 14 -------------- 1 file changed, 14 deletions(-) diff --git a/scrub/xfs_scrub_all.in b/scrub/xfs_scrub_all.in index f27251fa5..fc7a2e637 100644 --- a/scrub/xfs_scrub_all.in +++ b/scrub/xfs_scrub_all.in @@ -261,17 +261,6 @@ def main(): fs = find_mounts() - # Tail the journal if we ourselves aren't a service... - journalthread = None - if 'SERVICE_MODE' not in os.environ: - try: - cmd=['journalctl', '--no-pager', '-q', '-S', 'now', \ - '-f', '-u', 'xfs_scrub@*', '-o', \ - 'cat'] - journalthread = subprocess.Popen(cmd) - except: - pass - # Schedule scrub jobs... running_devs = set() killfuncs = set() @@ -308,9 +297,6 @@ def main(): while len(killfuncs) > 0: wait_for_termination(cond, killfuncs) - if journalthread is not None: - journalthread.terminate() - # See the service mode comments in xfs_scrub.c for why we do this. if 'SERVICE_MODE' in os.environ: time.sleep(2) From patchwork Tue Jul 30 01:15:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13746140 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7BEE310F9 for ; Tue, 30 Jul 2024 01:15:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722302112; cv=none; b=R+wGdCOLK3spB/1ov/fMmRBkBgbwO6qWI5+d5jKFNEIGobo0B2zhe78PpR2i2/yv6eSipjBbadnjnv+hTaC94gi17xjSrVXkMs6+Thjz1uxBj+d/zHlsjGAPr8snpsIqJYKGkDmmFS/6SrqpQJfEiLTBD9SO5rksi6A21MqhoZM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722302112; c=relaxed/simple; bh=KUxKz/Lq5pRVQQLoL4Zxq78HJXTDSvI77lLaoPK1jWM=; h=Date:Subject:From:To:Cc:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=D/nUTFLXDZjjgKwh4tA5DQMdCFHR3KLWrSUyZ7nPdc9yU7nyXO6zH40psNkg/9tUw3GRIsy4rwixrhtbQeqLZK/MqyUefr+z+sO3WecauVriuHb/5OTSe0eYMNc/52VWK67LJqqWuFfYFISaSBRXmaJLi/SaYveCqD8xOa3oDd8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=t8rTZzGo; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="t8rTZzGo" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 12690C32786; Tue, 30 Jul 2024 01:15:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1722302112; bh=KUxKz/Lq5pRVQQLoL4Zxq78HJXTDSvI77lLaoPK1jWM=; h=Date:Subject:From:To:Cc:In-Reply-To:References:From; b=t8rTZzGoz/V5qT89uI0kScU4rPAsECi/Ne7e+Jpwurn2xe5uydl2hd7l7lsLUTBSP MOrA4QL/awMEgfWTWgackvTCX0GWGswTVWEALeHAzkFYhlih9RVNdqdrQTUXaXHSIa wKMdcGsIZb7qWzgQY22s8Ap/aBPvUfN8dylRkly1FdXqZ2OXxjCUZaJCjCVfJbmXJV wwJafimZs3v4sO6TdmVauDLwynFixJSov5ra2VbV2NzA2kn9/QlDAEFYaONIoUxUCe qEX3kiOhcNLjWSr0fWQysScgX9jJhU/kB81dd01ATHqmbZXV7m4X+aenF3WKXziClS vvMWFrdaX/DHQ== Date: Mon, 29 Jul 2024 18:15:11 -0700 Subject: [PATCH 3/6] xfs_scrub_all: support metadata+media scans of all filesystems From: "Darrick J. Wong" To: djwong@kernel.org, cem@kernel.org Cc: Christoph Hellwig , linux-xfs@vger.kernel.org Message-ID: <172229849292.1350165.4310944468823799093.stgit@frogsfrogsfrogs> In-Reply-To: <172229849240.1350165.13200329618269649031.stgit@frogsfrogsfrogs> References: <172229849240.1350165.13200329618269649031.stgit@frogsfrogsfrogs> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-xfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Darrick J. Wong Add the necessary systemd services and control bits so that xfs_scrub_all can kick off a metadata+media scan of a filesystem. Signed-off-by: Darrick J. Wong Reviewed-by: Christoph Hellwig --- man/man8/xfs_scrub_all.8 | 5 +- scrub/Makefile | 4 + scrub/xfs_scrub_all.in | 27 +++++++-- scrub/xfs_scrub_fail.in | 13 +++- scrub/xfs_scrub_fail@.service.in | 2 - scrub/xfs_scrub_media@.service.in | 100 ++++++++++++++++++++++++++++++++ scrub/xfs_scrub_media_fail@.service.in | 76 ++++++++++++++++++++++++ 7 files changed, 214 insertions(+), 13 deletions(-) create mode 100644 scrub/xfs_scrub_media@.service.in create mode 100644 scrub/xfs_scrub_media_fail@.service.in diff --git a/man/man8/xfs_scrub_all.8 b/man/man8/xfs_scrub_all.8 index 74548802e..86a9b3ece 100644 --- a/man/man8/xfs_scrub_all.8 +++ b/man/man8/xfs_scrub_all.8 @@ -4,7 +4,7 @@ xfs_scrub_all \- scrub all mounted XFS filesystems .SH SYNOPSIS .B xfs_scrub_all [ -.B \-hV +.B \-hxV ] .SH DESCRIPTION .B xfs_scrub_all @@ -21,6 +21,9 @@ the same device simultaneously. .B \-h Display help. .TP +.B \-x +Read all file data extents to look for disk errors. +.TP .B \-V Prints the version number and exits. .SH EXIT CODE diff --git a/scrub/Makefile b/scrub/Makefile index 2050fe28f..5567ec061 100644 --- a/scrub/Makefile +++ b/scrub/Makefile @@ -9,6 +9,7 @@ include $(builddefs) SCRUB_PREREQS=$(HAVE_GETFSMAP) scrub_svcname=xfs_scrub@.service +scrub_media_svcname=xfs_scrub_media@.service ifeq ($(SCRUB_PREREQS),yes) LTCOMMAND = xfs_scrub @@ -22,6 +23,8 @@ INSTALL_SCRUB += install-systemd SYSTEMD_SERVICES=\ $(scrub_svcname) \ xfs_scrub_fail@.service \ + $(scrub_media_svcname) \ + xfs_scrub_media_fail@.service \ xfs_scrub_all.service \ xfs_scrub_all.timer \ system-xfs_scrub.slice @@ -113,6 +116,7 @@ xfs_scrub_all: xfs_scrub_all.in $(builddefs) @echo " [SED] $@" $(Q)$(SED) -e "s|@sbindir@|$(PKG_SBIN_DIR)|g" \ -e "s|@scrub_svcname@|$(scrub_svcname)|g" \ + -e "s|@scrub_media_svcname@|$(scrub_media_svcname)|g" \ -e "s|@pkg_version@|$(PKG_VERSION)|g" \ -e "s|@scrub_service_args@|$(XFS_SCRUB_SERVICE_ARGS)|g" \ -e "s|@scrub_args@|$(XFS_SCRUB_ARGS)|g" < $< > $@ diff --git a/scrub/xfs_scrub_all.in b/scrub/xfs_scrub_all.in index fc7a2e637..c1fd10986 100644 --- a/scrub/xfs_scrub_all.in +++ b/scrub/xfs_scrub_all.in @@ -19,6 +19,7 @@ from io import TextIOWrapper retcode = 0 terminate = False +scrub_media = False def DEVNULL(): '''Return /dev/null in subprocess writable format.''' @@ -88,11 +89,15 @@ def run_killable(cmd, stdout, killfuncs): # systemd doesn't like unit instance names with slashes in them, so it # replaces them with dashes when it invokes the service. Filesystem paths # need a special --path argument so that dashes do not get mangled. -def path_to_serviceunit(path): +def path_to_serviceunit(path, scrub_media): '''Convert a pathname into a systemd service unit name.''' - cmd = ['systemd-escape', '--template', '@scrub_svcname@', - '--path', path] + if scrub_media: + svcname = '@scrub_media_svcname@' + else: + svcname = '@scrub_svcname@' + cmd = ['systemd-escape', '--template', svcname, '--path', path] + try: proc = subprocess.Popen(cmd, stdout = subprocess.PIPE) proc.wait() @@ -153,7 +158,9 @@ def systemctl_start(unitname, killfuncs): def run_scrub(mnt, cond, running_devs, mntdevs, killfuncs): '''Run a scrub process.''' - global retcode, terminate + global retcode + global terminate + global scrub_media print("Scrubbing %s..." % mnt) sys.stdout.flush() @@ -164,7 +171,7 @@ def run_scrub(mnt, cond, running_devs, mntdevs, killfuncs): # Run per-mount systemd xfs_scrub service only if we ourselves # are running as a systemd service. - unitname = path_to_serviceunit(path) + unitname = path_to_serviceunit(path, scrub_media) if unitname is not None and 'SERVICE_MODE' in os.environ: ret = systemctl_start(unitname, killfuncs) if ret == 0 or ret == 1: @@ -183,6 +190,8 @@ def run_scrub(mnt, cond, running_devs, mntdevs, killfuncs): if 'SERVICE_MODE' in os.environ: cmd += '@scrub_service_args@'.split() cmd += '@scrub_args@'.split() + if scrub_media: + cmd += '-x' cmd += [mnt] ret = run_killable(cmd, None, killfuncs) if ret >= 0: @@ -247,18 +256,24 @@ def main(): a = (mnt, cond, running_devs, devs, killfuncs) thr = threading.Thread(target = run_scrub, args = a) thr.start() - global retcode, terminate + global retcode + global terminate + global scrub_media parser = argparse.ArgumentParser( \ description = "Scrub all mounted XFS filesystems.") parser.add_argument("-V", help = "Report version and exit.", \ action = "store_true") + parser.add_argument("-x", help = "Scrub file data after filesystem metadata.", \ + action = "store_true") args = parser.parse_args() if args.V: print("xfs_scrub_all version @pkg_version@") sys.exit(0) + scrub_media = args.x + fs = find_mounts() # Schedule scrub jobs... diff --git a/scrub/xfs_scrub_fail.in b/scrub/xfs_scrub_fail.in index 9437b0327..e420917f6 100755 --- a/scrub/xfs_scrub_fail.in +++ b/scrub/xfs_scrub_fail.in @@ -9,8 +9,11 @@ recipient="$1" test -z "${recipient}" && exit 0 -mntpoint="$2" +service="$2" +test -z "${service}" && exit 0 +mntpoint="$3" test -z "${mntpoint}" && exit 0 + hostname="$(hostname -f 2>/dev/null)" test -z "${hostname}" && hostname="${HOSTNAME}" @@ -21,16 +24,16 @@ if [ ! -x "${mailer}" ]; then fi # Turn the mountpoint into a properly escaped systemd instance name -scrub_svc="$(systemd-escape --template "@scrub_svcname@" --path "${mntpoint}")" +scrub_svc="$(systemd-escape --template "${service}@.service" --path "${mntpoint}")" (cat << ENDL To: $1 -From: -Subject: xfs_scrub failure on ${mntpoint} +From: <${service}@${hostname}> +Subject: ${service} failure on ${mntpoint} Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=UTF-8 -So sorry, the automatic xfs_scrub of ${mntpoint} on ${hostname} failed. +So sorry, the automatic ${service} of ${mntpoint} on ${hostname} failed. Please do not reply to this mesage. A log of what happened follows: diff --git a/scrub/xfs_scrub_fail@.service.in b/scrub/xfs_scrub_fail@.service.in index 2c879afd6..16077888d 100644 --- a/scrub/xfs_scrub_fail@.service.in +++ b/scrub/xfs_scrub_fail@.service.in @@ -10,7 +10,7 @@ Documentation=man:xfs_scrub(8) [Service] Type=oneshot Environment=EMAIL_ADDR=root -ExecStart=@pkg_libexec_dir@/xfs_scrub_fail "${EMAIL_ADDR}" %f +ExecStart=@pkg_libexec_dir@/xfs_scrub_fail "${EMAIL_ADDR}" xfs_scrub %f User=mail Group=mail SupplementaryGroups=systemd-journal diff --git a/scrub/xfs_scrub_media@.service.in b/scrub/xfs_scrub_media@.service.in new file mode 100644 index 000000000..e670748ce --- /dev/null +++ b/scrub/xfs_scrub_media@.service.in @@ -0,0 +1,100 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Copyright (c) 2018-2024 Oracle. All Rights Reserved. +# Author: Darrick J. Wong + +[Unit] +Description=Online XFS Metadata and Media Check for %f +OnFailure=xfs_scrub_media_fail@%i.service +Documentation=man:xfs_scrub(8) + +# Explicitly require the capabilities that this program needs +ConditionCapability=CAP_SYS_ADMIN +ConditionCapability=CAP_FOWNER +ConditionCapability=CAP_DAC_OVERRIDE +ConditionCapability=CAP_DAC_READ_SEARCH +ConditionCapability=CAP_SYS_RAWIO + +# Must be a mountpoint +ConditionPathIsMountPoint=%f +RequiresMountsFor=%f + +[Service] +Type=oneshot +Environment=SERVICE_MODE=1 +ExecStart=@sbindir@/xfs_scrub @scrub_service_args@ @scrub_args@ -M /tmp/scrub/ -x %f +SyslogIdentifier=%N + +# Run scrub with minimal CPU and IO priority so that nothing else will starve. +IOSchedulingClass=idle +CPUSchedulingPolicy=idle +CPUAccounting=true +Nice=19 + +# Create the service underneath the scrub background service slice so that we +# can control resource usage. +Slice=system-xfs_scrub.slice + +# No realtime CPU scheduling +RestrictRealtime=true + +# Dynamically create a user that isn't root +DynamicUser=true + +# Make the entire filesystem readonly and /home inaccessible, then bind mount +# the filesystem we're supposed to be checking into our private /tmp dir. +# 'norbind' means that we don't bind anything under that original mount. +ProtectSystem=strict +ProtectHome=yes +PrivateTmp=true +BindPaths=%f:/tmp/scrub:norbind + +# Don't let scrub complain about paths in /etc/projects that have been hidden +# by our sandboxing. scrub doesn't care about project ids anyway. +InaccessiblePaths=-/etc/projects + +# No network access +PrivateNetwork=true +ProtectHostname=true +RestrictAddressFamilies=none +IPAddressDeny=any + +# Don't let the program mess with the kernel configuration at all +ProtectKernelLogs=true +ProtectKernelModules=true +ProtectKernelTunables=true +ProtectControlGroups=true +ProtectProc=invisible +RestrictNamespaces=true + +# Hide everything in /proc, even /proc/mounts +ProcSubset=pid + +# Only allow the default personality Linux +LockPersonality=true + +# No writable memory pages +MemoryDenyWriteExecute=true + +# Don't let our mounts leak out to the host +PrivateMounts=true + +# Restrict system calls to the native arch and only enough to get things going +SystemCallArchitectures=native +SystemCallFilter=@system-service +SystemCallFilter=~@privileged +SystemCallFilter=~@resources +SystemCallFilter=~@mount + +# xfs_scrub needs these privileges to run, and no others +CapabilityBoundingSet=CAP_SYS_ADMIN CAP_FOWNER CAP_DAC_OVERRIDE CAP_DAC_READ_SEARCH CAP_SYS_RAWIO +AmbientCapabilities=CAP_SYS_ADMIN CAP_FOWNER CAP_DAC_OVERRIDE CAP_DAC_READ_SEARCH CAP_SYS_RAWIO +NoNewPrivileges=true + +# xfs_scrub doesn't create files +UMask=7777 + +# No access to hardware /dev files except for block devices +ProtectClock=true +DevicePolicy=closed +DeviceAllow=block-* diff --git a/scrub/xfs_scrub_media_fail@.service.in b/scrub/xfs_scrub_media_fail@.service.in new file mode 100644 index 000000000..97c0e0907 --- /dev/null +++ b/scrub/xfs_scrub_media_fail@.service.in @@ -0,0 +1,76 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Copyright (c) 2018-2024 Oracle. All Rights Reserved. +# Author: Darrick J. Wong + +[Unit] +Description=Online XFS Metadata and Media Check Failure Reporting for %f +Documentation=man:xfs_scrub(8) + +[Service] +Type=oneshot +Environment=EMAIL_ADDR=root +ExecStart=@pkg_libexec_dir@/xfs_scrub_fail "${EMAIL_ADDR}" xfs_scrub_media %f +User=mail +Group=mail +SupplementaryGroups=systemd-journal + +# Create the service underneath the scrub background service slice so that we +# can control resource usage. +Slice=system-xfs_scrub.slice + +# No realtime scheduling +RestrictRealtime=true + +# Make the entire filesystem readonly and /home inaccessible, then bind mount +# the filesystem we're supposed to be checking into our private /tmp dir. +ProtectSystem=full +ProtectHome=yes +PrivateTmp=true +RestrictSUIDSGID=true + +# Emailing reports requires network access, but not the ability to change the +# hostname. +ProtectHostname=true + +# Don't let the program mess with the kernel configuration at all +ProtectKernelLogs=true +ProtectKernelModules=true +ProtectKernelTunables=true +ProtectControlGroups=true +ProtectProc=invisible +RestrictNamespaces=true + +# Can't hide /proc because journalctl needs it to find various pieces of log +# information +#ProcSubset=pid + +# Only allow the default personality Linux +LockPersonality=true + +# No writable memory pages +MemoryDenyWriteExecute=true + +# Don't let our mounts leak out to the host +PrivateMounts=true + +# Restrict system calls to the native arch and only enough to get things going +SystemCallArchitectures=native +SystemCallFilter=@system-service +SystemCallFilter=~@privileged +SystemCallFilter=~@resources +SystemCallFilter=~@mount + +# xfs_scrub needs these privileges to run, and no others +CapabilityBoundingSet= +NoNewPrivileges=true + +# Failure reporting shouldn't create world-readable files +UMask=0077 + +# Clean up any IPC objects when this unit stops +RemoveIPC=true + +# No access to hardware device files +PrivateDevices=true +ProtectClock=true From patchwork Tue Jul 30 01:15:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13746141 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E0C5510F7 for ; Tue, 30 Jul 2024 01:15:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722302128; cv=none; b=YCbnbSJzTIqxFvm0QahnmlE+WgYA72Cqb/dE2O1q2nLDZ4/nMK3XsCbi/++0lSVAkh/F0zus7oGF051br55H4Hcn3oSgvtbceOPHjxUhUkVrteN3dSdo/CCNmeXEz1V+JZbz7WCruPA/RCfZtQKWLNh67wp7vLColvzqEuhuT4o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722302128; c=relaxed/simple; bh=iJpeO/pIiPIgxJyaNwu6OBED8hiD1CldNAX8kazPE5Q=; h=Date:Subject:From:To:Cc:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=pFV81zL4SUpQHc790pEuXKhcIL1Pij/k2qr55alZfenRCP9kX+dqeEmFBFgix3609pmpeYi+jsgFOW/0GAuA1Ks8X+g4BYmxC4bifAGfNf2aQZ4PXUAhIe5kw7vOB19qVvmIrKioEap3t40sEto7xKKxSniNlDRUW4zqpBjSAC8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=j798rMaX; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="j798rMaX" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B2072C32786; Tue, 30 Jul 2024 01:15:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1722302127; bh=iJpeO/pIiPIgxJyaNwu6OBED8hiD1CldNAX8kazPE5Q=; h=Date:Subject:From:To:Cc:In-Reply-To:References:From; b=j798rMaXWQhbfzcKliZdpcRN3QCPcHI4TgGd8IVpeNI1uhxc5fsDOuwdo+nDRCu9j FdT/90qT5ohS6nDyi4sk3DoG8kNQOvS5aeg5FvOEP1RT7BerY3FHaIRyKWHPdIf6Tw BcxQ1pn99LeyVZ0FFrTPJ4LguYfuSUna6KtgK6QahwjL40z8xrwBogwy/SqC20ZAQK DjgUYOrlNQAagTB1YsjbPSMz5S8tDfN9JDm0qaCrAlelhd/PvHXzV6N0dJ0c6JwloT ArOgKnTUXQeLBUE5R0w3s3HBHYfDHYSsm+FpVgLJ6R6wpdU4YONjVtTbyDQKrOwF1F w5MHWMoZotZZA== Date: Mon, 29 Jul 2024 18:15:27 -0700 Subject: [PATCH 4/6] xfs_scrub_all: enable periodic file data scrubs automatically From: "Darrick J. Wong" To: djwong@kernel.org, cem@kernel.org Cc: Christoph Hellwig , linux-xfs@vger.kernel.org Message-ID: <172229849307.1350165.8381886957483292431.stgit@frogsfrogsfrogs> In-Reply-To: <172229849240.1350165.13200329618269649031.stgit@frogsfrogsfrogs> References: <172229849240.1350165.13200329618269649031.stgit@frogsfrogsfrogs> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-xfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Darrick J. Wong Enhance xfs_scrub_all with the ability to initiate a file data scrub periodically. The user must specify the period, and they may optionally specify the path to a file that will record the last time the file data was scrubbed. Signed-off-by: Darrick J. Wong Reviewed-by: Christoph Hellwig --- debian/rules | 3 +- include/builddefs.in | 3 ++ man/man8/Makefile | 7 +++- man/man8/xfs_scrub_all.8.in | 15 ++++++++ scrub/Makefile | 3 ++ scrub/xfs_scrub_all.in | 76 +++++++++++++++++++++++++++++++++++++++- scrub/xfs_scrub_all.service.in | 6 ++- 7 files changed, 108 insertions(+), 5 deletions(-) rename man/man8/{xfs_scrub_all.8 => xfs_scrub_all.8.in} (63%) diff --git a/debian/rules b/debian/rules index 0db0ed8e7..69a79fc67 100755 --- a/debian/rules +++ b/debian/rules @@ -38,7 +38,8 @@ configure_options = \ --disable-ubsan \ --disable-addrsan \ --disable-threadsan \ - --enable-lto + --enable-lto \ + --localstatedir=/var options = export DEBUG=-DNDEBUG DISTRIBUTION=debian \ INSTALL_USER=root INSTALL_GROUP=root \ diff --git a/include/builddefs.in b/include/builddefs.in index 6ac36c149..734bd95ec 100644 --- a/include/builddefs.in +++ b/include/builddefs.in @@ -57,6 +57,9 @@ PKG_DOC_DIR = @datadir@/doc/@pkg_name@ PKG_LOCALE_DIR = @datadir@/locale PKG_DATA_DIR = @datadir@/@pkg_name@ MKFS_CFG_DIR = @datadir@/@pkg_name@/mkfs +PKG_STATE_DIR = @localstatedir@/lib/@pkg_name@ + +XFS_SCRUB_ALL_AUTO_MEDIA_SCAN_STAMP=$(PKG_STATE_DIR)/xfs_scrub_all_media.stamp CC = @cc@ BUILD_CC = @BUILD_CC@ diff --git a/man/man8/Makefile b/man/man8/Makefile index 272e45aeb..5be76ab72 100644 --- a/man/man8/Makefile +++ b/man/man8/Makefile @@ -11,11 +11,12 @@ ifneq ("$(ENABLE_SCRUB)","yes") MAN_PAGES = $(filter-out xfs_scrub%,$(shell echo *.$(MAN_SECTION))) else MAN_PAGES = $(shell echo *.$(MAN_SECTION)) + MAN_PAGES += xfs_scrub_all.8 endif MAN_PAGES += mkfs.xfs.8 MAN_DEST = $(PKG_MAN_DIR)/man$(MAN_SECTION) LSRCFILES = $(MAN_PAGES) -DIRT = mkfs.xfs.8 +DIRT = mkfs.xfs.8 xfs_scrub_all.8 default : $(MAN_PAGES) @@ -29,4 +30,8 @@ mkfs.xfs.8: mkfs.xfs.8.in @echo " [SED] $@" $(Q)$(SED) -e 's|@mkfs_cfg_dir@|$(MKFS_CFG_DIR)|g' < $^ > $@ +xfs_scrub_all.8: xfs_scrub_all.8.in + @echo " [SED] $@" + $(Q)$(SED) -e 's|@stampfile@|$(XFS_SCRUB_ALL_AUTO_MEDIA_SCAN_STAMP)|g' < $^ > $@ + install-dev : diff --git a/man/man8/xfs_scrub_all.8 b/man/man8/xfs_scrub_all.8.in similarity index 63% rename from man/man8/xfs_scrub_all.8 rename to man/man8/xfs_scrub_all.8.in index 86a9b3ece..0aa87e237 100644 --- a/man/man8/xfs_scrub_all.8 +++ b/man/man8/xfs_scrub_all.8.in @@ -18,6 +18,21 @@ operations can be run in parallel so long as no two scrubbers access the same device simultaneously. .SH OPTIONS .TP +.B \--auto-media-scan-interval +Automatically enable the file data scan (i.e. the +.B -x +flag) if it has not been run in the specified interval. +The interval must be a floating point number with an optional unit suffix. +Supported unit suffixes are +.IR y ", " q ", " mo ", " w ", " d ", " h ", " m ", and " s +for years, 90-day quarters, 30-day months, weeks, days, hours, minutes, and +seconds, respectively. +If no units are specified, the default is seconds. +.TP +.B \--auto-media-scan-stamp +Path to a file that will record the last time the media scan was run. +Defaults to @stampfile@. +.TP .B \-h Display help. .TP diff --git a/scrub/Makefile b/scrub/Makefile index 5567ec061..091a5c94b 100644 --- a/scrub/Makefile +++ b/scrub/Makefile @@ -118,6 +118,7 @@ xfs_scrub_all: xfs_scrub_all.in $(builddefs) -e "s|@scrub_svcname@|$(scrub_svcname)|g" \ -e "s|@scrub_media_svcname@|$(scrub_media_svcname)|g" \ -e "s|@pkg_version@|$(PKG_VERSION)|g" \ + -e "s|@stampfile@|$(XFS_SCRUB_ALL_AUTO_MEDIA_SCAN_STAMP)|g" \ -e "s|@scrub_service_args@|$(XFS_SCRUB_SERVICE_ARGS)|g" \ -e "s|@scrub_args@|$(XFS_SCRUB_ARGS)|g" < $< > $@ $(Q)chmod a+x $@ @@ -141,6 +142,7 @@ install: $(INSTALL_SCRUB) -e "s|@scrub_service_args@|$(XFS_SCRUB_SERVICE_ARGS)|g" \ -e "s|@scrub_args@|$(XFS_SCRUB_ARGS)|g" \ -e "s|@pkg_libexec_dir@|$(PKG_LIBEXEC_DIR)|g" \ + -e "s|@pkg_state_dir@|$(PKG_STATE_DIR)|g" \ < $< > $@ %.cron: %.cron.in $(builddefs) @@ -161,6 +163,7 @@ install-scrub: default $(INSTALL) -m 755 -d $(PKG_SBIN_DIR) $(LTINSTALL) -m 755 $(LTCOMMAND) $(PKG_SBIN_DIR) $(INSTALL) -m 755 $(XFS_SCRUB_ALL_PROG) $(PKG_SBIN_DIR) + $(INSTALL) -m 755 -d $(PKG_STATE_DIR) install-udev: $(UDEV_RULES) $(INSTALL) -m 755 -d $(UDEV_RULE_DIR) diff --git a/scrub/xfs_scrub_all.in b/scrub/xfs_scrub_all.in index c1fd10986..9dd6347fb 100644 --- a/scrub/xfs_scrub_all.in +++ b/scrub/xfs_scrub_all.in @@ -16,6 +16,10 @@ import os import argparse import signal from io import TextIOWrapper +from pathlib import Path +from datetime import timedelta +from datetime import datetime +from datetime import timezone retcode = 0 terminate = False @@ -250,6 +254,65 @@ def wait_for_termination(cond, killfuncs): fn() return True +def scan_interval(string): + '''Convert a textual scan interval argument into a time delta.''' + + if string.endswith('y'): + year = timedelta(seconds = 31556952) + return year * float(string[:-1]) + if string.endswith('q'): + return timedelta(days = 90 * float(string[:-1])) + if string.endswith('mo'): + return timedelta(days = 30 * float(string[:-2])) + if string.endswith('w'): + return timedelta(weeks = float(string[:-1])) + if string.endswith('d'): + return timedelta(days = float(string[:-1])) + if string.endswith('h'): + return timedelta(hours = float(string[:-1])) + if string.endswith('m'): + return timedelta(minutes = float(string[:-1])) + if string.endswith('s'): + return timedelta(seconds = float(string[:-1])) + return timedelta(seconds = int(string)) + +def utcnow(): + '''Create a representation of the time right now, in UTC.''' + + dt = datetime.utcnow() + return dt.replace(tzinfo = timezone.utc) + +def enable_automatic_media_scan(args): + '''Decide if we enable media scanning automatically.''' + already_enabled = args.x + + try: + interval = scan_interval(args.auto_media_scan_interval) + except Exception as e: + raise Exception('%s: Invalid media scan interval.' % \ + args.auto_media_scan_interval) + + p = Path(args.auto_media_scan_stamp) + if already_enabled: + res = True + else: + try: + last_run = p.stat().st_mtime + now = utcnow().timestamp() + res = last_run + interval.total_seconds() < now + except FileNotFoundError: + res = True + + if res: + # Truncate the stamp file to update its mtime + with p.open('w') as f: + pass + if not already_enabled: + print('Automatically enabling file data scrub.') + sys.stdout.flush() + + return res + def main(): '''Find mounts, schedule scrub runs.''' def thr(mnt, devs): @@ -266,13 +329,24 @@ def main(): action = "store_true") parser.add_argument("-x", help = "Scrub file data after filesystem metadata.", \ action = "store_true") + parser.add_argument("--auto-media-scan-interval", help = "Automatically scrub file data at this interval.", \ + default = None) + parser.add_argument("--auto-media-scan-stamp", help = "Stamp file for automatic file data scrub.", \ + default = '@stampfile@') args = parser.parse_args() if args.V: print("xfs_scrub_all version @pkg_version@") sys.exit(0) - scrub_media = args.x + if args.auto_media_scan_interval is not None: + try: + scrub_media = enable_automatic_media_scan(args) + except Exception as e: + print(e) + sys.exit(16) + else: + scrub_media = args.x fs = find_mounts() diff --git a/scrub/xfs_scrub_all.service.in b/scrub/xfs_scrub_all.service.in index 478cd8d05..9ecc3af0c 100644 --- a/scrub/xfs_scrub_all.service.in +++ b/scrub/xfs_scrub_all.service.in @@ -34,11 +34,13 @@ CapabilityBoundingSet= NoNewPrivileges=true RestrictSUIDSGID=true -# Make the entire filesystem readonly. We don't want to hide anything because -# we need to find all mounted XFS filesystems in the host. +# Make the entire filesystem readonly except for the media scan stamp file +# directory. We don't want to hide anything because we need to find all +# mounted XFS filesystems in the host. ProtectSystem=strict ProtectHome=read-only PrivateTmp=false +BindPaths=@pkg_state_dir@ # No network access except to the systemd control socket PrivateNetwork=true From patchwork Tue Jul 30 01:15:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13746142 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D354628EB for ; Tue, 30 Jul 2024 01:15:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722302143; cv=none; b=m7wOaKkxZlY1Ccx95upSLK0yVjPNaDlfA39lT0BheHx104oczO+dEyObtQyatSYVKhU7TTHIWjzHoMFqK2TWGr2DkDC087wkQKZTBNLDDG6MTrPcZQkrX2Kc8U2pDE8GhqIn192NRZZKNRvTabq7sln327YOvdKfAOMbaWrEJEA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722302143; c=relaxed/simple; bh=PoKvsvQx3iu+GW6wML8A7+WLfUMfZEr/9Rfh3UYJcyU=; h=Date:Subject:From:To:Cc:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=KfCOIFSxnyt0x2JxrAOnOusp1gSBkHRvajDlbN3ICd3s7WC8DbAYB5HSEyNDzenem+nv2ZR7Tj/UyFmIw5t4VCN/DZRuuTSn1uUn6jxVsp6me9ghGuf5piK6MfhWfBPynLmJzheF2cnLg9k8a6WkIaKNqtNG5ouSD1aGqC75K2Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=evdSIUWS; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="evdSIUWS" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5FE66C32786; Tue, 30 Jul 2024 01:15:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1722302143; bh=PoKvsvQx3iu+GW6wML8A7+WLfUMfZEr/9Rfh3UYJcyU=; h=Date:Subject:From:To:Cc:In-Reply-To:References:From; b=evdSIUWS1JAugRnKbiaD9Tvv89NKu0v+UCo2ptkz4ALeJNoBcbNZ2b/hAaEY8xxu2 u5wSMiocZyWcBIEBHVnkJDJlbjurNdgIc6kPCz0ScULTBMIoXuNnWxWgXvUIzCk6A+ XgP2+owLNtjHI44uL3X1YynToY62bqMTvY8Qfo8L+2JSgJq31ldbfQZlJHr6C57wH9 wx1UnI5BbrnoM0Qiw8Z4k5fSRnt/sU6TkaXXcCovZveZxlnCeixm6uhsehejmMcbZS k58YNf4n9iB+wMLrEK2vV2beuxnMnS2lggV6XKdqEqh5TcmPFZVhjG89Jtigpeptvk wuy5cblP+vaCA== Date: Mon, 29 Jul 2024 18:15:42 -0700 Subject: [PATCH 5/6] xfs_scrub_all: trigger automatic media scans once per month From: "Darrick J. Wong" To: djwong@kernel.org, cem@kernel.org Cc: Christoph Hellwig , linux-xfs@vger.kernel.org Message-ID: <172229849322.1350165.1379324570551390512.stgit@frogsfrogsfrogs> In-Reply-To: <172229849240.1350165.13200329618269649031.stgit@frogsfrogsfrogs> References: <172229849240.1350165.13200329618269649031.stgit@frogsfrogsfrogs> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-xfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Darrick J. Wong Teach the xfs_scrub_all background service to trigger an automatic scan of all file data once per month. Signed-off-by: Darrick J. Wong Reviewed-by: Christoph Hellwig --- scrub/Makefile | 8 +++++++- scrub/xfs_scrub_all.cron.in | 2 +- scrub/xfs_scrub_all.service.in | 2 +- 3 files changed, 9 insertions(+), 3 deletions(-) diff --git a/scrub/Makefile b/scrub/Makefile index 091a5c94b..0e09ed127 100644 --- a/scrub/Makefile +++ b/scrub/Makefile @@ -108,6 +108,9 @@ CFILES += unicrash.c LCFLAGS += -DHAVE_LIBICU $(LIBICU_CFLAGS) endif +# Automatically trigger a media scan once per month +XFS_SCRUB_ALL_AUTO_MEDIA_SCAN_INTERVAL=1mo + LDIRT = $(XFS_SCRUB_ALL_PROG) $(XFS_SCRUB_FAIL_PROG) *.service *.cron default: depend $(LTCOMMAND) $(XFS_SCRUB_ALL_PROG) $(XFS_SCRUB_FAIL_PROG) $(OPTIONAL_TARGETS) @@ -143,11 +146,14 @@ install: $(INSTALL_SCRUB) -e "s|@scrub_args@|$(XFS_SCRUB_ARGS)|g" \ -e "s|@pkg_libexec_dir@|$(PKG_LIBEXEC_DIR)|g" \ -e "s|@pkg_state_dir@|$(PKG_STATE_DIR)|g" \ + -e "s|@media_scan_interval@|$(XFS_SCRUB_ALL_AUTO_MEDIA_SCAN_INTERVAL)|g" \ < $< > $@ %.cron: %.cron.in $(builddefs) @echo " [SED] $@" - $(Q)$(SED) -e "s|@sbindir@|$(PKG_SBIN_DIR)|g" < $< > $@ + $(Q)$(SED) -e "s|@sbindir@|$(PKG_SBIN_DIR)|g" \ + -e "s|@media_scan_interval@|$(XFS_SCRUB_ALL_AUTO_MEDIA_SCAN_INTERVAL)|g" \ + < $< > $@ install-systemd: default $(SYSTEMD_SERVICES) $(INSTALL) -m 755 -d $(SYSTEMD_SYSTEM_UNIT_DIR) diff --git a/scrub/xfs_scrub_all.cron.in b/scrub/xfs_scrub_all.cron.in index e08a9daf9..e6633e091 100644 --- a/scrub/xfs_scrub_all.cron.in +++ b/scrub/xfs_scrub_all.cron.in @@ -3,4 +3,4 @@ # Copyright (C) 2018-2024 Oracle. All Rights Reserved. # Author: Darrick J. Wong # -10 3 * * 0 root test -e /run/systemd/system || @sbindir@/xfs_scrub_all +10 3 * * 0 root test -e /run/systemd/system || @sbindir@/xfs_scrub_all --auto-media-scan-interval @media_scan_interval@ diff --git a/scrub/xfs_scrub_all.service.in b/scrub/xfs_scrub_all.service.in index 9ecc3af0c..8ed682989 100644 --- a/scrub/xfs_scrub_all.service.in +++ b/scrub/xfs_scrub_all.service.in @@ -12,7 +12,7 @@ After=paths.target multi-user.target network.target network-online.target system [Service] Type=oneshot Environment=SERVICE_MODE=1 -ExecStart=@sbindir@/xfs_scrub_all +ExecStart=@sbindir@/xfs_scrub_all --auto-media-scan-interval @media_scan_interval@ SyslogIdentifier=xfs_scrub_all # Create the service underneath the scrub background service slice so that we From patchwork Tue Jul 30 01:15:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13746150 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6580010F7 for ; Tue, 30 Jul 2024 01:15:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722302159; cv=none; b=hXz2l34qOO6XQW7qjV3KxaT/3sTuF0Qs4OHKb1upsaI/ihuzc7UJ1wsNnHT5h1FYFKcnobKLAkZ7NW5S4ATX1LKePT7VTT+NEpvT2hxsyHQPIJI7bFCLoN+9VjuxqBmwQ0iag0wPXahQP47OApZwO18p7rS7AD1DoB05n05jC1k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722302159; c=relaxed/simple; bh=r7dKhAbCplgUcJAFoZFwoIDAJP6rc0yzpg/vuCm35R8=; h=Date:Subject:From:To:Cc:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=f1ssBAIm7OLlRftjaBmpwpGFjqB5gvzN7TWXhGjBnDPb2jFtZUOA52eJEK+AxSm71l2Or27N52ZmVPwZGMI34ibd2tmVqWHQ8MCodMDaPoGZ/WWqez498qyKqrGf9nykeFuaSM3eUdNDWdro4cIyCv7ThP0NUT/KjXn88RUAvwA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=fWeQmg8C; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="fWeQmg8C" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0AFA9C32786; Tue, 30 Jul 2024 01:15:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1722302159; bh=r7dKhAbCplgUcJAFoZFwoIDAJP6rc0yzpg/vuCm35R8=; h=Date:Subject:From:To:Cc:In-Reply-To:References:From; b=fWeQmg8CYeCSFCoBaPS7lNTZOjBdpF13H39+uDIF4h37FhAzsbHsFyIX2bF4fIzDm HlZ6AgWaPZ1DytUdYD05iNT1aVulNmJf102ywxaeA1N8bfNDHDn9zYVzOisG+6N4mT z4KPJi9fee1ksuFCYjCrFMTrZf+TXMwpiceEAntmOZDnQpXjkfJzPEHBUe4Co/Gzxb V/2rMC2StYKl9LBTwT30zmMpRsc0VnRcLXr/PwSXccAuNIMXn9cVi1kapUk8z/8xir Kx5dVpT/oihnD9G+BUdFUj9h4cBuALQwObmIOdbLChb2tIwsLuwPXIJHbjG98KdVC9 UOgqrv96x8iTQ== Date: Mon, 29 Jul 2024 18:15:58 -0700 Subject: [PATCH 6/6] xfs_scrub_all: failure reporting for the xfs_scrub_all job From: "Darrick J. Wong" To: djwong@kernel.org, cem@kernel.org Cc: Christoph Hellwig , linux-xfs@vger.kernel.org Message-ID: <172229849338.1350165.672580690717705898.stgit@frogsfrogsfrogs> In-Reply-To: <172229849240.1350165.13200329618269649031.stgit@frogsfrogsfrogs> References: <172229849240.1350165.13200329618269649031.stgit@frogsfrogsfrogs> User-Agent: StGit/0.19 Precedence: bulk X-Mailing-List: linux-xfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Darrick J. Wong Create a failure reporting service for when xfs_scrub_all fails. This shouldn't happen often, but let's report anyways. Signed-off-by: Darrick J. Wong Reviewed-by: Christoph Hellwig --- scrub/Makefile | 1 scrub/xfs_scrub_all.service.in | 1 scrub/xfs_scrub_all_fail.service.in | 71 +++++++++++++++++++++++++++++++++++ scrub/xfs_scrub_fail.in | 35 ++++++++++++++--- 4 files changed, 101 insertions(+), 7 deletions(-) create mode 100644 scrub/xfs_scrub_all_fail.service.in diff --git a/scrub/Makefile b/scrub/Makefile index 0e09ed127..7e6882450 100644 --- a/scrub/Makefile +++ b/scrub/Makefile @@ -26,6 +26,7 @@ SYSTEMD_SERVICES=\ $(scrub_media_svcname) \ xfs_scrub_media_fail@.service \ xfs_scrub_all.service \ + xfs_scrub_all_fail.service \ xfs_scrub_all.timer \ system-xfs_scrub.slice OPTIONAL_TARGETS += $(SYSTEMD_SERVICES) diff --git a/scrub/xfs_scrub_all.service.in b/scrub/xfs_scrub_all.service.in index 8ed682989..b86b787d2 100644 --- a/scrub/xfs_scrub_all.service.in +++ b/scrub/xfs_scrub_all.service.in @@ -5,6 +5,7 @@ [Unit] Description=Online XFS Metadata Check for All Filesystems +OnFailure=xfs_scrub_all_fail.service ConditionACPower=true Documentation=man:xfs_scrub_all(8) After=paths.target multi-user.target network.target network-online.target systemd-networkd.service NetworkManager.service connman.service diff --git a/scrub/xfs_scrub_all_fail.service.in b/scrub/xfs_scrub_all_fail.service.in new file mode 100644 index 000000000..53479db84 --- /dev/null +++ b/scrub/xfs_scrub_all_fail.service.in @@ -0,0 +1,71 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Copyright (c) 2018-2024 Oracle. All Rights Reserved. +# Author: Darrick J. Wong + +[Unit] +Description=Online XFS Metadata Check for All Filesystems Failure Reporting +Documentation=man:xfs_scrub_all(8) + +[Service] +Type=oneshot +Environment=EMAIL_ADDR=root +ExecStart=@pkg_libexec_dir@/xfs_scrub_fail "${EMAIL_ADDR}" xfs_scrub_all +User=mail +Group=mail +SupplementaryGroups=systemd-journal + +# No realtime scheduling +RestrictRealtime=true + +# Make the entire filesystem readonly and /home inaccessible. +ProtectSystem=full +ProtectHome=yes +PrivateTmp=true +RestrictSUIDSGID=true + +# Emailing reports requires network access, but not the ability to change the +# hostname. +ProtectHostname=true + +# Don't let the program mess with the kernel configuration at all +ProtectKernelLogs=true +ProtectKernelModules=true +ProtectKernelTunables=true +ProtectControlGroups=true +ProtectProc=invisible +RestrictNamespaces=true + +# Can't hide /proc because journalctl needs it to find various pieces of log +# information +#ProcSubset=pid + +# Only allow the default personality Linux +LockPersonality=true + +# No writable memory pages +MemoryDenyWriteExecute=true + +# Don't let our mounts leak out to the host +PrivateMounts=true + +# Restrict system calls to the native arch and only enough to get things going +SystemCallArchitectures=native +SystemCallFilter=@system-service +SystemCallFilter=~@privileged +SystemCallFilter=~@resources +SystemCallFilter=~@mount + +# xfs_scrub needs these privileges to run, and no others +CapabilityBoundingSet= +NoNewPrivileges=true + +# Failure reporting shouldn't create world-readable files +UMask=0077 + +# Clean up any IPC objects when this unit stops +RemoveIPC=true + +# No access to hardware device files +PrivateDevices=true +ProtectClock=true diff --git a/scrub/xfs_scrub_fail.in b/scrub/xfs_scrub_fail.in index e420917f6..089b438f0 100755 --- a/scrub/xfs_scrub_fail.in +++ b/scrub/xfs_scrub_fail.in @@ -5,14 +5,13 @@ # Copyright (C) 2018-2024 Oracle. All Rights Reserved. # Author: Darrick J. Wong -# Email logs of failed xfs_scrub unit runs +# Email logs of failed xfs_scrub and xfs_scrub_all unit runs recipient="$1" test -z "${recipient}" && exit 0 service="$2" test -z "${service}" && exit 0 mntpoint="$3" -test -z "${mntpoint}" && exit 0 hostname="$(hostname -f 2>/dev/null)" test -z "${hostname}" && hostname="${HOSTNAME}" @@ -23,11 +22,13 @@ if [ ! -x "${mailer}" ]; then exit 1 fi -# Turn the mountpoint into a properly escaped systemd instance name -scrub_svc="$(systemd-escape --template "${service}@.service" --path "${mntpoint}")" +fail_mail_mntpoint() { + local scrub_svc -(cat << ENDL -To: $1 + # Turn the mountpoint into a properly escaped systemd instance name + scrub_svc="$(systemd-escape --template "${service}@.service" --path "${mntpoint}")" + cat << ENDL +To: ${recipient} From: <${service}@${hostname}> Subject: ${service} failure on ${mntpoint} Content-Transfer-Encoding: 8bit @@ -38,5 +39,25 @@ Please do not reply to this mesage. A log of what happened follows: ENDL -systemctl status --full --lines 4294967295 "${scrub_svc}") | "${mailer}" -t -i + systemctl status --full --lines 4294967295 "${scrub_svc}" +} + +fail_mail() { + cat << ENDL +To: ${recipient} +From: <${service}@${hostname}> +Subject: ${service} failure + +So sorry, the automatic ${service} on ${hostname} failed. + +A log of what happened follows: +ENDL + systemctl status --full --lines 4294967295 "${service}" +} + +if [ -n "${mntpoint}" ]; then + fail_mail_mntpoint | "${mailer}" -t -i +else + fail_mail | "${mailer}" -t -i +fi exit "${PIPESTATUS[1]}"