From patchwork Wed Apr 12 03:45:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13208393 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AF5B2C7619A for ; Wed, 12 Apr 2023 03:45:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229609AbjDLDpI (ORCPT ); Tue, 11 Apr 2023 23:45:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37836 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229490AbjDLDpH (ORCPT ); Tue, 11 Apr 2023 23:45:07 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2F18E40D7; Tue, 11 Apr 2023 20:45:05 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id C20B36101C; Wed, 12 Apr 2023 03:45:04 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 236ABC433EF; Wed, 12 Apr 2023 03:45:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1681271104; bh=3jTdlLsyChWM8y6Aveuf5s29ciELZbaW/aXxDAg/ROc=; h=Date:Subject:From:To:Cc:From; b=HrcwzEZDay384BSyK/lW278fMMupHDNOmg0Extxl44wG23LftWvmn8RKmJH9kIz60 C2uFQVTKAsjB8I6r9Xm4q5jxnxVvxsqXpSGSNESo39Db6IKy+1Alsn7VgqPyFvHvw/ hH6ZZXpEoWnu0+LiB6iw+fpzX6GAImWErdYcnN5wO0m/pSFkSvPndQZQvja07dBSpF 7JdMkWfw//BiY9N5IUUtS+TEyZPlrMEMbG8OvI8npKOmQDf9Qct8mBPYpHQejgC2AM niSSrOpB5WbNGLCDQGqu/XmmAX5zvcu/h5MD8LZy06BK+yflRS1VOLde1/IpD70xAj 0CUmkKF1FheFw== Date: Tue, 11 Apr 2023 20:45:03 -0700 Subject: [GIT PULL 1/22] xfs: design documentation for online fsck From: "Darrick J. Wong" To: dchinner@fromorbit.com, djwong@kernel.org Cc: allison.henderson@oracle.com, catherine.hoang@oracle.com, chandan.babu@oracle.com, david@fromorbit.com, dchinner@redhat.com, hch@infradead.org, linux-fsdevel@vger.kernel.org, linux-xfs@vger.kernel.org, willy@infradead.org Message-ID: <168127093760.417736.12181322234550374115.stg-ugh@frogsfrogsfrogs> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-xfs@vger.kernel.org Hi Dave, Please pull this branch with changes for xfs. As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems. --D The following changes since commit 09a9639e56c01c7a00d6c0ca63f4c7c41abe075d: Linux 6.3-rc6 (2023-04-09 11:15:57 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/online-fsck-design-6.4_2023-04-11 for you to fetch changes up to 03786f0afb2ed5705a0478e14fea50a7f1a44f7e: xfs: document future directions of online fsck (2023-04-11 18:59:52 -0700) ---------------------------------------------------------------- xfs: design documentation for online fsck [v24.5] After six years of development and a nearly two year hiatus from patchbombing, I think it is time to resume the process of merging the online fsck feature into XFS. The full patchset comprises 105 separate patchsets that capture 470 patches across the kernel, xfsprogs, and fstests projects. I would like to merge this feature into upstream in time for the 2023 LTS kernel. As of 5.15 (aka last year's LTS), we have merged all generally useful infrastructure improvements into the regular filesystem. The only changes to the core filesystem that remain are the ones that are only useful to online fsck itself. In other words, the vast majority of the new code in the patchsets comprising the online fsck feature are is mostly self contained and can be turned off via Kconfig. Many of you readers might be wondering -- why have I chosen to make one large submission with 100+ patchsets comprising ~500 patches? Why didn't I merge small pieces of functionality bit by bit and revise common code as necessary? Well, the simple answer is that in the past six years, the fundamental algorithms have been revised repeatedly as I've built out the functionality. In other words, the codebase as it is now has the benefit that I now know every piece that's necessary to get the job done in a reasonable manner and within the constraints laid out by community reviews. I believe this has reduced code churn in mainline and freed up my time so that I can iterate faster. As a concession to the mail servers, I'm breaking up the submission into smaller pieces; I'm only pushing the design document and the revisions to the existing scrub code, which is the first 20%% of the patches. Also, I'm arbitrarily restarting the version numbering by reversioning all patchsets from version 22 to epoch 23, version 1. The big question to everyone reading this is: How might I convince you that there is more merit in merging the whole feature and dealing with the consequences than continuing to maintain it out of tree? --------- To prepare the XFS community and potential patch reviewers for the upstream submission of the online fsck feature, I decided to write a document capturing the broader picture behind the online repair development effort. The document begins by defining the problems that online fsck aims to solve and outlining specific use cases for the functionality. Using that as a base, the rest of the design document presents the high level algorithms that fulfill the goals set out at the start and the interactions between the large pieces of the system. Case studies round out the design documentation by adding the details of exactly how specific parts of the online fsck code integrate the algorithms with the filesystem. The goal of this effort is to help the XFS community understand how the gigantic online repair patchset works. The questions I submit to the community reviewers are: 1. As you read the design doc (and later the code), do you feel that you understand what's going on well enough to try to fix a bug if you found one? 2. What sorts of interactions between systems (or between scrub and the rest of the kernel) am I missing? 3. Do you feel confident enough in the implementation as it is now that the benefits of merging the feature (as EXPERIMENTAL) outweigh any potential disruptions to XFS at large? 4. Are there problematic interactions between subsystems that ought to be cleared up before merging? 5. Can I just merge all of this? I intend to commit this document to the kernel's documentation directory when we start merging the patchset, albeit without the links to git.kernel.org. A much more readable version of this is posted at: https://djwong.org/docs/xfs-online-fsck-design/ v2: add missing sections about: all the in-kernel data structures and new apis that the scrub and repair functions use; how xattrs and directories are checked; how space btree records are checked; and add more details to the parts where all these bits tie together. Proofread for verb tense inconsistencies and eliminate vague 'we' usage. Move all the discussion of what we can do with pageable kernel memory into a single source file and section. Document where log incompat feature locks fit into the locking model. v3: resync with 6.0, fix a few typos, begin discussion of the merging plan for this megapatchset. Signed-off-by: Darrick J. Wong ---------------------------------------------------------------- Darrick J. Wong (14): xfs: document the motivation for online fsck design xfs: document the general theory underlying online fsck design xfs: document the testing plan for online fsck xfs: document the user interface for online fsck xfs: document the filesystem metadata checking strategy xfs: document how online fsck deals with eventual consistency xfs: document pageable kernel memory xfs: document btree bulk loading xfs: document online file metadata repair code xfs: document full filesystem scans for online fsck xfs: document metadata file repair xfs: document directory tree repairs xfs: document the userspace fsck driver program xfs: document future directions of online fsck Documentation/filesystems/index.rst | 1 + .../filesystems/xfs-online-fsck-design.rst | 5315 ++++++++++++++++++++ .../filesystems/xfs-self-describing-metadata.rst | 1 + 3 files changed, 5317 insertions(+) create mode 100644 Documentation/filesystems/xfs-online-fsck-design.rst From patchwork Wed Apr 12 03:45:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13208394 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6941BC77B6E for ; Wed, 12 Apr 2023 03:45:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229555AbjDLDpW (ORCPT ); Tue, 11 Apr 2023 23:45:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37930 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229490AbjDLDpV (ORCPT ); Tue, 11 Apr 2023 23:45:21 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CD0AF30E0 for ; Tue, 11 Apr 2023 20:45:20 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 67EA76101C for ; Wed, 12 Apr 2023 03:45:20 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C316CC4339B; Wed, 12 Apr 2023 03:45:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1681271119; bh=zgq2GY3PFDK9wWxE5M8w4Ajvmbif3wuS8JM0owNeoIU=; h=Date:Subject:From:To:Cc:From; b=oI5cY9VCRtJaack3PjbEhhzghupHNLtLxmawEx4I6ZrmDAwsTNGKsdV6ts8FMNxu1 JXRVuESRFv5XgJit/BikHy9tqLc367mV7C+gCkHaEnT8t+rSapDr7bLB9WDfC+raYo ssEhrm1cNwY3yNkVDxjhWXOeSQgXLWqadQr2pRNMYZvxKpGFvQfOVQog/ORkw2O7YX ddmgh567Yc6i84evydNJX5EqtTqlj0uKa6Lb+Ilsai6vx6kUxZI4bEXUswGvI7D05W 55ABVURswarVHaymnLMdEDenXyNpAxhSCr5SeC4U04UVxKNruUyVNWwq4p98a5T6c/ 4j7XQ/8ZYta5w== Date: Tue, 11 Apr 2023 20:45:19 -0700 Subject: [GIT PULL 2/22] xfs: make intent items take a perag reference From: "Darrick J. Wong" To: dchinner@fromorbit.com, djwong@kernel.org Cc: dchinner@redhat.com, linux-xfs@vger.kernel.org Message-ID: <168127093858.417736.1801856127309869556.stg-ugh@frogsfrogsfrogs> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-xfs@vger.kernel.org Hi Dave, Please pull this branch with changes for xfs. As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems. --D The following changes since commit 03786f0afb2ed5705a0478e14fea50a7f1a44f7e: xfs: document future directions of online fsck (2023-04-11 18:59:52 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/intents-perag-refs-6.4_2023-04-11 for you to fetch changes up to 00e7b3bac1dc8961bd5aa9d39e79131c6bd81181: xfs: give xfs_refcount_intent its own perag reference (2023-04-11 18:59:55 -0700) ---------------------------------------------------------------- xfs: make intent items take a perag reference [v24.5] Now that we've cleaned up some code warts in the deferred work item processing code, let's make intent items take an active perag reference from their creation until they are finally freed by the defer ops machinery. This change facilitates the scrub drain in the next patchset and will make it easier for the future AG removal code to detect a busy AG in need of quiescing. Signed-off-by: Darrick J. Wong ---------------------------------------------------------------- Darrick J. Wong (5): xfs: give xfs_bmap_intent its own perag reference xfs: pass per-ag references to xfs_free_extent xfs: give xfs_extfree_intent its own perag reference xfs: give xfs_rmap_intent its own perag reference xfs: give xfs_refcount_intent its own perag reference fs/xfs/libxfs/xfs_ag.c | 6 ++--- fs/xfs/libxfs/xfs_alloc.c | 22 +++++++--------- fs/xfs/libxfs/xfs_alloc.h | 12 ++++++--- fs/xfs/libxfs/xfs_bmap.c | 1 + fs/xfs/libxfs/xfs_bmap.h | 4 +++ fs/xfs/libxfs/xfs_ialloc_btree.c | 7 +++-- fs/xfs/libxfs/xfs_refcount.c | 33 ++++++++++------------- fs/xfs/libxfs/xfs_refcount.h | 4 +++ fs/xfs/libxfs/xfs_refcount_btree.c | 5 ++-- fs/xfs/libxfs/xfs_rmap.c | 29 ++++++++------------ fs/xfs/libxfs/xfs_rmap.h | 4 +++ fs/xfs/scrub/repair.c | 3 ++- fs/xfs/xfs_bmap_item.c | 29 +++++++++++++++++++- fs/xfs/xfs_extfree_item.c | 54 +++++++++++++++++++++++++++----------- fs/xfs/xfs_refcount_item.c | 36 ++++++++++++++++++++++--- fs/xfs/xfs_rmap_item.c | 32 +++++++++++++++++++--- 16 files changed, 196 insertions(+), 85 deletions(-) From patchwork Wed Apr 12 03:45:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13208395 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 355A4C7619A for ; Wed, 12 Apr 2023 03:45:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229483AbjDLDpi (ORCPT ); Tue, 11 Apr 2023 23:45:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37962 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229451AbjDLDph (ORCPT ); Tue, 11 Apr 2023 23:45:37 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7206330E0 for ; Tue, 11 Apr 2023 20:45:36 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 0CA2262B68 for ; Wed, 12 Apr 2023 03:45:36 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 68154C433D2; Wed, 12 Apr 2023 03:45:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1681271135; bh=p018O0KDIHVz9AJeo+cUyOMtd6FJdvqNJEhGw02Q2N4=; h=Date:Subject:From:To:Cc:From; b=eZ3MQUt3BNZ1hm8kOohYWIg1VBVzMUv+6i5S96HooKrxQxSGqP0vmYHA8TgZ1RZhB uEUGLfW6TOnvhjRGxSny6UYzh8VwOLExYVrDCdpAJvnsxksRhSVXsJTDTKNXaBqqRF 98g6CDQm6S9ad3G8McE8BWgsofKQ/WwxXlk3RvmkAX7+KvDGenLkzZ7upKqVNv9GV0 dQrm/iqjvLBVYmkW3cjVAn4ydYi2Gd+/H69uSJPVsw3UKvih+Lld6jR5TXIs0IM0tX P5Q+UiIdRj02EoBqe+Ffsr5ZGnZfjf1BxJOH+q3OjsSOtIGeTArdOhO2uxueIVVene eJr2tnbqB6WcQ== Date: Tue, 11 Apr 2023 20:45:34 -0700 Subject: [GIT PULL 3/22] xfs: pass perag references around when possible From: "Darrick J. Wong" To: dchinner@fromorbit.com, djwong@kernel.org Cc: dchinner@redhat.com, linux-xfs@vger.kernel.org Message-ID: <168127093954.417736.329214140924049645.stg-ugh@frogsfrogsfrogs> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-xfs@vger.kernel.org Hi Dave, Please pull this branch with changes for xfs. As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems. --D The following changes since commit 00e7b3bac1dc8961bd5aa9d39e79131c6bd81181: xfs: give xfs_refcount_intent its own perag reference (2023-04-11 18:59:55 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/pass-perag-refs-6.4_2023-04-11 for you to fetch changes up to 9b2e5a234c89f097ec36f922763dfa1465dc06f8: xfs: create traced helper to get extra perag references (2023-04-11 18:59:55 -0700) ---------------------------------------------------------------- xfs: pass perag references around when possible [v24.5] Avoid the cost of perag radix tree lookups by passing around active perag references when possible. v24.2: rework some of the naming and whatnot so there's less opencoding Signed-off-by: Darrick J. Wong ---------------------------------------------------------------- Darrick J. Wong (1): xfs: create traced helper to get extra perag references fs/xfs/libxfs/xfs_ag.c | 13 +++++++++++++ fs/xfs/libxfs/xfs_ag.h | 1 + fs/xfs/libxfs/xfs_alloc_btree.c | 4 +--- fs/xfs/libxfs/xfs_ialloc_btree.c | 4 +--- fs/xfs/libxfs/xfs_refcount_btree.c | 5 +---- fs/xfs/libxfs/xfs_rmap_btree.c | 5 +---- fs/xfs/xfs_iunlink_item.c | 4 +--- fs/xfs/xfs_iwalk.c | 5 ++--- fs/xfs/xfs_trace.h | 1 + 9 files changed, 22 insertions(+), 20 deletions(-) From patchwork Wed Apr 12 03:45:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13208396 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 28760C7619A for ; Wed, 12 Apr 2023 03:45:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229490AbjDLDpy (ORCPT ); Tue, 11 Apr 2023 23:45:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37992 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229451AbjDLDpx (ORCPT ); Tue, 11 Apr 2023 23:45:53 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 03CFA40CA for ; Tue, 11 Apr 2023 20:45:52 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 9430D62B68 for ; Wed, 12 Apr 2023 03:45:51 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id F0C14C433D2; Wed, 12 Apr 2023 03:45:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1681271151; bh=+icNcZKOhk8CPbPYm7ED0zOfkpvhR/bdV8kPsGCvIM4=; h=Date:Subject:From:To:Cc:From; b=XgdKH9enEeP+AcJxFIVNe/17mSX2EAw1xG4bHtK4bybdCvU+YzIPMaKqPJ2L9QcWF fUN/lHQClXxxTEVtKhFihf99krjJD0yocepYVgnr/Ci9LLSSAmZETpL1m1an8w/iHa 1+pBYZVg/ZwuJaRLgeuzV1BeEkK2DCjkPd7dYQarwrMwmmkiRIOJne1KiwAQln413B 4gIfzXRj0bjwQCsRdslhrBxPZfw1woFsyAU0eZb7dyAF0+6V3O5TrTe0RSGNafKBcL r+xLzaZQ+RQi2QYg4lvLFM9iAu1VHf5j4isSGAU4q1IuErCTjEh1ehvBkm0r13IcT3 T9e34vDx2E62Q== Date: Tue, 11 Apr 2023 20:45:50 -0700 Subject: [GIT PULL 4/22] xfs_scrub: fix licensing and copyright notices From: "Darrick J. Wong" To: dchinner@fromorbit.com, djwong@kernel.org Cc: dchinner@redhat.com, linux-xfs@vger.kernel.org Message-ID: <168127094049.417736.4769433461213675106.stg-ugh@frogsfrogsfrogs> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-xfs@vger.kernel.org Hi Dave, Please pull this branch with changes for xfs. As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems. --D The following changes since commit 9b2e5a234c89f097ec36f922763dfa1465dc06f8: xfs: create traced helper to get extra perag references (2023-04-11 18:59:55 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/scrub-fix-legalese-6.4_2023-04-11 for you to fetch changes up to ecc73f8a58c7844b04186726f8699ba97cec2ef9: xfs: update copyright years for scrub/ files (2023-04-11 18:59:57 -0700) ---------------------------------------------------------------- xfs_scrub: fix licensing and copyright notices [v24.5] Fix various attribution problems in the xfs_scrub source code, such as the author's contact information, out of date SPDX tags, and a rough estimate of when the feature was under heavy development. The most egregious parts are the files that are missing license information completely. Signed-off-by: Darrick J. Wong ---------------------------------------------------------------- Darrick J. Wong (2): xfs: fix author and spdx headers on scrub/ files xfs: update copyright years for scrub/ files fs/xfs/scrub/agheader.c | 6 +++--- fs/xfs/scrub/agheader_repair.c | 6 +++--- fs/xfs/scrub/alloc.c | 6 +++--- fs/xfs/scrub/attr.c | 6 +++--- fs/xfs/scrub/attr.h | 4 ++-- fs/xfs/scrub/bitmap.c | 6 +++--- fs/xfs/scrub/bitmap.h | 6 +++--- fs/xfs/scrub/bmap.c | 6 +++--- fs/xfs/scrub/btree.c | 6 +++--- fs/xfs/scrub/btree.h | 6 +++--- fs/xfs/scrub/common.c | 6 +++--- fs/xfs/scrub/common.h | 6 +++--- fs/xfs/scrub/dabtree.c | 6 +++--- fs/xfs/scrub/dabtree.h | 6 +++--- fs/xfs/scrub/dir.c | 6 +++--- fs/xfs/scrub/fscounters.c | 4 ++-- fs/xfs/scrub/health.c | 6 +++--- fs/xfs/scrub/health.h | 6 +++--- fs/xfs/scrub/ialloc.c | 6 +++--- fs/xfs/scrub/inode.c | 6 +++--- fs/xfs/scrub/parent.c | 6 +++--- fs/xfs/scrub/quota.c | 6 +++--- fs/xfs/scrub/refcount.c | 6 +++--- fs/xfs/scrub/repair.c | 6 +++--- fs/xfs/scrub/repair.h | 6 +++--- fs/xfs/scrub/rmap.c | 6 +++--- fs/xfs/scrub/rtbitmap.c | 6 +++--- fs/xfs/scrub/scrub.c | 6 +++--- fs/xfs/scrub/scrub.h | 6 +++--- fs/xfs/scrub/symlink.c | 6 +++--- fs/xfs/scrub/trace.c | 6 +++--- fs/xfs/scrub/trace.h | 6 +++--- fs/xfs/scrub/xfs_scrub.h | 6 +++--- 33 files changed, 97 insertions(+), 97 deletions(-) From patchwork Wed Apr 12 03:46:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13208397 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5424C7619A for ; Wed, 12 Apr 2023 03:46:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229575AbjDLDqJ (ORCPT ); Tue, 11 Apr 2023 23:46:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38024 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229451AbjDLDqI (ORCPT ); Tue, 11 Apr 2023 23:46:08 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B0F1530E0 for ; Tue, 11 Apr 2023 20:46:07 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 4BEA362D90 for ; Wed, 12 Apr 2023 03:46:07 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A8AF5C433EF; Wed, 12 Apr 2023 03:46:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1681271166; bh=5jjg4cKduQSG67kiQ5yGAwss9+Nb+f522mP4Qh3s/UQ=; h=Date:Subject:From:To:Cc:From; b=escV3sTcX/AadC/afDEl48vgGMSqzdkU0C9Bw5L1fk6nVjCmxgdASVWzPe5KwHiNh 1t2+NfEWxALLXtDgVkOVY+7/xWRZ2clRqO0Lo/SrsqJLxNKjTx8ls1nlTlLc63r/O8 /YGd7HME1d0REXn7w7F63/utggbF1sgOhMuxl3RrhE+TiojXQknIbk5dLihqntEOmf +iEc95s0toLu7qJ5rwnRVBUcE2KxW9hvXbOZIW93jmDcNZBKu6bSpm7XXaZhO2AoSm 25Wd4rka/fGAawpVPw/GyvWT6YxDK7Ev0E5kdl9XiYk2ZoxAFg6py9zaFchelME1OS 6PCtrXf9Gm51Q== Date: Tue, 11 Apr 2023 20:46:06 -0700 Subject: [GIT PULL 5/22] xfs: drain deferred work items when scrubbing From: "Darrick J. Wong" To: dchinner@fromorbit.com, djwong@kernel.org Cc: dchinner@redhat.com, linux-xfs@vger.kernel.org Message-ID: <168127094157.417736.10965865343075972034.stg-ugh@frogsfrogsfrogs> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-xfs@vger.kernel.org Hi Dave, Please pull this branch with changes for xfs. As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems. --D The following changes since commit ecc73f8a58c7844b04186726f8699ba97cec2ef9: xfs: update copyright years for scrub/ files (2023-04-11 18:59:57 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/scrub-drain-intents-6.4_2023-04-11 for you to fetch changes up to 88accf17226733088923635b580779a3c86b6f23: xfs: scrub should use ECHRNG to signal that the drain is needed (2023-04-11 19:00:00 -0700) ---------------------------------------------------------------- xfs: drain deferred work items when scrubbing [v24.5] The design doc for XFS online fsck contains a long discussion of the eventual consistency models in use for XFS metadata. In that chapter, we note that it is possible for scrub to collide with a chain of deferred space metadata updates, and proposes a lightweight solution: The use of a pending-intents counter so that scrub can wait for the system to drain all chains. This patchset implements that scrub drain. The first patch implements the basic mechanism, and the subsequent patches reduce the runtime overhead by converting the implementation to use sloppy counters and introducing jump labels to avoid walking into scrub hooks when it isn't running. This last paradigm repeats elsewhere in this megaseries. v23.1: make intent items take an active ref to the perag structure and document why we bump and drop the intent counts when we do Signed-off-by: Darrick J. Wong ---------------------------------------------------------------- Darrick J. Wong (5): xfs: add a tracepoint to report incorrect extent refcounts xfs: allow queued AG intents to drain before scrubbing xfs: clean up scrub context if scrub setup returns -EDEADLOCK xfs: minimize overhead of drain wakeups by using jump labels xfs: scrub should use ECHRNG to signal that the drain is needed fs/xfs/Kconfig | 5 ++ fs/xfs/Makefile | 2 + fs/xfs/libxfs/xfs_ag.c | 4 ++ fs/xfs/libxfs/xfs_ag.h | 8 +++ fs/xfs/libxfs/xfs_defer.c | 6 +- fs/xfs/scrub/agheader.c | 9 +++ fs/xfs/scrub/alloc.c | 3 + fs/xfs/scrub/bmap.c | 3 + fs/xfs/scrub/btree.c | 1 + fs/xfs/scrub/common.c | 137 ++++++++++++++++++++++++++++++++++--- fs/xfs/scrub/common.h | 15 ++++ fs/xfs/scrub/dabtree.c | 1 + fs/xfs/scrub/fscounters.c | 7 ++ fs/xfs/scrub/health.c | 2 + fs/xfs/scrub/ialloc.c | 2 + fs/xfs/scrub/inode.c | 3 + fs/xfs/scrub/quota.c | 3 + fs/xfs/scrub/refcount.c | 9 ++- fs/xfs/scrub/repair.c | 3 + fs/xfs/scrub/rmap.c | 3 + fs/xfs/scrub/scrub.c | 63 ++++++++++++----- fs/xfs/scrub/scrub.h | 12 +++- fs/xfs/scrub/trace.h | 69 +++++++++++++++++++ fs/xfs/xfs_bmap_item.c | 12 +++- fs/xfs/xfs_drain.c | 166 +++++++++++++++++++++++++++++++++++++++++++++ fs/xfs/xfs_drain.h | 87 ++++++++++++++++++++++++ fs/xfs/xfs_extfree_item.c | 4 +- fs/xfs/xfs_linux.h | 1 + fs/xfs/xfs_refcount_item.c | 4 +- fs/xfs/xfs_rmap_item.c | 4 +- fs/xfs/xfs_trace.h | 71 +++++++++++++++++++ 31 files changed, 680 insertions(+), 39 deletions(-) create mode 100644 fs/xfs/xfs_drain.c create mode 100644 fs/xfs/xfs_drain.h From patchwork Wed Apr 12 03:46:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13208398 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8A32AC77B6E for ; Wed, 12 Apr 2023 03:46:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229588AbjDLDqZ (ORCPT ); Tue, 11 Apr 2023 23:46:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38050 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229451AbjDLDqY (ORCPT ); Tue, 11 Apr 2023 23:46:24 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5D26830E0 for ; Tue, 11 Apr 2023 20:46:23 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id ED83262D90 for ; Wed, 12 Apr 2023 03:46:22 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5006FC433D2; Wed, 12 Apr 2023 03:46:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1681271182; bh=pxK0DeFHR5lAfAIy65l7dDNU/8USpiP5ERz1sRzEL6c=; h=Date:Subject:From:To:Cc:From; b=B+pvLjG/5TQxxyLOb7/eeTjoNMeCCehd3/poSFt6WvmkJzbTOn9/ir9ioQWN1bmSW PWrF4lGD91tppp1zZ3v0WgOOwlcSvyVmsJYoOpBD8cpXym5Zxbeb+g6WtHkO8LYHKy 1bZ7zoqDeqBPxXFGnc8JMROx/SF/m6jMvmLmazHbnTMEVZgi2Y1iqX9kYk78y07h7N l2KOCx22i7jVuvecbuzx84dGwhBqVXUFRro4pz6gJFSiwxPCOMXjyWPDa7FQ8EgH/+ lr5e2IhB7+ohylc0rd/148V2P/P/5RkbzREnjTXLwFtKyyCQNEZVNeHcFrCe+WLoo3 t7+mgN6yrFYmA== Date: Tue, 11 Apr 2023 20:46:21 -0700 Subject: [GIT PULL 6/22] xfs: standardize btree record checking code From: "Darrick J. Wong" To: dchinner@fromorbit.com, djwong@kernel.org Cc: dchinner@redhat.com, linux-xfs@vger.kernel.org Message-ID: <168127094258.417736.16899254677069613479.stg-ugh@frogsfrogsfrogs> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-xfs@vger.kernel.org Hi Dave, Please pull this branch with changes for xfs. As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems. --D The following changes since commit 88accf17226733088923635b580779a3c86b6f23: xfs: scrub should use ECHRNG to signal that the drain is needed (2023-04-11 19:00:00 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/btree-complain-bad-records-6.4_2023-04-11 for you to fetch changes up to 6a3bd8fcf9afb47c703cb268f30f60aa2e7af86a: xfs: complain about bad file mapping records in the ondisk bmbt (2023-04-11 19:00:05 -0700) ---------------------------------------------------------------- xfs: standardize btree record checking code [v24.5] While I was cleaning things up for 6.1, I noticed that the btree _query_range and _query_all functions don't perform the same checking that the _get_rec functions perform. In fact, they don't perform /any/ sanity checking, which means that callers aren't warned about impossible records. Therefore, hoist the record validation and complaint logging code into separate functions, and call them from any place where we convert an ondisk record into an incore record. For online scrub, we can replace checking code with a call to the record checking functions in libxfs, thereby reducing the size of the codebase. Signed-off-by: Darrick J. Wong ---------------------------------------------------------------- Darrick J. Wong (8): xfs: standardize ondisk to incore conversion for free space btrees xfs: standardize ondisk to incore conversion for inode btrees xfs: standardize ondisk to incore conversion for refcount btrees xfs: return a failure address from xfs_rmap_irec_offset_unpack xfs: standardize ondisk to incore conversion for rmap btrees xfs: standardize ondisk to incore conversion for bmap btrees xfs: complain about bad records in query_range helpers xfs: complain about bad file mapping records in the ondisk bmbt fs/xfs/libxfs/xfs_alloc.c | 82 ++++++++++++++++++++++++---------- fs/xfs/libxfs/xfs_alloc.h | 6 +++ fs/xfs/libxfs/xfs_bmap.c | 31 ++++++++++++- fs/xfs/libxfs/xfs_bmap.h | 2 + fs/xfs/libxfs/xfs_ialloc.c | 77 +++++++++++++++++++++----------- fs/xfs/libxfs/xfs_ialloc.h | 2 + fs/xfs/libxfs/xfs_ialloc_btree.c | 2 +- fs/xfs/libxfs/xfs_ialloc_btree.h | 2 +- fs/xfs/libxfs/xfs_inode_fork.c | 3 +- fs/xfs/libxfs/xfs_refcount.c | 73 +++++++++++++++++++----------- fs/xfs/libxfs/xfs_refcount.h | 2 + fs/xfs/libxfs/xfs_rmap.c | 95 +++++++++++++++++++++++++--------------- fs/xfs/libxfs/xfs_rmap.h | 12 +++-- fs/xfs/scrub/alloc.c | 24 +++++----- fs/xfs/scrub/bmap.c | 6 +++ fs/xfs/scrub/ialloc.c | 24 ++-------- fs/xfs/scrub/refcount.c | 14 ++---- fs/xfs/scrub/rmap.c | 44 ++----------------- 18 files changed, 303 insertions(+), 198 deletions(-) From patchwork Wed Apr 12 03:46:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13208399 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E894C7619A for ; Wed, 12 Apr 2023 03:46:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229671AbjDLDqk (ORCPT ); Tue, 11 Apr 2023 23:46:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38080 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229451AbjDLDqj (ORCPT ); Tue, 11 Apr 2023 23:46:39 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ED69430E0 for ; Tue, 11 Apr 2023 20:46:38 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 8863562D90 for ; Wed, 12 Apr 2023 03:46:38 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E149AC433EF; Wed, 12 Apr 2023 03:46:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1681271198; bh=IjfYu1hDRFq38E8hyDr0r1vIcoFFiKjhU16MlSGiocs=; h=Date:Subject:From:To:Cc:From; b=MwdooraNgh/nPUoZlxLGjwFLlRorAONCo8SbRZE/Gb39sOgWeo5zj3dbOyR+7hTHa /iux00rgxCT0hXvGfakpMygYRIjBeVMxIEnJU6esvJ2twx9DH3QuU9OSSrZ6wd/PZE hgxFjcp1NsYVtaLJELXWlFbNADKKIqP2ZzX5baIwzLgv0r8GlS9WLnxsHX54Ssa1EZ qc3OKC86QpLZ9sV3ViK0NYm+u2e2G44nYNCpjLhrkzsohW/LtJX216MOpn4sR9IWS1 FBWNBMhJJPy3YgcKbBUNZ8JIdKgli6h3zrhnA5AKGAlwpQYDZxaSq1LRjUqAZG8pEA qYBM32/QvGS4w== Date: Tue, 11 Apr 2023 20:46:37 -0700 Subject: [GIT PULL 7/22] xfs: hoist scrub record checks into libxfs From: "Darrick J. Wong" To: dchinner@fromorbit.com, djwong@kernel.org Cc: dchinner@redhat.com, linux-xfs@vger.kernel.org Message-ID: <168127094360.417736.3162133789025574725.stg-ugh@frogsfrogsfrogs> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-xfs@vger.kernel.org Hi Dave, Please pull this branch with changes for xfs. As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems. --D The following changes since commit 6a3bd8fcf9afb47c703cb268f30f60aa2e7af86a: xfs: complain about bad file mapping records in the ondisk bmbt (2023-04-11 19:00:05 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/btree-hoist-scrub-checks-6.4_2023-04-11 for you to fetch changes up to de1a9ce225e93b22d189f8ffbce20074bc803121: xfs: hoist inode record alignment checks from scrub (2023-04-11 19:00:06 -0700) ---------------------------------------------------------------- xfs: hoist scrub record checks into libxfs [v24.5] There are a few things about btree records that scrub checked but the libxfs _get_rec functions didn't. Move these bits into libxfs so that everyone can benefit. Signed-off-by: Darrick J. Wong ---------------------------------------------------------------- Darrick J. Wong (3): xfs: hoist rmap record flag checks from scrub xfs: hoist rmap record flag checks from scrub xfs: hoist inode record alignment checks from scrub fs/xfs/libxfs/xfs_ialloc.c | 4 ++++ fs/xfs/libxfs/xfs_rmap.c | 27 +++++++++++++++++++++++++++ fs/xfs/scrub/ialloc.c | 6 ------ fs/xfs/scrub/rmap.c | 22 ---------------------- 4 files changed, 31 insertions(+), 28 deletions(-) From patchwork Wed Apr 12 03:46:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13208400 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5DD62C7619A for ; Wed, 12 Apr 2023 03:46:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229572AbjDLDq4 (ORCPT ); Tue, 11 Apr 2023 23:46:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38106 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229451AbjDLDqz (ORCPT ); Tue, 11 Apr 2023 23:46:55 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9B4AF30E0 for ; Tue, 11 Apr 2023 20:46:54 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 1D39862D90 for ; Wed, 12 Apr 2023 03:46:54 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7639CC433D2; Wed, 12 Apr 2023 03:46:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1681271213; bh=zJ+FyYRRAJjL7+SS4ho8rQ+Oj15PD5C3vHyPaP2tstY=; h=Date:Subject:From:To:Cc:From; b=uAjmmibylhHcnrKhYGeqwXcxBGtN3SiRaANjD/0aUrr03ZSgZ0vYeDpDFVJLYheMF EHzwGrqW0K+E9klBwGpET89ayPI13jEDblZ8lSUTI/eV8lX1eejSfNXjXCXabd1lcv 5xyBAq25zUFsLfP+9AUhnuJcDTX1oQwflwqYaRqOMPx6yq+xL7IsyHtixVr2t+0jQi TJ1P5Qp3wxi/+CR7CDddP41YfcfLfa0gL625q5aM4m+YWrt57BxhKupkariL7c+2TF YQDarYzjt56RlV1KPTDBQtS/FJVS7eWet2xzbDqT7YMVH8xnL7oeJ8alJphFh5/2Qh tnhfi7rKksf3w== Date: Tue, 11 Apr 2023 20:46:53 -0700 Subject: [GIT PULL 8/22] xfs: fix rmap btree key flag handling From: "Darrick J. Wong" To: dchinner@fromorbit.com, djwong@kernel.org Cc: dchinner@redhat.com, linux-xfs@vger.kernel.org Message-ID: <168127094461.417736.1444539165048108895.stg-ugh@frogsfrogsfrogs> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-xfs@vger.kernel.org Hi Dave, Please pull this branch with changes for xfs. As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems. --D The following changes since commit de1a9ce225e93b22d189f8ffbce20074bc803121: xfs: hoist inode record alignment checks from scrub (2023-04-11 19:00:06 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/rmap-btree-fix-key-handling-6.4_2023-04-11 for you to fetch changes up to 38384569a2a8a721623d80c5ae3bcf80614ab792: xfs: detect unwritten bit set in rmapbt node block keys (2023-04-11 19:00:07 -0700) ---------------------------------------------------------------- xfs: fix rmap btree key flag handling [v24.5] This series fixes numerous flag handling bugs in the rmapbt key code. The most serious transgression is that key comparisons completely strip out all flag bits from rm_offset, including the ones that participate in record lookups. The second problem is that for years we've been letting the unwritten flag (which is an attribute of a specific record and not part of the record key) escape from leaf records into key records. The solution to the second problem is to filter attribute flags when creating keys from records, and the solution to the first problem is to preserve *only* the flags used for key lookups. The ATTR and BMBT flags are a part of the lookup key, and the UNWRITTEN flag is a record attribute. This has worked for years without generating user complaints because ATTR and BMBT extents cannot be shared, so key comparisons succeed solely on rm_startblock. Only file data fork extents can be shared, and those records never set any of the three flag bits, so comparisons that dig into rm_owner and rm_offset work just fine. A filesystem written with an unpatched kernel and mounted on a patched kernel will work correctly because the ATTR/BMBT flags have been conveyed into keys correctly all along, and we still ignore the UNWRITTEN flag in any key record. This was what doomed my previous attempt to correct this problem in 2019. A filesystem written with a patched kernel and mounted on an unpatched kernel will also work correctly because unpatched kernels ignore all flags. With this patchset applied, the scrub code gains the ability to detect rmap btrees with incorrectly set attr and bmbt flags in the key records. After three years of testing, I haven't encountered any problems. Signed-off-by: Darrick J. Wong ---------------------------------------------------------------- Darrick J. Wong (2): xfs: fix rm_offset flag handling in rmap keys xfs: detect unwritten bit set in rmapbt node block keys fs/xfs/libxfs/xfs_rmap_btree.c | 40 +++++++++++++++++++++++-------- fs/xfs/scrub/btree.c | 10 ++++++++ fs/xfs/scrub/btree.h | 2 ++ fs/xfs/scrub/rmap.c | 53 ++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 95 insertions(+), 10 deletions(-) From patchwork Wed Apr 12 03:47:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13208401 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 00377C77B6E for ; Wed, 12 Apr 2023 03:47:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229678AbjDLDrM (ORCPT ); Tue, 11 Apr 2023 23:47:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38188 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229587AbjDLDrK (ORCPT ); Tue, 11 Apr 2023 23:47:10 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4F3114206 for ; Tue, 11 Apr 2023 20:47:10 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 9E78B62D90 for ; Wed, 12 Apr 2023 03:47:09 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0A90CC433EF; Wed, 12 Apr 2023 03:47:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1681271229; bh=Rio046g+lSIcm1a9pUS7+55+NKay54qdNwjGOCO8wy0=; h=Date:Subject:From:To:Cc:From; b=ePB0SJ+zGuQcjDcMBfV6MgKSgpwfzYzC9eNbSITGNdTD8tgp1lUHXE/TJa9U8i3P2 sD81dUDP8CCjfHZpzBANX6jcwAgTqgd/WcUmOex8lfK9BiLNmYgQRGM+3ANlYlufGv gmiDGlAzpr4UCBCICnubhDUIuVMGEfLUfmFTbWND7FaqJ088a9fjCtNXt5VWdu9jj9 z2kEKhl07D9mo30e4Lbu1rrauKrDrwnHrf28I70cPCl4JXt73cYa0WpjajHZw1v/fM uMNxS+9q0OID+Aiwsqwzy86nesv7+DEezqW0AOxK73A47oK9gMYZ+hRd+OXtmPdi8x jBeF7bQBxrasg== Date: Tue, 11 Apr 2023 20:47:08 -0700 Subject: [GIT PULL 9/22] xfs: enhance btree key scrubbing From: "Darrick J. Wong" To: dchinner@fromorbit.com, djwong@kernel.org Cc: dchinner@redhat.com, linux-xfs@vger.kernel.org Message-ID: <168127094565.417736.2463509060702882791.stg-ugh@frogsfrogsfrogs> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-xfs@vger.kernel.org Hi Dave, Please pull this branch with changes for xfs. As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems. --D The following changes since commit 38384569a2a8a721623d80c5ae3bcf80614ab792: xfs: detect unwritten bit set in rmapbt node block keys (2023-04-11 19:00:07 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/scrub-btree-key-enhancements-6.4_2023-04-11 for you to fetch changes up to 2bea8df0a52b05bc0dddd54e950ae37c83533b03: xfs: always scrub record/key order of interior records (2023-04-11 19:00:09 -0700) ---------------------------------------------------------------- xfs: enhance btree key scrubbing [v24.5] This series fixes the scrub btree block checker to ensure that the keys in the parent block accurately represent the block, and check the ordering of all interior key records. Signed-off-by: Darrick J. Wong ---------------------------------------------------------------- Darrick J. Wong (2): xfs: check btree keys reflect the child block xfs: always scrub record/key order of interior records fs/xfs/scrub/btree.c | 63 ++++++++++++++++++++++++++++++++++++++++++++++------ fs/xfs/scrub/btree.h | 8 ++++++- 2 files changed, 63 insertions(+), 8 deletions(-) From patchwork Wed Apr 12 03:47:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13208402 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82785C7619A for ; Wed, 12 Apr 2023 03:47:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229587AbjDLDr1 (ORCPT ); Tue, 11 Apr 2023 23:47:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38278 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229491AbjDLDr0 (ORCPT ); Tue, 11 Apr 2023 23:47:26 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 95E5630E0 for ; Tue, 11 Apr 2023 20:47:25 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 350A66101C for ; Wed, 12 Apr 2023 03:47:25 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 91152C433D2; Wed, 12 Apr 2023 03:47:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1681271244; bh=OKTPO4jv7zALxM6IwqIjSlQ9Quq+/5br6hUE+X99lzk=; h=Date:Subject:From:To:Cc:From; b=vF78qw4OKhta0FjFlK7y/IP7uyRdgvjzsedbfzbp33oFQs/aK08zzx7R2Vh4IctLc lJdOtUwb/FtY+9QRA5Yj9r7GVGEK2UvVUUyUIJiQxQIfzEIN1mupNrFg3KMs0zxU1v tIpfFa4a/QUOqizJK7M5RNCgv5AXlLImeOpYucQ3tGIexKb32j/PQG9cs+skwnqkb6 pEIaHUtvMRiXpuE/kp48fCXknftzQU6VlUdH2Y4XrQY46yTgQy3HZArj1v3e++70WG AaWk9M08QnSQbDhrgEYJc6orxqk8DVTP/NDw4fyf7AmAytaGOSuNAcr+K34ub6U5hv Q7OG8r8iA6lPg== Date: Tue, 11 Apr 2023 20:47:24 -0700 Subject: [GIT PULL 10/22] xfs: detect incorrect gaps in refcount btree From: "Darrick J. Wong" To: dchinner@fromorbit.com, djwong@kernel.org Cc: david@fromorbit.com, dchinner@redhat.com, linux-xfs@vger.kernel.org Message-ID: <168127094663.417736.1589396657136631142.stg-ugh@frogsfrogsfrogs> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-xfs@vger.kernel.org Hi Dave, Please pull this branch with changes for xfs. As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems. --D The following changes since commit 2bea8df0a52b05bc0dddd54e950ae37c83533b03: xfs: always scrub record/key order of interior records (2023-04-11 19:00:09 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/scrub-detect-refcount-gaps-6.4_2023-04-11 for you to fetch changes up to 7ac14fa2bd22e99a06ae16382b394f697cfe2b8a: xfs: ensure that all metadata and data blocks are not cow staging extents (2023-04-11 19:00:12 -0700) ---------------------------------------------------------------- xfs: detect incorrect gaps in refcount btree [v24.5] The next few patchsets address a deficiency in scrub that I found while QAing the refcount btree scrubber. If there's a gap between refcount records, we need to cross-reference that gap with the reverse mappings to ensure that there are no overlapping records in the rmap btree. If we find any, then the refcount btree is not consistent. This is not a property that is specific to the refcount btree; they all need to have this sort of keyspace scanning logic to detect inconsistencies. To do this accurately, we need to be able to scan the keyspace of a btree (which we already do) to be able to tell the caller if the keyspace is empty, sparse, or fully covered by records. The first few patches add the keyspace scanner to the generic btree code, along with the ability to mask off parts of btree keys because when we scan the rmapbt, we only care about space usage, not the owners. The final patch closes the scanning gap in the refcountbt scanner. v23.1: create helpers for the key extraction and comparison functions, improve documentation, and eliminate the ->mask_key indirect calls Signed-off-by: Darrick J. Wong ---------------------------------------------------------------- Darrick J. Wong (6): xfs: refactor converting btree irec to btree key xfs: refactor ->diff_two_keys callsites xfs: replace xfs_btree_has_record with a general keyspace scanner xfs: implement masked btree key comparisons for _has_records scans xfs: check the reference counts of gaps in the refcount btree xfs: ensure that all metadata and data blocks are not cow staging extents fs/xfs/libxfs/xfs_alloc.c | 11 +- fs/xfs/libxfs/xfs_alloc.h | 4 +- fs/xfs/libxfs/xfs_alloc_btree.c | 28 ++++- fs/xfs/libxfs/xfs_bmap_btree.c | 19 +++- fs/xfs/libxfs/xfs_btree.c | 208 +++++++++++++++++++++++++++---------- fs/xfs/libxfs/xfs_btree.h | 141 ++++++++++++++++++++++++- fs/xfs/libxfs/xfs_ialloc_btree.c | 22 +++- fs/xfs/libxfs/xfs_refcount.c | 11 +- fs/xfs/libxfs/xfs_refcount.h | 4 +- fs/xfs/libxfs/xfs_refcount_btree.c | 21 +++- fs/xfs/libxfs/xfs_rmap.c | 15 ++- fs/xfs/libxfs/xfs_rmap.h | 4 +- fs/xfs/libxfs/xfs_rmap_btree.c | 61 ++++++++--- fs/xfs/libxfs/xfs_types.h | 12 +++ fs/xfs/scrub/agheader.c | 5 + fs/xfs/scrub/alloc.c | 7 +- fs/xfs/scrub/bmap.c | 11 +- fs/xfs/scrub/btree.c | 24 ++--- fs/xfs/scrub/ialloc.c | 2 +- fs/xfs/scrub/inode.c | 1 + fs/xfs/scrub/refcount.c | 124 ++++++++++++++++++++-- fs/xfs/scrub/rmap.c | 6 +- fs/xfs/scrub/scrub.h | 2 + 23 files changed, 612 insertions(+), 131 deletions(-) From patchwork Wed Apr 12 03:47:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13208403 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E0CBC77B6E for ; Wed, 12 Apr 2023 03:47:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229506AbjDLDrn (ORCPT ); Tue, 11 Apr 2023 23:47:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38322 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229491AbjDLDrm (ORCPT ); Tue, 11 Apr 2023 23:47:42 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 39EAD30E0 for ; Tue, 11 Apr 2023 20:47:41 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id CCFD062CAC for ; Wed, 12 Apr 2023 03:47:40 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 33795C433D2; Wed, 12 Apr 2023 03:47:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1681271260; bh=TS6b2HixXPNkpr+JNBX9+RPj9tj8UxDgLVwThxPE9XM=; h=Date:Subject:From:To:Cc:From; b=pEM8wiWlxfdClHdlYdMdcmtaXkfTg1vl9VsjU4tpCYMyKy553Sy28atv2e8rasZLS cah3qnSXImHz9e6zi5jUrvry9lOYSHcx321uO2BPer5Ht2pH7tg7PweG8aXnvMsVmt p5WHGICI9G+P/RD1/kH/63stLCUx54jGmWAlyDw1mmBbYD4dSheTsGWlCpexnmhZHW n2k5Fz1p3L2Qy7mm5EK/5H/UXN8DpnvJnxRlMr3bg36HPRsHWYdQb8Fi0w7iShNTiw xfzISVsKvFxduUvtbO6vSMZBHSSjzy1BJfQI3MklGUehVW1TmWlzBKURNelMZtjpIu Kfo6UyXBlZTtA== Date: Tue, 11 Apr 2023 20:47:39 -0700 Subject: [GIT PULL 11/22] xfs: detect incorrect gaps in inode btree From: "Darrick J. Wong" To: dchinner@fromorbit.com, djwong@kernel.org Cc: dchinner@redhat.com, linux-xfs@vger.kernel.org Message-ID: <168127094761.417736.17953865523353006184.stg-ugh@frogsfrogsfrogs> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-xfs@vger.kernel.org Hi Dave, Please pull this branch with changes for xfs. As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems. --D The following changes since commit 7ac14fa2bd22e99a06ae16382b394f697cfe2b8a: xfs: ensure that all metadata and data blocks are not cow staging extents (2023-04-11 19:00:12 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/scrub-detect-inobt-gaps-6.4_2023-04-11 for you to fetch changes up to efc0845f5d3e253f7f46a60b66a94c3164d76ee3: xfs: convert xfs_ialloc_has_inodes_at_extent to return keyfill scan results (2023-04-11 19:00:15 -0700) ---------------------------------------------------------------- xfs: detect incorrect gaps in inode btree [v24.5] This series continues the corrections for a couple of problems I found in the inode btree scrubber. The first problem is that we don't directly check the inobt records have a direct correspondence with the finobt records, and vice versa. The second problem occurs on filesystems with sparse inode chunks -- the cross-referencing we do detects sparseness, but it doesn't actually check the consistency between the inobt hole records and the rmap data. Signed-off-by: Darrick J. Wong ---------------------------------------------------------------- Darrick J. Wong (4): xfs: remove pointless shadow variable from xfs_difree_inobt xfs: clean up broken eearly-exit code in the inode btree scrubber xfs: directly cross-reference the inode btrees with each other xfs: convert xfs_ialloc_has_inodes_at_extent to return keyfill scan results fs/xfs/libxfs/xfs_ialloc.c | 84 ++++++++------ fs/xfs/libxfs/xfs_ialloc.h | 5 +- fs/xfs/scrub/ialloc.c | 268 ++++++++++++++++++++++++++++++++++++--------- 3 files changed, 269 insertions(+), 88 deletions(-) From patchwork Wed Apr 12 03:47:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13208404 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98789C7619A for ; Wed, 12 Apr 2023 03:47:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229523AbjDLDr6 (ORCPT ); Tue, 11 Apr 2023 23:47:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38366 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229491AbjDLDr5 (ORCPT ); Tue, 11 Apr 2023 23:47:57 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C771D40CA for ; Tue, 11 Apr 2023 20:47:56 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 64BDE629DF for ; Wed, 12 Apr 2023 03:47:56 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id BB646C433D2; Wed, 12 Apr 2023 03:47:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1681271275; bh=V0/bDNx5Xw75DenmDTdK4NZSRhSGUoPyXsyRq7XgEX8=; h=Date:Subject:From:To:Cc:From; b=WRIH0wQ1jVh5jVeF2FQn+uTdG3JoSwk4MekQhkzn33k6/qdqYRcK58tGXfMiCnxbi hxbYwuCw8LqqqdZRB5/67oHweC/v8DJKs9nuOo/DXxRuwiXx/EkDWlBmW3rURt3lQB nmE+Qg+ope4x7qeTgU4XuwSW2oBqhB7NhiwksQGyK1oL0ZDNHK8hxHVPjm9J1OcOTX V33L+6RBkxRKXYhoAOshC1+h46vpX4qb7IrvogEouPBm9/2fPkWM74zGVe/QhTzlgs dSqCIbhQ7eAXc6XQEoymYHKAkxE8fKOi40cN8ZIXLkk3N6DuUXf2L0atc6docCeuUV QIIVt/dQsTiTw== Date: Tue, 11 Apr 2023 20:47:55 -0700 Subject: [GIT PULL 12/22] xfs: detect incorrect gaps in rmap btree From: "Darrick J. Wong" To: dchinner@fromorbit.com, djwong@kernel.org Cc: dchinner@redhat.com, linux-xfs@vger.kernel.org Message-ID: <168127094858.417736.10571726062895792086.stg-ugh@frogsfrogsfrogs> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-xfs@vger.kernel.org Hi Dave, Please pull this branch with changes for xfs. As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems. --D The following changes since commit efc0845f5d3e253f7f46a60b66a94c3164d76ee3: xfs: convert xfs_ialloc_has_inodes_at_extent to return keyfill scan results (2023-04-11 19:00:15 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/scrub-detect-rmapbt-gaps-6.4_2023-04-11 for you to fetch changes up to 30f8ee5e7e0ccce396dff209c6cbce49d0d7e167: xfs: ensure that single-owner file blocks are not owned by others (2023-04-11 19:00:16 -0700) ---------------------------------------------------------------- xfs: detect incorrect gaps in rmap btree [v24.5] Following in the theme of the last two patchsets, this one strengthens the rmap btree record checking so that scrub can count the number of space records that map to a given owner and that do not map to a given owner. This enables us to determine exclusive ownership of space that can't be shared. Signed-off-by: Darrick J. Wong ---------------------------------------------------------------- Darrick J. Wong (2): xfs: teach scrub to check for sole ownership of metadata objects xfs: ensure that single-owner file blocks are not owned by others fs/xfs/libxfs/xfs_rmap.c | 198 ++++++++++++++++++++++++++++++++--------------- fs/xfs/libxfs/xfs_rmap.h | 18 ++++- fs/xfs/scrub/agheader.c | 10 +-- fs/xfs/scrub/bmap.c | 14 +++- fs/xfs/scrub/btree.c | 2 +- fs/xfs/scrub/ialloc.c | 4 +- fs/xfs/scrub/inode.c | 2 +- fs/xfs/scrub/rmap.c | 45 ++++++----- fs/xfs/scrub/scrub.h | 2 +- 9 files changed, 198 insertions(+), 97 deletions(-) From patchwork Wed Apr 12 03:48:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13208405 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B85CC77B6E for ; Wed, 12 Apr 2023 03:48:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229514AbjDLDsO (ORCPT ); Tue, 11 Apr 2023 23:48:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38524 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229491AbjDLDsN (ORCPT ); Tue, 11 Apr 2023 23:48:13 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 63E4D10C0 for ; Tue, 11 Apr 2023 20:48:12 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 0205C629DF for ; Wed, 12 Apr 2023 03:48:12 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5BAEEC433D2; Wed, 12 Apr 2023 03:48:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1681271291; bh=l+G402CTO76jw1ugCH1eYB+2StZ2m0rcSphemHWlg7Y=; h=Date:Subject:From:To:Cc:From; b=q8IesSaxu+8Bfh4kwH05UG2H/iOlTjWQCnOHTmTla3fZlnJmX6z2c+LdEjJ1p8xKt TgwUc0AuvY4qxQhc37u64IsnJraopssSHdq3vBFkVD2VMOc1HmjTRCRQ7JS7S5uJ5G hq1Ve55+ky62r88sI+24EYMGdrM4/4VXhi1d4nDryCm1GhfEbd/uj0nKRXNKBs6I/2 xaesPaCRSVqgkGO/UneinTNzNNZvWBTTRfLRPV6BG8qefY/gg9mFe9+kaJbOPf6+rD Z+YCSwUBDDdZOG1J+fkGJWslNyZ79NL0otE+8U+5G3jYoODMdJPVlS1vxwA9iPR+eq f8qYKlh7H+LFg== Date: Tue, 11 Apr 2023 20:48:10 -0700 Subject: [GIT PULL 13/22] xfs: fix iget/irele usage in online fsck From: "Darrick J. Wong" To: dchinner@fromorbit.com, djwong@kernel.org Cc: dchinner@redhat.com, hch@lst.de, linux-xfs@vger.kernel.org Message-ID: <168127094955.417736.8034002722203014684.stg-ugh@frogsfrogsfrogs> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-xfs@vger.kernel.org Hi Dave, Please pull this branch with changes for xfs. As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems. --D The following changes since commit 30f8ee5e7e0ccce396dff209c6cbce49d0d7e167: xfs: ensure that single-owner file blocks are not owned by others (2023-04-11 19:00:16 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/scrub-iget-fixes-6.4_2023-04-11 for you to fetch changes up to 1fc7a0597d237c17b6501f8c33b76d3eaaae9079: xfs: don't take the MMAPLOCK when scrubbing file metadata (2023-04-11 19:00:22 -0700) ---------------------------------------------------------------- xfs: fix iget/irele usage in online fsck [v24.5] This patchset fixes a handful of problems relating to how we get and release incore inodes in the online scrub code. The first patch fixes how we handle DONTCACHE -- our reasons for setting (or clearing it) depend entirely on the runtime environment at irele time. Hence we can refactor iget and irele to use our own wrappers that set that context appropriately. The second patch fixes a race between the iget call in the inode core scrubber and other writer threads that are allocating or freeing inodes in the same AG by changing the behavior of xchk_iget (and the inode core scrub setup function) to return either an incore inode or the AGI buffer so that we can be sure that the inode cannot disappear on us. The final patch elides MMAPLOCK from scrub paths when possible. It did not fit anywhere else. Signed-off-by: Darrick J. Wong ---------------------------------------------------------------- Darrick J. Wong (12): xfs: use the directory name hash function for dir scrubbing xfs: streamline the directory iteration code for scrub xfs: xfs_iget in the directory scrubber needs to use UNTRUSTED xfs: always check the existence of a dirent's child inode xfs: remove xchk_parent_count_parent_dentries xfs: simplify xchk_parent_validate xfs: fix parent pointer scrub racing with subdirectory reparenting xfs: manage inode DONTCACHE status at irele time xfs: fix an inode lookup race in xchk_get_inode xfs: rename xchk_get_inode -> xchk_iget_for_scrubbing xfs: retain the AGI when we can't iget an inode to scrub the core xfs: don't take the MMAPLOCK when scrubbing file metadata fs/xfs/Makefile | 1 + fs/xfs/scrub/bmap.c | 9 +- fs/xfs/scrub/common.c | 328 ++++++++++++++++++++++++++++++++---------- fs/xfs/scrub/common.h | 11 +- fs/xfs/scrub/dir.c | 240 ++++++++++--------------------- fs/xfs/scrub/inode.c | 191 +++++++++++++++++++++---- fs/xfs/scrub/parent.c | 308 +++++++++++++--------------------------- fs/xfs/scrub/readdir.c | 375 +++++++++++++++++++++++++++++++++++++++++++++++++ fs/xfs/scrub/readdir.h | 19 +++ fs/xfs/scrub/scrub.c | 2 +- fs/xfs/xfs_icache.c | 3 +- fs/xfs/xfs_icache.h | 11 +- 12 files changed, 1009 insertions(+), 489 deletions(-) create mode 100644 fs/xfs/scrub/readdir.c create mode 100644 fs/xfs/scrub/readdir.h From patchwork Wed Apr 12 03:48:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13208406 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 851EBC7619A for ; Wed, 12 Apr 2023 03:48:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229491AbjDLDs3 (ORCPT ); Tue, 11 Apr 2023 23:48:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38584 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229458AbjDLDs3 (ORCPT ); Tue, 11 Apr 2023 23:48:29 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2DBC010C0 for ; Tue, 11 Apr 2023 20:48:28 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id BC38A629DF for ; Wed, 12 Apr 2023 03:48:27 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 24230C433EF; Wed, 12 Apr 2023 03:48:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1681271307; bh=Ba12Kihk+Xd3tJm7QCt9zgvBJ+yVNjskTCKJu5/eKTc=; h=Date:Subject:From:To:Cc:From; b=O/uZ+nbT/b83WSyvOIPOXmKs2R68T8bfXfKOGdhyAO4bl7Y4XC/NMzVbPEqAKAtLp tMK3tJZnOvYQMQc8GkLu09glLjK3fZ/ZRWBoPTCC29zru60jcegSopV5s90XDHynTU hMSfN5HhYIOEAo8Jdd1TnZlCgQbVdT6N5SaxoHXstjTwKZlGXPZVfJKRKWD7bHErzv a80DDdkQ0mZI8AMgLYp5jvSA//VgCyCpgxUhP0V9DyqGEWc8IfKr1BEern75p77QpK iIahDe4wHCHEulgx1YPhIbD1juqqvPLC4un3u9nsJTrKO/rLYnrMpyOhnzDv5KaO0P CSyLw/8wcYWLQ== Date: Tue, 11 Apr 2023 20:48:26 -0700 Subject: [GIT PULL 14/22] xfs: fix bugs in parent pointer checking From: "Darrick J. Wong" To: dchinner@fromorbit.com, djwong@kernel.org Cc: linux-xfs@vger.kernel.org Message-ID: <168127095051.417736.2174858080826643116.stg-ugh@frogsfrogsfrogs> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-xfs@vger.kernel.org Hi Dave, Please pull this branch with changes for xfs. As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems. --D The following changes since commit 0916056eba4fd816f8042a3960597c316ea10256: xfs: fix parent pointer scrub racing with subdirectory reparenting (2023-04-11 19:00:20 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/scrub-parent-fixes-6.4_2023-04-11 for you to fetch changes up to 0916056eba4fd816f8042a3960597c316ea10256: xfs: fix parent pointer scrub racing with subdirectory reparenting (2023-04-11 19:00:20 -0700) ---------------------------------------------------------------- xfs: fix bugs in parent pointer checking [v24.5] Jan Kara pointed out that the VFS doesn't take i_rwsem of a child subdirectory that is being moved from one parent to another. Upon deeper analysis, I realized that this was the source of a very hard to trigger false corruption report in the parent pointer checking code. Now that we've refactored how directory walks work in scrub, we can also get rid of all the unnecessary and broken locking to make parent pointer scrubbing work properly. Signed-off-by: Darrick J. Wong ---------------------------------------------------------------- From patchwork Wed Apr 12 03:48:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13208407 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 80087C7619A for ; Wed, 12 Apr 2023 03:48:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229492AbjDLDss (ORCPT ); Tue, 11 Apr 2023 23:48:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38610 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229458AbjDLDso (ORCPT ); Tue, 11 Apr 2023 23:48:44 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D548E10C0 for ; Tue, 11 Apr 2023 20:48:43 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 70A5A62B68 for ; Wed, 12 Apr 2023 03:48:43 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id CA1ABC433EF; Wed, 12 Apr 2023 03:48:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1681271322; bh=k0SgcYTg9L7NAEBJjQxuVztTF//Bx4yEJtdfdd4AvE4=; h=Date:Subject:From:To:Cc:From; b=duFs0MOenSxM4a1IvhmkMC82P2vjxHvGBdjzl3BFM/DYIb3Q6PHhNfzCxT5P315JW obZuKT94KWJ3QRFfWFpfkqwy6EuGF2Vy3HXaSt2r/W9f4T426PrIsnM9QKH5Ry4/TL 9gsUs9upr5ehPAVOlU6SxfahrtXDMyMkM8UFQiRSiIbxel9bfjDtAK3uAT8k1Gguz+ rUC2Nixmg/Q7b/K2t6cCsyX92KbPklnKLil64i8IQR+ykDwQkx3LzLX2MPoGYSJebH hG/+3r+HFHnwNaMbkvl4dxV4PPXMtdulkeAxfwIwFrHj5g787wieCLRHWTLlMxTE33 3iUNfKU0GLp9w== Date: Tue, 11 Apr 2023 20:48:42 -0700 Subject: [GIT PULL 15/22] xfs: fix iget usage in directory scrub From: "Darrick J. Wong" To: dchinner@fromorbit.com, djwong@kernel.org Cc: linux-xfs@vger.kernel.org Message-ID: <168127095149.417736.13949355066335699103.stg-ugh@frogsfrogsfrogs> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-xfs@vger.kernel.org Hi Dave, Please pull this branch with changes for xfs. As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems. --D The following changes since commit 6bb9209ceebb07fd07cec25af04eed1809c654de: xfs: always check the existence of a dirent's child inode (2023-04-11 19:00:18 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/scrub-dir-iget-fixes-6.4_2023-04-11 for you to fetch changes up to 6bb9209ceebb07fd07cec25af04eed1809c654de: xfs: always check the existence of a dirent's child inode (2023-04-11 19:00:18 -0700) ---------------------------------------------------------------- xfs: fix iget usage in directory scrub [v24.5] In this series, we fix some problems with how the directory scrubber grabs child inodes. First, we want to reduce EDEADLOCK returns by replacing fixed-iteration loops with interruptible trylock loops. Second, we add UNTRUSTED to the child iget call so that we can detect a dirent that points to an unallocated inode. Third, we fix a bug where we weren't checking the inode pointed to by dotdot entries at all. Signed-off-by: Darrick J. Wong ---------------------------------------------------------------- From patchwork Wed Apr 12 03:48:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13208408 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B85EAC77B6E for ; Wed, 12 Apr 2023 03:49:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229527AbjDLDtD (ORCPT ); Tue, 11 Apr 2023 23:49:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38646 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229458AbjDLDtA (ORCPT ); Tue, 11 Apr 2023 23:49:00 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6423410C0 for ; Tue, 11 Apr 2023 20:48:59 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 08F426298F for ; Wed, 12 Apr 2023 03:48:59 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 63665C4339B; Wed, 12 Apr 2023 03:48:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1681271338; bh=kFKu42g/lIpvNTJZ67kOTIUcJ2pseGD+MYr5gMf/xo0=; h=Date:Subject:From:To:Cc:From; b=M6+dYkFvz13CG7VLofDrAauDq+46DZzz0lfcKWjFk9kYWYqEGPPj74O7UEExRMlNg 5qldsNQrFCCMuBdjHJFP8dtaz+tueR4vp0BS6lsK0PJeBbpUefFnSEvY6o49N3ADY5 kmT5zc+eVZgTCDh13+rUQln69dTklqoZTrFVk8VWlVJlgWIZYx27B7n3SP2poQ+xsf a0HEGdT6cEB4l2LWGGy4rM7T5e7tIffnfKmLz/R9YcjS78FfdmetcjXqe5/UAuFCux xg9dYjJmD70BzSERr2r/+AwayBDVPDq7l5wPga7UskCbSpgXose/IjrFtVPl6wtNHO tC6ouCn3PwXrw== Date: Tue, 11 Apr 2023 20:48:58 -0700 Subject: [GIT PULL 16/22] xfs: merge bmap records for faster scrubs From: "Darrick J. Wong" To: dchinner@fromorbit.com, djwong@kernel.org Cc: dchinner@redhat.com, linux-xfs@vger.kernel.org Message-ID: <168127095245.417736.9350032118598729884.stg-ugh@frogsfrogsfrogs> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-xfs@vger.kernel.org Hi Dave, Please pull this branch with changes for xfs. As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems. --D The following changes since commit 6bb9209ceebb07fd07cec25af04eed1809c654de: xfs: always check the existence of a dirent's child inode (2023-04-11 19:00:18 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/scrub-merge-bmap-records-6.4_2023-04-11 for you to fetch changes up to 1e59fdb7d6157ff685a250e0873a015a2b16a4f2: xfs: don't call xchk_bmap_check_rmaps for btree-format file forks (2023-04-11 19:00:26 -0700) ---------------------------------------------------------------- xfs: merge bmap records for faster scrubs [v24.5] I started looking into performance problems with the data fork scrubber in generic/333, and noticed a few things that needed improving. First, due to design reasons, it's possible for file forks btrees to contain multiple contiguous mappings to the same physical space. Instead of checking each ondisk mapping individually, it's much faster to combine them when possible and check the combined mapping because that's fewer trips through the rmap btree, and we can drop this check-around behavior that it does when an rmapbt lookup produces a record that starts before or ends after a particular bmbt mapping. Second, I noticed that the bmbt scrubber decides to walk every reverse mapping in the filesystem if the file fork is in btree format. This is very costly, and only necessary if the inode repair code had to zap a fork to convince iget to work. Constraining the full-rmap scan to this one case means we can skip it for normal files, which drives the runtime of this test from 8 hours down to 45 minutes (observed with realtime reflink and rebuild-all mode.) Signed-off-by: Darrick J. Wong ---------------------------------------------------------------- Darrick J. Wong (14): xfs: remove xchk_parent_count_parent_dentries xfs: simplify xchk_parent_validate xfs: fix parent pointer scrub racing with subdirectory reparenting xfs: manage inode DONTCACHE status at irele time xfs: fix an inode lookup race in xchk_get_inode xfs: rename xchk_get_inode -> xchk_iget_for_scrubbing xfs: retain the AGI when we can't iget an inode to scrub the core xfs: don't take the MMAPLOCK when scrubbing file metadata xfs: change bmap scrubber to store the previous mapping xfs: accumulate iextent records when checking bmap xfs: split xchk_bmap_xref_rmap into two functions xfs: alert the user about data/attr fork mappings that could be merged xfs: split the xchk_bmap_check_rmaps into a predicate xfs: don't call xchk_bmap_check_rmaps for btree-format file forks fs/xfs/libxfs/xfs_bmap.h | 2 +- fs/xfs/scrub/bmap.c | 388 ++++++++++++++++++++++++++++++----------------- fs/xfs/scrub/common.c | 328 ++++++++++++++++++++++++++++++--------- fs/xfs/scrub/common.h | 11 +- fs/xfs/scrub/dir.c | 14 +- fs/xfs/scrub/inode.c | 191 +++++++++++++++++++---- fs/xfs/scrub/parent.c | 237 ++++++++++------------------- fs/xfs/scrub/scrub.c | 2 +- fs/xfs/xfs_icache.c | 3 +- fs/xfs/xfs_icache.h | 11 +- 10 files changed, 765 insertions(+), 422 deletions(-) From patchwork Wed Apr 12 03:49:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13208409 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3B8AC77B6E for ; Wed, 12 Apr 2023 03:49:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229456AbjDLDt0 (ORCPT ); Tue, 11 Apr 2023 23:49:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38966 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229696AbjDLDtX (ORCPT ); Tue, 11 Apr 2023 23:49:23 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2F7024C2C for ; Tue, 11 Apr 2023 20:49:15 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 9A65D62D90 for ; Wed, 12 Apr 2023 03:49:14 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 02E37C433EF; Wed, 12 Apr 2023 03:49:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1681271354; bh=Uv+TZMzDGXZPHuHxIbJ1NvfvAk0l5MaI6q+K9knhyus=; h=Date:Subject:From:To:Cc:From; b=hM2COnbJEYs/G4rNoqBfbI3k8bLb/wD8SFztSIb5RHiNTk4YRPXpRhwh8N2ldKSac XskmbMksm3QDHdarRO9t1Ai5ZGguMFtsWdz0lbVmIywnYmLZw7U+XWaY5Qaq1af6ZD IhMMb2jpAV3zhn1CmSLCPaBlJCa6BJ6xcxNvjAxqBjl+ux9HDvAKBxTk3+iEU0GYpc AyRiYa3boJPs4ufqpnq6U39YQBKU408383k9cXGHi21MLsji6sOF5DBgdbwLC3m0Ck /jWj+EPbZk/KDvVhfxqvhXwmObMDhRGCoDPMqjMeF4XA4XI2eEsvaEiE2R78UWs12F tq+3NhvTJvp0g== Date: Tue, 11 Apr 2023 20:49:13 -0700 Subject: [GIT PULL 17/22] xfs: detect mergeable and overlapping btree records From: "Darrick J. Wong" To: dchinner@fromorbit.com, djwong@kernel.org Cc: dchinner@redhat.com, linux-xfs@vger.kernel.org Message-ID: <168127095343.417736.18039725981807061339.stg-ugh@frogsfrogsfrogs> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-xfs@vger.kernel.org Hi Dave, Please pull this branch with changes for xfs. As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems. --D The following changes since commit 1e59fdb7d6157ff685a250e0873a015a2b16a4f2: xfs: don't call xchk_bmap_check_rmaps for btree-format file forks (2023-04-11 19:00:26 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/scrub-detect-mergeable-records-6.4_2023-04-11 for you to fetch changes up to 1c1646afc96783702f92356846d6e47e0bbd6b11: xfs: check for reverse mapping records that could be merged (2023-04-11 19:00:28 -0700) ---------------------------------------------------------------- xfs: detect mergeable and overlapping btree records [v24.5] While I was doing differential fuzz analysis between xfs_scrub and xfs_repair, I noticed that xfs_repair was only partially effective at detecting btree records that can be merged, and xfs_scrub totally didn't notice at all. For every interval btree type except for the bmbt, there should never exist two adjacent records with adjacent keyspaces because the blockcount field is always large enough to span the entire keyspace of the domain. This is because the free space, rmap, and refcount btrees have a blockcount field large enough to store the maximum AG length, and there can never be an allocation larger than an AG. The bmbt is a different story due to its ondisk encoding where the blockcount is only 21 bits wide. Because AGs can span up to 2^31 blocks and the RT volume can span up to 2^52 blocks, a preallocation of 2^22 blocks will be expressed as two records of 2^21 length. We don't opportunistically combine records when doing bmbt operations, which is why the fsck tools have never complained about this scenario. Offline repair is partially effective at detecting mergeable records because I taught it to do that for the rmap and refcount btrees. This series enhances the free space, rmap, and refcount scrubbers to detect mergeable records. For the bmbt, it will flag the file as being eligible for an optimization to shrink the size of the data structure. The last patch in this set also enhances the rmap scrubber to detect records that overlap incorrectly. This check is done automatically for non-overlapping btree types, but we have to do it separately for the rmapbt because there are constraints on which allocation types are allowed to overlap. Signed-off-by: Darrick J. Wong ---------------------------------------------------------------- Darrick J. Wong (4): xfs: flag free space btree records that could be merged xfs: flag refcount btree records that could be merged xfs: check overlapping rmap btree records xfs: check for reverse mapping records that could be merged fs/xfs/scrub/alloc.c | 29 ++++++++++- fs/xfs/scrub/refcount.c | 44 +++++++++++++++++ fs/xfs/scrub/rmap.c | 126 +++++++++++++++++++++++++++++++++++++++++++++++- 3 files changed, 196 insertions(+), 3 deletions(-) From patchwork Wed Apr 12 03:49:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13208410 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 65B41C7619A for ; Wed, 12 Apr 2023 03:49:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229507AbjDLDtc (ORCPT ); Tue, 11 Apr 2023 23:49:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39192 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229458AbjDLDtb (ORCPT ); Tue, 11 Apr 2023 23:49:31 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A6FC440C1 for ; Tue, 11 Apr 2023 20:49:30 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 38C6D62D91 for ; Wed, 12 Apr 2023 03:49:30 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8E52FC433EF; Wed, 12 Apr 2023 03:49:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1681271369; bh=ogJWvQIKL0uYE3EJ8MYU4KLD/z1maE7uiyQ2o92oOuU=; h=Date:Subject:From:To:Cc:From; b=jkpI+UB5etu2dV5pKJZXke8NiL4xDw3CThHDsm98trdmL5YhwOoDUmndDyLZlGyWu WSsbEDwPwxJV8PuN8ZqAE9ORz6kxBtQKu5/CiiX6iAPSFsahfXRrxwTLs49HdHaqGc Dcr33NdMD8Z67k1WoarPraxFONIZbE2HTAkIibjtGXdoR2gBYt3zlzXf5ONO5sjdi0 EB49gFEx5D12JYAy9u/FFOKIkoVhR2NVHgJV4VuTZhfGMeUyJiESc94BLgXN6Cf5p7 GQpuxl35zcBZZux+lVTyt7QPQROVoV4kc6LZK2WqhsR5d8B84PxFtQ5n4yqgtovZhd eK13saFN6gi1w== Date: Tue, 11 Apr 2023 20:49:29 -0700 Subject: [GIT PULL 18/22] xfs: clean up memory management in xattr scrub From: "Darrick J. Wong" To: dchinner@fromorbit.com, djwong@kernel.org Cc: dchinner@redhat.com, linux-xfs@vger.kernel.org Message-ID: <168127095443.417736.15033228784686723319.stg-ugh@frogsfrogsfrogs> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-xfs@vger.kernel.org Hi Dave, Please pull this branch with changes for xfs. As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems. --D The following changes since commit 1c1646afc96783702f92356846d6e47e0bbd6b11: xfs: check for reverse mapping records that could be merged (2023-04-11 19:00:28 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/scrub-fix-xattr-memory-mgmt-6.4_2023-04-11 for you to fetch changes up to 44af6c7e59b12d740809cf25a60c9f90f03e6d20: xfs: don't load local xattr values during scrub (2023-04-11 19:00:35 -0700) ---------------------------------------------------------------- xfs: clean up memory management in xattr scrub [v24.5] Currently, the extended attribute scrubber uses a single VLA to store all the context information needed in various parts of the scrubber code. This includes xattr leaf block space usage bitmaps, and the value buffer used to check the correctness of remote xattr value block headers. We try to minimize the insanity through the use of helper functions, but this is a memory management nightmare. Clean this up by making the bitmap and value pointers explicit members of struct xchk_xattr_buf. Second, strengthen the xattr checking by teaching it to look for overlapping data structures in the shortform attr data. Signed-off-by: Darrick J. Wong ---------------------------------------------------------------- Darrick J. Wong (12): xfs: xattr scrub should ensure one namespace bit per name xfs: don't shadow @leaf in xchk_xattr_block xfs: remove unnecessary dstmap in xattr scrubber xfs: split freemap from xchk_xattr_buf.buf xfs: split usedmap from xchk_xattr_buf.buf xfs: split valuebuf from xchk_xattr_buf.buf xfs: remove flags argument from xchk_setup_xattr_buf xfs: move xattr scrub buffer allocation to top level function xfs: check used space of shortform xattr structures xfs: clean up xattr scrub initialization xfs: only allocate free space bitmap for xattr scrub if needed xfs: don't load local xattr values during scrub fs/xfs/scrub/attr.c | 306 ++++++++++++++++++++++++++++++++++++--------------- fs/xfs/scrub/attr.h | 60 ++-------- fs/xfs/scrub/scrub.c | 3 + fs/xfs/scrub/scrub.h | 10 ++ 4 files changed, 239 insertions(+), 140 deletions(-) From patchwork Wed Apr 12 03:49:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13208411 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53492C7619A for ; Wed, 12 Apr 2023 03:49:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229696AbjDLDts (ORCPT ); Tue, 11 Apr 2023 23:49:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39254 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229604AbjDLDtr (ORCPT ); Tue, 11 Apr 2023 23:49:47 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3EBDA10C0 for ; Tue, 11 Apr 2023 20:49:46 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id CDF0C629DF for ; Wed, 12 Apr 2023 03:49:45 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2CA6BC433D2; Wed, 12 Apr 2023 03:49:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1681271385; bh=mq4ChGg2tCK4P1Cw1VoitjDRRBFPFFbMgsqMkm3xvYw=; h=Date:Subject:From:To:Cc:From; b=GRRssJAZIRBPcoAMaZlB6VVxccKCWPsMA2rJg7Pm7lQu2/KcWlRTr3uCnV/u6Ah5Z ZQQPIojBjnZinZ0VzRC3rU8h6J7z1eDJE+0d7C61XRT+/dNDOyPxZbUCi2py6CvuK6 V/ptnITqX2U8+kRTv3gflYrPhpGlrLNOAKmwlpdKUMGXRSgxMBD8+jN11bsoRov7Ab m6bAwZyaEnwX+ai/epwbi4EvDJs9mApBNGNf5PItPqCBCdaUL1rdtYWyTAjsl3u5Gg HHcy0c4l7F39wQE2s02xadHjWxZ5lbW3/oghLMXW186iBxwm4POrwlEssRYvAVfayY tNQOlgfWJFXvQ== Date: Tue, 11 Apr 2023 20:49:44 -0700 Subject: [GIT PULL 19/22] xfs: rework online fsck incore bitmap From: "Darrick J. Wong" To: dchinner@fromorbit.com, djwong@kernel.org Cc: dchinner@redhat.com, linux-xfs@vger.kernel.org Message-ID: <168127095543.417736.17628317256060611866.stg-ugh@frogsfrogsfrogs> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-xfs@vger.kernel.org Hi Dave, Please pull this branch with changes for xfs. As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems. --D The following changes since commit 44af6c7e59b12d740809cf25a60c9f90f03e6d20: xfs: don't load local xattr values during scrub (2023-04-11 19:00:35 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/repair-bitmap-rework-6.4_2023-04-11 for you to fetch changes up to 6772fcc8890ae34595253fcfb8196c1aea65e111: xfs: convert xbitmap to interval tree (2023-04-11 19:00:36 -0700) ---------------------------------------------------------------- xfs: rework online fsck incore bitmap [v24.5] In this series, we make some changes to the incore bitmap code: First, we shorten the prefix to 'xbitmap'. Then, we rework some utility functions for later use by online repair and clarify how the walk functions are supposed to be used. Finally, we use all these new pieces to convert the incore bitmap to use an interval tree instead of linked lists. This lifts the limitation that callers had to be careful not to set a range that was already set; and gets us ready for the btree rebuilder functions needing to be able to set bits in a bitmap and generate maximal contiguous extents for the set ranges. Signed-off-by: Darrick J. Wong ---------------------------------------------------------------- Darrick J. Wong (3): xfs: remove the for_each_xbitmap_ helpers xfs: drop the _safe behavior from the xbitmap foreach macro xfs: convert xbitmap to interval tree fs/xfs/scrub/agheader_repair.c | 99 ++++++----- fs/xfs/scrub/bitmap.c | 367 +++++++++++++++++++++++++---------------- fs/xfs/scrub/bitmap.h | 33 ++-- fs/xfs/scrub/repair.c | 104 ++++++------ 4 files changed, 358 insertions(+), 245 deletions(-) From patchwork Wed Apr 12 03:50:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13208412 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4F76AC77B6E for ; Wed, 12 Apr 2023 03:50:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229458AbjDLDuD (ORCPT ); Tue, 11 Apr 2023 23:50:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39370 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229546AbjDLDuC (ORCPT ); Tue, 11 Apr 2023 23:50:02 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D499A40D7 for ; Tue, 11 Apr 2023 20:50:01 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 69709629DF for ; Wed, 12 Apr 2023 03:50:01 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C7163C433D2; Wed, 12 Apr 2023 03:50:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1681271400; bh=eg2PzQ/bAS9Le06OYNQPLcSWbbhtonLngp4I16kgIBQ=; h=Date:Subject:From:To:Cc:From; b=QBPII7zQMf73m8AMqbNACvPhkKBXUmhFn1fjgV/FvZmRu+0Cvc8+IjRCdNJc8jBPE j/3AGZYwd4Zx03FfbUH80Or3kY+HIJSaw1fgPYi+sh6qyr9N9bhhfBeeYuvLZR/z4o IOE+JswVsYePKWS1jxDSxzKQjkAtjzgISKdTl4jQoGru4TKcIdBCs9y4Wjln4SAZlt xj0Ds4xkflctE3AI35PVfIivCR234wc/UhhyWuYHIdDfftm990fTCM17U8oVDufXFw ERtRiKZowhOJJfsptrMXBGyz9FNs5Vra1WqUyqwyKwSD8IxkSY6O7V1DnhDbFwCCko S3P0JX6twUOBQ== Date: Tue, 11 Apr 2023 20:50:00 -0700 Subject: [GIT PULL 20/22] xfs: strengthen rmapbt scrubbing From: "Darrick J. Wong" To: dchinner@fromorbit.com, djwong@kernel.org Cc: dchinner@redhat.com, linux-xfs@vger.kernel.org Message-ID: <168127095646.417736.15195904792324204986.stg-ugh@frogsfrogsfrogs> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-xfs@vger.kernel.org Hi Dave, Please pull this branch with changes for xfs. As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems. --D The following changes since commit 6772fcc8890ae34595253fcfb8196c1aea65e111: xfs: convert xbitmap to interval tree (2023-04-11 19:00:36 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/scrub-strengthen-rmap-checking-6.4_2023-04-11 for you to fetch changes up to 4f5e304248ab4939e9aef58244041c194f01f0b5: xfs: cross-reference rmap records with refcount btrees (2023-04-11 19:00:39 -0700) ---------------------------------------------------------------- xfs: strengthen rmapbt scrubbing [v24.5] This series strengthens space allocation record cross referencing by using AG block bitmaps to compute the difference between space used according to the rmap records and the primary metadata, and reports cross-referencing errors for any discrepancies. Signed-off-by: Darrick J. Wong ---------------------------------------------------------------- Darrick J. Wong (5): xfs: introduce bitmap type for AG blocks xfs: cross-reference rmap records with ag btrees xfs: cross-reference rmap records with free space btrees xfs: cross-reference rmap records with inode btrees xfs: cross-reference rmap records with refcount btrees fs/xfs/Makefile | 2 +- fs/xfs/scrub/bitmap.c | 55 ++++++++++ fs/xfs/scrub/bitmap.h | 72 +++++++++++++ fs/xfs/scrub/repair.h | 1 + fs/xfs/scrub/rmap.c | 283 +++++++++++++++++++++++++++++++++++++++++++++++++- 5 files changed, 411 insertions(+), 2 deletions(-) From patchwork Wed Apr 12 03:50:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13208413 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EEB35C7619A for ; Wed, 12 Apr 2023 03:50:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229484AbjDLDuU (ORCPT ); Tue, 11 Apr 2023 23:50:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39440 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229546AbjDLDuS (ORCPT ); Tue, 11 Apr 2023 23:50:18 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 678FF40C1 for ; Tue, 11 Apr 2023 20:50:17 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 02D6762D89 for ; Wed, 12 Apr 2023 03:50:17 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5B835C433D2; Wed, 12 Apr 2023 03:50:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1681271416; bh=HVD7RK79pZ3fAGWRmAXOZEsi1ZPbB7HRN8nEYVoB600=; h=Date:Subject:From:To:Cc:From; b=aeYIl9VOMj3xK5XKQYsUTwOFayI+X62UxDNYdzNzK4RZzFYIrbiVSAtTAAAFS3/ni 7m6L/qJhBVYdJkhV6heFVDGoaRJdrazGYpxGtTVoKDrnZXALOwOdwftjq2pqW0OSBh VDsrxRMLfIUIqcBfqDF5iIql4WLjXsMCBj43FAAxE4b4wgog12ELJZlHOehg+HF30F NGuaIEiSPOUJfzEzGb6D0jCJgICQZ220tTIVtU9r5vXjlF0PTSxBabb8QSwJnwM/uy Zy61mx5G4ya1O78aK3H8Z+i/9R3oLjKX6isU7lledJgEaXqetEpI3DBas7WIcc6Zwr 5YOefHBjPzcTA== Date: Tue, 11 Apr 2023 20:50:15 -0700 Subject: [GIT PULL 21/22] xfs: fix ascii-ci problems, then kill it From: "Darrick J. Wong" To: dchinner@fromorbit.com, djwong@kernel.org Cc: david@fromorbit.com, hch@infradead.org, hch@lst.de, linux-xfs@vger.kernel.org Message-ID: <168127095745.417736.210471032478306085.stg-ugh@frogsfrogsfrogs> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-xfs@vger.kernel.org Hi Dave, Please pull this branch with changes for xfs. As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems. --D The following changes since commit 4f5e304248ab4939e9aef58244041c194f01f0b5: xfs: cross-reference rmap records with refcount btrees (2023-04-11 19:00:39 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/fix-asciici-bugs-6.4_2023-04-11 for you to fetch changes up to 7ba83850ca2691865713b307ed001bde5fddb084: xfs: deprecate the ascii-ci feature (2023-04-11 19:05:19 -0700) ---------------------------------------------------------------- xfs: fix ascii-ci problems, then kill it [v2] Last week, I was fiddling around with the metadump name obfuscation code while writing a debugger command to generate directories full of names that all have the same hash name. I had a few questions about how well all that worked with ascii-ci mode, and discovered a nasty discrepancy between the kernel and glibc's implementations of the tolower() function. I discovered that I could create a directory that is large enough to require separate leaf index blocks. The hashes stored in the dabtree use the ascii-ci specific hash function, which uses a library function to convert the name to lowercase before hashing. If the kernel and C library's versions of tolower do not behave exactly identically, xfs_ascii_ci_hashname will not produce the same results for the same inputs. xfs_repair will deem the leaf information corrupt and rebuild the directory. After that, lookups in the kernel will fail because the hash index doesn't work. The kernel's tolower function will convert extended ascii uppercase letters (e.g. A-with-umlaut) to extended ascii lowercase letters (e.g. a-with-umlaut), whereas glibc's will only do that if you force LANG to ascii. Tiny embedded libc implementations just plain won't do it at all, and the result is a mess. Stabilize the behavior of the hash function by encoding the name transformation function in libxfs, add it to the selftest, and fix all the userspace tools, none of which handle this transformation correctly. The v1 series generated a /lot/ of discussion, in which several things became very clear: (1) Linus is not enamored of case folding of any kind; (2) Dave and Christoph don't seem to agree on whether the feature is supposed to work for 7-bit ascii or latin1; (3) it trashes UTF8 encoded names if those happen to show up; and (4) I don't want to maintain this mess any longer than I have to. Kill it in 2030. v2: rename the functions to make it clear we're moving away from the letters t, o, l, o, w, e, and r; and deprecate the whole feature once we've fixed the bugs and added tests. Signed-off-by: Darrick J. Wong ---------------------------------------------------------------- Darrick J. Wong (3): xfs: stabilize the dirent name transformation function used for ascii-ci dir hash computation xfs: test the ascii case-insensitive hash xfs: deprecate the ascii-ci feature Documentation/admin-guide/xfs.rst | 1 + fs/xfs/Kconfig | 27 +++++ fs/xfs/libxfs/xfs_dir2.c | 5 +- fs/xfs/libxfs/xfs_dir2.h | 31 ++++++ fs/xfs/xfs_dahash_test.c | 211 ++++++++++++++++++++------------------ fs/xfs/xfs_super.c | 13 +++ 6 files changed, 186 insertions(+), 102 deletions(-) From patchwork Wed Apr 12 03:50:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Darrick J. Wong" X-Patchwork-Id: 13208414 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 03051C77B73 for ; Wed, 12 Apr 2023 03:50:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229546AbjDLDue (ORCPT ); Tue, 11 Apr 2023 23:50:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39474 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229517AbjDLDue (ORCPT ); Tue, 11 Apr 2023 23:50:34 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0A10E10C0; Tue, 11 Apr 2023 20:50:33 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 9801E62CAC; Wed, 12 Apr 2023 03:50:32 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 01009C433D2; Wed, 12 Apr 2023 03:50:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1681271432; bh=fu1hYI5jT4tiB4AU2l5sPWhTCilqWNeftIP5svSQfSk=; h=Date:Subject:From:To:Cc:From; b=hc1dgI+vbzzROsUuwgbg8MqTqNW8K295MIAVp+fbpU/MvsAMppUmSYc015F4/MKs/ 7LZ0GB+JJsXF0mKyvHL1xk0botyjzTsvGgKFH0t39VGmYwVoehas/ZPDFv6ISbn4xx shxpSIrven8ASVAjOuMDXlO8IsJU3HnCn2HxqAUZvjKbOuRjuIPolKeS48R+rvpd7M 3Jgf6FfA/LtTBD8EkXxF5XRYUPoEx8hDQKigIilM3ZlQn0Kw+WdEo8jFPC47MZw5F7 44lKA5QETDNwB+kNxsJJSHSig8yNH/isnfuCs8Xf30Jp04W1GBwi3jUA1p2Ogjh8DB zZ/arnY9ewBlw== Date: Tue, 11 Apr 2023 20:50:31 -0700 Subject: [GIT PULL 22/22] xfs: rollup of various bug fixes From: "Darrick J. Wong" To: dchinner@fromorbit.com, djwong@kernel.org Cc: dchinner@redhat.com, linux-xfs@vger.kernel.org, stable@vger.kernel.org, syzbot+6ae213503fb12e87934f@syzkaller.appspotmail.com, yebin10@huawei.com Message-ID: <168127095854.417736.18149025693130053322.stg-ugh@frogsfrogsfrogs> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-xfs@vger.kernel.org Hi Dave, Please pull this branch with changes for xfs. As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems. --D The following changes since commit 7ba83850ca2691865713b307ed001bde5fddb084: xfs: deprecate the ascii-ci feature (2023-04-11 19:05:19 -0700) are available in the Git repository at: git://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/fix-bugs-6.4_2023-04-11 for you to fetch changes up to 4fe9cd8a34a1f934e5f936d4245e19300b52d440: xfs: fix BUG_ON in xfs_getbmap() (2023-04-11 19:48:08 -0700) ---------------------------------------------------------------- xfs: rollup of various bug fixes This is a rollup of miscellaneous bug fixes. Signed-off-by: Darrick J. Wong ---------------------------------------------------------------- Darrick J. Wong (2): xfs: _{attr,data}_map_shared should take ILOCK_EXCL until iread_extents is completely done xfs: verify buffer contents when we skip log replay Dave Chinner (2): xfs: remove WARN when dquot cache insertion fails xfs: don't consider future format versions valid Ye Bin (1): xfs: fix BUG_ON in xfs_getbmap() fs/xfs/libxfs/xfs_bmap.c | 6 ++++++ fs/xfs/libxfs/xfs_inode_fork.c | 16 +++++++++++++++- fs/xfs/libxfs/xfs_inode_fork.h | 6 ++++-- fs/xfs/libxfs/xfs_sb.c | 11 ++++++----- fs/xfs/xfs_bmap_util.c | 14 ++++++-------- fs/xfs/xfs_buf_item_recover.c | 10 ++++++++++ fs/xfs/xfs_dquot.c | 1 - 7 files changed, 47 insertions(+), 17 deletions(-)