Hi Chandan,
Please pull this branch with changes for xfs for 6.6-rc1.
As usual, I did a test-merge with the main upstream branch as of a few
minutes ago, and didn't see any conflicts. Please let me know if you
encounter any problems.
--D
The following changes since commit 014ad53732d2bac34d21a251f3622a4da516e21b:
xfs: use per-AG bitmaps to reap unused AG metadata blocks during repair (2023-08-10 07:48:04 -0700)
are available in the Git repository at:
https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/big-array-6.6_2023-08-10
for you to fetch changes up to 764018caa99f7629cefc92257a26b83289a674f3:
xfs: improve xfarray quicksort pivot (2023-08-10 07:48:07 -0700)
----------------------------------------------------------------
xfs: stage repair information in pageable memory [v26.1]
In general, online repair of an indexed record set walks the filesystem
looking for records. These records are sorted and bulk-loaded into a
new btree. To make this happen without pinning gigabytes of metadata in
memory, first create an abstraction ('xfile') of memfd files so that
kernel code can access paged memory, and then an array abstraction
('xfarray') based on xfiles so that online repair can create an array of
new records without pinning memory.
These two data storage abstractions are critical for repair of space
metadata -- the memory used is pageable, which helps us avoid pinning
kernel memory and driving OOM problems; and they are byte-accessible
enough that we can use them like (very slow and programmatic) memory
buffers.
Later patchsets will build on this functionality to provide blob storage
and btrees.
This has been running on the djcloud for years with no problems. Enjoy!
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
----------------------------------------------------------------
Darrick J. Wong (7):
xfs: create a big array data structure
xfs: enable sorting of xfile-backed arrays
xfs: convert xfarray insertion sort to heapsort using scratchpad memory
xfs: teach xfile to pass back direct-map pages to caller
xfs: speed up xfarray sort by sorting xfile page contents directly
xfs: cache pages used for xfarray quicksort convergence
xfs: improve xfarray quicksort pivot
fs/xfs/Kconfig | 1 +
fs/xfs/Makefile | 2 +
fs/xfs/scrub/trace.c | 4 +-
fs/xfs/scrub/trace.h | 260 ++++++++++++
fs/xfs/scrub/xfarray.c | 1083 ++++++++++++++++++++++++++++++++++++++++++++++++
fs/xfs/scrub/xfarray.h | 141 +++++++
fs/xfs/scrub/xfile.c | 420 +++++++++++++++++++
fs/xfs/scrub/xfile.h | 77 ++++
8 files changed, 1987 insertions(+), 1 deletion(-)
create mode 100644 fs/xfs/scrub/xfarray.c
create mode 100644 fs/xfs/scrub/xfarray.h
create mode 100644 fs/xfs/scrub/xfile.c
create mode 100644 fs/xfs/scrub/xfile.h
Hi Chandan, Please pull this branch with changes for xfs for 6.6-rc1. As usual, I did a test-merge with the main upstream branch as of a few minutes ago, and didn't see any conflicts. Please let me know if you encounter any problems. --D The following changes since commit 014ad53732d2bac34d21a251f3622a4da516e21b: xfs: use per-AG bitmaps to reap unused AG metadata blocks during repair (2023-08-10 07:48:04 -0700) are available in the Git repository at: https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfs-linux.git tags/big-array-6.6_2023-08-10 for you to fetch changes up to 764018caa99f7629cefc92257a26b83289a674f3: xfs: improve xfarray quicksort pivot (2023-08-10 07:48:07 -0700) ---------------------------------------------------------------- xfs: stage repair information in pageable memory [v26.1] In general, online repair of an indexed record set walks the filesystem looking for records. These records are sorted and bulk-loaded into a new btree. To make this happen without pinning gigabytes of metadata in memory, first create an abstraction ('xfile') of memfd files so that kernel code can access paged memory, and then an array abstraction ('xfarray') based on xfiles so that online repair can create an array of new records without pinning memory. These two data storage abstractions are critical for repair of space metadata -- the memory used is pageable, which helps us avoid pinning kernel memory and driving OOM problems; and they are byte-accessible enough that we can use them like (very slow and programmatic) memory buffers. Later patchsets will build on this functionality to provide blob storage and btrees. This has been running on the djcloud for years with no problems. Enjoy! Signed-off-by: Darrick J. Wong <djwong@kernel.org> ---------------------------------------------------------------- Darrick J. Wong (7): xfs: create a big array data structure xfs: enable sorting of xfile-backed arrays xfs: convert xfarray insertion sort to heapsort using scratchpad memory xfs: teach xfile to pass back direct-map pages to caller xfs: speed up xfarray sort by sorting xfile page contents directly xfs: cache pages used for xfarray quicksort convergence xfs: improve xfarray quicksort pivot fs/xfs/Kconfig | 1 + fs/xfs/Makefile | 2 + fs/xfs/scrub/trace.c | 4 +- fs/xfs/scrub/trace.h | 260 ++++++++++++ fs/xfs/scrub/xfarray.c | 1083 ++++++++++++++++++++++++++++++++++++++++++++++++ fs/xfs/scrub/xfarray.h | 141 +++++++ fs/xfs/scrub/xfile.c | 420 +++++++++++++++++++ fs/xfs/scrub/xfile.h | 77 ++++ 8 files changed, 1987 insertions(+), 1 deletion(-) create mode 100644 fs/xfs/scrub/xfarray.c create mode 100644 fs/xfs/scrub/xfarray.h create mode 100644 fs/xfs/scrub/xfile.c create mode 100644 fs/xfs/scrub/xfile.h