[00/45,v3] xfs: consolidated log and optimisation changes

Message ID	20210305051143.182133-1-david@fromorbit.com (mailing list archive)
Headers	show Return-Path: <linux-xfs-owner@kernel.org> From: Dave Chinner <david@fromorbit.com> To: linux-xfs@vger.kernel.org Subject: [PATCH 00/45 v3] xfs: consolidated log and optimisation changes Date: Fri, 5 Mar 2021 16:10:58 +1100 Message-Id: <20210305051143.182133-1-david@fromorbit.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk
Series	xfs: consolidated log and optimisation changes \| expand [00/45,v3] xfs: consolidated log and optimisation changes [01/45] xfs: initialise attr fork on inode create [02/45] xfs: log stripe roundoff is a property of the log [03/45] xfs: separate CIL commit record IO [04/45] xfs: remove xfs_blkdev_issue_flush [05/45] xfs: async blkdev cache flush [06/45] xfs: CIL checkpoint flushes caches unconditionally [07/45] xfs: remove need_start_rec parameter from xlog_write() [08/45] xfs: journal IO cache flush reductions [09/45] xfs: Fix CIL throttle hang when CIL space used going backwards [10/45] xfs: reduce buffer log item shadow allocations [11/45] xfs: xfs_buf_item_size_segment() needs to pass segment offset [12/45] xfs: optimise xfs_buf_item_size/format for contiguous regions [13/45] xfs: xfs_log_force_lsn isn't passed a LSN [14/45] xfs: AIL needs asynchronous CIL forcing [15/45] xfs: CIL work is serialised, not pipelined [16/45] xfs: type verification is expensive [17/45] xfs: No need for inode number error injection in __xfs_dir3_data_check [18/45] xfs: reduce debug overhead of dir leaf/node checks [19/45] xfs: factor out the CIL transaction header building [20/45] xfs: only CIL pushes require a start record [21/45] xfs: embed the xlog_op_header in the unmount record [22/45] xfs: embed the xlog_op_header in the commit record [23/45] xfs: log tickets don't need log client id [24/45] xfs: move log iovec alignment to preparation function [25/45] xfs: reserve space and initialise xlog_op_header in item formatting [26/45] xfs: log ticket region debug is largely useless [27/45] xfs: pass lv chain length into xlog_write() [28/45] xfs: introduce xlog_write_single() [29/45] xfs:_introduce xlog_write_partial() [30/45] xfs: xlog_write() no longer needs contwr state [31/45] xfs: CIL context doesn't need to count iovecs [32/45] xfs: use the CIL space used counter for emptiness checks [33/45] xfs: lift init CIL reservation out of xc_cil_lock [34/45] xfs: rework per-iclog header CIL reservation [35/45] xfs: introduce per-cpu CIL tracking sructure [36/45] xfs: implement percpu cil space used calculation [37/45] xfs: track CIL ticket reservation in percpu structure [38/45] xfs: convert CIL busy extents to per-cpu [39/45] xfs: Add order IDs to log items in CIL [40/45] xfs: convert CIL to unordered per cpu lists [41/45] xfs: move CIL ordering to the logvec chain [42/45] xfs: __percpu_counter_compare() inode count debug too expensive [43/45] xfs: avoid cil push lock if possible [44/45] xfs: xlog_sync() manually adjusts grant head space [45/45] xfs: expanding delayed logging design with background material

Message ID

20210305051143.182133-1-david@fromorbit.com (mailing list archive)

Headers

From: Dave Chinner <david@fromorbit.com>
To: linux-xfs@vger.kernel.org
Subject: [PATCH 00/45 v3] xfs: consolidated log and optimisation changes
Date: Fri,  5 Mar 2021 16:10:58 +1100
Message-Id: <20210305051143.182133-1-david@fromorbit.com>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: bulk

Series

xfs: consolidated log and optimisation changes | expand

Message

Dave Chinner March 5, 2021, 5:10 a.m. UTC

Hi folks,

This is a consolidated patchset of all the outstanding patches I've
sent out recently. Previous versions and sub-series descriptions
can be found here:

https://lore.kernel.org/linux-xfs/20210226050158.GW4662@dread.disaster.area/T/#mf72a4f6acc05d117adec3fea5f6fee83432ecfd4
https://lore.kernel.org/linux-xfs/20210223033442.3267258-1-david@fromorbit.com/T/#m80819d7f7940e5f216f3219baf4f98ed35643d13
https://lore.kernel.org/linux-xfs/20210223053212.3287398-1-david@fromorbit.com/T/#mad305a62ab3532493c83b4c615f21fbaf9a87ae0
https://lore.kernel.org/linux-xfs/20210223044636.3280862-1-david@fromorbit.com/T/#m791941d515fd437bf07aa42d35df5ddb124cb80f
https://lore.kernel.org/linux-xfs/20210224063459.3436852-1-david@fromorbit.com/T/#mcb037e1495e6bb4f3289e52d581e53efe2d29765
https://lore.kernel.org/linux-xfs/20210223054748.3292734-1-david@fromorbit.com/T/#md88d8f88657f1e33008f864677921f54b4d64a2f
https://lore.kernel.org/linux-xfs/20210225033725.3558450-1-david@fromorbit.com/T/#mb6a91780514d5abe4d8b04ed1d78a8f8d681f101

The changes are largely just a rebase onto a current TOT kernel, bug
fixes and modifications from review comments. These are itemised in
the change log below. It runs though the fstests auto group fine on
e multiple machines here and performance test numbers are largely
unchanged from previous versions.

I've added a couple of new patches to the end of the series, the
most notable being an update to the delayed logging design document
to include descriptions of transaction types, log space accounting
and how we use relogging to ensure rolling transactions do not
deadlock on log space.

Cheers,

Dave.

Version 3:
- rebase onto 5.12-rc1+
- aggregate many small dependent patchsets in one large one.
- simplify xlog_wait_on_iclog_lsn() back to just a call to xlog_wait_on_iclog()
- remove xfs_blkdev_issue_flush() instead of moving and renaming it.
- pass bio to xfs_flush_bdev_async() so it doesn't need allocation.
- skip cache flush in xfs_flush_bdev_async() if the underlying queue does not
require it.
- fixed whitespace in xfs_flush_bdev_async()
- remove the implicit external log's data device cache flush code and replace it
with an explicit flush in the unmount record write so that it works the same
as the new CIL checkpoint cache pre-flush mechanism. This mechanism now
guarantees metadata vs journal ordering for both internal and external logs.
- updated various commit messages
- fixed incorrect/unintended changes to xfs_log_force() behaviour
- typedef uint64_t xfs_csn_t; and conversion.
- removed stray trace_printk()s that were used for debugging.
- fixed minor formatting details.
- uninlined xlog_prepare_iovec()
- fixed up "lv chain vector and size calculation" commit message to reflect we
are only calculating and passin gin the vector byte count.
- reworked the loop in xlog_write_single() based on Christoph's suggestion. Much
cleaner!
- added patch to pass log ticket down to xlog_sync() so that it accounts the
roundoff to the log ticket rather than directly modifying grant heads. Grant
heads are hot, so every little bit helps.
- added patch to update delayed logging design doc with background material on
how transactions and log space accounting works in XFS.

Version 2:
- fix ticket reservation roundoff to include 2 roundoffs
- removed stale copied comment from roundoff initialisation.
- clarified "separation" to mean "separation for ordering purposes" in commit
message.
- added comment that newly activated, clean, empty iclogs have a LSN of 0 so are
captured by the "iclog lsn < start_lsn" case that avoids needing to wait
before releasing the commit iclog to be written.
- added async cache flush infrastructure
- convert CIL checkpoint push work it issue an unconditional metadata device
cache flush rather than asking the first iclog write to issue it via
REQ_PREFLUSH.
- cleaned up xlog_write() to remove a redundant parameter and prepare the logic
for setting flags on the iclog based on the type of operational data is being
written to the log.
- added XLOG_ICL_NEED_FUA flag to complement the NEED_FLUSH flag, allowing
callers to issue explicit flushes and clear the NEED_FLUSH flag before the
iclog is written without dropping the REQ_FUA requirement in /dev/null...
- added CIL commit-in-start-iclog optimisation that clears the NEED_FLUSH flag
to avoid an unnecessary cache flush when issuing the iclog.
- fixed typo in CIL throttle bugfix comment.
- fixed trailing whitespace in commit message.