| Message ID | 20210803083403.GI2757197@dread.disaster.area (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
| Series | [alternative] xfs: per-cpu deferred inode inactivation queues | expand |
On Tue, Aug 03, 2021 at 06:34:03PM +1000, Dave Chinner wrote: > > From: Dave Chinner <dchinner@redhat.com> > > Move inode inactivation to background work contexts so that it no > longer runs in the context that releases the final reference to an > inode. This will allow process work that ends up blocking on > inactivation to continue doing work while the filesytem processes > the inactivation in the background. > > A typical demonstration of this is unlinking an inode with lots of > extents. The extents are removed during inactivation, so this blocks > the process that unlinked the inode from the directory structure. By > moving the inactivation to the background process, the userspace > applicaiton can keep working (e.g. unlinking the next inode in the > directory) while the inactivation work on the previous inode is > done by a different CPU. > > The implementation of the queue is relatively simple. We use a > per-cpu lockless linked list (llist) to queue inodes for > inactivation without requiring serialisation mechanisms, and a work > item to allow the queue to be processed by a CPU bound worker > thread. We also keep a count of the queue depth so that we can > trigger work after a number of deferred inactivations have been > queued. > > The use of a bound workqueue with a single work depth allows the > workqueue to run one work item per CPU. We queue the work item on > the CPU we are currently running on, and so this essentially gives > us affine per-cpu worker threads for the per-cpu queues. THis > maintains the effective CPU affinity that occurs within XFS at the > AG level due to all objects in a directory being local to an AG. > Hence inactivation work tends to run on the same CPU that last > accessed all the objects that inactivation accesses and this > maintains hot CPU caches for unlink workloads. > > A depth of 32 inodes was chosen to match the number of inodes in an > inode cluster buffer. This hopefully allows sequential > allocation/unlink behaviours to defering inactivation of all the > inodes in a single cluster buffer at a time, further helping > maintain hot CPU and buffer cache accesses while running > inactivations. > > A hard per-cpu queue throttle of 256 inode has been set to avoid > runaway queuing when inodes that take a long to time inactivate are > being processed. For example, when unlinking inodes with large > numbers of extents that can take a lot of processing to free. > > Signed-off-by: Dave Chinner <dchinner@redhat.com> > --- > > Hi Darrick, > > This is the current version of the per-cpu deferred queues updated > to replace patch 3 in this series. There are no performance > regressions that I've measured with this, and most of fstests is > passing. There are some failures that I haven't looked at yet - > g/055, g/102, g/219, g/226, g/233, and so on. THese tests did not Yeah, I saw all of those emitting various problems that all trace back to ENOSPC or EDQUOT. The various "bang on inodegc sooner than later" patches in this series fix those problems; I think rework will simplify the code changes a lot, since all we need to do now is force the queue_work and the flush_work when space/quota/memory are low. generic/219 - output mismatch (see /var/tmp/fstests/generic/219.out.bad) --- tests/generic/219.out 2021-05-13 11:47:55.683860312 -0700 +++ /var/tmp/fstests/generic/219.out.bad 2021-08-03 10:49:26.554855113 -0700 @@ -34,4 +34,4 @@ Size: 49152 Filetype: Regular File File: "SCRATCH_MNT/mmap" Size: 49152 Filetype: Regular File -Usage OK (type=g) +Too many blocks used (type=g) ... (Run 'diff -u /tmp/fstests/tests/generic/219.out /var/tmp/fstests/generic/219.out.bad' to see the entire diff) generic/371 - output mismatch (see /var/tmp/fstests/generic/371.out.bad) --- tests/generic/371.out 2021-05-13 11:47:55.712860228 -0700 +++ /var/tmp/fstests/generic/371.out.bad 2021-08-03 10:58:10.161397704 -0700 @@ -1,2 +1,198 @@ QA output created by 371 Silence is golden +fallocate: No space left on device +fallocate: No space left on device +fallocate: No space left on device +fallocate: No space left on device +fallocate: No space left on device ... (Run 'diff -u /tmp/fstests/tests/generic/371.out /var/tmp/fstests/generic/371.out.bad' to see the entire diff) generic/427 [failed, exit status 1]- output mismatch (see /var/tmp/fstests/generic/427.out.bad) --- tests/generic/427.out 2021-05-13 11:47:55.723860196 -0700 +++ /var/tmp/fstests/generic/427.out.bad 2021-08-03 11:00:27.076995641 -0700 @@ -1,2 +1,2 @@ QA output created by 427 -Success, all done. +open: No space left on device ... (Run 'diff -u /tmp/fstests/tests/generic/427.out /var/tmp/fstests/generic/427.out.bad' to see the entire diff) generic/511 - output mismatch (see /var/tmp/fstests/generic/511.out.bad) --- tests/generic/511.out 2021-05-13 11:47:55.738860153 -0700 +++ /var/tmp/fstests/generic/511.out.bad 2021-08-03 11:03:29.524335265 -0700 @@ -1,2 +1,5 @@ QA output created by 511 +touch: cannot touch '/opt/a': No space left on device +Seed set to 1 +/opt/a: No space left on device Silence is golden ... (Run 'diff -u /tmp/fstests/tests/generic/511.out /var/tmp/fstests/generic/511.out.bad' to see the entire diff) generic/531 _check_dmesg: something found in dmesg (see /var/tmp/fstests/generic/531.dmesg) - output mismatch (see /var/tmp/fstests/generic/531.out.bad) --- tests/generic/531.out 2021-05-13 11:47:55.741860145 -0700 +++ /var/tmp/fstests/generic/531.out.bad 2021-08-03 11:04:51.329004913 -0700 @@ -1,2 +1,9 @@ QA output created by 531 +open?: Input/output error +open?: Input/output error +open?: Input/output error +open?: No such file or directory +open?: Input/output error +open?: Input/output error ... (Run 'diff -u /tmp/fstests/tests/generic/531.out /var/tmp/fstests/generic/531.out.bad' to see the entire diff) generic/536 - output mismatch (see /var/tmp/fstests/generic/536.out.bad) --- tests/generic/536.out 2021-05-13 11:47:55.742860142 -0700 +++ /var/tmp/fstests/generic/536.out.bad 2021-08-03 11:04:58.832711203 -0700 @@ -1,3 +1,5 @@ QA output created by 536 file.1 +hexdump: /opt/file.1: No such file or directory file.2 +hexdump: /opt/file.2: No such file or directory ... (Run 'diff -u /tmp/fstests/tests/generic/536.out /var/tmp/fstests/generic/536.out.bad' to see the entire diff) generic/603 - output mismatch (see /var/tmp/fstests/generic/603.out.bad) --- tests/generic/603.out 2021-05-13 11:47:55.755860104 -0700 +++ /var/tmp/fstests/generic/603.out.bad 2021-08-03 11:07:23.155347585 -0700 @@ -18,11 +18,15 @@ ### Initialize files, and their mode and ownership --- Test block quota --- Write 225 blocks... +pwrite: Disk quota exceeded Rewrite 250 blocks plus 1 byte, over the block softlimit... +pwrite: Disk quota exceeded Try to write 1 one more block after grace... ... (Run 'diff -u /tmp/fstests/tests/generic/603.out /var/tmp/fstests/generic/603.out.bad' to see the entire diff) I noticed that xfs/264 seems to get hung up on xfs_buftarg_wait when the dmsetup suspend freezes the fs and we try to quiesce the log. Unfortunately, that hanged my -g quick test. :/ > fail with my original "hack the queue onto the end of the series" > patch - there were zero regressions from that patch so clearly some > of the fixes later in this patch series are still necessary. Or I > screwed up/missed a flush location that those tests would have > otherwise triggered. I suspect patch 19(?) that triggers an inodegc > flush from the blockgc flush at ENOSPC might be one of the missing > pieces... > > Hence I don't think these failures have to do with the relative lack > of throttling, low space management or memory pressure detection. > More tests are failing on my 16GB test VM than the 512MB test VM, > and in general I haven't seen memory pressure have any impact on > this queuing mechanism at all. No surprises there, more RAM enables more laziness. > I suspect that means most of the rest of the patchset is not > necessary for inodegc management. I haven't yet gone through them > to see which ones address the failures I'm seeing, so that's the > next step here. > > It would be good if you can run this through you test setups for > this patchset to see if it behaves well in those situations. If it > reproduces the same failures as Im seeing, then maybe by the time > I'm awake again you've worked out which remaining bits of the > patchset are still required.... Will do. --D > Cheers, > > Dave. > > fs/xfs/scrub/common.c | 7 + > fs/xfs/xfs_icache.c | 338 +++++++++++++++++++++++++++++++++++------------ > fs/xfs/xfs_icache.h | 5 + > fs/xfs/xfs_inode.h | 20 ++- > fs/xfs/xfs_log_recover.c | 7 + > fs/xfs/xfs_mount.c | 26 +++- > fs/xfs/xfs_mount.h | 34 ++++- > fs/xfs/xfs_super.c | 111 +++++++++++++++- > fs/xfs/xfs_trace.h | 50 ++++++- > 9 files changed, 505 insertions(+), 93 deletions(-) > > diff --git a/fs/xfs/scrub/common.c b/fs/xfs/scrub/common.c > index 8558ca05e11d..06b697f72f23 100644 > --- a/fs/xfs/scrub/common.c > +++ b/fs/xfs/scrub/common.c > @@ -884,6 +884,7 @@ xchk_stop_reaping( > { > sc->flags |= XCHK_REAPING_DISABLED; > xfs_blockgc_stop(sc->mp); > + xfs_inodegc_stop(sc->mp); > } > > /* Restart background reaping of resources. */ > @@ -891,6 +892,12 @@ void > xchk_start_reaping( > struct xfs_scrub *sc) > { > + /* > + * Readonly filesystems do not perform inactivation, so there's no > + * need to restart the worker. > + */ > + if (!(sc->mp->m_flags & XFS_MOUNT_RDONLY)) > + xfs_inodegc_start(sc->mp); > xfs_blockgc_start(sc->mp); > sc->flags &= ~XCHK_REAPING_DISABLED; > } > diff --git a/fs/xfs/xfs_icache.c b/fs/xfs/xfs_icache.c > index 709507cc83ae..b1c2cab3c690 100644 > --- a/fs/xfs/xfs_icache.c > +++ b/fs/xfs/xfs_icache.c > @@ -213,7 +213,7 @@ xfs_blockgc_queue( > { > rcu_read_lock(); > if (radix_tree_tagged(&pag->pag_ici_root, XFS_ICI_BLOCKGC_TAG)) > - queue_delayed_work(pag->pag_mount->m_gc_workqueue, > + queue_delayed_work(pag->pag_mount->m_blockgc_wq, > &pag->pag_blockgc_work, > msecs_to_jiffies(xfs_blockgc_secs * 1000)); > rcu_read_unlock(); > @@ -292,86 +292,6 @@ xfs_perag_clear_inode_tag( > trace_xfs_perag_clear_inode_tag(mp, pag->pag_agno, tag, _RET_IP_); > } > > -#ifdef DEBUG > -static void > -xfs_check_delalloc( > - struct xfs_inode *ip, > - int whichfork) > -{ > - struct xfs_ifork *ifp = XFS_IFORK_PTR(ip, whichfork); > - struct xfs_bmbt_irec got; > - struct xfs_iext_cursor icur; > - > - if (!ifp || !xfs_iext_lookup_extent(ip, ifp, 0, &icur, &got)) > - return; > - do { > - if (isnullstartblock(got.br_startblock)) { > - xfs_warn(ip->i_mount, > - "ino %llx %s fork has delalloc extent at [0x%llx:0x%llx]", > - ip->i_ino, > - whichfork == XFS_DATA_FORK ? "data" : "cow", > - got.br_startoff, got.br_blockcount); > - } > - } while (xfs_iext_next_extent(ifp, &icur, &got)); > -} > -#else > -#define xfs_check_delalloc(ip, whichfork) do { } while (0) > -#endif > - > -/* > - * We set the inode flag atomically with the radix tree tag. > - * Once we get tag lookups on the radix tree, this inode flag > - * can go away. > - */ > -void > -xfs_inode_mark_reclaimable( > - struct xfs_inode *ip) > -{ > - struct xfs_mount *mp = ip->i_mount; > - struct xfs_perag *pag; > - bool need_inactive = xfs_inode_needs_inactive(ip); > - > - if (!need_inactive) { > - /* Going straight to reclaim, so drop the dquots. */ > - xfs_qm_dqdetach(ip); > - } else { > - xfs_inactive(ip); > - } > - > - if (!XFS_FORCED_SHUTDOWN(mp) && ip->i_delayed_blks) { > - xfs_check_delalloc(ip, XFS_DATA_FORK); > - xfs_check_delalloc(ip, XFS_COW_FORK); > - ASSERT(0); > - } > - > - XFS_STATS_INC(mp, vn_reclaim); > - > - /* > - * We should never get here with one of the reclaim flags already set. > - */ > - ASSERT_ALWAYS(!xfs_iflags_test(ip, XFS_IRECLAIMABLE)); > - ASSERT_ALWAYS(!xfs_iflags_test(ip, XFS_IRECLAIM)); > - > - /* > - * We always use background reclaim here because even if the inode is > - * clean, it still may be under IO and hence we have wait for IO > - * completion to occur before we can reclaim the inode. The background > - * reclaim path handles this more efficiently than we can here, so > - * simply let background reclaim tear down all inodes. > - */ > - pag = xfs_perag_get(mp, XFS_INO_TO_AGNO(mp, ip->i_ino)); > - spin_lock(&pag->pag_ici_lock); > - spin_lock(&ip->i_flags_lock); > - > - xfs_perag_set_inode_tag(pag, XFS_INO_TO_AGINO(mp, ip->i_ino), > - XFS_ICI_RECLAIM_TAG); > - __xfs_iflags_set(ip, XFS_IRECLAIMABLE); > - > - spin_unlock(&ip->i_flags_lock); > - spin_unlock(&pag->pag_ici_lock); > - xfs_perag_put(pag); > -} > - > static inline void > xfs_inew_wait( > struct xfs_inode *ip) > @@ -569,6 +489,15 @@ xfs_iget_cache_hit( > if (ip->i_flags & (XFS_INEW | XFS_IRECLAIM)) > goto out_skip; > > + if (ip->i_flags & XFS_NEED_INACTIVE) { > + /* Unlinked inodes cannot be re-grabbed. */ > + if (VFS_I(ip)->i_nlink == 0) { > + error = -ENOENT; > + goto out_error; > + } > + goto out_inodegc_flush; > + } > + > /* > * Check the inode free state is valid. This also detects lookup > * racing with unlinks. > @@ -616,6 +545,12 @@ xfs_iget_cache_hit( > spin_unlock(&ip->i_flags_lock); > rcu_read_unlock(); > return error; > + > +out_inodegc_flush: > + spin_unlock(&ip->i_flags_lock); > + rcu_read_unlock(); > + xfs_inodegc_flush(mp); > + return -EAGAIN; > } > > static int > @@ -943,6 +878,7 @@ xfs_reclaim_inode( > > xfs_iflags_clear(ip, XFS_IFLUSHING); > reclaim: > + trace_xfs_inode_reclaiming(ip); > > /* > * Because we use RCU freeing we need to ensure the inode always appears > @@ -1420,6 +1356,8 @@ xfs_blockgc_start( > > /* Don't try to run block gc on an inode that's in any of these states. */ > #define XFS_BLOCKGC_NOGRAB_IFLAGS (XFS_INEW | \ > + XFS_NEED_INACTIVE | \ > + XFS_INACTIVATING | \ > XFS_IRECLAIMABLE | \ > XFS_IRECLAIM) > /* > @@ -1794,3 +1732,241 @@ xfs_icwalk( > return last_error; > BUILD_BUG_ON(XFS_ICWALK_PRIVATE_FLAGS & XFS_ICWALK_FLAGS_VALID); > } > + > +#ifdef DEBUG > +static void > +xfs_check_delalloc( > + struct xfs_inode *ip, > + int whichfork) > +{ > + struct xfs_ifork *ifp = XFS_IFORK_PTR(ip, whichfork); > + struct xfs_bmbt_irec got; > + struct xfs_iext_cursor icur; > + > + if (!ifp || !xfs_iext_lookup_extent(ip, ifp, 0, &icur, &got)) > + return; > + do { > + if (isnullstartblock(got.br_startblock)) { > + xfs_warn(ip->i_mount, > + "ino %llx %s fork has delalloc extent at [0x%llx:0x%llx]", > + ip->i_ino, > + whichfork == XFS_DATA_FORK ? "data" : "cow", > + got.br_startoff, got.br_blockcount); > + } > + } while (xfs_iext_next_extent(ifp, &icur, &got)); > +} > +#else > +#define xfs_check_delalloc(ip, whichfork) do { } while (0) > +#endif > + > +/* Schedule the inode for reclaim. */ > +static void > +xfs_inodegc_set_reclaimable( > + struct xfs_inode *ip) > +{ > + struct xfs_mount *mp = ip->i_mount; > + struct xfs_perag *pag; > + > + if (!XFS_FORCED_SHUTDOWN(mp) && ip->i_delayed_blks) { > + xfs_check_delalloc(ip, XFS_DATA_FORK); > + xfs_check_delalloc(ip, XFS_COW_FORK); > + ASSERT(0); > + } > + > + pag = xfs_perag_get(mp, XFS_INO_TO_AGNO(mp, ip->i_ino)); > + spin_lock(&pag->pag_ici_lock); > + spin_lock(&ip->i_flags_lock); > + > + trace_xfs_inode_set_reclaimable(ip); > + ip->i_flags &= ~(XFS_NEED_INACTIVE | XFS_INACTIVATING); > + ip->i_flags |= XFS_IRECLAIMABLE; > + xfs_perag_set_inode_tag(pag, XFS_INO_TO_AGINO(mp, ip->i_ino), > + XFS_ICI_RECLAIM_TAG); > + > + spin_unlock(&ip->i_flags_lock); > + spin_unlock(&pag->pag_ici_lock); > + xfs_perag_put(pag); > +} > + > +/* > + * Free all speculative preallocations and possibly even the inode itself. > + * This is the last chance to make changes to an otherwise unreferenced file > + * before incore reclamation happens. > + */ > +static void > +xfs_inodegc_inactivate( > + struct xfs_inode *ip) > +{ > + struct xfs_mount *mp = ip->i_mount; > + > + /* > + * Inactivation isn't supposed to run when the fs is frozen because > + * we don't want kernel threads to block on transaction allocation. > + */ > + ASSERT(mp->m_super->s_writers.frozen < SB_FREEZE_FS); > + > + trace_xfs_inode_inactivating(ip); > + xfs_inactive(ip); > + xfs_inodegc_set_reclaimable(ip); > +} > + > +void > +xfs_inodegc_worker( > + struct work_struct *work) > +{ > + struct xfs_inodegc *gc = container_of(work, struct xfs_inodegc, > + work); > + struct llist_node *node = llist_del_all(&gc->list); > + struct xfs_inode *ip, *n; > + > + trace_xfs_inodegc_worker(NULL, __return_address); > + > + WRITE_ONCE(gc->items, 0); > + llist_for_each_entry_safe(ip, n, node, i_gclist) { > + xfs_iflags_set(ip, XFS_INACTIVATING); > + xfs_inodegc_inactivate(ip); > + } > +} > + > +/* > + * Force all currently queued inode inactivation work to run immediately, and > + * wait for the work to finish. Two pass - queue all the work first pass, wait > + * for it in a second pass. > + */ > +void > +xfs_inodegc_flush( > + struct xfs_mount *mp) > +{ > + struct xfs_inodegc *gc; > + int cpu; > + > + trace_xfs_inodegc_flush(mp, __return_address); > + > + for_each_online_cpu(cpu) { > + gc = per_cpu_ptr(mp->m_inodegc, cpu); > + queue_work_on(cpu, mp->m_inodegc_wq, &gc->work); > + } > + > + for_each_online_cpu(cpu) { > + gc = per_cpu_ptr(mp->m_inodegc, cpu); > + flush_work(&gc->work); > + } > +} > + > +/* > + * Flush all the pending work and then disable the inode inactivation background > + * workers and wait for them to stop. > + */ > +void > +xfs_inodegc_stop( > + struct xfs_mount *mp) > +{ > + struct xfs_inodegc *gc; > + int cpu; > + > + if (!test_and_clear_bit(XFS_OPFLAG_INODEGC_RUNNING_BIT, &mp->m_opflags)) > + return; > + > + xfs_inodegc_flush(mp); > + > + for_each_online_cpu(cpu) { > + gc = per_cpu_ptr(mp->m_inodegc, cpu); > + cancel_work_sync(&gc->work); > + } > + trace_xfs_inodegc_stop(mp, __return_address); > +} > + > +/* > + * Enable the inode inactivation background workers and schedule deferred inode > + * inactivation work if there is any. > + */ > +void > +xfs_inodegc_start( > + struct xfs_mount *mp) > +{ > + struct xfs_inodegc *gc; > + int cpu; > + > + if (test_and_set_bit(XFS_OPFLAG_INODEGC_RUNNING_BIT, &mp->m_opflags)) > + return; > + > + trace_xfs_inodegc_start(mp, __return_address); > + for_each_online_cpu(cpu) { > + gc = per_cpu_ptr(mp->m_inodegc, cpu); > + if (!llist_empty(&gc->list)) > + queue_work_on(cpu, mp->m_inodegc_wq, &gc->work); > + } > +} > + > +/* > + * Queue a background inactivation worker if there are inodes that need to be > + * inactivated and higher level xfs code hasn't disabled the background > + * workers. > + */ > +static void > +xfs_inodegc_queue( > + struct xfs_inode *ip) > +{ > + struct xfs_mount *mp = ip->i_mount; > + struct xfs_inodegc *gc; > + int items; > + > + trace_xfs_inode_set_need_inactive(ip); > + spin_lock(&ip->i_flags_lock); > + ip->i_flags |= XFS_NEED_INACTIVE; > + spin_unlock(&ip->i_flags_lock); > + > + gc = get_cpu_ptr(mp->m_inodegc); > + llist_add(&ip->i_gclist, &gc->list); > + items = READ_ONCE(gc->items); > + WRITE_ONCE(gc->items, items + 1); > + put_cpu_ptr(gc); > + > + if (!test_bit(XFS_OPFLAG_INODEGC_RUNNING_BIT, &mp->m_opflags)) > + return; > + if (items > 32) { > + trace_xfs_inodegc_queue(mp, __return_address); > + queue_work(mp->m_inodegc_wq, &gc->work); > + } > + /* throttle */ > + if (items > 256) { > + trace_xfs_inodegc_throttle(mp, __return_address); > + flush_work(&gc->work); > + } > +} > + > +/* > + * We set the inode flag atomically with the radix tree tag. Once we get tag > + * lookups on the radix tree, this inode flag can go away. > + * > + * We always use background reclaim here because even if the inode is clean, it > + * still may be under IO and hence we have wait for IO completion to occur > + * before we can reclaim the inode. The background reclaim path handles this > + * more efficiently than we can here, so simply let background reclaim tear down > + * all inodes. > + */ > +void > +xfs_inode_mark_reclaimable( > + struct xfs_inode *ip) > +{ > + struct xfs_mount *mp = ip->i_mount; > + bool need_inactive; > + > + XFS_STATS_INC(mp, vn_reclaim); > + > + /* > + * We should never get here with any of the reclaim flags already set. > + */ > + ASSERT_ALWAYS(!xfs_iflags_test(ip, XFS_ALL_IRECLAIM_FLAGS)); > + > + need_inactive = xfs_inode_needs_inactive(ip); > + if (need_inactive) { > + xfs_inodegc_queue(ip); > + return; > + } > + > + /* Going straight to reclaim, so drop the dquots. */ > + xfs_qm_dqdetach(ip); > + xfs_inodegc_set_reclaimable(ip); > +} > + > diff --git a/fs/xfs/xfs_icache.h b/fs/xfs/xfs_icache.h > index d0062ebb3f7a..c1dfc909a5b0 100644 > --- a/fs/xfs/xfs_icache.h > +++ b/fs/xfs/xfs_icache.h > @@ -74,4 +74,9 @@ int xfs_icache_inode_is_allocated(struct xfs_mount *mp, struct xfs_trans *tp, > void xfs_blockgc_stop(struct xfs_mount *mp); > void xfs_blockgc_start(struct xfs_mount *mp); > > +void xfs_inodegc_worker(struct work_struct *work); > +void xfs_inodegc_flush(struct xfs_mount *mp); > +void xfs_inodegc_stop(struct xfs_mount *mp); > +void xfs_inodegc_start(struct xfs_mount *mp); > + > #endif > diff --git a/fs/xfs/xfs_inode.h b/fs/xfs/xfs_inode.h > index e3137bbc7b14..1f62b481d8c5 100644 > --- a/fs/xfs/xfs_inode.h > +++ b/fs/xfs/xfs_inode.h > @@ -42,6 +42,7 @@ typedef struct xfs_inode { > mrlock_t i_lock; /* inode lock */ > mrlock_t i_mmaplock; /* inode mmap IO lock */ > atomic_t i_pincount; /* inode pin count */ > + struct llist_node i_gclist; /* deferred inactivation list */ > > /* > * Bitsets of inode metadata that have been checked and/or are sick. > @@ -240,6 +241,7 @@ static inline bool xfs_inode_has_bigtime(struct xfs_inode *ip) > #define __XFS_IPINNED_BIT 8 /* wakeup key for zero pin count */ > #define XFS_IPINNED (1 << __XFS_IPINNED_BIT) > #define XFS_IEOFBLOCKS (1 << 9) /* has the preallocblocks tag set */ > +#define XFS_NEED_INACTIVE (1 << 10) /* see XFS_INACTIVATING below */ > /* > * If this unlinked inode is in the middle of recovery, don't let drop_inode > * truncate and free the inode. This can happen if we iget the inode during > @@ -248,6 +250,21 @@ static inline bool xfs_inode_has_bigtime(struct xfs_inode *ip) > #define XFS_IRECOVERY (1 << 11) > #define XFS_ICOWBLOCKS (1 << 12)/* has the cowblocks tag set */ > > +/* > + * If we need to update on-disk metadata before this IRECLAIMABLE inode can be > + * freed, then NEED_INACTIVE will be set. Once we start the updates, the > + * INACTIVATING bit will be set to keep iget away from this inode. After the > + * inactivation completes, both flags will be cleared and the inode is a > + * plain old IRECLAIMABLE inode. > + */ > +#define XFS_INACTIVATING (1 << 13) > + > +/* All inode state flags related to inode reclaim. */ > +#define XFS_ALL_IRECLAIM_FLAGS (XFS_IRECLAIMABLE | \ > + XFS_IRECLAIM | \ > + XFS_NEED_INACTIVE | \ > + XFS_INACTIVATING) > + > /* > * Per-lifetime flags need to be reset when re-using a reclaimable inode during > * inode lookup. This prevents unintended behaviour on the new inode from > @@ -255,7 +272,8 @@ static inline bool xfs_inode_has_bigtime(struct xfs_inode *ip) > */ > #define XFS_IRECLAIM_RESET_FLAGS \ > (XFS_IRECLAIMABLE | XFS_IRECLAIM | \ > - XFS_IDIRTY_RELEASE | XFS_ITRUNCATED) > + XFS_IDIRTY_RELEASE | XFS_ITRUNCATED | XFS_NEED_INACTIVE | \ > + XFS_INACTIVATING) > > /* > * Flags for inode locking. > diff --git a/fs/xfs/xfs_log_recover.c b/fs/xfs/xfs_log_recover.c > index 1721fce2ec94..a98d2429d795 100644 > --- a/fs/xfs/xfs_log_recover.c > +++ b/fs/xfs/xfs_log_recover.c > @@ -2786,6 +2786,13 @@ xlog_recover_process_iunlinks( > } > xfs_buf_rele(agibp); > } > + > + /* > + * Flush the pending unlinked inodes to ensure that the inactivations > + * are fully completed on disk and the incore inodes can be reclaimed > + * before we signal that recovery is complete. > + */ > + xfs_inodegc_flush(mp); > } > > STATIC void > diff --git a/fs/xfs/xfs_mount.c b/fs/xfs/xfs_mount.c > index baf7b323cb15..1f7e9a608f38 100644 > --- a/fs/xfs/xfs_mount.c > +++ b/fs/xfs/xfs_mount.c > @@ -514,7 +514,8 @@ xfs_check_summary_counts( > * Flush and reclaim dirty inodes in preparation for unmount. Inodes and > * internal inode structures can be sitting in the CIL and AIL at this point, > * so we need to unpin them, write them back and/or reclaim them before unmount > - * can proceed. > + * can proceed. In other words, callers are required to have inactivated all > + * inodes. > * > * An inode cluster that has been freed can have its buffer still pinned in > * memory because the transaction is still sitting in a iclog. The stale inodes > @@ -546,6 +547,7 @@ xfs_unmount_flush_inodes( > mp->m_flags |= XFS_MOUNT_UNMOUNTING; > > xfs_ail_push_all_sync(mp->m_ail); > + xfs_inodegc_stop(mp); > cancel_delayed_work_sync(&mp->m_reclaim_work); > xfs_reclaim_inodes(mp); > xfs_health_unmount(mp); > @@ -782,6 +784,9 @@ xfs_mountfs( > if (error) > goto out_log_dealloc; > > + /* Enable background inode inactivation workers. */ > + xfs_inodegc_start(mp); > + > /* > * Get and sanity-check the root inode. > * Save the pointer to it in the mount structure. > @@ -942,6 +947,15 @@ xfs_mountfs( > xfs_irele(rip); > /* Clean out dquots that might be in memory after quotacheck. */ > xfs_qm_unmount(mp); > + > + /* > + * Inactivate all inodes that might still be in memory after a log > + * intent recovery failure so that reclaim can free them. Metadata > + * inodes and the root directory shouldn't need inactivation, but the > + * mount failed for some reason, so pull down all the state and flee. > + */ > + xfs_inodegc_flush(mp); > + > /* > * Flush all inode reclamation work and flush the log. > * We have to do this /after/ rtunmount and qm_unmount because those > @@ -989,6 +1003,16 @@ xfs_unmountfs( > uint64_t resblks; > int error; > > + /* > + * Perform all on-disk metadata updates required to inactivate inodes > + * that the VFS evicted earlier in the unmount process. Freeing inodes > + * and discarding CoW fork preallocations can cause shape changes to > + * the free inode and refcount btrees, respectively, so we must finish > + * this before we discard the metadata space reservations. Metadata > + * inodes and the root directory do not require inactivation. > + */ > + xfs_inodegc_flush(mp); > + > xfs_blockgc_stop(mp); > xfs_fs_unreserve_ag_blocks(mp); > xfs_qm_unmount_quotas(mp); > diff --git a/fs/xfs/xfs_mount.h b/fs/xfs/xfs_mount.h > index c78b63fe779a..470013a48c17 100644 > --- a/fs/xfs/xfs_mount.h > +++ b/fs/xfs/xfs_mount.h > @@ -56,6 +56,15 @@ struct xfs_error_cfg { > long retry_timeout; /* in jiffies, -1 = infinite */ > }; > > +/* > + * Per-cpu deferred inode inactivation GC lists. > + */ > +struct xfs_inodegc { > + struct llist_head list; > + struct work_struct work; > + int items; > +}; > + > /* > * The struct xfsmount layout is optimised to separate read-mostly variables > * from variables that are frequently modified. We put the read-mostly variables > @@ -82,6 +91,8 @@ typedef struct xfs_mount { > xfs_buftarg_t *m_ddev_targp; /* saves taking the address */ > xfs_buftarg_t *m_logdev_targp;/* ptr to log device */ > xfs_buftarg_t *m_rtdev_targp; /* ptr to rt device */ > + void __percpu *m_inodegc; /* percpu inodegc structures */ > + > /* > * Optional cache of rt summary level per bitmap block with the > * invariant that m_rsum_cache[bbno] <= the minimum i for which > @@ -94,8 +105,9 @@ typedef struct xfs_mount { > struct workqueue_struct *m_unwritten_workqueue; > struct workqueue_struct *m_cil_workqueue; > struct workqueue_struct *m_reclaim_workqueue; > - struct workqueue_struct *m_gc_workqueue; > struct workqueue_struct *m_sync_workqueue; > + struct workqueue_struct *m_blockgc_wq; > + struct workqueue_struct *m_inodegc_wq; > > int m_bsize; /* fs logical block size */ > uint8_t m_blkbit_log; /* blocklog + NBBY */ > @@ -154,6 +166,13 @@ typedef struct xfs_mount { > uint8_t m_rt_checked; > uint8_t m_rt_sick; > > + /* > + * This atomic bitset controls flags that alter the behavior of the > + * filesystem. Use only the atomic bit helper functions here; see > + * XFS_OPFLAG_* for information about the actual flags. > + */ > + unsigned long m_opflags; > + > /* > * End of read-mostly variables. Frequently written variables and locks > * should be placed below this comment from now on. The first variable > @@ -258,6 +277,19 @@ typedef struct xfs_mount { > #define XFS_MOUNT_DAX_ALWAYS (1ULL << 26) > #define XFS_MOUNT_DAX_NEVER (1ULL << 27) > > +/* > + * Operation flags -- each entry here is a bit index into m_opflags and is > + * not itself a flag value. Use the atomic bit functions to access. > + */ > +enum xfs_opflag_bits { > + /* > + * If set, background inactivation worker threads will be scheduled to > + * process queued inodegc work. If not, queued inodes remain in memory > + * waiting to be processed. > + */ > + XFS_OPFLAG_INODEGC_RUNNING_BIT = 0, > +}; > + > /* > * Max and min values for mount-option defined I/O > * preallocation sizes. > diff --git a/fs/xfs/xfs_super.c b/fs/xfs/xfs_super.c > index ef89a9a3ba9e..913d54eb4929 100644 > --- a/fs/xfs/xfs_super.c > +++ b/fs/xfs/xfs_super.c > @@ -508,21 +508,29 @@ xfs_init_mount_workqueues( > if (!mp->m_reclaim_workqueue) > goto out_destroy_cil; > > - mp->m_gc_workqueue = alloc_workqueue("xfs-gc/%s", > + mp->m_blockgc_wq = alloc_workqueue("xfs-blockgc/%s", > WQ_SYSFS | WQ_UNBOUND | WQ_FREEZABLE | WQ_MEM_RECLAIM, > 0, mp->m_super->s_id); > - if (!mp->m_gc_workqueue) > + if (!mp->m_blockgc_wq) > goto out_destroy_reclaim; > > + mp->m_inodegc_wq = alloc_workqueue("xfs-inodegc/%s", > + XFS_WQFLAGS(WQ_FREEZABLE | WQ_MEM_RECLAIM), > + 1, mp->m_super->s_id); > + if (!mp->m_inodegc_wq) > + goto out_destroy_blockgc; > + > mp->m_sync_workqueue = alloc_workqueue("xfs-sync/%s", > XFS_WQFLAGS(WQ_FREEZABLE), 0, mp->m_super->s_id); > if (!mp->m_sync_workqueue) > - goto out_destroy_eofb; > + goto out_destroy_inodegc; > > return 0; > > -out_destroy_eofb: > - destroy_workqueue(mp->m_gc_workqueue); > +out_destroy_inodegc: > + destroy_workqueue(mp->m_inodegc_wq); > +out_destroy_blockgc: > + destroy_workqueue(mp->m_blockgc_wq); > out_destroy_reclaim: > destroy_workqueue(mp->m_reclaim_workqueue); > out_destroy_cil: > @@ -540,7 +548,8 @@ xfs_destroy_mount_workqueues( > struct xfs_mount *mp) > { > destroy_workqueue(mp->m_sync_workqueue); > - destroy_workqueue(mp->m_gc_workqueue); > + destroy_workqueue(mp->m_blockgc_wq); > + destroy_workqueue(mp->m_inodegc_wq); > destroy_workqueue(mp->m_reclaim_workqueue); > destroy_workqueue(mp->m_cil_workqueue); > destroy_workqueue(mp->m_unwritten_workqueue); > @@ -702,6 +711,8 @@ xfs_fs_sync_fs( > { > struct xfs_mount *mp = XFS_M(sb); > > + trace_xfs_fs_sync_fs(mp, __return_address); > + > /* > * Doing anything during the async pass would be counterproductive. > */ > @@ -718,6 +729,25 @@ xfs_fs_sync_fs( > flush_delayed_work(&mp->m_log->l_work); > } > > + /* > + * Flush all deferred inode inactivation work so that the free space > + * counters will reflect recent deletions. Do not force the log again > + * because log recovery can restart the inactivation from the info that > + * we just wrote into the ondisk log. > + * > + * For regular operation this isn't strictly necessary since we aren't > + * required to guarantee that unlinking frees space immediately, but > + * that is how XFS historically behaved. > + * > + * If, however, the filesystem is at FREEZE_PAGEFAULTS, this is our > + * last chance to complete the inactivation work before the filesystem > + * freezes and the log is quiesced. The background worker will not > + * activate again until the fs is thawed because the VFS won't evict > + * any more inodes until freeze_super drops s_umount and we disable the > + * worker in xfs_fs_freeze. > + */ > + xfs_inodegc_flush(mp); > + > return 0; > } > > @@ -832,6 +862,17 @@ xfs_fs_freeze( > */ > flags = memalloc_nofs_save(); > xfs_blockgc_stop(mp); > + > + /* > + * Stop the inodegc background worker. freeze_super already flushed > + * all pending inodegc work when it sync'd the filesystem after setting > + * SB_FREEZE_PAGEFAULTS, and it holds s_umount, so we know that inodes > + * cannot enter xfs_fs_destroy_inode until the freeze is complete. > + * If the filesystem is read-write, inactivated inodes will queue but > + * the worker will not run until the filesystem thaws or unmounts. > + */ > + xfs_inodegc_stop(mp); > + > xfs_save_resvblks(mp); > ret = xfs_log_quiesce(mp); > memalloc_nofs_restore(flags); > @@ -847,6 +888,14 @@ xfs_fs_unfreeze( > xfs_restore_resvblks(mp); > xfs_log_work_queue(mp); > xfs_blockgc_start(mp); > + > + /* > + * Don't reactivate the inodegc worker on a readonly filesystem because > + * inodes are sent directly to reclaim. > + */ > + if (!(mp->m_flags & XFS_MOUNT_RDONLY)) > + xfs_inodegc_start(mp); > + > return 0; > } > > @@ -972,6 +1021,35 @@ xfs_destroy_percpu_counters( > percpu_counter_destroy(&mp->m_delalloc_blks); > } > > +static int > +xfs_inodegc_init_percpu( > + struct xfs_mount *mp) > +{ > + struct xfs_inodegc *gc; > + int cpu; > + > + mp->m_inodegc = alloc_percpu(struct xfs_inodegc); > + if (!mp->m_inodegc) > + return -ENOMEM; > + > + for_each_possible_cpu(cpu) { > + gc = per_cpu_ptr(mp->m_inodegc, cpu); > + init_llist_head(&gc->list); > + gc->items = 0; > + INIT_WORK(&gc->work, xfs_inodegc_worker); > + } > + return 0; > +} > + > +static void > +xfs_inodegc_free_percpu( > + struct xfs_mount *mp) > +{ > + if (!mp->m_inodegc) > + return; > + free_percpu(mp->m_inodegc); > +} > + > static void > xfs_fs_put_super( > struct super_block *sb) > @@ -988,6 +1066,7 @@ xfs_fs_put_super( > > xfs_freesb(mp); > free_percpu(mp->m_stats.xs_stats); > + xfs_inodegc_free_percpu(mp); > xfs_destroy_percpu_counters(mp); > xfs_destroy_mount_workqueues(mp); > xfs_close_devices(mp); > @@ -1359,11 +1438,15 @@ xfs_fs_fill_super( > if (error) > goto out_destroy_workqueues; > > + error = xfs_inodegc_init_percpu(mp); > + if (error) > + goto out_destroy_counters; > + > /* Allocate stats memory before we do operations that might use it */ > mp->m_stats.xs_stats = alloc_percpu(struct xfsstats); > if (!mp->m_stats.xs_stats) { > error = -ENOMEM; > - goto out_destroy_counters; > + goto out_destroy_inodegc; > } > > error = xfs_readsb(mp, flags); > @@ -1566,6 +1649,8 @@ xfs_fs_fill_super( > xfs_freesb(mp); > out_free_stats: > free_percpu(mp->m_stats.xs_stats); > + out_destroy_inodegc: > + xfs_inodegc_free_percpu(mp); > out_destroy_counters: > xfs_destroy_percpu_counters(mp); > out_destroy_workqueues: > @@ -1649,6 +1734,9 @@ xfs_remount_rw( > if (error && error != -ENOSPC) > return error; > > + /* Re-enable the background inode inactivation worker. */ > + xfs_inodegc_start(mp); > + > return 0; > } > > @@ -1671,6 +1759,15 @@ xfs_remount_ro( > return error; > } > > + /* > + * Stop the inodegc background worker. xfs_fs_reconfigure already > + * flushed all pending inodegc work when it sync'd the filesystem. > + * The VFS holds s_umount, so we know that inodes cannot enter > + * xfs_fs_destroy_inode during a remount operation. In readonly mode > + * we send inodes straight to reclaim, so no inodes will be queued. > + */ > + xfs_inodegc_stop(mp); > + > /* Free the per-AG metadata reservation pool. */ > error = xfs_fs_unreserve_ag_blocks(mp); > if (error) { > diff --git a/fs/xfs/xfs_trace.h b/fs/xfs/xfs_trace.h > index 19260291ff8b..c2fac46a029b 100644 > --- a/fs/xfs/xfs_trace.h > +++ b/fs/xfs/xfs_trace.h > @@ -157,6 +157,45 @@ DEFINE_PERAG_REF_EVENT(xfs_perag_put); > DEFINE_PERAG_REF_EVENT(xfs_perag_set_inode_tag); > DEFINE_PERAG_REF_EVENT(xfs_perag_clear_inode_tag); > > +DECLARE_EVENT_CLASS(xfs_fs_class, > + TP_PROTO(struct xfs_mount *mp, void *caller_ip), > + TP_ARGS(mp, caller_ip), > + TP_STRUCT__entry( > + __field(dev_t, dev) > + __field(unsigned long long, mflags) > + __field(unsigned long, opflags) > + __field(unsigned long, sbflags) > + __field(void *, caller_ip) > + ), > + TP_fast_assign( > + if (mp) { > + __entry->dev = mp->m_super->s_dev; > + __entry->mflags = mp->m_flags; > + __entry->opflags = mp->m_opflags; > + __entry->sbflags = mp->m_super->s_flags; > + } > + __entry->caller_ip = caller_ip; > + ), > + TP_printk("dev %d:%d m_flags 0x%llx m_opflags 0x%lx s_flags 0x%lx caller %pS", > + MAJOR(__entry->dev), MINOR(__entry->dev), > + __entry->mflags, > + __entry->opflags, > + __entry->sbflags, > + __entry->caller_ip) > +); > + > +#define DEFINE_FS_EVENT(name) \ > +DEFINE_EVENT(xfs_fs_class, name, \ > + TP_PROTO(struct xfs_mount *mp, void *caller_ip), \ > + TP_ARGS(mp, caller_ip)) > +DEFINE_FS_EVENT(xfs_inodegc_flush); > +DEFINE_FS_EVENT(xfs_inodegc_start); > +DEFINE_FS_EVENT(xfs_inodegc_stop); > +DEFINE_FS_EVENT(xfs_inodegc_worker); > +DEFINE_FS_EVENT(xfs_inodegc_queue); > +DEFINE_FS_EVENT(xfs_inodegc_throttle); > +DEFINE_FS_EVENT(xfs_fs_sync_fs); > + > DECLARE_EVENT_CLASS(xfs_ag_class, > TP_PROTO(struct xfs_mount *mp, xfs_agnumber_t agno), > TP_ARGS(mp, agno), > @@ -616,14 +655,17 @@ DECLARE_EVENT_CLASS(xfs_inode_class, > TP_STRUCT__entry( > __field(dev_t, dev) > __field(xfs_ino_t, ino) > + __field(unsigned long, iflags) > ), > TP_fast_assign( > __entry->dev = VFS_I(ip)->i_sb->s_dev; > __entry->ino = ip->i_ino; > + __entry->iflags = ip->i_flags; > ), > - TP_printk("dev %d:%d ino 0x%llx", > + TP_printk("dev %d:%d ino 0x%llx iflags 0x%lx", > MAJOR(__entry->dev), MINOR(__entry->dev), > - __entry->ino) > + __entry->ino, > + __entry->iflags) > ) > > #define DEFINE_INODE_EVENT(name) \ > @@ -667,6 +709,10 @@ DEFINE_INODE_EVENT(xfs_inode_free_eofblocks_invalid); > DEFINE_INODE_EVENT(xfs_inode_set_cowblocks_tag); > DEFINE_INODE_EVENT(xfs_inode_clear_cowblocks_tag); > DEFINE_INODE_EVENT(xfs_inode_free_cowblocks_invalid); > +DEFINE_INODE_EVENT(xfs_inode_set_reclaimable); > +DEFINE_INODE_EVENT(xfs_inode_reclaiming); > +DEFINE_INODE_EVENT(xfs_inode_set_need_inactive); > +DEFINE_INODE_EVENT(xfs_inode_inactivating); > > /* > * ftrace's __print_symbolic requires that all enum values be wrapped in the
diff --git a/fs/xfs/scrub/common.c b/fs/xfs/scrub/common.c index 8558ca05e11d..06b697f72f23 100644 --- a/fs/xfs/scrub/common.c +++ b/fs/xfs/scrub/common.c @@ -884,6 +884,7 @@ xchk_stop_reaping( { sc->flags |= XCHK_REAPING_DISABLED; xfs_blockgc_stop(sc->mp); + xfs_inodegc_stop(sc->mp); } /* Restart background reaping of resources. */ @@ -891,6 +892,12 @@ void xchk_start_reaping( struct xfs_scrub *sc) { + /* + * Readonly filesystems do not perform inactivation, so there's no + * need to restart the worker. + */ + if (!(sc->mp->m_flags & XFS_MOUNT_RDONLY)) + xfs_inodegc_start(sc->mp); xfs_blockgc_start(sc->mp); sc->flags &= ~XCHK_REAPING_DISABLED; } diff --git a/fs/xfs/xfs_icache.c b/fs/xfs/xfs_icache.c index 709507cc83ae..b1c2cab3c690 100644 --- a/fs/xfs/xfs_icache.c +++ b/fs/xfs/xfs_icache.c @@ -213,7 +213,7 @@ xfs_blockgc_queue( { rcu_read_lock(); if (radix_tree_tagged(&pag->pag_ici_root, XFS_ICI_BLOCKGC_TAG)) - queue_delayed_work(pag->pag_mount->m_gc_workqueue, + queue_delayed_work(pag->pag_mount->m_blockgc_wq, &pag->pag_blockgc_work, msecs_to_jiffies(xfs_blockgc_secs * 1000)); rcu_read_unlock(); @@ -292,86 +292,6 @@ xfs_perag_clear_inode_tag( trace_xfs_perag_clear_inode_tag(mp, pag->pag_agno, tag, _RET_IP_); } -#ifdef DEBUG -static void -xfs_check_delalloc( - struct xfs_inode *ip, - int whichfork) -{ - struct xfs_ifork *ifp = XFS_IFORK_PTR(ip, whichfork); - struct xfs_bmbt_irec got; - struct xfs_iext_cursor icur; - - if (!ifp || !xfs_iext_lookup_extent(ip, ifp, 0, &icur, &got)) - return; - do { - if (isnullstartblock(got.br_startblock)) { - xfs_warn(ip->i_mount, - "ino %llx %s fork has delalloc extent at [0x%llx:0x%llx]", - ip->i_ino, - whichfork == XFS_DATA_FORK ? "data" : "cow", - got.br_startoff, got.br_blockcount); - } - } while (xfs_iext_next_extent(ifp, &icur, &got)); -} -#else -#define xfs_check_delalloc(ip, whichfork) do { } while (0) -#endif - -/* - * We set the inode flag atomically with the radix tree tag. - * Once we get tag lookups on the radix tree, this inode flag - * can go away. - */ -void -xfs_inode_mark_reclaimable( - struct xfs_inode *ip) -{ - struct xfs_mount *mp = ip->i_mount; - struct xfs_perag *pag; - bool need_inactive = xfs_inode_needs_inactive(ip); - - if (!need_inactive) { - /* Going straight to reclaim, so drop the dquots. */ - xfs_qm_dqdetach(ip); - } else { - xfs_inactive(ip); - } - - if (!XFS_FORCED_SHUTDOWN(mp) && ip->i_delayed_blks) { - xfs_check_delalloc(ip, XFS_DATA_FORK); - xfs_check_delalloc(ip, XFS_COW_FORK); - ASSERT(0); - } - - XFS_STATS_INC(mp, vn_reclaim); - - /* - * We should never get here with one of the reclaim flags already set. - */ - ASSERT_ALWAYS(!xfs_iflags_test(ip, XFS_IRECLAIMABLE)); - ASSERT_ALWAYS(!xfs_iflags_test(ip, XFS_IRECLAIM)); - - /* - * We always use background reclaim here because even if the inode is - * clean, it still may be under IO and hence we have wait for IO - * completion to occur before we can reclaim the inode. The background - * reclaim path handles this more efficiently than we can here, so - * simply let background reclaim tear down all inodes. - */ - pag = xfs_perag_get(mp, XFS_INO_TO_AGNO(mp, ip->i_ino)); - spin_lock(&pag->pag_ici_lock); - spin_lock(&ip->i_flags_lock); - - xfs_perag_set_inode_tag(pag, XFS_INO_TO_AGINO(mp, ip->i_ino), - XFS_ICI_RECLAIM_TAG); - __xfs_iflags_set(ip, XFS_IRECLAIMABLE); - - spin_unlock(&ip->i_flags_lock); - spin_unlock(&pag->pag_ici_lock); - xfs_perag_put(pag); -} - static inline void xfs_inew_wait( struct xfs_inode *ip) @@ -569,6 +489,15 @@ xfs_iget_cache_hit( if (ip->i_flags & (XFS_INEW | XFS_IRECLAIM)) goto out_skip; + if (ip->i_flags & XFS_NEED_INACTIVE) { + /* Unlinked inodes cannot be re-grabbed. */ + if (VFS_I(ip)->i_nlink == 0) { + error = -ENOENT; + goto out_error; + } + goto out_inodegc_flush; + } + /* * Check the inode free state is valid. This also detects lookup * racing with unlinks. @@ -616,6 +545,12 @@ xfs_iget_cache_hit( spin_unlock(&ip->i_flags_lock); rcu_read_unlock(); return error; + +out_inodegc_flush: + spin_unlock(&ip->i_flags_lock); + rcu_read_unlock(); + xfs_inodegc_flush(mp); + return -EAGAIN; } static int @@ -943,6 +878,7 @@ xfs_reclaim_inode( xfs_iflags_clear(ip, XFS_IFLUSHING); reclaim: + trace_xfs_inode_reclaiming(ip); /* * Because we use RCU freeing we need to ensure the inode always appears @@ -1420,6 +1356,8 @@ xfs_blockgc_start( /* Don't try to run block gc on an inode that's in any of these states. */ #define XFS_BLOCKGC_NOGRAB_IFLAGS (XFS_INEW | \ + XFS_NEED_INACTIVE | \ + XFS_INACTIVATING | \ XFS_IRECLAIMABLE | \ XFS_IRECLAIM) /* @@ -1794,3 +1732,241 @@ xfs_icwalk( return last_error; BUILD_BUG_ON(XFS_ICWALK_PRIVATE_FLAGS & XFS_ICWALK_FLAGS_VALID); } + +#ifdef DEBUG +static void +xfs_check_delalloc( + struct xfs_inode *ip, + int whichfork) +{ + struct xfs_ifork *ifp = XFS_IFORK_PTR(ip, whichfork); + struct xfs_bmbt_irec got; + struct xfs_iext_cursor icur; + + if (!ifp || !xfs_iext_lookup_extent(ip, ifp, 0, &icur, &got)) + return; + do { + if (isnullstartblock(got.br_startblock)) { + xfs_warn(ip->i_mount, + "ino %llx %s fork has delalloc extent at [0x%llx:0x%llx]", + ip->i_ino, + whichfork == XFS_DATA_FORK ? "data" : "cow", + got.br_startoff, got.br_blockcount); + } + } while (xfs_iext_next_extent(ifp, &icur, &got)); +} +#else +#define xfs_check_delalloc(ip, whichfork) do { } while (0) +#endif + +/* Schedule the inode for reclaim. */ +static void +xfs_inodegc_set_reclaimable( + struct xfs_inode *ip) +{ + struct xfs_mount *mp = ip->i_mount; + struct xfs_perag *pag; + + if (!XFS_FORCED_SHUTDOWN(mp) && ip->i_delayed_blks) { + xfs_check_delalloc(ip, XFS_DATA_FORK); + xfs_check_delalloc(ip, XFS_COW_FORK); + ASSERT(0); + } + + pag = xfs_perag_get(mp, XFS_INO_TO_AGNO(mp, ip->i_ino)); + spin_lock(&pag->pag_ici_lock); + spin_lock(&ip->i_flags_lock); + + trace_xfs_inode_set_reclaimable(ip); + ip->i_flags &= ~(XFS_NEED_INACTIVE | XFS_INACTIVATING); + ip->i_flags |= XFS_IRECLAIMABLE; + xfs_perag_set_inode_tag(pag, XFS_INO_TO_AGINO(mp, ip->i_ino), + XFS_ICI_RECLAIM_TAG); + + spin_unlock(&ip->i_flags_lock); + spin_unlock(&pag->pag_ici_lock); + xfs_perag_put(pag); +} + +/* + * Free all speculative preallocations and possibly even the inode itself. + * This is the last chance to make changes to an otherwise unreferenced file + * before incore reclamation happens. + */ +static void +xfs_inodegc_inactivate( + struct xfs_inode *ip) +{ + struct xfs_mount *mp = ip->i_mount; + + /* + * Inactivation isn't supposed to run when the fs is frozen because + * we don't want kernel threads to block on transaction allocation. + */ + ASSERT(mp->m_super->s_writers.frozen < SB_FREEZE_FS); + + trace_xfs_inode_inactivating(ip); + xfs_inactive(ip); + xfs_inodegc_set_reclaimable(ip); +} + +void +xfs_inodegc_worker( + struct work_struct *work) +{ + struct xfs_inodegc *gc = container_of(work, struct xfs_inodegc, + work); + struct llist_node *node = llist_del_all(&gc->list); + struct xfs_inode *ip, *n; + + trace_xfs_inodegc_worker(NULL, __return_address); + + WRITE_ONCE(gc->items, 0); + llist_for_each_entry_safe(ip, n, node, i_gclist) { + xfs_iflags_set(ip, XFS_INACTIVATING); + xfs_inodegc_inactivate(ip); + } +} + +/* + * Force all currently queued inode inactivation work to run immediately, and + * wait for the work to finish. Two pass - queue all the work first pass, wait + * for it in a second pass. + */ +void +xfs_inodegc_flush( + struct xfs_mount *mp) +{ + struct xfs_inodegc *gc; + int cpu; + + trace_xfs_inodegc_flush(mp, __return_address); + + for_each_online_cpu(cpu) { + gc = per_cpu_ptr(mp->m_inodegc, cpu); + queue_work_on(cpu, mp->m_inodegc_wq, &gc->work); + } + + for_each_online_cpu(cpu) { + gc = per_cpu_ptr(mp->m_inodegc, cpu); + flush_work(&gc->work); + } +} + +/* + * Flush all the pending work and then disable the inode inactivation background + * workers and wait for them to stop. + */ +void +xfs_inodegc_stop( + struct xfs_mount *mp) +{ + struct xfs_inodegc *gc; + int cpu; + + if (!test_and_clear_bit(XFS_OPFLAG_INODEGC_RUNNING_BIT, &mp->m_opflags)) + return; + + xfs_inodegc_flush(mp); + + for_each_online_cpu(cpu) { + gc = per_cpu_ptr(mp->m_inodegc, cpu); + cancel_work_sync(&gc->work); + } + trace_xfs_inodegc_stop(mp, __return_address); +} + +/* + * Enable the inode inactivation background workers and schedule deferred inode + * inactivation work if there is any. + */ +void +xfs_inodegc_start( + struct xfs_mount *mp) +{ + struct xfs_inodegc *gc; + int cpu; + + if (test_and_set_bit(XFS_OPFLAG_INODEGC_RUNNING_BIT, &mp->m_opflags)) + return; + + trace_xfs_inodegc_start(mp, __return_address); + for_each_online_cpu(cpu) { + gc = per_cpu_ptr(mp->m_inodegc, cpu); + if (!llist_empty(&gc->list)) + queue_work_on(cpu, mp->m_inodegc_wq, &gc->work); + } +} + +/* + * Queue a background inactivation worker if there are inodes that need to be + * inactivated and higher level xfs code hasn't disabled the background + * workers. + */ +static void +xfs_inodegc_queue( + struct xfs_inode *ip) +{ + struct xfs_mount *mp = ip->i_mount; + struct xfs_inodegc *gc; + int items; + + trace_xfs_inode_set_need_inactive(ip); + spin_lock(&ip->i_flags_lock); + ip->i_flags |= XFS_NEED_INACTIVE; + spin_unlock(&ip->i_flags_lock); + + gc = get_cpu_ptr(mp->m_inodegc); + llist_add(&ip->i_gclist, &gc->list); + items = READ_ONCE(gc->items); + WRITE_ONCE(gc->items, items + 1); + put_cpu_ptr(gc); + + if (!test_bit(XFS_OPFLAG_INODEGC_RUNNING_BIT, &mp->m_opflags)) + return; + if (items > 32) { + trace_xfs_inodegc_queue(mp, __return_address); + queue_work(mp->m_inodegc_wq, &gc->work); + } + /* throttle */ + if (items > 256) { + trace_xfs_inodegc_throttle(mp, __return_address); + flush_work(&gc->work); + } +} + +/* + * We set the inode flag atomically with the radix tree tag. Once we get tag + * lookups on the radix tree, this inode flag can go away. + * + * We always use background reclaim here because even if the inode is clean, it + * still may be under IO and hence we have wait for IO completion to occur + * before we can reclaim the inode. The background reclaim path handles this + * more efficiently than we can here, so simply let background reclaim tear down + * all inodes. + */ +void +xfs_inode_mark_reclaimable( + struct xfs_inode *ip) +{ + struct xfs_mount *mp = ip->i_mount; + bool need_inactive; + + XFS_STATS_INC(mp, vn_reclaim); + + /* + * We should never get here with any of the reclaim flags already set. + */ + ASSERT_ALWAYS(!xfs_iflags_test(ip, XFS_ALL_IRECLAIM_FLAGS)); + + need_inactive = xfs_inode_needs_inactive(ip); + if (need_inactive) { + xfs_inodegc_queue(ip); + return; + } + + /* Going straight to reclaim, so drop the dquots. */ + xfs_qm_dqdetach(ip); + xfs_inodegc_set_reclaimable(ip); +} + diff --git a/fs/xfs/xfs_icache.h b/fs/xfs/xfs_icache.h index d0062ebb3f7a..c1dfc909a5b0 100644 --- a/fs/xfs/xfs_icache.h +++ b/fs/xfs/xfs_icache.h @@ -74,4 +74,9 @@ int xfs_icache_inode_is_allocated(struct xfs_mount *mp, struct xfs_trans *tp, void xfs_blockgc_stop(struct xfs_mount *mp); void xfs_blockgc_start(struct xfs_mount *mp); +void xfs_inodegc_worker(struct work_struct *work); +void xfs_inodegc_flush(struct xfs_mount *mp); +void xfs_inodegc_stop(struct xfs_mount *mp); +void xfs_inodegc_start(struct xfs_mount *mp); + #endif diff --git a/fs/xfs/xfs_inode.h b/fs/xfs/xfs_inode.h index e3137bbc7b14..1f62b481d8c5 100644 --- a/fs/xfs/xfs_inode.h +++ b/fs/xfs/xfs_inode.h @@ -42,6 +42,7 @@ typedef struct xfs_inode { mrlock_t i_lock; /* inode lock */ mrlock_t i_mmaplock; /* inode mmap IO lock */ atomic_t i_pincount; /* inode pin count */ + struct llist_node i_gclist; /* deferred inactivation list */ /* * Bitsets of inode metadata that have been checked and/or are sick. @@ -240,6 +241,7 @@ static inline bool xfs_inode_has_bigtime(struct xfs_inode *ip) #define __XFS_IPINNED_BIT 8 /* wakeup key for zero pin count */ #define XFS_IPINNED (1 << __XFS_IPINNED_BIT) #define XFS_IEOFBLOCKS (1 << 9) /* has the preallocblocks tag set */ +#define XFS_NEED_INACTIVE (1 << 10) /* see XFS_INACTIVATING below */ /* * If this unlinked inode is in the middle of recovery, don't let drop_inode * truncate and free the inode. This can happen if we iget the inode during @@ -248,6 +250,21 @@ static inline bool xfs_inode_has_bigtime(struct xfs_inode *ip) #define XFS_IRECOVERY (1 << 11) #define XFS_ICOWBLOCKS (1 << 12)/* has the cowblocks tag set */ +/* + * If we need to update on-disk metadata before this IRECLAIMABLE inode can be + * freed, then NEED_INACTIVE will be set. Once we start the updates, the + * INACTIVATING bit will be set to keep iget away from this inode. After the + * inactivation completes, both flags will be cleared and the inode is a + * plain old IRECLAIMABLE inode. + */ +#define XFS_INACTIVATING (1 << 13) + +/* All inode state flags related to inode reclaim. */ +#define XFS_ALL_IRECLAIM_FLAGS (XFS_IRECLAIMABLE | \ + XFS_IRECLAIM | \ + XFS_NEED_INACTIVE | \ + XFS_INACTIVATING) + /* * Per-lifetime flags need to be reset when re-using a reclaimable inode during * inode lookup. This prevents unintended behaviour on the new inode from @@ -255,7 +272,8 @@ static inline bool xfs_inode_has_bigtime(struct xfs_inode *ip) */ #define XFS_IRECLAIM_RESET_FLAGS \ (XFS_IRECLAIMABLE | XFS_IRECLAIM | \ - XFS_IDIRTY_RELEASE | XFS_ITRUNCATED) + XFS_IDIRTY_RELEASE | XFS_ITRUNCATED | XFS_NEED_INACTIVE | \ + XFS_INACTIVATING) /* * Flags for inode locking. diff --git a/fs/xfs/xfs_log_recover.c b/fs/xfs/xfs_log_recover.c index 1721fce2ec94..a98d2429d795 100644 --- a/fs/xfs/xfs_log_recover.c +++ b/fs/xfs/xfs_log_recover.c @@ -2786,6 +2786,13 @@ xlog_recover_process_iunlinks( } xfs_buf_rele(agibp); } + + /* + * Flush the pending unlinked inodes to ensure that the inactivations + * are fully completed on disk and the incore inodes can be reclaimed + * before we signal that recovery is complete. + */ + xfs_inodegc_flush(mp); } STATIC void diff --git a/fs/xfs/xfs_mount.c b/fs/xfs/xfs_mount.c index baf7b323cb15..1f7e9a608f38 100644 --- a/fs/xfs/xfs_mount.c +++ b/fs/xfs/xfs_mount.c @@ -514,7 +514,8 @@ xfs_check_summary_counts( * Flush and reclaim dirty inodes in preparation for unmount. Inodes and * internal inode structures can be sitting in the CIL and AIL at this point, * so we need to unpin them, write them back and/or reclaim them before unmount - * can proceed. + * can proceed. In other words, callers are required to have inactivated all + * inodes. * * An inode cluster that has been freed can have its buffer still pinned in * memory because the transaction is still sitting in a iclog. The stale inodes @@ -546,6 +547,7 @@ xfs_unmount_flush_inodes( mp->m_flags |= XFS_MOUNT_UNMOUNTING; xfs_ail_push_all_sync(mp->m_ail); + xfs_inodegc_stop(mp); cancel_delayed_work_sync(&mp->m_reclaim_work); xfs_reclaim_inodes(mp); xfs_health_unmount(mp); @@ -782,6 +784,9 @@ xfs_mountfs( if (error) goto out_log_dealloc; + /* Enable background inode inactivation workers. */ + xfs_inodegc_start(mp); + /* * Get and sanity-check the root inode. * Save the pointer to it in the mount structure. @@ -942,6 +947,15 @@ xfs_mountfs( xfs_irele(rip); /* Clean out dquots that might be in memory after quotacheck. */ xfs_qm_unmount(mp); + + /* + * Inactivate all inodes that might still be in memory after a log + * intent recovery failure so that reclaim can free them. Metadata + * inodes and the root directory shouldn't need inactivation, but the + * mount failed for some reason, so pull down all the state and flee. + */ + xfs_inodegc_flush(mp); + /* * Flush all inode reclamation work and flush the log. * We have to do this /after/ rtunmount and qm_unmount because those @@ -989,6 +1003,16 @@ xfs_unmountfs( uint64_t resblks; int error; + /* + * Perform all on-disk metadata updates required to inactivate inodes + * that the VFS evicted earlier in the unmount process. Freeing inodes + * and discarding CoW fork preallocations can cause shape changes to + * the free inode and refcount btrees, respectively, so we must finish + * this before we discard the metadata space reservations. Metadata + * inodes and the root directory do not require inactivation. + */ + xfs_inodegc_flush(mp); + xfs_blockgc_stop(mp); xfs_fs_unreserve_ag_blocks(mp); xfs_qm_unmount_quotas(mp); diff --git a/fs/xfs/xfs_mount.h b/fs/xfs/xfs_mount.h index c78b63fe779a..470013a48c17 100644 --- a/fs/xfs/xfs_mount.h +++ b/fs/xfs/xfs_mount.h @@ -56,6 +56,15 @@ struct xfs_error_cfg { long retry_timeout; /* in jiffies, -1 = infinite */ }; +/* + * Per-cpu deferred inode inactivation GC lists. + */ +struct xfs_inodegc { + struct llist_head list; + struct work_struct work; + int items; +}; + /* * The struct xfsmount layout is optimised to separate read-mostly variables * from variables that are frequently modified. We put the read-mostly variables @@ -82,6 +91,8 @@ typedef struct xfs_mount { xfs_buftarg_t *m_ddev_targp; /* saves taking the address */ xfs_buftarg_t *m_logdev_targp;/* ptr to log device */ xfs_buftarg_t *m_rtdev_targp; /* ptr to rt device */ + void __percpu *m_inodegc; /* percpu inodegc structures */ + /* * Optional cache of rt summary level per bitmap block with the * invariant that m_rsum_cache[bbno] <= the minimum i for which @@ -94,8 +105,9 @@ typedef struct xfs_mount { struct workqueue_struct *m_unwritten_workqueue; struct workqueue_struct *m_cil_workqueue; struct workqueue_struct *m_reclaim_workqueue; - struct workqueue_struct *m_gc_workqueue; struct workqueue_struct *m_sync_workqueue; + struct workqueue_struct *m_blockgc_wq; + struct workqueue_struct *m_inodegc_wq; int m_bsize; /* fs logical block size */ uint8_t m_blkbit_log; /* blocklog + NBBY */ @@ -154,6 +166,13 @@ typedef struct xfs_mount { uint8_t m_rt_checked; uint8_t m_rt_sick; + /* + * This atomic bitset controls flags that alter the behavior of the + * filesystem. Use only the atomic bit helper functions here; see + * XFS_OPFLAG_* for information about the actual flags. + */ + unsigned long m_opflags; + /* * End of read-mostly variables. Frequently written variables and locks * should be placed below this comment from now on. The first variable @@ -258,6 +277,19 @@ typedef struct xfs_mount { #define XFS_MOUNT_DAX_ALWAYS (1ULL << 26) #define XFS_MOUNT_DAX_NEVER (1ULL << 27) +/* + * Operation flags -- each entry here is a bit index into m_opflags and is + * not itself a flag value. Use the atomic bit functions to access. + */ +enum xfs_opflag_bits { + /* + * If set, background inactivation worker threads will be scheduled to + * process queued inodegc work. If not, queued inodes remain in memory + * waiting to be processed. + */ + XFS_OPFLAG_INODEGC_RUNNING_BIT = 0, +}; + /* * Max and min values for mount-option defined I/O * preallocation sizes. diff --git a/fs/xfs/xfs_super.c b/fs/xfs/xfs_super.c index ef89a9a3ba9e..913d54eb4929 100644 --- a/fs/xfs/xfs_super.c +++ b/fs/xfs/xfs_super.c @@ -508,21 +508,29 @@ xfs_init_mount_workqueues( if (!mp->m_reclaim_workqueue) goto out_destroy_cil; - mp->m_gc_workqueue = alloc_workqueue("xfs-gc/%s", + mp->m_blockgc_wq = alloc_workqueue("xfs-blockgc/%s", WQ_SYSFS | WQ_UNBOUND | WQ_FREEZABLE | WQ_MEM_RECLAIM, 0, mp->m_super->s_id); - if (!mp->m_gc_workqueue) + if (!mp->m_blockgc_wq) goto out_destroy_reclaim; + mp->m_inodegc_wq = alloc_workqueue("xfs-inodegc/%s", + XFS_WQFLAGS(WQ_FREEZABLE | WQ_MEM_RECLAIM), + 1, mp->m_super->s_id); + if (!mp->m_inodegc_wq) + goto out_destroy_blockgc; + mp->m_sync_workqueue = alloc_workqueue("xfs-sync/%s", XFS_WQFLAGS(WQ_FREEZABLE), 0, mp->m_super->s_id); if (!mp->m_sync_workqueue) - goto out_destroy_eofb; + goto out_destroy_inodegc; return 0; -out_destroy_eofb: - destroy_workqueue(mp->m_gc_workqueue); +out_destroy_inodegc: + destroy_workqueue(mp->m_inodegc_wq); +out_destroy_blockgc: + destroy_workqueue(mp->m_blockgc_wq); out_destroy_reclaim: destroy_workqueue(mp->m_reclaim_workqueue); out_destroy_cil: @@ -540,7 +548,8 @@ xfs_destroy_mount_workqueues( struct xfs_mount *mp) { destroy_workqueue(mp->m_sync_workqueue); - destroy_workqueue(mp->m_gc_workqueue); + destroy_workqueue(mp->m_blockgc_wq); + destroy_workqueue(mp->m_inodegc_wq); destroy_workqueue(mp->m_reclaim_workqueue); destroy_workqueue(mp->m_cil_workqueue); destroy_workqueue(mp->m_unwritten_workqueue); @@ -702,6 +711,8 @@ xfs_fs_sync_fs( { struct xfs_mount *mp = XFS_M(sb); + trace_xfs_fs_sync_fs(mp, __return_address); + /* * Doing anything during the async pass would be counterproductive. */ @@ -718,6 +729,25 @@ xfs_fs_sync_fs( flush_delayed_work(&mp->m_log->l_work); } + /* + * Flush all deferred inode inactivation work so that the free space + * counters will reflect recent deletions. Do not force the log again + * because log recovery can restart the inactivation from the info that + * we just wrote into the ondisk log. + * + * For regular operation this isn't strictly necessary since we aren't + * required to guarantee that unlinking frees space immediately, but + * that is how XFS historically behaved. + * + * If, however, the filesystem is at FREEZE_PAGEFAULTS, this is our + * last chance to complete the inactivation work before the filesystem + * freezes and the log is quiesced. The background worker will not + * activate again until the fs is thawed because the VFS won't evict + * any more inodes until freeze_super drops s_umount and we disable the + * worker in xfs_fs_freeze. + */ + xfs_inodegc_flush(mp); + return 0; } @@ -832,6 +862,17 @@ xfs_fs_freeze( */ flags = memalloc_nofs_save(); xfs_blockgc_stop(mp); + + /* + * Stop the inodegc background worker. freeze_super already flushed + * all pending inodegc work when it sync'd the filesystem after setting + * SB_FREEZE_PAGEFAULTS, and it holds s_umount, so we know that inodes + * cannot enter xfs_fs_destroy_inode until the freeze is complete. + * If the filesystem is read-write, inactivated inodes will queue but + * the worker will not run until the filesystem thaws or unmounts. + */ + xfs_inodegc_stop(mp); + xfs_save_resvblks(mp); ret = xfs_log_quiesce(mp); memalloc_nofs_restore(flags); @@ -847,6 +888,14 @@ xfs_fs_unfreeze( xfs_restore_resvblks(mp); xfs_log_work_queue(mp); xfs_blockgc_start(mp); + + /* + * Don't reactivate the inodegc worker on a readonly filesystem because + * inodes are sent directly to reclaim. + */ + if (!(mp->m_flags & XFS_MOUNT_RDONLY)) + xfs_inodegc_start(mp); + return 0; } @@ -972,6 +1021,35 @@ xfs_destroy_percpu_counters( percpu_counter_destroy(&mp->m_delalloc_blks); } +static int +xfs_inodegc_init_percpu( + struct xfs_mount *mp) +{ + struct xfs_inodegc *gc; + int cpu; + + mp->m_inodegc = alloc_percpu(struct xfs_inodegc); + if (!mp->m_inodegc) + return -ENOMEM; + + for_each_possible_cpu(cpu) { + gc = per_cpu_ptr(mp->m_inodegc, cpu); + init_llist_head(&gc->list); + gc->items = 0; + INIT_WORK(&gc->work, xfs_inodegc_worker); + } + return 0; +} + +static void +xfs_inodegc_free_percpu( + struct xfs_mount *mp) +{ + if (!mp->m_inodegc) + return; + free_percpu(mp->m_inodegc); +} + static void xfs_fs_put_super( struct super_block *sb) @@ -988,6 +1066,7 @@ xfs_fs_put_super( xfs_freesb(mp); free_percpu(mp->m_stats.xs_stats); + xfs_inodegc_free_percpu(mp); xfs_destroy_percpu_counters(mp); xfs_destroy_mount_workqueues(mp); xfs_close_devices(mp); @@ -1359,11 +1438,15 @@ xfs_fs_fill_super( if (error) goto out_destroy_workqueues; + error = xfs_inodegc_init_percpu(mp); + if (error) + goto out_destroy_counters; + /* Allocate stats memory before we do operations that might use it */ mp->m_stats.xs_stats = alloc_percpu(struct xfsstats); if (!mp->m_stats.xs_stats) { error = -ENOMEM; - goto out_destroy_counters; + goto out_destroy_inodegc; } error = xfs_readsb(mp, flags); @@ -1566,6 +1649,8 @@ xfs_fs_fill_super( xfs_freesb(mp); out_free_stats: free_percpu(mp->m_stats.xs_stats); + out_destroy_inodegc: + xfs_inodegc_free_percpu(mp); out_destroy_counters: xfs_destroy_percpu_counters(mp); out_destroy_workqueues: @@ -1649,6 +1734,9 @@ xfs_remount_rw( if (error && error != -ENOSPC) return error; + /* Re-enable the background inode inactivation worker. */ + xfs_inodegc_start(mp); + return 0; } @@ -1671,6 +1759,15 @@ xfs_remount_ro( return error; } + /* + * Stop the inodegc background worker. xfs_fs_reconfigure already + * flushed all pending inodegc work when it sync'd the filesystem. + * The VFS holds s_umount, so we know that inodes cannot enter + * xfs_fs_destroy_inode during a remount operation. In readonly mode + * we send inodes straight to reclaim, so no inodes will be queued. + */ + xfs_inodegc_stop(mp); + /* Free the per-AG metadata reservation pool. */ error = xfs_fs_unreserve_ag_blocks(mp); if (error) { diff --git a/fs/xfs/xfs_trace.h b/fs/xfs/xfs_trace.h index 19260291ff8b..c2fac46a029b 100644 --- a/fs/xfs/xfs_trace.h +++ b/fs/xfs/xfs_trace.h @@ -157,6 +157,45 @@ DEFINE_PERAG_REF_EVENT(xfs_perag_put); DEFINE_PERAG_REF_EVENT(xfs_perag_set_inode_tag); DEFINE_PERAG_REF_EVENT(xfs_perag_clear_inode_tag); +DECLARE_EVENT_CLASS(xfs_fs_class, + TP_PROTO(struct xfs_mount *mp, void *caller_ip), + TP_ARGS(mp, caller_ip), + TP_STRUCT__entry( + __field(dev_t, dev) + __field(unsigned long long, mflags) + __field(unsigned long, opflags) + __field(unsigned long, sbflags) + __field(void *, caller_ip) + ), + TP_fast_assign( + if (mp) { + __entry->dev = mp->m_super->s_dev; + __entry->mflags = mp->m_flags; + __entry->opflags = mp->m_opflags; + __entry->sbflags = mp->m_super->s_flags; + } + __entry->caller_ip = caller_ip; + ), + TP_printk("dev %d:%d m_flags 0x%llx m_opflags 0x%lx s_flags 0x%lx caller %pS", + MAJOR(__entry->dev), MINOR(__entry->dev), + __entry->mflags, + __entry->opflags, + __entry->sbflags, + __entry->caller_ip) +); + +#define DEFINE_FS_EVENT(name) \ +DEFINE_EVENT(xfs_fs_class, name, \ + TP_PROTO(struct xfs_mount *mp, void *caller_ip), \ + TP_ARGS(mp, caller_ip)) +DEFINE_FS_EVENT(xfs_inodegc_flush); +DEFINE_FS_EVENT(xfs_inodegc_start); +DEFINE_FS_EVENT(xfs_inodegc_stop); +DEFINE_FS_EVENT(xfs_inodegc_worker); +DEFINE_FS_EVENT(xfs_inodegc_queue); +DEFINE_FS_EVENT(xfs_inodegc_throttle); +DEFINE_FS_EVENT(xfs_fs_sync_fs); + DECLARE_EVENT_CLASS(xfs_ag_class, TP_PROTO(struct xfs_mount *mp, xfs_agnumber_t agno), TP_ARGS(mp, agno), @@ -616,14 +655,17 @@ DECLARE_EVENT_CLASS(xfs_inode_class, TP_STRUCT__entry( __field(dev_t, dev) __field(xfs_ino_t, ino) + __field(unsigned long, iflags) ), TP_fast_assign( __entry->dev = VFS_I(ip)->i_sb->s_dev; __entry->ino = ip->i_ino; + __entry->iflags = ip->i_flags; ), - TP_printk("dev %d:%d ino 0x%llx", + TP_printk("dev %d:%d ino 0x%llx iflags 0x%lx", MAJOR(__entry->dev), MINOR(__entry->dev), - __entry->ino) + __entry->ino, + __entry->iflags) ) #define DEFINE_INODE_EVENT(name) \ @@ -667,6 +709,10 @@ DEFINE_INODE_EVENT(xfs_inode_free_eofblocks_invalid); DEFINE_INODE_EVENT(xfs_inode_set_cowblocks_tag); DEFINE_INODE_EVENT(xfs_inode_clear_cowblocks_tag); DEFINE_INODE_EVENT(xfs_inode_free_cowblocks_invalid); +DEFINE_INODE_EVENT(xfs_inode_set_reclaimable); +DEFINE_INODE_EVENT(xfs_inode_reclaiming); +DEFINE_INODE_EVENT(xfs_inode_set_need_inactive); +DEFINE_INODE_EVENT(xfs_inode_inactivating); /* * ftrace's __print_symbolic requires that all enum values be wrapped in the