| Message ID | 20170125221739.GA33995@bfoster.bfoster (mailing list archive) |
|---|---|
| State | Superseded, archived |
| Headers | show |
Hello, Dne 25.1.2017 v 23:17 Brian Foster napsal(a): > On Tue, Jan 24, 2017 at 02:17:36PM +0100, Martin Svec wrote: >> Hello, >> >> Dne 23.1.2017 v 14:44 Brian Foster napsal(a): >>> On Mon, Jan 23, 2017 at 10:44:20AM +0100, Martin Svec wrote: >>>> Hello Dave, >>>> >>>> Any updates on this? It's a bit annoying to workaround the bug by increasing RAM just because of the >>>> initial quotacheck. >>>> >>> Note that Dave is away on a bit of an extended vacation[1]. It looks >>> like he was in the process of fishing through the code to spot any >>> potential problems related to quotacheck+reclaim. I see you've cc'd him >>> directly so we'll see if we get a response wrt to if he got anywhere >>> with that... >>> >>> Skimming back through this thread, it looks like we have an issue where >>> quota check is not quite reliable in the event of reclaim, and you >>> appear to be reproducing this due to a probably unique combination of >>> large inode count and low memory. >>> >>> Is my understanding correct that you've reproduced this on more recent >>> kernels than the original report? >> Yes, I repeated the tests using 4.9.3 kernel on another VM where we hit this issue. >> >> Configuration: >> * vSphere 5.5 virtual machine, 2 vCPUs, virtual disks residing on iSCSI VMFS datastore >> * Debian Jessie 64 bit webserver, vanilla kernel 4.9.3 >> * 180 GB XFS data disk mounted as /www >> >> Quotacheck behavior depends on assigned RAM: >> * 2 or less GiB: mount /www leads to a storm of OOM kills including shell, ttys etc., so the system >> becomes unusable. >> * 3 GiB: mount /www task hangs in the same way as I reported in earlier in this thread. >> * 4 or more GiB: mount /www succeeds. >> > I was able to reproduce the quotacheck OOM situation on latest kernels. > This problem actually looks like a regression as of commit 17c12bcd3 > ("xfs: when replaying bmap operations, don't let unlinked inodes get > reaped"), but I don't think that patch is the core problem. That patch > pulled up setting MS_ACTIVE on the superblock from after XFS runs > quotacheck to before it (for other reasons), which has a side effect of > causing inodes to be placed onto the lru once they are released. Before > this change, all inodes were immediately marked for reclaim once > released from quotacheck because the superblock had not been set active. > > The problem here is first that quotacheck issues a bulkstat and thus > grabs and releases every inode in the fs. The quotacheck occurs at mount > time, which means we still hold the s_umount lock and thus the shrinker > cannot run even though it is registered. Therefore, we basically just > populate the lru until we've consumed too much memory and blow up. > > I think the solution here is to preserve the quotacheck behavior prior > to commit 17c12bcd3 via something like the following: > > --- a/fs/xfs/xfs_qm.c > +++ b/fs/xfs/xfs_qm.c > @@ -1177,7 +1177,7 @@ xfs_qm_dqusage_adjust( > * the case in all other instances. It's OK that we do this because > * quotacheck is done only at mount time. > */ > - error = xfs_iget(mp, NULL, ino, 0, XFS_ILOCK_EXCL, &ip); > + error = xfs_iget(mp, NULL, ino, XFS_IGET_DONTCACHE, XFS_ILOCK_EXCL, &ip); > if (error) { > *res = BULKSTAT_RV_NOTHING; > return error; > > ... which allows quotacheck to run as normal in my quick tests. Could > you try this on your more recent kernel tests and see whether you still > reproduce any problems? The above patch fixes OOM issues and reduces overall memory consumption during quotacheck. However, it does not fix the original xfs_qm_flush_one() freezing. I'm still able to reproduce it with 1 GB of RAM or lower. Tested with 4.9.5 kernel. If it makes sense to you, I can rsync the whole filesystem to a new XFS volume and repeat the tests. At least, that could tell us if the problem depends on a particular state of on-disk metadata structures or it's a general property of the given filesystem tree. Martin -- To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Jan 26, 2017 at 06:46:42PM +0100, Martin Svec wrote: > Hello, > > Dne 25.1.2017 v 23:17 Brian Foster napsal(a): > > On Tue, Jan 24, 2017 at 02:17:36PM +0100, Martin Svec wrote: > >> Hello, > >> > >> Dne 23.1.2017 v 14:44 Brian Foster napsal(a): > >>> On Mon, Jan 23, 2017 at 10:44:20AM +0100, Martin Svec wrote: > >>>> Hello Dave, > >>>> > >>>> Any updates on this? It's a bit annoying to workaround the bug by increasing RAM just because of the > >>>> initial quotacheck. > >>>> > >>> Note that Dave is away on a bit of an extended vacation[1]. It looks > >>> like he was in the process of fishing through the code to spot any > >>> potential problems related to quotacheck+reclaim. I see you've cc'd him > >>> directly so we'll see if we get a response wrt to if he got anywhere > >>> with that... > >>> > >>> Skimming back through this thread, it looks like we have an issue where > >>> quota check is not quite reliable in the event of reclaim, and you > >>> appear to be reproducing this due to a probably unique combination of > >>> large inode count and low memory. > >>> > >>> Is my understanding correct that you've reproduced this on more recent > >>> kernels than the original report? > >> Yes, I repeated the tests using 4.9.3 kernel on another VM where we hit this issue. > >> > >> Configuration: > >> * vSphere 5.5 virtual machine, 2 vCPUs, virtual disks residing on iSCSI VMFS datastore > >> * Debian Jessie 64 bit webserver, vanilla kernel 4.9.3 > >> * 180 GB XFS data disk mounted as /www > >> > >> Quotacheck behavior depends on assigned RAM: > >> * 2 or less GiB: mount /www leads to a storm of OOM kills including shell, ttys etc., so the system > >> becomes unusable. > >> * 3 GiB: mount /www task hangs in the same way as I reported in earlier in this thread. > >> * 4 or more GiB: mount /www succeeds. > >> > > I was able to reproduce the quotacheck OOM situation on latest kernels. > > This problem actually looks like a regression as of commit 17c12bcd3 > > ("xfs: when replaying bmap operations, don't let unlinked inodes get > > reaped"), but I don't think that patch is the core problem. That patch > > pulled up setting MS_ACTIVE on the superblock from after XFS runs > > quotacheck to before it (for other reasons), which has a side effect of > > causing inodes to be placed onto the lru once they are released. Before > > this change, all inodes were immediately marked for reclaim once > > released from quotacheck because the superblock had not been set active. > > > > The problem here is first that quotacheck issues a bulkstat and thus > > grabs and releases every inode in the fs. The quotacheck occurs at mount > > time, which means we still hold the s_umount lock and thus the shrinker > > cannot run even though it is registered. Therefore, we basically just > > populate the lru until we've consumed too much memory and blow up. > > > > I think the solution here is to preserve the quotacheck behavior prior > > to commit 17c12bcd3 via something like the following: > > > > --- a/fs/xfs/xfs_qm.c > > +++ b/fs/xfs/xfs_qm.c > > @@ -1177,7 +1177,7 @@ xfs_qm_dqusage_adjust( > > * the case in all other instances. It's OK that we do this because > > * quotacheck is done only at mount time. > > */ > > - error = xfs_iget(mp, NULL, ino, 0, XFS_ILOCK_EXCL, &ip); > > + error = xfs_iget(mp, NULL, ino, XFS_IGET_DONTCACHE, XFS_ILOCK_EXCL, &ip); > > if (error) { > > *res = BULKSTAT_RV_NOTHING; > > return error; > > > > ... which allows quotacheck to run as normal in my quick tests. Could > > you try this on your more recent kernel tests and see whether you still > > reproduce any problems? > > The above patch fixes OOM issues and reduces overall memory consumption during quotacheck. However, > it does not fix the original xfs_qm_flush_one() freezing. I'm still able to reproduce it with 1 GB > of RAM or lower. Tested with 4.9.5 kernel. > Ok, thanks. I'll get that fix posted shortly. I hadn't tried reducing RAM any further. I dropped my vm down to 1GB and I don't reproduce a hang. If I drop to 512MB, the mount actually crashes due to what looks like the problem that djwong just fixed[1]. With that one liner applied, it does look like I've hit a mount hang in the quotacheck path. Note that I'm also running into OOM issues again though, probably due to legitimately not having enough RAM for this vm. Anyways, I'll see if I can dig anything out of that... FWIW, this is all on the latest for-next (4.10.0-rc5). [1] http://www.spinics.net/lists/linux-xfs/msg03869.html > If it makes sense to you, I can rsync the whole filesystem to a new XFS volume and repeat the tests. > At least, that could tell us if the problem depends on a particular state of on-disk metadata > structures or it's a general property of the given filesystem tree. > That couldn't hurt, thanks. Brian > Martin > > -- > To unsubscribe from this list: send the line "unsubscribe linux-xfs" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Dne 26.1.2017 v 20:12 Brian Foster napsal(a): > On Thu, Jan 26, 2017 at 06:46:42PM +0100, Martin Svec wrote: >> Hello, >> >> Dne 25.1.2017 v 23:17 Brian Foster napsal(a): >>> On Tue, Jan 24, 2017 at 02:17:36PM +0100, Martin Svec wrote: >>>> Hello, >>>> >>>> Dne 23.1.2017 v 14:44 Brian Foster napsal(a): >>>>> On Mon, Jan 23, 2017 at 10:44:20AM +0100, Martin Svec wrote: >>>>>> Hello Dave, >>>>>> >>>>>> Any updates on this? It's a bit annoying to workaround the bug by increasing RAM just because of the >>>>>> initial quotacheck. >>>>>> >>>>> Note that Dave is away on a bit of an extended vacation[1]. It looks >>>>> like he was in the process of fishing through the code to spot any >>>>> potential problems related to quotacheck+reclaim. I see you've cc'd him >>>>> directly so we'll see if we get a response wrt to if he got anywhere >>>>> with that... >>>>> >>>>> Skimming back through this thread, it looks like we have an issue where >>>>> quota check is not quite reliable in the event of reclaim, and you >>>>> appear to be reproducing this due to a probably unique combination of >>>>> large inode count and low memory. >>>>> >>>>> Is my understanding correct that you've reproduced this on more recent >>>>> kernels than the original report? >>>> Yes, I repeated the tests using 4.9.3 kernel on another VM where we hit this issue. >>>> >>>> Configuration: >>>> * vSphere 5.5 virtual machine, 2 vCPUs, virtual disks residing on iSCSI VMFS datastore >>>> * Debian Jessie 64 bit webserver, vanilla kernel 4.9.3 >>>> * 180 GB XFS data disk mounted as /www >>>> >>>> Quotacheck behavior depends on assigned RAM: >>>> * 2 or less GiB: mount /www leads to a storm of OOM kills including shell, ttys etc., so the system >>>> becomes unusable. >>>> * 3 GiB: mount /www task hangs in the same way as I reported in earlier in this thread. >>>> * 4 or more GiB: mount /www succeeds. >>>> >>> I was able to reproduce the quotacheck OOM situation on latest kernels. >>> This problem actually looks like a regression as of commit 17c12bcd3 >>> ("xfs: when replaying bmap operations, don't let unlinked inodes get >>> reaped"), but I don't think that patch is the core problem. That patch >>> pulled up setting MS_ACTIVE on the superblock from after XFS runs >>> quotacheck to before it (for other reasons), which has a side effect of >>> causing inodes to be placed onto the lru once they are released. Before >>> this change, all inodes were immediately marked for reclaim once >>> released from quotacheck because the superblock had not been set active. >>> >>> The problem here is first that quotacheck issues a bulkstat and thus >>> grabs and releases every inode in the fs. The quotacheck occurs at mount >>> time, which means we still hold the s_umount lock and thus the shrinker >>> cannot run even though it is registered. Therefore, we basically just >>> populate the lru until we've consumed too much memory and blow up. >>> >>> I think the solution here is to preserve the quotacheck behavior prior >>> to commit 17c12bcd3 via something like the following: >>> >>> --- a/fs/xfs/xfs_qm.c >>> +++ b/fs/xfs/xfs_qm.c >>> @@ -1177,7 +1177,7 @@ xfs_qm_dqusage_adjust( >>> * the case in all other instances. It's OK that we do this because >>> * quotacheck is done only at mount time. >>> */ >>> - error = xfs_iget(mp, NULL, ino, 0, XFS_ILOCK_EXCL, &ip); >>> + error = xfs_iget(mp, NULL, ino, XFS_IGET_DONTCACHE, XFS_ILOCK_EXCL, &ip); >>> if (error) { >>> *res = BULKSTAT_RV_NOTHING; >>> return error; >>> >>> ... which allows quotacheck to run as normal in my quick tests. Could >>> you try this on your more recent kernel tests and see whether you still >>> reproduce any problems? >> The above patch fixes OOM issues and reduces overall memory consumption during quotacheck. However, >> it does not fix the original xfs_qm_flush_one() freezing. I'm still able to reproduce it with 1 GB >> of RAM or lower. Tested with 4.9.5 kernel. >> > Ok, thanks. I'll get that fix posted shortly. > > I hadn't tried reducing RAM any further. I dropped my vm down to 1GB and > I don't reproduce a hang. If I drop to 512MB, the mount actually crashes > due to what looks like the problem that djwong just fixed[1]. > > With that one liner applied, it does look like I've hit a mount hang in > the quotacheck path. Note that I'm also running into OOM issues again > though, probably due to legitimately not having enough RAM for this vm. > Anyways, I'll see if I can dig anything out of that... > > FWIW, this is all on the latest for-next (4.10.0-rc5). > > [1] http://www.spinics.net/lists/linux-xfs/msg03869.html > >> If it makes sense to you, I can rsync the whole filesystem to a new XFS volume and repeat the tests. >> At least, that could tell us if the problem depends on a particular state of on-disk metadata >> structures or it's a general property of the given filesystem tree. >> > That couldn't hurt, thanks. > Well, after rsync to a fresh non-resized XFS volume, I still hit the mount hang with 1GB RAM. Martin -- To unsubscribe from this list: send the line "unsubscribe linux-xfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
--- a/fs/xfs/xfs_qm.c +++ b/fs/xfs/xfs_qm.c @@ -1177,7 +1177,7 @@ xfs_qm_dqusage_adjust( * the case in all other instances. It's OK that we do this because * quotacheck is done only at mount time. */ - error = xfs_iget(mp, NULL, ino, 0, XFS_ILOCK_EXCL, &ip); + error = xfs_iget(mp, NULL, ino, XFS_IGET_DONTCACHE, XFS_ILOCK_EXCL, &ip); if (error) { *res = BULKSTAT_RV_NOTHING; return error;