[2/6] ocfs2: Fix quota recovery failure on unmount
diff mbox

Message ID 20180228111802.23967-3-jack@suse.cz
State New
Headers show

Commit Message

Jan Kara Feb. 28, 2018, 11:17 a.m. UTC
When filesystem is unmounted while there is still some recovery work
going on, it can happen that quotas get disabled before quota recovery
is complete resulting in failed quota recovery and inconsistent quota
accounting. Move disabling of recovery in ocfs2_dismount_volume() before
disabling of quotas to fix this race.

Signed-off-by: Jan Kara <jack@suse.cz>
---
 fs/ocfs2/super.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

Comments

Shichangkuo March 19, 2018, 9:02 a.m. UTC | #1
Hi Jan,
    This patch will cause kernel crashes while umounting ocfs2.
    After parsing the core file, I find the root reason is as follows:
    osb->recovery_map is freed when recovery thread exited, but following functions still access it by calling ocfs2_inode_lock.
    So, we should disable quotas between ocfs2_truncate_log_shutdown and ocfs2_recovery_exit.
    Am I right?

Thanks
Changkuo

>>>

> When filesystem is unmounted while there is still some recovery work going

> on, it can happen that quotas get disabled before quota recovery is complete

> resulting in failed quota recovery and inconsistent quota accounting. Move

> disabling of recovery in ocfs2_dismount_volume() before disabling of quotas

> to fix this race.

>

> Signed-off-by: Jan Kara <jack@suse.cz>

> ---

>  fs/ocfs2/super.c | 10 +++++++---

>  1 file changed, 7 insertions(+), 3 deletions(-)

>

> diff --git a/fs/ocfs2/super.c b/fs/ocfs2/super.c index

> ffa4952d432b..14c3d5ee6e24 100644

> --- a/fs/ocfs2/super.c

> +++ b/fs/ocfs2/super.c

> @@ -1904,6 +1904,13 @@ static void ocfs2_dismount_volume(struct

> super_block *sb, int mnt_err)

>       /* Orphan scan should be stopped as early as possible */

>       ocfs2_orphan_scan_stop(osb);

>

> +     /*

> +      * This will disable recovery and flush any recovery work. This needs

> +      * to happen before disabling quotas as quota recovery needs quotas

> +      * enabled.

> +      */

> +     ocfs2_recovery_exit(osb);

> +

>       ocfs2_disable_quotas(osb);

>

>       /* All dquots should be freed by now */ @@ -1915,9 +1922,6 @@

> static void ocfs2_dismount_volume(struct super_block *sb, int mnt_err)

>

>       ocfs2_truncate_log_shutdown(osb);

>

> -     /* This will disable recovery and flush any recovery work. */

> -     ocfs2_recovery_exit(osb);

> -

>       ocfs2_journal_shutdown(osb);

>

>       ocfs2_sync_blockdev(sb);

> --

> 2.13.6


-------------------------------------------------------------------------------------------------------------------------------------
本邮件及其附件含有新华三技术有限公司的保密信息,仅限于发送给上面地址中列出
的个人或群组。禁止任何其他人以任何形式使用(包括但不限于全部或部分地泄露、复制、
或散发)本邮件中的信息。如果您错收了本邮件,请您立即电话或邮件通知发件人并删除本
邮件!
This e-mail and its attachments contain confidential information from New H3C, which is
intended only for the person or entity whose address is listed above. Any use of the
information contained herein in any way (including, but not limited to, total or partial
disclosure, reproduction, or dissemination) by persons other than the intended
recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender
by phone or email immediately and delete it!
Jan Kara March 21, 2018, 11:09 a.m. UTC | #2
Hi Changkuo!

On Mon 19-03-18 09:02:42, Shichangkuo wrote:
> Hi Jan,
>     This patch will cause kernel crashes while umounting ocfs2.
>     After parsing the core file, I find the root reason is as follows:
>     osb->recovery_map is freed when recovery thread exited, but following functions still access it by calling ocfs2_inode_lock.
>     So, we should disable quotas between ocfs2_truncate_log_shutdown and ocfs2_recovery_exit.
>     Am I right?

Thanks a lot for testing and the analysis. Sadly fixing this is not that
easy. We have to have recovery thread still working so that we can acquire
locks to sync and disable quotas. OTOH we need to stop recovery thread from
racing of quota recovery with disabling of quotas. Drat. I need to go back
and rethink this...

								Honza

> 
> Thanks
> Changkuo
> 
> >>>
> > When filesystem is unmounted while there is still some recovery work going
> > on, it can happen that quotas get disabled before quota recovery is complete
> > resulting in failed quota recovery and inconsistent quota accounting. Move
> > disabling of recovery in ocfs2_dismount_volume() before disabling of quotas
> > to fix this race.
> >
> > Signed-off-by: Jan Kara <jack@suse.cz>
> > ---
> >  fs/ocfs2/super.c | 10 +++++++---
> >  1 file changed, 7 insertions(+), 3 deletions(-)
> >
> > diff --git a/fs/ocfs2/super.c b/fs/ocfs2/super.c index
> > ffa4952d432b..14c3d5ee6e24 100644
> > --- a/fs/ocfs2/super.c
> > +++ b/fs/ocfs2/super.c
> > @@ -1904,6 +1904,13 @@ static void ocfs2_dismount_volume(struct
> > super_block *sb, int mnt_err)
> >       /* Orphan scan should be stopped as early as possible */
> >       ocfs2_orphan_scan_stop(osb);
> >
> > +     /*
> > +      * This will disable recovery and flush any recovery work. This needs
> > +      * to happen before disabling quotas as quota recovery needs quotas
> > +      * enabled.
> > +      */
> > +     ocfs2_recovery_exit(osb);
> > +
> >       ocfs2_disable_quotas(osb);
> >
> >       /* All dquots should be freed by now */ @@ -1915,9 +1922,6 @@
> > static void ocfs2_dismount_volume(struct super_block *sb, int mnt_err)
> >
> >       ocfs2_truncate_log_shutdown(osb);
> >
> > -     /* This will disable recovery and flush any recovery work. */
> > -     ocfs2_recovery_exit(osb);
> > -
> >       ocfs2_journal_shutdown(osb);
> >
> >       ocfs2_sync_blockdev(sb);
> > --
> > 2.13.6
> 
> -------------------------------------------------------------------------------------------------------------------------------------
> 本邮件及其附件含有新华三技术有限公司的保密信息,仅限于发送给上面地址中列出
> 的个人或群组。禁止任何其他人以任何形式使用(包括但不限于全部或部分地泄露、复制、
> 或散发)本邮件中的信息。如果您错收了本邮件,请您立即电话或邮件通知发件人并删除本
> 邮件!
> This e-mail and its attachments contain confidential information from New H3C, which is
> intended only for the person or entity whose address is listed above. Any use of the
> information contained herein in any way (including, but not limited to, total or partial
> disclosure, reproduction, or dissemination) by persons other than the intended
> recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender
> by phone or email immediately and delete it!

Patch
diff mbox

diff --git a/fs/ocfs2/super.c b/fs/ocfs2/super.c
index ffa4952d432b..14c3d5ee6e24 100644
--- a/fs/ocfs2/super.c
+++ b/fs/ocfs2/super.c
@@ -1904,6 +1904,13 @@  static void ocfs2_dismount_volume(struct super_block *sb, int mnt_err)
 	/* Orphan scan should be stopped as early as possible */
 	ocfs2_orphan_scan_stop(osb);
 
+	/*
+	 * This will disable recovery and flush any recovery work. This needs
+	 * to happen before disabling quotas as quota recovery needs quotas
+	 * enabled.
+	 */
+	ocfs2_recovery_exit(osb);
+
 	ocfs2_disable_quotas(osb);
 
 	/* All dquots should be freed by now */
@@ -1915,9 +1922,6 @@  static void ocfs2_dismount_volume(struct super_block *sb, int mnt_err)
 
 	ocfs2_truncate_log_shutdown(osb);
 
-	/* This will disable recovery and flush any recovery work. */
-	ocfs2_recovery_exit(osb);
-
 	ocfs2_journal_shutdown(osb);
 
 	ocfs2_sync_blockdev(sb);