nfs: Fix nfsi->nrequests count error on nfs_inode_remove_request
diff mbox series

Message ID 1569479378-128669-1-git-send-email-zhangxiaoxu5@huawei.com
State New
Headers show
Series
  • nfs: Fix nfsi->nrequests count error on nfs_inode_remove_request
Related show

Commit Message

zhangxiaoxu (A) Sept. 26, 2019, 6:29 a.m. UTC
When xfstests testing, there are some WARNING as below:

WARNING: CPU: 0 PID: 6235 at fs/nfs/inode.c:122 nfs_clear_inode+0x9c/0xd8
Modules linked in:
CPU: 0 PID: 6235 Comm: umount.nfs
Hardware name: linux,dummy-virt (DT)
pstate: 60000005 (nZCv daif -PAN -UAO)
pc : nfs_clear_inode+0x9c/0xd8
lr : nfs_evict_inode+0x60/0x78
sp : fffffc000f68fc00
x29: fffffc000f68fc00 x28: fffffe00c53155c0
x27: fffffe00c5315000 x26: fffffc0009a63748
x25: fffffc000f68fd18 x24: fffffc000bfaaf40
x23: fffffc000936d3c0 x22: fffffe00c4ff5e20
x21: fffffc000bfaaf40 x20: fffffe00c4ff5d10
x19: fffffc000c056000 x18: 000000000000003c
x17: 0000000000000000 x16: 0000000000000000
x15: 0000000000000040 x14: 0000000000000228
x13: fffffc000c3a2000 x12: 0000000000000045
x11: 0000000000000000 x10: 0000000000000000
x9 : 0000000000000000 x8 : 0000000000000000
x7 : 0000000000000000 x6 : fffffc00084b027c
x5 : fffffc0009a64000 x4 : fffffe00c0e77400
x3 : fffffc000c0563a8 x2 : fffffffffffffffb
x1 : 000000000000764e x0 : 0000000000000001
Call trace:
 nfs_clear_inode+0x9c/0xd8
 nfs_evict_inode+0x60/0x78
 evict+0x108/0x380
 dispose_list+0x70/0xa0
 evict_inodes+0x194/0x210
 generic_shutdown_super+0xb0/0x220
 nfs_kill_super+0x40/0x88
 deactivate_locked_super+0xb4/0x120
 deactivate_super+0x144/0x160
 cleanup_mnt+0x98/0x148
 __cleanup_mnt+0x38/0x50
 task_work_run+0x114/0x160
 do_notify_resume+0x2f8/0x308
 work_pending+0x8/0x14

The nrequest should be increased/decreased only if PG_INODE_REF flag
was setted.

But in the nfs_inode_remove_request function, it maybe decrease when
no PG_INODE_REF flag, this maybe lead nrequests count error.

Reported-by: Hulk Robot <hulkci@huawei.com>
Signed-off-by: ZhangXiaoxu <zhangxiaoxu5@huawei.com>
---
 fs/nfs/write.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

Comments

zhangxiaoxu (A) Oct. 5, 2019, 1:51 a.m. UTC | #1
ping.

On 2019/9/26 14:29, ZhangXiaoxu wrote:
> When xfstests testing, there are some WARNING as below:
> 
> WARNING: CPU: 0 PID: 6235 at fs/nfs/inode.c:122 nfs_clear_inode+0x9c/0xd8
> Modules linked in:
> CPU: 0 PID: 6235 Comm: umount.nfs
> Hardware name: linux,dummy-virt (DT)
> pstate: 60000005 (nZCv daif -PAN -UAO)
> pc : nfs_clear_inode+0x9c/0xd8
> lr : nfs_evict_inode+0x60/0x78
> sp : fffffc000f68fc00
> x29: fffffc000f68fc00 x28: fffffe00c53155c0
> x27: fffffe00c5315000 x26: fffffc0009a63748
> x25: fffffc000f68fd18 x24: fffffc000bfaaf40
> x23: fffffc000936d3c0 x22: fffffe00c4ff5e20
> x21: fffffc000bfaaf40 x20: fffffe00c4ff5d10
> x19: fffffc000c056000 x18: 000000000000003c
> x17: 0000000000000000 x16: 0000000000000000
> x15: 0000000000000040 x14: 0000000000000228
> x13: fffffc000c3a2000 x12: 0000000000000045
> x11: 0000000000000000 x10: 0000000000000000
> x9 : 0000000000000000 x8 : 0000000000000000
> x7 : 0000000000000000 x6 : fffffc00084b027c
> x5 : fffffc0009a64000 x4 : fffffe00c0e77400
> x3 : fffffc000c0563a8 x2 : fffffffffffffffb
> x1 : 000000000000764e x0 : 0000000000000001
> Call trace:
>   nfs_clear_inode+0x9c/0xd8
>   nfs_evict_inode+0x60/0x78
>   evict+0x108/0x380
>   dispose_list+0x70/0xa0
>   evict_inodes+0x194/0x210
>   generic_shutdown_super+0xb0/0x220
>   nfs_kill_super+0x40/0x88
>   deactivate_locked_super+0xb4/0x120
>   deactivate_super+0x144/0x160
>   cleanup_mnt+0x98/0x148
>   __cleanup_mnt+0x38/0x50
>   task_work_run+0x114/0x160
>   do_notify_resume+0x2f8/0x308
>   work_pending+0x8/0x14
> 
> The nrequest should be increased/decreased only if PG_INODE_REF flag
> was setted.
> 
> But in the nfs_inode_remove_request function, it maybe decrease when
> no PG_INODE_REF flag, this maybe lead nrequests count error.
> 
> Reported-by: Hulk Robot <hulkci@huawei.com>
> Signed-off-by: ZhangXiaoxu <zhangxiaoxu5@huawei.com>
> ---
>   fs/nfs/write.c | 5 +++--
>   1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/fs/nfs/write.c b/fs/nfs/write.c
> index 85ca495..52cab65 100644
> --- a/fs/nfs/write.c
> +++ b/fs/nfs/write.c
> @@ -786,7 +786,6 @@ static void nfs_inode_remove_request(struct nfs_page *req)
>   	struct nfs_inode *nfsi = NFS_I(inode);
>   	struct nfs_page *head;
>   
> -	atomic_long_dec(&nfsi->nrequests);
>   	if (nfs_page_group_sync_on_bit(req, PG_REMOVE)) {
>   		head = req->wb_head;
>   
> @@ -799,8 +798,10 @@ static void nfs_inode_remove_request(struct nfs_page *req)
>   		spin_unlock(&mapping->private_lock);
>   	}
>   
> -	if (test_and_clear_bit(PG_INODE_REF, &req->wb_flags))
> +	if (test_and_clear_bit(PG_INODE_REF, &req->wb_flags)) {
>   		nfs_release_request(req);
> +		atomic_long_dec(&nfsi->nrequests);
> +	}
>   }
>   
>   static void
>
Trond Myklebust Oct. 5, 2019, 2:07 p.m. UTC | #2
On Sat, 2019-10-05 at 09:51 +0800, zhangxiaoxu (A) wrote:
> ping.
> 
> On 2019/9/26 14:29, ZhangXiaoxu wrote:
> > When xfstests testing, there are some WARNING as below:
> > 
> > WARNING: CPU: 0 PID: 6235 at fs/nfs/inode.c:122
> > nfs_clear_inode+0x9c/0xd8
> > Modules linked in:
> > CPU: 0 PID: 6235 Comm: umount.nfs
> > Hardware name: linux,dummy-virt (DT)
> > pstate: 60000005 (nZCv daif -PAN -UAO)
> > pc : nfs_clear_inode+0x9c/0xd8
> > lr : nfs_evict_inode+0x60/0x78
> > sp : fffffc000f68fc00
> > x29: fffffc000f68fc00 x28: fffffe00c53155c0
> > x27: fffffe00c5315000 x26: fffffc0009a63748
> > x25: fffffc000f68fd18 x24: fffffc000bfaaf40
> > x23: fffffc000936d3c0 x22: fffffe00c4ff5e20
> > x21: fffffc000bfaaf40 x20: fffffe00c4ff5d10
> > x19: fffffc000c056000 x18: 000000000000003c
> > x17: 0000000000000000 x16: 0000000000000000
> > x15: 0000000000000040 x14: 0000000000000228
> > x13: fffffc000c3a2000 x12: 0000000000000045
> > x11: 0000000000000000 x10: 0000000000000000
> > x9 : 0000000000000000 x8 : 0000000000000000
> > x7 : 0000000000000000 x6 : fffffc00084b027c
> > x5 : fffffc0009a64000 x4 : fffffe00c0e77400
> > x3 : fffffc000c0563a8 x2 : fffffffffffffffb
> > x1 : 000000000000764e x0 : 0000000000000001
> > Call trace:
> >   nfs_clear_inode+0x9c/0xd8
> >   nfs_evict_inode+0x60/0x78
> >   evict+0x108/0x380
> >   dispose_list+0x70/0xa0
> >   evict_inodes+0x194/0x210
> >   generic_shutdown_super+0xb0/0x220
> >   nfs_kill_super+0x40/0x88
> >   deactivate_locked_super+0xb4/0x120
> >   deactivate_super+0x144/0x160
> >   cleanup_mnt+0x98/0x148
> >   __cleanup_mnt+0x38/0x50
> >   task_work_run+0x114/0x160
> >   do_notify_resume+0x2f8/0x308
> >   work_pending+0x8/0x14
> > 
> > The nrequest should be increased/decreased only if PG_INODE_REF
> > flag
> > was setted.
> > 
> > But in the nfs_inode_remove_request function, it maybe decrease
> > when
> > no PG_INODE_REF flag, this maybe lead nrequests count error.
> > 
> > Reported-by: Hulk Robot <hulkci@huawei.com>
> > Signed-off-by: ZhangXiaoxu <zhangxiaoxu5@huawei.com>
> > ---
> >   fs/nfs/write.c | 5 +++--
> >   1 file changed, 3 insertions(+), 2 deletions(-)
> > 
> > diff --git a/fs/nfs/write.c b/fs/nfs/write.c
> > index 85ca495..52cab65 100644
> > --- a/fs/nfs/write.c
> > +++ b/fs/nfs/write.c
> > @@ -786,7 +786,6 @@ static void nfs_inode_remove_request(struct
> > nfs_page *req)
> >   	struct nfs_inode *nfsi = NFS_I(inode);
> >   	struct nfs_page *head;
> >   
> > -	atomic_long_dec(&nfsi->nrequests);
> >   	if (nfs_page_group_sync_on_bit(req, PG_REMOVE)) {
> >   		head = req->wb_head;
> >   
> > @@ -799,8 +798,10 @@ static void nfs_inode_remove_request(struct
> > nfs_page *req)
> >   		spin_unlock(&mapping->private_lock);
> >   	}
> >   
> > -	if (test_and_clear_bit(PG_INODE_REF, &req->wb_flags))
> > +	if (test_and_clear_bit(PG_INODE_REF, &req->wb_flags)) {
> >   		nfs_release_request(req);
> > +		atomic_long_dec(&nfsi->nrequests);
> > +	}
> >   }
> >   
> >   static void
> > 

Hmm... What about nfs_page_group_init()? That also bumps nfsi-
>nrequests.
Trond Myklebust Oct. 5, 2019, 2:35 p.m. UTC | #3
On Sat, 2019-10-05 at 14:07 +0000, Trond Myklebust wrote:
> On Sat, 2019-10-05 at 09:51 +0800, zhangxiaoxu (A) wrote:
> > ping.
> > 
> > On 2019/9/26 14:29, ZhangXiaoxu wrote:
> > > When xfstests testing, there are some WARNING as below:
> > > 
> > > WARNING: CPU: 0 PID: 6235 at fs/nfs/inode.c:122
> > > nfs_clear_inode+0x9c/0xd8
> > > Modules linked in:
> > > CPU: 0 PID: 6235 Comm: umount.nfs
> > > Hardware name: linux,dummy-virt (DT)
> > > pstate: 60000005 (nZCv daif -PAN -UAO)
> > > pc : nfs_clear_inode+0x9c/0xd8
> > > lr : nfs_evict_inode+0x60/0x78
> > > sp : fffffc000f68fc00
> > > x29: fffffc000f68fc00 x28: fffffe00c53155c0
> > > x27: fffffe00c5315000 x26: fffffc0009a63748
> > > x25: fffffc000f68fd18 x24: fffffc000bfaaf40
> > > x23: fffffc000936d3c0 x22: fffffe00c4ff5e20
> > > x21: fffffc000bfaaf40 x20: fffffe00c4ff5d10
> > > x19: fffffc000c056000 x18: 000000000000003c
> > > x17: 0000000000000000 x16: 0000000000000000
> > > x15: 0000000000000040 x14: 0000000000000228
> > > x13: fffffc000c3a2000 x12: 0000000000000045
> > > x11: 0000000000000000 x10: 0000000000000000
> > > x9 : 0000000000000000 x8 : 0000000000000000
> > > x7 : 0000000000000000 x6 : fffffc00084b027c
> > > x5 : fffffc0009a64000 x4 : fffffe00c0e77400
> > > x3 : fffffc000c0563a8 x2 : fffffffffffffffb
> > > x1 : 000000000000764e x0 : 0000000000000001
> > > Call trace:
> > >   nfs_clear_inode+0x9c/0xd8
> > >   nfs_evict_inode+0x60/0x78
> > >   evict+0x108/0x380
> > >   dispose_list+0x70/0xa0
> > >   evict_inodes+0x194/0x210
> > >   generic_shutdown_super+0xb0/0x220
> > >   nfs_kill_super+0x40/0x88
> > >   deactivate_locked_super+0xb4/0x120
> > >   deactivate_super+0x144/0x160
> > >   cleanup_mnt+0x98/0x148
> > >   __cleanup_mnt+0x38/0x50
> > >   task_work_run+0x114/0x160
> > >   do_notify_resume+0x2f8/0x308
> > >   work_pending+0x8/0x14
> > > 
> > > The nrequest should be increased/decreased only if PG_INODE_REF
> > > flag
> > > was setted.
> > > 
> > > But in the nfs_inode_remove_request function, it maybe decrease
> > > when
> > > no PG_INODE_REF flag, this maybe lead nrequests count error.
> > > 
> > > Reported-by: Hulk Robot <hulkci@huawei.com>
> > > Signed-off-by: ZhangXiaoxu <zhangxiaoxu5@huawei.com>
> > > ---
> > >   fs/nfs/write.c | 5 +++--
> > >   1 file changed, 3 insertions(+), 2 deletions(-)
> > > 
> > > diff --git a/fs/nfs/write.c b/fs/nfs/write.c
> > > index 85ca495..52cab65 100644
> > > --- a/fs/nfs/write.c
> > > +++ b/fs/nfs/write.c
> > > @@ -786,7 +786,6 @@ static void nfs_inode_remove_request(struct
> > > nfs_page *req)
> > >   	struct nfs_inode *nfsi = NFS_I(inode);
> > >   	struct nfs_page *head;
> > >   
> > > -	atomic_long_dec(&nfsi->nrequests);
> > >   	if (nfs_page_group_sync_on_bit(req, PG_REMOVE)) {
> > >   		head = req->wb_head;
> > >   
> > > @@ -799,8 +798,10 @@ static void nfs_inode_remove_request(struct
> > > nfs_page *req)
> > >   		spin_unlock(&mapping->private_lock);
> > >   	}
> > >   
> > > -	if (test_and_clear_bit(PG_INODE_REF, &req->wb_flags))
> > > +	if (test_and_clear_bit(PG_INODE_REF, &req->wb_flags)) {
> > >   		nfs_release_request(req);
> > > +		atomic_long_dec(&nfsi->nrequests);
> > > +	}
> > >   }
> > >   
> > >   static void
> > > 
> 
> Hmm... What about nfs_page_group_init()? That also bumps nfsi-
> > nrequests.

Ah... Never mind, that's just copying the PG_INODE_REF flag to the new
subrequest.

However nfs_lock_and_join_requests() looks like it does need to change
to something like the following:

        /* Postpone destruction of this request */
-       if (test_and_clear_bit(PG_REMOVE, &head->wb_flags)) {
-               set_bit(PG_INODE_REF, &head->wb_flags);
+       if (test_and_clear_bit(PG_REMOVE, &head->wb_flags) &&
+           !test_and_set_bit(PG_INODE_REF, &head->wb_flags)) {
                kref_get(&head->wb_kref);
                atomic_long_inc(&NFS_I(inode)->nrequests);
        }


Do you agree?
zhangxiaoxu (A) Oct. 8, 2019, 2 a.m. UTC | #4
Thanks for your review.

I think PG_REMOVE and PG_INODE_REF can't be both setted except in
'nfs_inode_remove_request' function.

nfs_inode_remove_request
	// maybe set PG_REMOVE here.
	nfs_page_group_sync_on_bit(req, PG_REMOVE)
		nfs_page_group_lock(req);
		nfs_page_group_sync_on_bit_locked
			WARN_ON_ONCE(test_and_set_bit(bit, &req->wb_flags));
		nfs_page_group_unlock(req);
	// But also clear the PG_INODE_REF flag.
	test_and_clear_bit(PG_INODE_REF, &req->wb_flags)

'nfs_lock_and_join_requests' also need the PG_HEADLOCK flag:

nfs_lock_and_join_requests
	nfs_page_group_lock(head);
		test_and_clear_bit(PG_REMOVE, &head->wb_flags)
	nfs_page_group_unlock(head);


在 2019/10/5 22:35, Trond Myklebust 写道:
> However nfs_lock_and_join_requests() looks like it does need to change
> to something like the following:
> 
>          /* Postpone destruction of this request */
> -       if (test_and_clear_bit(PG_REMOVE, &head->wb_flags)) {
> -               set_bit(PG_INODE_REF, &head->wb_flags);
> +       if (test_and_clear_bit(PG_REMOVE, &head->wb_flags) &&
> +           !test_and_set_bit(PG_INODE_REF, &head->wb_flags)) {
>                  kref_get(&head->wb_kref);
>                  atomic_long_inc(&NFS_I(inode)->nrequests);
>          }
> 
> 
> Do you agree?
Trond Myklebust Oct. 8, 2019, 12:31 p.m. UTC | #5
On Tue, 2019-10-08 at 10:00 +0800, zhangxiaoxu (A) wrote:
> Thanks for your review.
> 
> I think PG_REMOVE and PG_INODE_REF can't be both setted except in
> 'nfs_inode_remove_request' function.
> 
> nfs_inode_remove_request
> 	// maybe set PG_REMOVE here.
> 	nfs_page_group_sync_on_bit(req, PG_REMOVE)
> 		nfs_page_group_lock(req);
> 		nfs_page_group_sync_on_bit_locked
> 			WARN_ON_ONCE(test_and_set_bit(bit, &req-
> >wb_flags));
> 		nfs_page_group_unlock(req);
> 	// But also clear the PG_INODE_REF flag.
> 	test_and_clear_bit(PG_INODE_REF, &req->wb_flags)
> 
> 'nfs_lock_and_join_requests' also need the PG_HEADLOCK flag:
> 
> nfs_lock_and_join_requests
> 	nfs_page_group_lock(head);
> 		test_and_clear_bit(PG_REMOVE, &head->wb_flags)
> 	nfs_page_group_unlock(head);

Fair enough, and the race between nfs_inode_remove_request() and
nfs_lock_and_join_requests() with the setting of PG_REMOVE and the
clearing on PG_INODE_REF is prevented by the presence of the PG_LOCK.

OK. I agree with that argument. Thanks for the explanation!

Cheers
  Trond

Patch
diff mbox series

diff --git a/fs/nfs/write.c b/fs/nfs/write.c
index 85ca495..52cab65 100644
--- a/fs/nfs/write.c
+++ b/fs/nfs/write.c
@@ -786,7 +786,6 @@  static void nfs_inode_remove_request(struct nfs_page *req)
 	struct nfs_inode *nfsi = NFS_I(inode);
 	struct nfs_page *head;
 
-	atomic_long_dec(&nfsi->nrequests);
 	if (nfs_page_group_sync_on_bit(req, PG_REMOVE)) {
 		head = req->wb_head;
 
@@ -799,8 +798,10 @@  static void nfs_inode_remove_request(struct nfs_page *req)
 		spin_unlock(&mapping->private_lock);
 	}
 
-	if (test_and_clear_bit(PG_INODE_REF, &req->wb_flags))
+	if (test_and_clear_bit(PG_INODE_REF, &req->wb_flags)) {
 		nfs_release_request(req);
+		atomic_long_dec(&nfsi->nrequests);
+	}
 }
 
 static void