nfs NULL-dereferencing in net-next
diff mbox

Message ID 817e43c5-d88d-e616-7074-5715de29d319@Netapp.com
State New
Headers show

Commit Message

Schumaker, Anna Oct. 26, 2016, 2:40 p.m. UTC
On 10/25/2016 01:19 PM, Yotam Gigi wrote:
> 
>> -----Original Message-----
>> From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org] On
>> Behalf Of Jakub Kicinski
>> Sent: Monday, October 17, 2016 10:20 PM
>> To: Andy Adamson <andros@netapp.com>; Anna Schumaker
>> <Anna.Schumaker@Netapp.com>; linux-nfs@vger.kernel.org
>> Cc: netdev@vger.kernel.org; Trond Myklebust <Trond.Myklebust@netapp.com>
>> Subject: nfs NULL-dereferencing in net-next
>>
>> Hi!
>>
>> I'm hitting this reliably on net-next, HEAD at 3f3177bb680f
>> ("fsl/fman: fix error return code in mac_probe()").
> 
> 
> I see the same thing. It happens constantly on some of my machines, making them
> completely unusable.
> 
> I bisected it and got to the commit:
> 
> commit 04ea1b3e6d8ed4978bb608c1748530af3de8c274
> Author: Andy Adamson <andros@netapp.com>
> Date:   Fri Sep 9 09:22:27 2016 -0400
> 
>     NFS add xprt switch addrs test to match client
>     
>     Signed-off-by: Andy Adamson <andros@netapp.com>
>     Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>

Thanks for reporting on this everyone!  Does this patch help?

>From 96376ca1dd4077a1d341bdcb9cc86426ee3844f1 Mon Sep 17 00:00:00 2001
From: Anna Schumaker <Anna.Schumaker@Netapp.com>
Date: Wed, 26 Oct 2016 10:33:31 -0400
Subject: [PATCH] SUNRPC: Fix suspicious RCU usage

We need to hold the rcu_read_lock() when calling rcu_dereference(),
otherwise we can't guarantee that the object being dereferenced still
exists.

Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>
---
 net/sunrpc/clnt.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

--
2.10.1

> 
> 
>>
>> [   23.409633] BUG: unable to handle kernel NULL pointer dereference at
>> 0000000000000172
>> [   23.418716] IP: [<ffffffffc041776c>] rpc_clnt_xprt_switch_has_addr+0xc/0x40
>> [sunrpc]
>> [   23.427574] PGD 859020067 [   23.430472] PUD 858f2d067
>> PMD 0 [   23.434311]
>> [   23.436133] Oops: 0000 [#1] PREEMPT SMP
>> [   23.440506] Modules linked in: nfsv4 ip6table_filter ip6_tables iptable_filter
>> ip_tables ebtable_nat ebtables x_tables intel_ri
>> [   23.505915] CPU: 1 PID: 1067 Comm: mount.nfs Not tainted 4.8.0-perf-13951-
>> g3f3177bb680f #51
>> [   23.515363] Hardware name: Dell Inc. PowerEdge T630/0W9WXC, BIOS 1.2.10
>> 03/10/2015
>> [   23.523937] task: ffff983e9086ea00 task.stack: ffffac6c0a57c000
>> [   23.530641] RIP: 0010:[<ffffffffc041776c>]  [<ffffffffc041776c>]
>> rpc_clnt_xprt_switch_has_addr+0xc/0x40 [sunrpc]
>> [   23.542229] RSP: 0018:ffffac6c0a57fb28  EFLAGS: 00010a97
>> [   23.548255] RAX: 00000000c80214ac RBX: ffff983e97c7b000 RCX: ffff983e9b3bc180
>> [   23.556320] RDX: 0000000000000001 RSI: ffff983e9928ed28 RDI: ffffffffffffffea
>> [   23.564386] RBP: ffffac6c0a57fb38 R08: ffff983e97090630 R09: ffff983e9928ed30
>> [   23.572452] R10: ffffac6c0a57fba0 R11: 0000000000000010 R12: ffffac6c0a57fba0
>> [   23.580517] R13: ffff983e9928ed28 R14: 0000000000000000 R15: ffff983e91360560
>> [   23.588585] FS:  00007f4c348aa880(0000) GS:ffff983e9f240000(0000)
>> knlGS:0000000000000000
>> [   23.597742] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
>> [   23.604251] CR2: 0000000000000172 CR3: 0000000850a5f000 CR4:
>> 00000000001406e0
>> [   23.612316] Stack:
>> [   23.614648]  ffff983e97c7b000 ffffac6c0a57fba0 ffffac6c0a57fb90 ffffffffc04d38c3
>> [   23.623331]  ffff983e91360500 ffff983e9928ed30 ffffffffc0b9e560
>> ffff983e913605b8
>> [   23.632016]  ffff983e9882e800 ffff983e9882e800 ffffac6c0a57fc30 ffffac6c0a57fdb8
>> [   23.640706] Call Trace:
>> [   23.643535]  [<ffffffffc04d38c3>] nfs_get_client+0x123/0x340 [nfs]
>> [   23.650542]  [<ffffffffc0b8f070>] nfs4_set_client+0x80/0xb0 [nfsv4]
>> [   23.657642]  [<ffffffffc0b90305>] nfs4_create_server+0x115/0x2a0 [nfsv4]
>> [   23.665230]  [<ffffffffc0b888ce>] nfs4_remote_mount+0x2e/0x60 [nfsv4]
>> [   23.672519]  [<ffffffffba1e590a>] mount_fs+0x3a/0x160
>> [   23.678254]  [<ffffffffba201a5e>] ? alloc_vfsmnt+0x19e/0x230
>> [   23.684669]  [<ffffffffba201b57>] vfs_kern_mount+0x67/0x110
>> [   23.690990]  [<ffffffffc0b887f4>] nfs_do_root_mount+0x84/0xc0 [nfsv4]
>> [   23.698284]  [<ffffffffc0b88b97>] nfs4_try_mount+0x37/0x50 [nfsv4]
>> [   23.705287]  [<ffffffffc04dfbd1>] nfs_fs_mount+0x2d1/0xa70 [nfs]
>> [   23.712092]  [<ffffffffba3a6228>] ? find_next_bit+0x18/0x20
>> [   23.718413]  [<ffffffffc04deac0>] ? nfs_remount+0x3c0/0x3c0 [nfs]
>> [   23.725316]  [<ffffffffc04dedb0>] ? nfs_clone_super+0x130/0x130 [nfs]
>> [   23.732606]  [<ffffffffba1e590a>] mount_fs+0x3a/0x160
>> [   23.738340]  [<ffffffffba201a5e>] ? alloc_vfsmnt+0x19e/0x230
>> [   23.744755]  [<ffffffffba201b57>] vfs_kern_mount+0x67/0x110
>> [   23.751071]  [<ffffffffba2041df>] do_mount+0x1bf/0xc70
>> [   23.756904]  [<ffffffffba203e9b>] ? copy_mount_options+0xbb/0x220
>> [   23.763803]  [<ffffffffba204fa3>] SyS_mount+0x83/0xd0
>> [   23.769538]  [<ffffffffba6f1ea4>] entry_SYSCALL_64_fastpath+0x17/0x98
>> [   23.776817] Code: 01 00 48 8b 93 f8 04 00 00 44 89 e6 48 c7 c7 98 b2 43 c0 e8 9f 0d d4
>> f9 eb c0 0f 1f 44 00 00 0f 1f 44 00 00
>> [   23.802909] RIP  [<ffffffffc041776c>] rpc_clnt_xprt_switch_has_addr+0xc/0x40
>> [sunrpc]
>> [   23.811857]  RSP <ffffac6c0a57fb28>
>> [   23.815839] CR2: 0000000000000172
>> [   23.819629] ---[ end trace 9958eca92c9eeafe ]---
>> [   23.827345] note: mount.nfs[1067] exited with preempt_count 1
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Comments

Jakub Kicinski Oct. 26, 2016, 6:08 p.m. UTC | #1
On Wed, 26 Oct 2016 16:15:24 +0000, Yotam Gigi wrote:
> >-----Original Message-----
> >From: Anna Schumaker [mailto:Anna.Schumaker@Netapp.com]
> >Sent: Wednesday, October 26, 2016 5:40 PM
> >To: Yotam Gigi <yotamg@mellanox.com>; Jakub Kicinski <kubakici@wp.pl>; Andy
> >Adamson <andros@netapp.com>; Anna Schumaker
> ><Anna.Schumaker@Netapp.com>; linux-nfs@vger.kernel.org
> >Cc: netdev@vger.kernel.org; Trond Myklebust <Trond.Myklebust@netapp.com>;
> >Yotam Gigi <yotam.gi@gmail.com>; mlxsw <mlxsw@mellanox.com>
> >Subject: Re: nfs NULL-dereferencing in net-next
> >
> >On 10/25/2016 01:19 PM, Yotam Gigi wrote:  
> >>  
> >>> -----Original Message-----
> >>> From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org]  
> >On  
> >>> Behalf Of Jakub Kicinski
> >>> Sent: Monday, October 17, 2016 10:20 PM
> >>> To: Andy Adamson <andros@netapp.com>; Anna Schumaker
> >>> <Anna.Schumaker@Netapp.com>; linux-nfs@vger.kernel.org
> >>> Cc: netdev@vger.kernel.org; Trond Myklebust  
> ><Trond.Myklebust@netapp.com>  
> >>> Subject: nfs NULL-dereferencing in net-next
> >>>
> >>> Hi!
> >>>
> >>> I'm hitting this reliably on net-next, HEAD at 3f3177bb680f
> >>> ("fsl/fman: fix error return code in mac_probe()").  
> >>
> >>
> >> I see the same thing. It happens constantly on some of my machines, making  
> >them  
> >> completely unusable.
> >>
> >> I bisected it and got to the commit:
> >>
> >> commit 04ea1b3e6d8ed4978bb608c1748530af3de8c274
> >> Author: Andy Adamson <andros@netapp.com>
> >> Date:   Fri Sep 9 09:22:27 2016 -0400
> >>
> >>     NFS add xprt switch addrs test to match client
> >>
> >>     Signed-off-by: Andy Adamson <andros@netapp.com>
> >>     Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>  
> >
> >Thanks for reporting on this everyone!  Does this patch help?  
> 
> Actually, I still see the same bug with the same trace.

I rebuild the latest net-next and I'm not seeing the trace any more...
I'm only seeing this (with or without your patch):

[   23.465877] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0
[   23.473784] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0
[   23.588890] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0
[   23.596746] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0
[   23.781574] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0
[   23.789599] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0

HTH
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Schumaker, Anna Oct. 26, 2016, 6:17 p.m. UTC | #2
On 10/26/2016 02:08 PM, Jakub Kicinski wrote:
> On Wed, 26 Oct 2016 16:15:24 +0000, Yotam Gigi wrote:
>>> -----Original Message-----
>>> From: Anna Schumaker [mailto:Anna.Schumaker@Netapp.com]
>>> Sent: Wednesday, October 26, 2016 5:40 PM
>>> To: Yotam Gigi <yotamg@mellanox.com>; Jakub Kicinski <kubakici@wp.pl>; Andy
>>> Adamson <andros@netapp.com>; Anna Schumaker
>>> <Anna.Schumaker@Netapp.com>; linux-nfs@vger.kernel.org
>>> Cc: netdev@vger.kernel.org; Trond Myklebust <Trond.Myklebust@netapp.com>;
>>> Yotam Gigi <yotam.gi@gmail.com>; mlxsw <mlxsw@mellanox.com>
>>> Subject: Re: nfs NULL-dereferencing in net-next
>>>
>>> On 10/25/2016 01:19 PM, Yotam Gigi wrote:  
>>>>  
>>>>> -----Original Message-----
>>>>> From: netdev-owner@vger.kernel.org [mailto:netdev-owner@vger.kernel.org]  
>>> On  
>>>>> Behalf Of Jakub Kicinski
>>>>> Sent: Monday, October 17, 2016 10:20 PM
>>>>> To: Andy Adamson <andros@netapp.com>; Anna Schumaker
>>>>> <Anna.Schumaker@Netapp.com>; linux-nfs@vger.kernel.org
>>>>> Cc: netdev@vger.kernel.org; Trond Myklebust  
>>> <Trond.Myklebust@netapp.com>  
>>>>> Subject: nfs NULL-dereferencing in net-next
>>>>>
>>>>> Hi!
>>>>>
>>>>> I'm hitting this reliably on net-next, HEAD at 3f3177bb680f
>>>>> ("fsl/fman: fix error return code in mac_probe()").  
>>>>
>>>>
>>>> I see the same thing. It happens constantly on some of my machines, making  
>>> them  
>>>> completely unusable.
>>>>
>>>> I bisected it and got to the commit:
>>>>
>>>> commit 04ea1b3e6d8ed4978bb608c1748530af3de8c274
>>>> Author: Andy Adamson <andros@netapp.com>
>>>> Date:   Fri Sep 9 09:22:27 2016 -0400
>>>>
>>>>     NFS add xprt switch addrs test to match client
>>>>
>>>>     Signed-off-by: Andy Adamson <andros@netapp.com>
>>>>     Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>  
>>>
>>> Thanks for reporting on this everyone!  Does this patch help?  
>>
>> Actually, I still see the same bug with the same trace.

Well, it was worth a shot.  I'll keep poking at it.

> 
> I rebuild the latest net-next and I'm not seeing the trace any more...
> I'm only seeing this (with or without your patch):
> 
> [   23.465877] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0
> [   23.473784] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0
> [   23.588890] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0
> [   23.596746] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0
> [   23.781574] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0
> [   23.789599] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0

Interesting, I get that too when I try to use NFS v4.1.  It's weird that the crash would stop happening like that, so maybe something is racy in this area.

Thanks for testing, Yotam and Jakub!
Anna

> 
> HTH
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Yotam Gigi Oct. 27, 2016, 6:50 a.m. UTC | #3
DQoNCj4tLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPkZyb206IEFubmEgU2NodW1ha2VyIFtt
YWlsdG86QW5uYS5TY2h1bWFrZXJATmV0YXBwLmNvbV0NCj5TZW50OiBXZWRuZXNkYXksIE9jdG9i
ZXIgMjYsIDIwMTYgOToxNyBQTQ0KPlRvOiBKYWt1YiBLaWNpbnNraSA8a3ViYWtpY2lAd3AucGw+
DQo+Q2M6IFlvdGFtIEdpZ2kgPHlvdGFtZ0BtZWxsYW5veC5jb20+OyBBbmR5IEFkYW1zb24gPGFu
ZHJvc0BuZXRhcHAuY29tPjsNCj5saW51eC1uZnNAdmdlci5rZXJuZWwub3JnOyBuZXRkZXZAdmdl
ci5rZXJuZWwub3JnOyBUcm9uZCBNeWtsZWJ1c3QNCj48VHJvbmQuTXlrbGVidXN0QG5ldGFwcC5j
b20+OyBZb3RhbSBHaWdpIDx5b3RhbS5naUBnbWFpbC5jb20+OyBtbHhzdw0KPjxtbHhzd0BtZWxs
YW5veC5jb20+DQo+U3ViamVjdDogUmU6IG5mcyBOVUxMLWRlcmVmZXJlbmNpbmcgaW4gbmV0LW5l
eHQNCj4NCj5PbiAxMC8yNi8yMDE2IDAyOjA4IFBNLCBKYWt1YiBLaWNpbnNraSB3cm90ZToNCj4+
IE9uIFdlZCwgMjYgT2N0IDIwMTYgMTY6MTU6MjQgKzAwMDAsIFlvdGFtIEdpZ2kgd3JvdGU6DQo+
Pj4+IC0tLS0tT3JpZ2luYWwgTWVzc2FnZS0tLS0tDQo+Pj4+IEZyb206IEFubmEgU2NodW1ha2Vy
IFttYWlsdG86QW5uYS5TY2h1bWFrZXJATmV0YXBwLmNvbV0NCj4+Pj4gU2VudDogV2VkbmVzZGF5
LCBPY3RvYmVyIDI2LCAyMDE2IDU6NDAgUE0NCj4+Pj4gVG86IFlvdGFtIEdpZ2kgPHlvdGFtZ0Bt
ZWxsYW5veC5jb20+OyBKYWt1YiBLaWNpbnNraSA8a3ViYWtpY2lAd3AucGw+Ow0KPkFuZHkNCj4+
Pj4gQWRhbXNvbiA8YW5kcm9zQG5ldGFwcC5jb20+OyBBbm5hIFNjaHVtYWtlcg0KPj4+PiA8QW5u
YS5TY2h1bWFrZXJATmV0YXBwLmNvbT47IGxpbnV4LW5mc0B2Z2VyLmtlcm5lbC5vcmcNCj4+Pj4g
Q2M6IG5ldGRldkB2Z2VyLmtlcm5lbC5vcmc7IFRyb25kIE15a2xlYnVzdA0KPjxUcm9uZC5NeWts
ZWJ1c3RAbmV0YXBwLmNvbT47DQo+Pj4+IFlvdGFtIEdpZ2kgPHlvdGFtLmdpQGdtYWlsLmNvbT47
IG1seHN3IDxtbHhzd0BtZWxsYW5veC5jb20+DQo+Pj4+IFN1YmplY3Q6IFJlOiBuZnMgTlVMTC1k
ZXJlZmVyZW5jaW5nIGluIG5ldC1uZXh0DQo+Pj4+DQo+Pj4+IE9uIDEwLzI1LzIwMTYgMDE6MTkg
UE0sIFlvdGFtIEdpZ2kgd3JvdGU6DQo+Pj4+Pg0KPj4+Pj4+IC0tLS0tT3JpZ2luYWwgTWVzc2Fn
ZS0tLS0tDQo+Pj4+Pj4gRnJvbTogbmV0ZGV2LW93bmVyQHZnZXIua2VybmVsLm9yZyBbbWFpbHRv
Om5ldGRldi0NCj5vd25lckB2Z2VyLmtlcm5lbC5vcmddDQo+Pj4+IE9uDQo+Pj4+Pj4gQmVoYWxm
IE9mIEpha3ViIEtpY2luc2tpDQo+Pj4+Pj4gU2VudDogTW9uZGF5LCBPY3RvYmVyIDE3LCAyMDE2
IDEwOjIwIFBNDQo+Pj4+Pj4gVG86IEFuZHkgQWRhbXNvbiA8YW5kcm9zQG5ldGFwcC5jb20+OyBB
bm5hIFNjaHVtYWtlcg0KPj4+Pj4+IDxBbm5hLlNjaHVtYWtlckBOZXRhcHAuY29tPjsgbGludXgt
bmZzQHZnZXIua2VybmVsLm9yZw0KPj4+Pj4+IENjOiBuZXRkZXZAdmdlci5rZXJuZWwub3JnOyBU
cm9uZCBNeWtsZWJ1c3QNCj4+Pj4gPFRyb25kLk15a2xlYnVzdEBuZXRhcHAuY29tPg0KPj4+Pj4+
IFN1YmplY3Q6IG5mcyBOVUxMLWRlcmVmZXJlbmNpbmcgaW4gbmV0LW5leHQNCj4+Pj4+Pg0KPj4+
Pj4+IEhpIQ0KPj4+Pj4+DQo+Pj4+Pj4gSSdtIGhpdHRpbmcgdGhpcyByZWxpYWJseSBvbiBuZXQt
bmV4dCwgSEVBRCBhdCAzZjMxNzdiYjY4MGYNCj4+Pj4+PiAoImZzbC9mbWFuOiBmaXggZXJyb3Ig
cmV0dXJuIGNvZGUgaW4gbWFjX3Byb2JlKCkiKS4NCj4+Pj4+DQo+Pj4+Pg0KPj4+Pj4gSSBzZWUg
dGhlIHNhbWUgdGhpbmcuIEl0IGhhcHBlbnMgY29uc3RhbnRseSBvbiBzb21lIG9mIG15IG1hY2hp
bmVzLCBtYWtpbmcNCj4+Pj4gdGhlbQ0KPj4+Pj4gY29tcGxldGVseSB1bnVzYWJsZS4NCj4+Pj4+
DQo+Pj4+PiBJIGJpc2VjdGVkIGl0IGFuZCBnb3QgdG8gdGhlIGNvbW1pdDoNCj4+Pj4+DQo+Pj4+
PiBjb21taXQgMDRlYTFiM2U2ZDhlZDQ5NzhiYjYwOGMxNzQ4NTMwYWYzZGU4YzI3NA0KPj4+Pj4g
QXV0aG9yOiBBbmR5IEFkYW1zb24gPGFuZHJvc0BuZXRhcHAuY29tPg0KPj4+Pj4gRGF0ZTogICBG
cmkgU2VwIDkgMDk6MjI6MjcgMjAxNiAtMDQwMA0KPj4+Pj4NCj4+Pj4+ICAgICBORlMgYWRkIHhw
cnQgc3dpdGNoIGFkZHJzIHRlc3QgdG8gbWF0Y2ggY2xpZW50DQo+Pj4+Pg0KPj4+Pj4gICAgIFNp
Z25lZC1vZmYtYnk6IEFuZHkgQWRhbXNvbiA8YW5kcm9zQG5ldGFwcC5jb20+DQo+Pj4+PiAgICAg
U2lnbmVkLW9mZi1ieTogQW5uYSBTY2h1bWFrZXIgPEFubmEuU2NodW1ha2VyQE5ldGFwcC5jb20+
DQo+Pj4+DQo+Pj4+IFRoYW5rcyBmb3IgcmVwb3J0aW5nIG9uIHRoaXMgZXZlcnlvbmUhICBEb2Vz
IHRoaXMgcGF0Y2ggaGVscD8NCj4+Pg0KPj4+IEFjdHVhbGx5LCBJIHN0aWxsIHNlZSB0aGUgc2Ft
ZSBidWcgd2l0aCB0aGUgc2FtZSB0cmFjZS4NCj4NCj5XZWxsLCBpdCB3YXMgd29ydGggYSBzaG90
LiAgSSdsbCBrZWVwIHBva2luZyBhdCBpdC4NCj4NCj4+DQo+PiBJIHJlYnVpbGQgdGhlIGxhdGVz
dCBuZXQtbmV4dCBhbmQgSSdtIG5vdCBzZWVpbmcgdGhlIHRyYWNlIGFueSBtb3JlLi4uDQo+PiBJ
J20gb25seSBzZWVpbmcgdGhpcyAod2l0aCBvciB3aXRob3V0IHlvdXIgcGF0Y2gpOg0KPj4NCj4+
IFsgICAyMy40NjU4NzddIE5GUzogc2V0X3BuZnNfbGF5b3V0ZHJpdmVyOiBjbF9leGNoYW5nZV9m
bGFncyAweDANCj4+IFsgICAyMy40NzM3ODRdIE5GUzogc2V0X3BuZnNfbGF5b3V0ZHJpdmVyOiBj
bF9leGNoYW5nZV9mbGFncyAweDANCj4+IFsgICAyMy41ODg4OTBdIE5GUzogc2V0X3BuZnNfbGF5
b3V0ZHJpdmVyOiBjbF9leGNoYW5nZV9mbGFncyAweDANCj4+IFsgICAyMy41OTY3NDZdIE5GUzog
c2V0X3BuZnNfbGF5b3V0ZHJpdmVyOiBjbF9leGNoYW5nZV9mbGFncyAweDANCj4+IFsgICAyMy43
ODE1NzRdIE5GUzogc2V0X3BuZnNfbGF5b3V0ZHJpdmVyOiBjbF9leGNoYW5nZV9mbGFncyAweDAN
Cj4+IFsgICAyMy43ODk1OTldIE5GUzogc2V0X3BuZnNfbGF5b3V0ZHJpdmVyOiBjbF9leGNoYW5n
ZV9mbGFncyAweDANCj4NCj5JbnRlcmVzdGluZywgSSBnZXQgdGhhdCB0b28gd2hlbiBJIHRyeSB0
byB1c2UgTkZTIHY0LjEuICBJdCdzIHdlaXJkIHRoYXQgdGhlIGNyYXNoIHdvdWxkDQo+c3RvcCBo
YXBwZW5pbmcgbGlrZSB0aGF0LCBzbyBtYXliZSBzb21ldGhpbmcgaXMgcmFjeSBpbiB0aGlzIGFy
ZWEuDQo+DQo+VGhhbmtzIGZvciB0ZXN0aW5nLCBZb3RhbSBhbmQgSmFrdWIhDQo+QW5uYQ0KDQpJ
IGp1c3QgZm91bmQgb3V0IHRoYXQgaXQgaGFwcGVucyBvbiBhbnkgb2YgbXkgbWFjaGluZXMsIG9u
Y2UgSSBwdXQgdHdvIG5mcyBlbnRyaWVzIGluDQpteSBmc3RhYi4gSWYgSSBwdXQgb25seSBvbmUs
IEkgZG9uJ3Qgc2VlIHRoZSBwcm9ibGVtLiANCg0KSSBob3BlIGl0IG1pZ2h0IGJlIGhlbHBmdWwg
OikNCg0KPg0KPj4NCj4+IEhUSA0KPj4NCg0K
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Jakub Kicinski Oct. 31, 2016, 3:25 p.m. UTC | #4
On Thu, 27 Oct 2016 06:50:22 +0000, Yotam Gigi wrote:
> >-----Original Message-----
> >From: Anna Schumaker [mailto:Anna.Schumaker@Netapp.com]
> >Sent: Wednesday, October 26, 2016 9:17 PM
> >To: Jakub Kicinski <kubakici@wp.pl>
> >Cc: Yotam Gigi <yotamg@mellanox.com>; Andy Adamson <andros@netapp.com>;
> >linux-nfs@vger.kernel.org; netdev@vger.kernel.org; Trond Myklebust
> ><Trond.Myklebust@netapp.com>; Yotam Gigi <yotam.gi@gmail.com>; mlxsw
> ><mlxsw@mellanox.com>
> >Subject: Re: nfs NULL-dereferencing in net-next
> >
> >On 10/26/2016 02:08 PM, Jakub Kicinski wrote:  
> >> On Wed, 26 Oct 2016 16:15:24 +0000, Yotam Gigi wrote:  
> >>>> -----Original Message-----
> >>>> From: Anna Schumaker [mailto:Anna.Schumaker@Netapp.com]
> >>>> Sent: Wednesday, October 26, 2016 5:40 PM
> >>>> To: Yotam Gigi <yotamg@mellanox.com>; Jakub Kicinski <kubakici@wp.pl>;  
> >Andy  
> >>>> Adamson <andros@netapp.com>; Anna Schumaker
> >>>> <Anna.Schumaker@Netapp.com>; linux-nfs@vger.kernel.org
> >>>> Cc: netdev@vger.kernel.org; Trond Myklebust  
> ><Trond.Myklebust@netapp.com>;  
> >>>> Yotam Gigi <yotam.gi@gmail.com>; mlxsw <mlxsw@mellanox.com>
> >>>> Subject: Re: nfs NULL-dereferencing in net-next
> >>>>
> >>>> On 10/25/2016 01:19 PM, Yotam Gigi wrote:  
> >>>>>  
> >>>>>> -----Original Message-----
> >>>>>> From: netdev-owner@vger.kernel.org [mailto:netdev-  
> >owner@vger.kernel.org]  
> >>>> On  
> >>>>>> Behalf Of Jakub Kicinski
> >>>>>> Sent: Monday, October 17, 2016 10:20 PM
> >>>>>> To: Andy Adamson <andros@netapp.com>; Anna Schumaker
> >>>>>> <Anna.Schumaker@Netapp.com>; linux-nfs@vger.kernel.org
> >>>>>> Cc: netdev@vger.kernel.org; Trond Myklebust  
> >>>> <Trond.Myklebust@netapp.com>  
> >>>>>> Subject: nfs NULL-dereferencing in net-next
> >>>>>>
> >>>>>> Hi!
> >>>>>>
> >>>>>> I'm hitting this reliably on net-next, HEAD at 3f3177bb680f
> >>>>>> ("fsl/fman: fix error return code in mac_probe()").  
> >>>>>
> >>>>>
> >>>>> I see the same thing. It happens constantly on some of my machines, making  
> >>>> them  
> >>>>> completely unusable.
> >>>>>
> >>>>> I bisected it and got to the commit:
> >>>>>
> >>>>> commit 04ea1b3e6d8ed4978bb608c1748530af3de8c274
> >>>>> Author: Andy Adamson <andros@netapp.com>
> >>>>> Date:   Fri Sep 9 09:22:27 2016 -0400
> >>>>>
> >>>>>     NFS add xprt switch addrs test to match client
> >>>>>
> >>>>>     Signed-off-by: Andy Adamson <andros@netapp.com>
> >>>>>     Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>  
> >>>>
> >>>> Thanks for reporting on this everyone!  Does this patch help?  
> >>>
> >>> Actually, I still see the same bug with the same trace.  
> >
> >Well, it was worth a shot.  I'll keep poking at it.
> >  
> >>
> >> I rebuild the latest net-next and I'm not seeing the trace any more...
> >> I'm only seeing this (with or without your patch):
> >>
> >> [   23.465877] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0
> >> [   23.473784] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0
> >> [   23.588890] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0
> >> [   23.596746] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0
> >> [   23.781574] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0
> >> [   23.789599] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0  
> >
> >Interesting, I get that too when I try to use NFS v4.1.  It's weird that the crash would
> >stop happening like that, so maybe something is racy in this area.
> >
> >Thanks for testing, Yotam and Jakub!
> >Anna  
> 
> I just found out that it happens on any of my machines, once I put two nfs entries in
> my fstab. If I put only one, I don't see the problem. 
> 
> I hope it might be helpful :)

Hi Anna,

any updates on this one?  The crash came back half an hour after I
reported that it was gone...

Over the weekend David Miller rebased net-next on top of 4.9.0-rc3 and
the bug is still there :(  FWIW I also have multiple nfs mounts on my
setup, 2 in fstab and one in a startup script.  Following Yotam I
dropped one of the fstab entries and things seem to be working (even
though I still have multiple mounts, the other one just comes a bit
later).
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Schumaker, Anna Nov. 3, 2016, 1:06 p.m. UTC | #5
On 10/31/2016 11:25 AM, Jakub Kicinski wrote:
> On Thu, 27 Oct 2016 06:50:22 +0000, Yotam Gigi wrote:
>>> -----Original Message-----
>>> From: Anna Schumaker [mailto:Anna.Schumaker@Netapp.com]
>>> Sent: Wednesday, October 26, 2016 9:17 PM
>>> To: Jakub Kicinski <kubakici@wp.pl>
>>> Cc: Yotam Gigi <yotamg@mellanox.com>; Andy Adamson <andros@netapp.com>;
>>> linux-nfs@vger.kernel.org; netdev@vger.kernel.org; Trond Myklebust
>>> <Trond.Myklebust@netapp.com>; Yotam Gigi <yotam.gi@gmail.com>; mlxsw
>>> <mlxsw@mellanox.com>
>>> Subject: Re: nfs NULL-dereferencing in net-next
>>>
>>> On 10/26/2016 02:08 PM, Jakub Kicinski wrote:  
>>>> On Wed, 26 Oct 2016 16:15:24 +0000, Yotam Gigi wrote:  
>>>>>> -----Original Message-----
>>>>>> From: Anna Schumaker [mailto:Anna.Schumaker@Netapp.com]
>>>>>> Sent: Wednesday, October 26, 2016 5:40 PM
>>>>>> To: Yotam Gigi <yotamg@mellanox.com>; Jakub Kicinski <kubakici@wp.pl>;  
>>> Andy  
>>>>>> Adamson <andros@netapp.com>; Anna Schumaker
>>>>>> <Anna.Schumaker@Netapp.com>; linux-nfs@vger.kernel.org
>>>>>> Cc: netdev@vger.kernel.org; Trond Myklebust  
>>> <Trond.Myklebust@netapp.com>;  
>>>>>> Yotam Gigi <yotam.gi@gmail.com>; mlxsw <mlxsw@mellanox.com>
>>>>>> Subject: Re: nfs NULL-dereferencing in net-next
>>>>>>
>>>>>> On 10/25/2016 01:19 PM, Yotam Gigi wrote:  
>>>>>>>  
>>>>>>>> -----Original Message-----
>>>>>>>> From: netdev-owner@vger.kernel.org [mailto:netdev-  
>>> owner@vger.kernel.org]  
>>>>>> On  
>>>>>>>> Behalf Of Jakub Kicinski
>>>>>>>> Sent: Monday, October 17, 2016 10:20 PM
>>>>>>>> To: Andy Adamson <andros@netapp.com>; Anna Schumaker
>>>>>>>> <Anna.Schumaker@Netapp.com>; linux-nfs@vger.kernel.org
>>>>>>>> Cc: netdev@vger.kernel.org; Trond Myklebust  
>>>>>> <Trond.Myklebust@netapp.com>  
>>>>>>>> Subject: nfs NULL-dereferencing in net-next
>>>>>>>>
>>>>>>>> Hi!
>>>>>>>>
>>>>>>>> I'm hitting this reliably on net-next, HEAD at 3f3177bb680f
>>>>>>>> ("fsl/fman: fix error return code in mac_probe()").  
>>>>>>>
>>>>>>>
>>>>>>> I see the same thing. It happens constantly on some of my machines, making  
>>>>>> them  
>>>>>>> completely unusable.
>>>>>>>
>>>>>>> I bisected it and got to the commit:
>>>>>>>
>>>>>>> commit 04ea1b3e6d8ed4978bb608c1748530af3de8c274
>>>>>>> Author: Andy Adamson <andros@netapp.com>
>>>>>>> Date:   Fri Sep 9 09:22:27 2016 -0400
>>>>>>>
>>>>>>>     NFS add xprt switch addrs test to match client
>>>>>>>
>>>>>>>     Signed-off-by: Andy Adamson <andros@netapp.com>
>>>>>>>     Signed-off-by: Anna Schumaker <Anna.Schumaker@Netapp.com>  
>>>>>>
>>>>>> Thanks for reporting on this everyone!  Does this patch help?  
>>>>>
>>>>> Actually, I still see the same bug with the same trace.  
>>>
>>> Well, it was worth a shot.  I'll keep poking at it.
>>>  
>>>>
>>>> I rebuild the latest net-next and I'm not seeing the trace any more...
>>>> I'm only seeing this (with or without your patch):
>>>>
>>>> [   23.465877] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0
>>>> [   23.473784] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0
>>>> [   23.588890] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0
>>>> [   23.596746] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0
>>>> [   23.781574] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0
>>>> [   23.789599] NFS: set_pnfs_layoutdriver: cl_exchange_flags 0x0  
>>>
>>> Interesting, I get that too when I try to use NFS v4.1.  It's weird that the crash would
>>> stop happening like that, so maybe something is racy in this area.
>>>
>>> Thanks for testing, Yotam and Jakub!
>>> Anna  
>>
>> I just found out that it happens on any of my machines, once I put two nfs entries in
>> my fstab. If I put only one, I don't see the problem. 
>>
>> I hope it might be helpful :)
> 
> Hi Anna,
> 
> any updates on this one?  The crash came back half an hour after I
> reported that it was gone...

Aww, I was hoping that patch would work.  It still seemed to fix some issues for me when mounting multiple servers, so I'm planning to keep it.  Unfortunately I'm out of town this week, so I haven't had much of a chance to keep poking at this issue.  I should be able to get back to it next week!

Thanks for the update!
Anna

> 
> Over the weekend David Miller rebased net-next on top of 4.9.0-rc3 and
> the bug is still there :(  FWIW I also have multiple nfs mounts on my
> setup, 2 in fstab and one in a startup script.  Following Yotam I
> dropped one of the fstab entries and things seem to be working (even
> though I still have multiple mounts, the other one just comes a bit
> later).
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Olaf Hering Nov. 10, 2016, 2:47 p.m. UTC | #6
On Thu, Nov 03, Anna Schumaker wrote:

> Aww, I was hoping that patch would work.  It still seemed to fix some
> issues for me when mounting multiple servers, so I'm planning to keep
> it.  Unfortunately I'm out of town this week, so I haven't had much of
> a chance to keep poking at this issue.  I should be able to get back
> to it next week!

Is this supposed to be fixed already? I get an oops in
rpc_clnt_xprt_switch_has_addr+0xc/0x40 with 4.9.0-rc4.

Olaf
Schumaker, Anna Nov. 10, 2016, 3:02 p.m. UTC | #7
On 11/10/2016 09:47 AM, Olaf Hering wrote:
> On Thu, Nov 03, Anna Schumaker wrote:
> 
>> Aww, I was hoping that patch would work.  It still seemed to fix some
>> issues for me when mounting multiple servers, so I'm planning to keep
>> it.  Unfortunately I'm out of town this week, so I haven't had much of
>> a chance to keep poking at this issue.  I should be able to get back
>> to it next week!
> 
> Is this supposed to be fixed already? I get an oops in
> rpc_clnt_xprt_switch_has_addr+0xc/0x40 with 4.9.0-rc4.

Not yet, sorry.  I'm waiting on one bugfix patch before sending my pull request.

Thanks for waiting patiently,
Anna

> 
> Olaf
> 

--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Patch
diff mbox

diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
index 34dd7b2..62a4827 100644
--- a/net/sunrpc/clnt.c
+++ b/net/sunrpc/clnt.c
@@ -2753,14 +2753,18 @@  EXPORT_SYMBOL_GPL(rpc_cap_max_reconnect_timeout);

 void rpc_clnt_xprt_switch_put(struct rpc_clnt *clnt)
 {
+       rcu_read_lock();
        xprt_switch_put(rcu_dereference(clnt->cl_xpi.xpi_xpswitch));
+       rcu_read_unlock();
 }
 EXPORT_SYMBOL_GPL(rpc_clnt_xprt_switch_put);

 void rpc_clnt_xprt_switch_add_xprt(struct rpc_clnt *clnt, struct rpc_xprt *xprt)
 {
+       rcu_read_lock();
        rpc_xprt_switch_add_xprt(rcu_dereference(clnt->cl_xpi.xpi_xpswitch),
                                 xprt);
+       rcu_read_unlock();
 }
 EXPORT_SYMBOL_GPL(rpc_clnt_xprt_switch_add_xprt);

@@ -2770,9 +2774,8 @@  bool rpc_clnt_xprt_switch_has_addr(struct rpc_clnt *clnt,
        struct rpc_xprt_switch *xps;
        bool ret;

-       xps = rcu_dereference(clnt->cl_xpi.xpi_xpswitch);
-
        rcu_read_lock();
+       xps = rcu_dereference(clnt->cl_xpi.xpi_xpswitch);
        ret = rpc_xprt_switch_has_addr(xps, sap);
        rcu_read_unlock();
        return ret;