Message ID | 20220523045707.1704761-1-liuyacan@corp.netease.com (mailing list archive) |
---|---|
State | Accepted |
Commit | 75c1edf23b95a9c66923d9269d8e86e4dbde151f |
Delegated to: | Netdev Maintainers |
Headers | show |
Series | [v2,net] net/smc: postpone sk_refcnt increment in connect() | expand |
Hello: This patch was applied to netdev/net.git (master) by David S. Miller <davem@davemloft.net>: On Mon, 23 May 2022 12:57:07 +0800 you wrote: > From: liuyacan <liuyacan@corp.netease.com> > > Same trigger condition as commit 86434744. When setsockopt runs > in parallel to a connect(), and switch the socket into fallback > mode. Then the sk_refcnt is incremented in smc_connect(), but > its state stay in SMC_INIT (NOT SMC_ACTIVE). This cause the > corresponding sk_refcnt decrement in __smc_release() will not be > performed. > > [...] Here is the summary with links: - [v2,net] net/smc: postpone sk_refcnt increment in connect() https://git.kernel.org/netdev/net/c/75c1edf23b95 You are awesome, thank you!
On 23/05/2022 06:57, liuyacan@corp.netease.com wrote: > From: liuyacan <liuyacan@corp.netease.com> > > Same trigger condition as commit 86434744. When setsockopt runs > in parallel to a connect(), and switch the socket into fallback > mode. Then the sk_refcnt is incremented in smc_connect(), but > its state stay in SMC_INIT (NOT SMC_ACTIVE). This cause the > corresponding sk_refcnt decrement in __smc_release() will not be > performed. > > Fixes: 86434744fedf ("net/smc: add fallback check to connect()") > Signed-off-by: liuyacan <liuyacan@corp.netease.com> > --- > net/smc/af_smc.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/net/smc/af_smc.c b/net/smc/af_smc.c > index fce16b9d6..45a24d242 100644 > --- a/net/smc/af_smc.c > +++ b/net/smc/af_smc.c > @@ -1564,9 +1564,9 @@ static int smc_connect(struct socket *sock, struct sockaddr *addr, > if (rc && rc != -EINPROGRESS) > goto out; > > - sock_hold(&smc->sk); /* sock put in passive closing */ > if (smc->use_fallback) > goto out; > + sock_hold(&smc->sk); /* sock put in passive closing */ > if (flags & O_NONBLOCK) { > if (queue_work(smc_hs_wq, &smc->connect_work)) > smc->connect_nonblock = 1; This is a rather unusual problem that can come up when fallback=true BEFORE smc_connect() is called. But nevertheless, it is a problem. Right now I am not sure if it is okay when we NOT hold a ref to smc->sk during all fallback processing. This change also conflicts with a patch that is already on net-next (3aba1030). With the new patch on net-next it would also be possible to detect in __smc_release() that the socket is in state sk->sk_state == SMC_INIT but the sock->state is SS_CONNECTING or SS_CONNECTED and call sock_put() in this case. What do you think?
> This is a rather unusual problem that can come up when fallback=true BEFORE smc_connect() > is called. But nevertheless, it is a problem. > > Right now I am not sure if it is okay when we NOT hold a ref to smc->sk during all fallback > processing. This change also conflicts with a patch that is already on net-next (3aba1030). Do you mean put the ref to smc->sk during all fallback processing unconditionally and remove the fallback branch sock_put() in __smc_release()? > With the new patch on net-next it would also be possible to detect in __smc_release() that > the socket is in state sk->sk_state == SMC_INIT but the sock->state is SS_CONNECTING or > SS_CONNECTED and call sock_put() in this case. > What do you think? Oh, I didn't notice this patch on net-next. Emm, I think I need to do some testing with this patch. Thank you.
On 23/05/2022 16:19, liuyacan@corp.netease.com wrote: >> This is a rather unusual problem that can come up when fallback=true BEFORE smc_connect() >> is called. But nevertheless, it is a problem. >> >> Right now I am not sure if it is okay when we NOT hold a ref to smc->sk during all fallback >> processing. This change also conflicts with a patch that is already on net-next (3aba1030). > > Do you mean put the ref to smc->sk during all fallback processing unconditionally and remove > the fallback branch sock_put() in __smc_release()? What I had in mind was to eventually call sock_put() in __smc_release() even if sk->sk_state == SMC_INIT (currently the extra check in the if() for sk->sk_state != SMC_INIT prevents the sock_put()), but only when it is sure that we actually reached the sock_hold() in smc_connect() before. But maybe we find out that the sock_hold() is not needed for fallback sockets, I don't know...
> >> This is a rather unusual problem that can come up when fallback=true BEFORE smc_connect() > >> is called. But nevertheless, it is a problem. > >> > >> Right now I am not sure if it is okay when we NOT hold a ref to smc->sk during all fallback > >> processing. This change also conflicts with a patch that is already on net-next (3aba1030). > > > > Do you mean put the ref to smc->sk during all fallback processing unconditionally and remove > > the fallback branch sock_put() in __smc_release()? > > What I had in mind was to eventually call sock_put() in __smc_release() even if sk->sk_state == SMC_INIT > (currently the extra check in the if() for sk->sk_state != SMC_INIT prevents the sock_put()), but only > when it is sure that we actually reached the sock_hold() in smc_connect() before. > > But maybe we find out that the sock_hold() is not needed for fallback sockets, I don't know... I do think the sock_hold()/sock_put() for smc->sk is a bit complicated, Emm, I'm not sure if it can be simplified.. In fact, I'm sure there must be another ref count issue in my environment,but I haven't caught it yet.
On Mon, May 23, 2022 at 11:21:19PM +0800, liuyacan@corp.netease.com wrote: > > >> This is a rather unusual problem that can come up when fallback=true BEFORE smc_connect() > > >> is called. But nevertheless, it is a problem. > > >> > > >> Right now I am not sure if it is okay when we NOT hold a ref to smc->sk during all fallback > > >> processing. This change also conflicts with a patch that is already on net-next (3aba1030). > > > > > > Do you mean put the ref to smc->sk during all fallback processing unconditionally and remove > > > the fallback branch sock_put() in __smc_release()? > > > > What I had in mind was to eventually call sock_put() in __smc_release() even if sk->sk_state == SMC_INIT > > (currently the extra check in the if() for sk->sk_state != SMC_INIT prevents the sock_put()), but only > > when it is sure that we actually reached the sock_hold() in smc_connect() before. > > > > But maybe we find out that the sock_hold() is not needed for fallback sockets, I don't know... > > I do think the sock_hold()/sock_put() for smc->sk is a bit complicated, Emm, I'm not sure if it > can be simplified.. > > In fact, I'm sure there must be another ref count issue in my environment,but I haven't caught it yet. I am wondering the issue of this ref count. If it is convenient, would you like to provide some more details? syzkaller has reported some issues about ref count, but syzkaller and others' bot don't have RDMA devices, they cannot cover most of the code routines in SMC. We are working on it to provide SMC fuzz test with RDMA environment. So it's very nice to have real world issues. Thanks, Tony Lu
On 23/05/2022 17:21, liuyacan@corp.netease.com wrote: >>>> This is a rather unusual problem that can come up when fallback=true BEFORE smc_connect() >>>> is called. But nevertheless, it is a problem. >>>> >>>> Right now I am not sure if it is okay when we NOT hold a ref to smc->sk during all fallback >>>> processing. This change also conflicts with a patch that is already on net-next (3aba1030). >>> >>> Do you mean put the ref to smc->sk during all fallback processing unconditionally and remove >>> the fallback branch sock_put() in __smc_release()? >> >> What I had in mind was to eventually call sock_put() in __smc_release() even if sk->sk_state == SMC_INIT >> (currently the extra check in the if() for sk->sk_state != SMC_INIT prevents the sock_put()), but only >> when it is sure that we actually reached the sock_hold() in smc_connect() before. >> >> But maybe we find out that the sock_hold() is not needed for fallback sockets, I don't know... > > I do think the sock_hold()/sock_put() for smc->sk is a bit complicated, Emm, I'm not sure if it > can be simplified.. > > In fact, I'm sure there must be another ref count issue in my environment,but I haven't caught it yet. > Can you check my latest mail from a minute ago in thread "Re: [PATCH net-next v2] net/smc: align the connect behaviour with TCP" I think this answer also affects our discussion.
> > > >> This is a rather unusual problem that can come up when fallback=true BEFORE smc_connect() > > > >> is called. But nevertheless, it is a problem. > > > >> > > > >> Right now I am not sure if it is okay when we NOT hold a ref to smc->sk during all fallback > > > >> processing. This change also conflicts with a patch that is already on net-next (3aba1030). > > > > > > > > Do you mean put the ref to smc->sk during all fallback processing unconditionally and remove > > > > the fallback branch sock_put() in __smc_release()? > > > > > > What I had in mind was to eventually call sock_put() in __smc_release() even if sk->sk_state == SMC_INIT > > > (currently the extra check in the if() for sk->sk_state != SMC_INIT prevents the sock_put()), but only > > > when it is sure that we actually reached the sock_hold() in smc_connect() before. > > > > > > But maybe we find out that the sock_hold() is not needed for fallback sockets, I don't know... > > > > I do think the sock_hold()/sock_put() for smc->sk is a bit complicated, Emm, I'm not sure if it > > can be simplified.. > > > > In fact, I'm sure there must be another ref count issue in my environment,but I haven't caught it yet. > > I am wondering the issue of this ref count. If it is convenient, would > you like to provide some more details? > > syzkaller has reported some issues about ref count, but syzkaller and > others' bot don't have RDMA devices, they cannot cover most of the code > routines in SMC. We are working on it to provide SMC fuzz test with RDMA > environment. So it's very nice to have real world issues. > > Thanks, > Tony Lu I have encountered two types of problems. However, I cannot reproduce it stably. case 1. After closing the app (>> TIME_WAIT), 'lsmod' shows that the smc module ref count is still greater than 0. case 2 [rare]. 'lsmod' shows smc module ref count is less than 0. Some clues of case 2 are as follows: kernel: [67166.688386] ------------[ cut here ]------------ kernel: [67166.693658] cache_from_obj: Wrong slab cache. SMC but object is from SMC kernel: [67166.701136] WARNING: CPU: 47 PID: 176961 at mm/slab.h:469 kmem_cache_free+0x329/0x410 ...... kernel: [67166.846819] CPU: 47 PID: 176961 Comm: redis-server Kdump: loaded Tainted: G R B OE 5.10.0-0.bpo.9-amd64 #1 Debian 5.10.70-1~bpo10+1 kernel: [67166.860915] Hardware name: Inspur SA5280M6/SA5280M6, BIOS 06.00.01 10/09/2021 kernel: [67166.868747] RIP: 0010:kmem_cache_free+0x329/0x410 kernel: [67166.874168] Code: ff 0f 0b 48 8d b8 f0 9d 02 00 e9 e4 fe ff ff 48 8b 57 60 49 8b 4f 60 48 c7 c6 30 86 63 a4 48 c7 c7 f8 e6 8f a4 e8 89 63 5c 00 <0f> 0b 48 89 de 4c 89 ff e8 1a ad ff ff 48 8b 0d 63 34 ef 00 e9 49 kernel: [67166.894360] RSP: 0018:ffffbd450f527e18 EFLAGS: 00010286 kernel: [67166.900306] RAX: 0000000000000000 RBX: ffffa00fa4548d00 RCX: 0000000000000000 kernel: [67166.908169] RDX: ffffa04c7f7e8760 RSI: ffffa04c7f7d8a00 RDI: ffffa04c7f7d8a00 kernel: [67166.916027] RBP: ffffa01024548d00 R08: 0000000000000000 R09: c0000000ffffbfff kernel: [67166.923860] R10: 0000000000000001 R11: ffffbd450f527c20 R12: 0000000000000000 kernel: [67166.931713] R13: 0000000000000000 R14: ffffa00fa4548f28 R15: ffffa02d3366bf00 kernel: [67166.939564] FS: 00007fe131c80f40(0000) GS:ffffa04c7f7c0000(0000) knlGS:0000000000000000 kernel: [67166.948361] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 kernel: [67166.954817] CR2: 00007fe12f477000 CR3: 00000004874be003 CR4: 0000000000770ee0 kernel: [67166.962662] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 kernel: [67166.970498] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 kernel: [67166.978306] PKRU: 55555554 kernel: [67166.981695] Call Trace: kernel: [67166.985017] __sk_destruct+0x12c/0x1e0 kernel: [67166.989449] smc_release+0x19a/0x230 [smc] kernel: [67166.994325] __sock_release+0x3d/0xa0 kernel: [67166.998656] sock_close+0x11/0x20 kernel: [67167.002637] __fput+0x93/0x240 kernel: [67167.006347] task_work_run+0x76/0xb0 kernel: [67167.010569] exit_to_user_mode_prepare+0x129/0x130 kernel: [67167.016000] syscall_exit_to_user_mode+0x28/0x140 kernel: [67167.021339] entry_SYSCALL_64_after_hwframe+0x44/0xa9
diff --git a/net/smc/af_smc.c b/net/smc/af_smc.c index fce16b9d6..45a24d242 100644 --- a/net/smc/af_smc.c +++ b/net/smc/af_smc.c @@ -1564,9 +1564,9 @@ static int smc_connect(struct socket *sock, struct sockaddr *addr, if (rc && rc != -EINPROGRESS) goto out; - sock_hold(&smc->sk); /* sock put in passive closing */ if (smc->use_fallback) goto out; + sock_hold(&smc->sk); /* sock put in passive closing */ if (flags & O_NONBLOCK) { if (queue_work(smc_hs_wq, &smc->connect_work)) smc->connect_nonblock = 1;