| Message ID | 20241024015543.568476-1-liujian56@huawei.com (mailing list archive) |
|---|---|
| State | New |
| Delegated to: | Netdev Maintainers |
| Headers | show |
| Series | [net] sunrpc: fix one UAF issue caused by sunrpc kernel tcp socket | expand |
On Thu, 2024-10-24 at 09:55 +0800, Liu Jian wrote: > BUG: KASAN: slab-use-after-free in > tcp_write_timer_handler+0x156/0x3e0 > Read of size 1 at addr ffff888111f322cd by task swapper/0/0 > > CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.12.0-rc4-dirty #7 > Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 > Call Trace: > <IRQ> > dump_stack_lvl+0x68/0xa0 > print_address_description.constprop.0+0x2c/0x3d0 > print_report+0xb4/0x270 > kasan_report+0xbd/0xf0 > tcp_write_timer_handler+0x156/0x3e0 > tcp_write_timer+0x66/0x170 > call_timer_fn+0xfb/0x1d0 > __run_timers+0x3f8/0x480 > run_timer_softirq+0x9b/0x100 > handle_softirqs+0x153/0x390 > __irq_exit_rcu+0x103/0x120 > irq_exit_rcu+0xe/0x20 > sysvec_apic_timer_interrupt+0x76/0x90 > </IRQ> > <TASK> > asm_sysvec_apic_timer_interrupt+0x1a/0x20 > RIP: 0010:default_idle+0xf/0x20 > Code: 4c 01 c7 4c 29 c2 e9 72 ff ff ff 90 90 90 90 90 90 90 90 90 90 > 90 90 > 90 90 90 90 f3 0f 1e fa 66 90 0f 00 2d 33 f8 25 00 fb f4 <fa> c3 cc > cc cc > cc 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 > RSP: 0018:ffffffffa2007e28 EFLAGS: 00000242 > RAX: 00000000000f3b31 RBX: 1ffffffff4400fc7 RCX: ffffffffa09c3196 > RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff9f00590f > RBP: 0000000000000000 R08: 0000000000000001 R09: ffffed102360835d > R10: ffff88811b041aeb R11: 0000000000000001 R12: 0000000000000000 > R13: ffffffffa202d7c0 R14: 0000000000000000 R15: 00000000000147d0 > default_idle_call+0x6b/0xa0 > cpuidle_idle_call+0x1af/0x1f0 > do_idle+0xbc/0x130 > cpu_startup_entry+0x33/0x40 > rest_init+0x11f/0x210 > start_kernel+0x39a/0x420 > x86_64_start_reservations+0x18/0x30 > x86_64_start_kernel+0x97/0xa0 > common_startup_64+0x13e/0x141 > </TASK> > > Allocated by task 595: > kasan_save_stack+0x24/0x50 > kasan_save_track+0x14/0x30 > __kasan_slab_alloc+0x87/0x90 > kmem_cache_alloc_noprof+0x12b/0x3f0 > copy_net_ns+0x94/0x380 > create_new_namespaces+0x24c/0x500 > unshare_nsproxy_namespaces+0x75/0xf0 > ksys_unshare+0x24e/0x4f0 > __x64_sys_unshare+0x1f/0x30 > do_syscall_64+0x70/0x180 > entry_SYSCALL_64_after_hwframe+0x76/0x7e > > Freed by task 100: > kasan_save_stack+0x24/0x50 > kasan_save_track+0x14/0x30 > kasan_save_free_info+0x3b/0x60 > __kasan_slab_free+0x54/0x70 > kmem_cache_free+0x156/0x5d0 > cleanup_net+0x5d3/0x670 > process_one_work+0x776/0xa90 > worker_thread+0x2e2/0x560 > kthread+0x1a8/0x1f0 > ret_from_fork+0x34/0x60 > ret_from_fork_asm+0x1a/0x30 > > Reproduction script: > > mkdir -p /mnt/nfsshare > mkdir -p /mnt/nfs/netns_1 > mkfs.ext4 /dev/sdb > mount /dev/sdb /mnt/nfsshare > systemctl restart nfs-server > chmod 777 /mnt/nfsshare > exportfs -i -o rw,no_root_squash *:/mnt/nfsshare > > ip netns add netns_1 > ip link add name veth_1_peer type veth peer veth_1 > ifconfig veth_1_peer 11.11.0.254 up > ip link set veth_1 netns netns_1 > ip netns exec netns_1 ifconfig veth_1 11.11.0.1 > > ip netns exec netns_1 /root/iptables -A OUTPUT -d 11.11.0.254 -p tcp > \ > --tcp-flags FIN FIN -j DROP > > (note: In my environment, a DESTROY_CLIENTID operation is always sent > immediately, breaking the nfs tcp connection.) > ip netns exec netns_1 timeout -s 9 300 mount -t nfs -o > proto=tcp,vers=4.1 \ > 11.11.0.254:/mnt/nfsshare /mnt/nfs/netns_1 > > ip netns del netns_1 > > The reason here is that the tcp socket in netns_1 (nfs side) has been > shutdown and closed (done in xs_destroy), but the FIN message (with > ack) > is discarded, and the nfsd side keeps sending retransmission > messages. > As a result, when the tcp sock in netns_1 processes the received > message, > it sends the message (FIN message) in the sending queue, and the tcp > timer > is re-established. When the network namespace is deleted, the net > structure > accessed by tcp's timer handler function causes problems. > > The modification here aborts the TCP connection directly in > xs_destroy(). > > Fixes: 26abe14379f8 ("net: Modify sk_alloc to not reference count the > netns of kernel sockets.") > Signed-off-by: Liu Jian <liujian56@huawei.com> > --- > net/sunrpc/xprtsock.c | 3 +++ > 1 file changed, 3 insertions(+) > > diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c > index 0e1691316f42..91ee3484155a 100644 > --- a/net/sunrpc/xprtsock.c > +++ b/net/sunrpc/xprtsock.c > @@ -1287,6 +1287,9 @@ static void xs_reset_transport(struct sock_xprt > *transport) > release_sock(sk); > mutex_unlock(&transport->recv_mutex); > > + if (sk->sk_prot == &tcp_prot) > + tcp_abort(sk, ECONNABORTED); We've already called kernel_sock_shutdown(sock, SHUT_RDWR), and we're about to close the socket. Why on earth should we need a hack like the above in order to abort the connection at this point? This would appear to be a networking layer bug, and not an RPC level problem. > + > trace_rpc_socket_close(xprt, sock); > __fput_sync(filp); >
On Thu, 2024-10-24 at 02:20 +0000, Trond Myklebust wrote: > On Thu, 2024-10-24 at 09:55 +0800, Liu Jian wrote: > > BUG: KASAN: slab-use-after-free in > > tcp_write_timer_handler+0x156/0x3e0 > > Read of size 1 at addr ffff888111f322cd by task swapper/0/0 > > > > CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.12.0-rc4-dirty > > #7 > > Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0- > > 1 > > Call Trace: > > <IRQ> > > dump_stack_lvl+0x68/0xa0 > > print_address_description.constprop.0+0x2c/0x3d0 > > print_report+0xb4/0x270 > > kasan_report+0xbd/0xf0 > > tcp_write_timer_handler+0x156/0x3e0 > > tcp_write_timer+0x66/0x170 > > call_timer_fn+0xfb/0x1d0 > > __run_timers+0x3f8/0x480 > > run_timer_softirq+0x9b/0x100 > > handle_softirqs+0x153/0x390 > > __irq_exit_rcu+0x103/0x120 > > irq_exit_rcu+0xe/0x20 > > sysvec_apic_timer_interrupt+0x76/0x90 > > </IRQ> > > <TASK> > > asm_sysvec_apic_timer_interrupt+0x1a/0x20 > > RIP: 0010:default_idle+0xf/0x20 > > Code: 4c 01 c7 4c 29 c2 e9 72 ff ff ff 90 90 90 90 90 90 90 90 90 > > 90 > > 90 90 > > 90 90 90 90 f3 0f 1e fa 66 90 0f 00 2d 33 f8 25 00 fb f4 <fa> c3 > > cc > > cc cc > > cc 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 > > RSP: 0018:ffffffffa2007e28 EFLAGS: 00000242 > > RAX: 00000000000f3b31 RBX: 1ffffffff4400fc7 RCX: ffffffffa09c3196 > > RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff9f00590f > > RBP: 0000000000000000 R08: 0000000000000001 R09: ffffed102360835d > > R10: ffff88811b041aeb R11: 0000000000000001 R12: 0000000000000000 > > R13: ffffffffa202d7c0 R14: 0000000000000000 R15: 00000000000147d0 > > default_idle_call+0x6b/0xa0 > > cpuidle_idle_call+0x1af/0x1f0 > > do_idle+0xbc/0x130 > > cpu_startup_entry+0x33/0x40 > > rest_init+0x11f/0x210 > > start_kernel+0x39a/0x420 > > x86_64_start_reservations+0x18/0x30 > > x86_64_start_kernel+0x97/0xa0 > > common_startup_64+0x13e/0x141 > > </TASK> > > > > Allocated by task 595: > > kasan_save_stack+0x24/0x50 > > kasan_save_track+0x14/0x30 > > __kasan_slab_alloc+0x87/0x90 > > kmem_cache_alloc_noprof+0x12b/0x3f0 > > copy_net_ns+0x94/0x380 > > create_new_namespaces+0x24c/0x500 > > unshare_nsproxy_namespaces+0x75/0xf0 > > ksys_unshare+0x24e/0x4f0 > > __x64_sys_unshare+0x1f/0x30 > > do_syscall_64+0x70/0x180 > > entry_SYSCALL_64_after_hwframe+0x76/0x7e > > > > Freed by task 100: > > kasan_save_stack+0x24/0x50 > > kasan_save_track+0x14/0x30 > > kasan_save_free_info+0x3b/0x60 > > __kasan_slab_free+0x54/0x70 > > kmem_cache_free+0x156/0x5d0 > > cleanup_net+0x5d3/0x670 > > process_one_work+0x776/0xa90 > > worker_thread+0x2e2/0x560 > > kthread+0x1a8/0x1f0 > > ret_from_fork+0x34/0x60 > > ret_from_fork_asm+0x1a/0x30 > > > > Reproduction script: > > > > mkdir -p /mnt/nfsshare > > mkdir -p /mnt/nfs/netns_1 > > mkfs.ext4 /dev/sdb > > mount /dev/sdb /mnt/nfsshare > > systemctl restart nfs-server > > chmod 777 /mnt/nfsshare > > exportfs -i -o rw,no_root_squash *:/mnt/nfsshare > > > > ip netns add netns_1 > > ip link add name veth_1_peer type veth peer veth_1 > > ifconfig veth_1_peer 11.11.0.254 up > > ip link set veth_1 netns netns_1 > > ip netns exec netns_1 ifconfig veth_1 11.11.0.1 > > > > ip netns exec netns_1 /root/iptables -A OUTPUT -d 11.11.0.254 -p > > tcp > > \ > > --tcp-flags FIN FIN -j DROP > > > > (note: In my environment, a DESTROY_CLIENTID operation is always > > sent > > immediately, breaking the nfs tcp connection.) > > ip netns exec netns_1 timeout -s 9 300 mount -t nfs -o > > proto=tcp,vers=4.1 \ > > 11.11.0.254:/mnt/nfsshare /mnt/nfs/netns_1 > > > > ip netns del netns_1 > > > > The reason here is that the tcp socket in netns_1 (nfs side) has > > been > > shutdown and closed (done in xs_destroy), but the FIN message (with > > ack) > > is discarded, and the nfsd side keeps sending retransmission > > messages. > > As a result, when the tcp sock in netns_1 processes the received > > message, > > it sends the message (FIN message) in the sending queue, and the > > tcp > > timer > > is re-established. When the network namespace is deleted, the net > > structure > > accessed by tcp's timer handler function causes problems. > > > > The modification here aborts the TCP connection directly in > > xs_destroy(). > > > > Fixes: 26abe14379f8 ("net: Modify sk_alloc to not reference count > > the > > netns of kernel sockets.") > > Signed-off-by: Liu Jian <liujian56@huawei.com> > > --- > > net/sunrpc/xprtsock.c | 3 +++ > > 1 file changed, 3 insertions(+) > > > > diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c > > index 0e1691316f42..91ee3484155a 100644 > > --- a/net/sunrpc/xprtsock.c > > +++ b/net/sunrpc/xprtsock.c > > @@ -1287,6 +1287,9 @@ static void xs_reset_transport(struct > > sock_xprt > > *transport) > > release_sock(sk); > > mutex_unlock(&transport->recv_mutex); > > > > + if (sk->sk_prot == &tcp_prot) > > + tcp_abort(sk, ECONNABORTED); > > We've already called kernel_sock_shutdown(sock, SHUT_RDWR), and we're > about to close the socket. Why on earth should we need a hack like > the > above in order to abort the connection at this point? > > This would appear to be a networking layer bug, and not an RPC level > problem. > To put this differently: if a use after free can occur in the kernel when the RPC layer closes a TCP socket and then exits the network namespace, then can't that occur when a userland application does the same? If not, then what prevents it from happening? > > + > > trace_rpc_socket_close(xprt, sock); > > __fput_sync(filp); > >
在 2024/10/24 20:57, Trond Myklebust 写道: > On Thu, 2024-10-24 at 02:20 +0000, Trond Myklebust wrote: >> On Thu, 2024-10-24 at 09:55 +0800, Liu Jian wrote: >>> BUG: KASAN: slab-use-after-free in >>> tcp_write_timer_handler+0x156/0x3e0 >>> Read of size 1 at addr ffff888111f322cd by task swapper/0/0 >>> >>> CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.12.0-rc4-dirty >>> #7 >>> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0- >>> 1 >>> Call Trace: >>> <IRQ> >>> dump_stack_lvl+0x68/0xa0 >>> print_address_description.constprop.0+0x2c/0x3d0 >>> print_report+0xb4/0x270 >>> kasan_report+0xbd/0xf0 >>> tcp_write_timer_handler+0x156/0x3e0 >>> tcp_write_timer+0x66/0x170 >>> call_timer_fn+0xfb/0x1d0 >>> __run_timers+0x3f8/0x480 >>> run_timer_softirq+0x9b/0x100 >>> handle_softirqs+0x153/0x390 >>> __irq_exit_rcu+0x103/0x120 >>> irq_exit_rcu+0xe/0x20 >>> sysvec_apic_timer_interrupt+0x76/0x90 >>> </IRQ> >>> <TASK> >>> asm_sysvec_apic_timer_interrupt+0x1a/0x20 >>> RIP: 0010:default_idle+0xf/0x20 >>> Code: 4c 01 c7 4c 29 c2 e9 72 ff ff ff 90 90 90 90 90 90 90 90 90 >>> 90 >>> 90 90 >>> 90 90 90 90 f3 0f 1e fa 66 90 0f 00 2d 33 f8 25 00 fb f4 <fa> c3 >>> cc >>> cc cc >>> cc 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 >>> RSP: 0018:ffffffffa2007e28 EFLAGS: 00000242 >>> RAX: 00000000000f3b31 RBX: 1ffffffff4400fc7 RCX: ffffffffa09c3196 >>> RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff9f00590f >>> RBP: 0000000000000000 R08: 0000000000000001 R09: ffffed102360835d >>> R10: ffff88811b041aeb R11: 0000000000000001 R12: 0000000000000000 >>> R13: ffffffffa202d7c0 R14: 0000000000000000 R15: 00000000000147d0 >>> default_idle_call+0x6b/0xa0 >>> cpuidle_idle_call+0x1af/0x1f0 >>> do_idle+0xbc/0x130 >>> cpu_startup_entry+0x33/0x40 >>> rest_init+0x11f/0x210 >>> start_kernel+0x39a/0x420 >>> x86_64_start_reservations+0x18/0x30 >>> x86_64_start_kernel+0x97/0xa0 >>> common_startup_64+0x13e/0x141 >>> </TASK> >>> >>> Allocated by task 595: >>> kasan_save_stack+0x24/0x50 >>> kasan_save_track+0x14/0x30 >>> __kasan_slab_alloc+0x87/0x90 >>> kmem_cache_alloc_noprof+0x12b/0x3f0 >>> copy_net_ns+0x94/0x380 >>> create_new_namespaces+0x24c/0x500 >>> unshare_nsproxy_namespaces+0x75/0xf0 >>> ksys_unshare+0x24e/0x4f0 >>> __x64_sys_unshare+0x1f/0x30 >>> do_syscall_64+0x70/0x180 >>> entry_SYSCALL_64_after_hwframe+0x76/0x7e >>> >>> Freed by task 100: >>> kasan_save_stack+0x24/0x50 >>> kasan_save_track+0x14/0x30 >>> kasan_save_free_info+0x3b/0x60 >>> __kasan_slab_free+0x54/0x70 >>> kmem_cache_free+0x156/0x5d0 >>> cleanup_net+0x5d3/0x670 >>> process_one_work+0x776/0xa90 >>> worker_thread+0x2e2/0x560 >>> kthread+0x1a8/0x1f0 >>> ret_from_fork+0x34/0x60 >>> ret_from_fork_asm+0x1a/0x30 >>> >>> Reproduction script: >>> >>> mkdir -p /mnt/nfsshare >>> mkdir -p /mnt/nfs/netns_1 >>> mkfs.ext4 /dev/sdb >>> mount /dev/sdb /mnt/nfsshare >>> systemctl restart nfs-server >>> chmod 777 /mnt/nfsshare >>> exportfs -i -o rw,no_root_squash *:/mnt/nfsshare >>> >>> ip netns add netns_1 >>> ip link add name veth_1_peer type veth peer veth_1 >>> ifconfig veth_1_peer 11.11.0.254 up >>> ip link set veth_1 netns netns_1 >>> ip netns exec netns_1 ifconfig veth_1 11.11.0.1 >>> >>> ip netns exec netns_1 /root/iptables -A OUTPUT -d 11.11.0.254 -p >>> tcp >>> \ >>> --tcp-flags FIN FIN -j DROP >>> >>> (note: In my environment, a DESTROY_CLIENTID operation is always >>> sent >>> immediately, breaking the nfs tcp connection.) >>> ip netns exec netns_1 timeout -s 9 300 mount -t nfs -o >>> proto=tcp,vers=4.1 \ >>> 11.11.0.254:/mnt/nfsshare /mnt/nfs/netns_1 >>> >>> ip netns del netns_1 >>> >>> The reason here is that the tcp socket in netns_1 (nfs side) has >>> been >>> shutdown and closed (done in xs_destroy), but the FIN message (with >>> ack) >>> is discarded, and the nfsd side keeps sending retransmission >>> messages. >>> As a result, when the tcp sock in netns_1 processes the received >>> message, >>> it sends the message (FIN message) in the sending queue, and the >>> tcp >>> timer >>> is re-established. When the network namespace is deleted, the net >>> structure >>> accessed by tcp's timer handler function causes problems. >>> >>> The modification here aborts the TCP connection directly in >>> xs_destroy(). >>> >>> Fixes: 26abe14379f8 ("net: Modify sk_alloc to not reference count >>> the >>> netns of kernel sockets.") >>> Signed-off-by: Liu Jian <liujian56@huawei.com> >>> --- >>> net/sunrpc/xprtsock.c | 3 +++ >>> 1 file changed, 3 insertions(+) >>> >>> diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c >>> index 0e1691316f42..91ee3484155a 100644 >>> --- a/net/sunrpc/xprtsock.c >>> +++ b/net/sunrpc/xprtsock.c >>> @@ -1287,6 +1287,9 @@ static void xs_reset_transport(struct >>> sock_xprt >>> *transport) >>> release_sock(sk); >>> mutex_unlock(&transport->recv_mutex); >>> >>> + if (sk->sk_prot == &tcp_prot) >>> + tcp_abort(sk, ECONNABORTED); >> We've already called kernel_sock_shutdown(sock, SHUT_RDWR), and we're >> about to close the socket. Why on earth should we need a hack like >> the >> above in order to abort the connection at this point? >> >> This would appear to be a networking layer bug, and not an RPC level >> problem. >> > To put this differently: if a use after free can occur in the kernel > when the RPC layer closes a TCP socket and then exits the network > namespace, then can't that occur when a userland application does the > same? > > If not, then what prevents it from happening? The socket created by the userspace program obtains the reference counting of the namespace, but the kernel socket does not. There's some discussion here: https://lore.kernel.org/all/CANn89iJE5anTbyLJ0TdGAqGsE+GichY3YzQECjNUVMz=G3bcQg@mail.gmail.com/ >>> + >>> trace_rpc_socket_close(xprt, sock); >>> __fput_sync(filp); >>>
On Thu, 2024-10-24 at 21:40 +0800, liujian (CE) wrote: > > > 在 2024/10/24 20:57, Trond Myklebust 写道: > > On Thu, 2024-10-24 at 02:20 +0000, Trond Myklebust wrote: > > > On Thu, 2024-10-24 at 09:55 +0800, Liu Jian wrote: > > > > BUG: KASAN: slab-use-after-free in > > > > tcp_write_timer_handler+0x156/0x3e0 > > > > Read of size 1 at addr ffff888111f322cd by task swapper/0/0 > > > > > > > > CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.12.0-rc4- > > > > dirty > > > > #7 > > > > Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS > > > > 1.15.0- > > > > 1 > > > > Call Trace: > > > > <IRQ> > > > > dump_stack_lvl+0x68/0xa0 > > > > print_address_description.constprop.0+0x2c/0x3d0 > > > > print_report+0xb4/0x270 > > > > kasan_report+0xbd/0xf0 > > > > tcp_write_timer_handler+0x156/0x3e0 > > > > tcp_write_timer+0x66/0x170 > > > > call_timer_fn+0xfb/0x1d0 > > > > __run_timers+0x3f8/0x480 > > > > run_timer_softirq+0x9b/0x100 > > > > handle_softirqs+0x153/0x390 > > > > __irq_exit_rcu+0x103/0x120 > > > > irq_exit_rcu+0xe/0x20 > > > > sysvec_apic_timer_interrupt+0x76/0x90 > > > > </IRQ> > > > > <TASK> > > > > asm_sysvec_apic_timer_interrupt+0x1a/0x20 > > > > RIP: 0010:default_idle+0xf/0x20 > > > > Code: 4c 01 c7 4c 29 c2 e9 72 ff ff ff 90 90 90 90 90 90 90 90 > > > > 90 > > > > 90 > > > > 90 90 > > > > 90 90 90 90 f3 0f 1e fa 66 90 0f 00 2d 33 f8 25 00 fb f4 <fa> > > > > c3 > > > > cc > > > > cc cc > > > > cc 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 > > > > RSP: 0018:ffffffffa2007e28 EFLAGS: 00000242 > > > > RAX: 00000000000f3b31 RBX: 1ffffffff4400fc7 RCX: > > > > ffffffffa09c3196 > > > > RDX: 0000000000000000 RSI: 0000000000000000 RDI: > > > > ffffffff9f00590f > > > > RBP: 0000000000000000 R08: 0000000000000001 R09: > > > > ffffed102360835d > > > > R10: ffff88811b041aeb R11: 0000000000000001 R12: > > > > 0000000000000000 > > > > R13: ffffffffa202d7c0 R14: 0000000000000000 R15: > > > > 00000000000147d0 > > > > default_idle_call+0x6b/0xa0 > > > > cpuidle_idle_call+0x1af/0x1f0 > > > > do_idle+0xbc/0x130 > > > > cpu_startup_entry+0x33/0x40 > > > > rest_init+0x11f/0x210 > > > > start_kernel+0x39a/0x420 > > > > x86_64_start_reservations+0x18/0x30 > > > > x86_64_start_kernel+0x97/0xa0 > > > > common_startup_64+0x13e/0x141 > > > > </TASK> > > > > > > > > Allocated by task 595: > > > > kasan_save_stack+0x24/0x50 > > > > kasan_save_track+0x14/0x30 > > > > __kasan_slab_alloc+0x87/0x90 > > > > kmem_cache_alloc_noprof+0x12b/0x3f0 > > > > copy_net_ns+0x94/0x380 > > > > create_new_namespaces+0x24c/0x500 > > > > unshare_nsproxy_namespaces+0x75/0xf0 > > > > ksys_unshare+0x24e/0x4f0 > > > > __x64_sys_unshare+0x1f/0x30 > > > > do_syscall_64+0x70/0x180 > > > > entry_SYSCALL_64_after_hwframe+0x76/0x7e > > > > > > > > Freed by task 100: > > > > kasan_save_stack+0x24/0x50 > > > > kasan_save_track+0x14/0x30 > > > > kasan_save_free_info+0x3b/0x60 > > > > __kasan_slab_free+0x54/0x70 > > > > kmem_cache_free+0x156/0x5d0 > > > > cleanup_net+0x5d3/0x670 > > > > process_one_work+0x776/0xa90 > > > > worker_thread+0x2e2/0x560 > > > > kthread+0x1a8/0x1f0 > > > > ret_from_fork+0x34/0x60 > > > > ret_from_fork_asm+0x1a/0x30 > > > > > > > > Reproduction script: > > > > > > > > mkdir -p /mnt/nfsshare > > > > mkdir -p /mnt/nfs/netns_1 > > > > mkfs.ext4 /dev/sdb > > > > mount /dev/sdb /mnt/nfsshare > > > > systemctl restart nfs-server > > > > chmod 777 /mnt/nfsshare > > > > exportfs -i -o rw,no_root_squash *:/mnt/nfsshare > > > > > > > > ip netns add netns_1 > > > > ip link add name veth_1_peer type veth peer veth_1 > > > > ifconfig veth_1_peer 11.11.0.254 up > > > > ip link set veth_1 netns netns_1 > > > > ip netns exec netns_1 ifconfig veth_1 11.11.0.1 > > > > > > > > ip netns exec netns_1 /root/iptables -A OUTPUT -d 11.11.0.254 - > > > > p > > > > tcp > > > > \ > > > > --tcp-flags FIN FIN -j DROP > > > > > > > > (note: In my environment, a DESTROY_CLIENTID operation is > > > > always > > > > sent > > > > immediately, breaking the nfs tcp connection.) > > > > ip netns exec netns_1 timeout -s 9 300 mount -t nfs -o > > > > proto=tcp,vers=4.1 \ > > > > 11.11.0.254:/mnt/nfsshare /mnt/nfs/netns_1 > > > > > > > > ip netns del netns_1 > > > > > > > > The reason here is that the tcp socket in netns_1 (nfs side) > > > > has > > > > been > > > > shutdown and closed (done in xs_destroy), but the FIN message > > > > (with > > > > ack) > > > > is discarded, and the nfsd side keeps sending retransmission > > > > messages. > > > > As a result, when the tcp sock in netns_1 processes the > > > > received > > > > message, > > > > it sends the message (FIN message) in the sending queue, and > > > > the > > > > tcp > > > > timer > > > > is re-established. When the network namespace is deleted, the > > > > net > > > > structure > > > > accessed by tcp's timer handler function causes problems. > > > > > > > > The modification here aborts the TCP connection directly in > > > > xs_destroy(). > > > > > > > > Fixes: 26abe14379f8 ("net: Modify sk_alloc to not reference > > > > count > > > > the > > > > netns of kernel sockets.") > > > > Signed-off-by: Liu Jian <liujian56@huawei.com> > > > > --- > > > > net/sunrpc/xprtsock.c | 3 +++ > > > > 1 file changed, 3 insertions(+) > > > > > > > > diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c > > > > index 0e1691316f42..91ee3484155a 100644 > > > > --- a/net/sunrpc/xprtsock.c > > > > +++ b/net/sunrpc/xprtsock.c > > > > @@ -1287,6 +1287,9 @@ static void xs_reset_transport(struct > > > > sock_xprt > > > > *transport) > > > > release_sock(sk); > > > > mutex_unlock(&transport->recv_mutex); > > > > > > > > + if (sk->sk_prot == &tcp_prot) > > > > + tcp_abort(sk, ECONNABORTED); > > > We've already called kernel_sock_shutdown(sock, SHUT_RDWR), and > > > we're > > > about to close the socket. Why on earth should we need a hack > > > like > > > the > > > above in order to abort the connection at this point? > > > > > > This would appear to be a networking layer bug, and not an RPC > > > level > > > problem. > > > > > To put this differently: if a use after free can occur in the > > kernel > > when the RPC layer closes a TCP socket and then exits the network > > namespace, then can't that occur when a userland application does > > the > > same? > > > > If not, then what prevents it from happening? > The socket created by the userspace program obtains the reference > counting of the namespace, but the kernel socket does not. > > There's some discussion here: > https://lore.kernel.org/all/CANn89iJE5anTbyLJ0TdGAqGsE+GichY3YzQECjNUVMz=G3bcQg@mail.gmail.com/ OK... So then it looks to me as if NFS, SMB, AFS, and any other networked filesystem that can be started from inside a container is going to need to do the same thing that rds appears to be doing. Should there perhaps be a helper function in the networking layer for this? > > > > + > > > > trace_rpc_socket_close(xprt, sock); > > > > __fput_sync(filp); > > > > > >
diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c index 0e1691316f42..91ee3484155a 100644 --- a/net/sunrpc/xprtsock.c +++ b/net/sunrpc/xprtsock.c @@ -1287,6 +1287,9 @@ static void xs_reset_transport(struct sock_xprt *transport) release_sock(sk); mutex_unlock(&transport->recv_mutex); + if (sk->sk_prot == &tcp_prot) + tcp_abort(sk, ECONNABORTED); + trace_rpc_socket_close(xprt, sock); __fput_sync(filp);
BUG: KASAN: slab-use-after-free in tcp_write_timer_handler+0x156/0x3e0 Read of size 1 at addr ffff888111f322cd by task swapper/0/0 CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.12.0-rc4-dirty #7 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 Call Trace: <IRQ> dump_stack_lvl+0x68/0xa0 print_address_description.constprop.0+0x2c/0x3d0 print_report+0xb4/0x270 kasan_report+0xbd/0xf0 tcp_write_timer_handler+0x156/0x3e0 tcp_write_timer+0x66/0x170 call_timer_fn+0xfb/0x1d0 __run_timers+0x3f8/0x480 run_timer_softirq+0x9b/0x100 handle_softirqs+0x153/0x390 __irq_exit_rcu+0x103/0x120 irq_exit_rcu+0xe/0x20 sysvec_apic_timer_interrupt+0x76/0x90 </IRQ> <TASK> asm_sysvec_apic_timer_interrupt+0x1a/0x20 RIP: 0010:default_idle+0xf/0x20 Code: 4c 01 c7 4c 29 c2 e9 72 ff ff ff 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 66 90 0f 00 2d 33 f8 25 00 fb f4 <fa> c3 cc cc cc cc 66 66 2e 0f 1f 84 00 00 00 00 00 90 90 90 90 90 RSP: 0018:ffffffffa2007e28 EFLAGS: 00000242 RAX: 00000000000f3b31 RBX: 1ffffffff4400fc7 RCX: ffffffffa09c3196 RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffff9f00590f RBP: 0000000000000000 R08: 0000000000000001 R09: ffffed102360835d R10: ffff88811b041aeb R11: 0000000000000001 R12: 0000000000000000 R13: ffffffffa202d7c0 R14: 0000000000000000 R15: 00000000000147d0 default_idle_call+0x6b/0xa0 cpuidle_idle_call+0x1af/0x1f0 do_idle+0xbc/0x130 cpu_startup_entry+0x33/0x40 rest_init+0x11f/0x210 start_kernel+0x39a/0x420 x86_64_start_reservations+0x18/0x30 x86_64_start_kernel+0x97/0xa0 common_startup_64+0x13e/0x141 </TASK> Allocated by task 595: kasan_save_stack+0x24/0x50 kasan_save_track+0x14/0x30 __kasan_slab_alloc+0x87/0x90 kmem_cache_alloc_noprof+0x12b/0x3f0 copy_net_ns+0x94/0x380 create_new_namespaces+0x24c/0x500 unshare_nsproxy_namespaces+0x75/0xf0 ksys_unshare+0x24e/0x4f0 __x64_sys_unshare+0x1f/0x30 do_syscall_64+0x70/0x180 entry_SYSCALL_64_after_hwframe+0x76/0x7e Freed by task 100: kasan_save_stack+0x24/0x50 kasan_save_track+0x14/0x30 kasan_save_free_info+0x3b/0x60 __kasan_slab_free+0x54/0x70 kmem_cache_free+0x156/0x5d0 cleanup_net+0x5d3/0x670 process_one_work+0x776/0xa90 worker_thread+0x2e2/0x560 kthread+0x1a8/0x1f0 ret_from_fork+0x34/0x60 ret_from_fork_asm+0x1a/0x30 Reproduction script: mkdir -p /mnt/nfsshare mkdir -p /mnt/nfs/netns_1 mkfs.ext4 /dev/sdb mount /dev/sdb /mnt/nfsshare systemctl restart nfs-server chmod 777 /mnt/nfsshare exportfs -i -o rw,no_root_squash *:/mnt/nfsshare ip netns add netns_1 ip link add name veth_1_peer type veth peer veth_1 ifconfig veth_1_peer 11.11.0.254 up ip link set veth_1 netns netns_1 ip netns exec netns_1 ifconfig veth_1 11.11.0.1 ip netns exec netns_1 /root/iptables -A OUTPUT -d 11.11.0.254 -p tcp \ --tcp-flags FIN FIN -j DROP (note: In my environment, a DESTROY_CLIENTID operation is always sent immediately, breaking the nfs tcp connection.) ip netns exec netns_1 timeout -s 9 300 mount -t nfs -o proto=tcp,vers=4.1 \ 11.11.0.254:/mnt/nfsshare /mnt/nfs/netns_1 ip netns del netns_1 The reason here is that the tcp socket in netns_1 (nfs side) has been shutdown and closed (done in xs_destroy), but the FIN message (with ack) is discarded, and the nfsd side keeps sending retransmission messages. As a result, when the tcp sock in netns_1 processes the received message, it sends the message (FIN message) in the sending queue, and the tcp timer is re-established. When the network namespace is deleted, the net structure accessed by tcp's timer handler function causes problems. The modification here aborts the TCP connection directly in xs_destroy(). Fixes: 26abe14379f8 ("net: Modify sk_alloc to not reference count the netns of kernel sockets.") Signed-off-by: Liu Jian <liujian56@huawei.com> --- net/sunrpc/xprtsock.c | 3 +++ 1 file changed, 3 insertions(+)