diff mbox series

[net] inet: fully convert sk->sk_rx_dst to RCU rules

Message ID 20211220143330.680945-1-eric.dumazet@gmail.com (mailing list archive)
State Accepted
Commit 8f905c0e7354ef261360fb7535ea079b1082c105
Delegated to: Netdev Maintainers
Headers show
Series [net] inet: fully convert sk->sk_rx_dst to RCU rules | expand

Checks

Context Check Description
netdev/tree_selection success Clearly marked for net
netdev/fixes_present success Fixes tag present in non-next series
netdev/subject_prefix success Link
netdev/cover_letter success Single patches do not need cover letters
netdev/patch_count success Link
netdev/header_inline success No static functions without inline keyword in header files
netdev/build_32bit success Errors and warnings before: 3043 this patch: 3043
netdev/cc_maintainers warning 2 maintainers not CCed: yoshfuji@linux-ipv6.org dsahern@kernel.org
netdev/build_clang success Errors and warnings before: 412 this patch: 412
netdev/module_param success Was 0 now: 0
netdev/verify_signedoff success Signed-off-by tag matches author and committer
netdev/verify_fixes success Fixes tag looks correct
netdev/build_allmodconfig_warn success Errors and warnings before: 3177 this patch: 3177
netdev/checkpatch warning WARNING: Possible repeated word: 'Google'
netdev/kdoc success Errors and warnings before: 0 this patch: 0
netdev/source_inline success Was 0 now: 0

Commit Message

Eric Dumazet Dec. 20, 2021, 2:33 p.m. UTC
From: Eric Dumazet <edumazet@google.com>

syzbot reported various issues around early demux,
one being included in this changelog [1]

sk->sk_rx_dst is using RCU protection without clearly
documenting it.

And following sequences in tcp_v4_do_rcv()/tcp_v6_do_rcv()
are not following standard RCU rules.

[a]    dst_release(dst);
[b]    sk->sk_rx_dst = NULL;

They look wrong because a delete operation of RCU protected
pointer is supposed to clear the pointer before
the call_rcu()/synchronize_rcu() guarding actual memory freeing.

In some cases indeed, dst could be freed before [b] is done.

We could cheat by clearing sk_rx_dst before calling
dst_release(), but this seems the right time to stick
to standard RCU annotations and debugging facilities.

[1]
BUG: KASAN: use-after-free in dst_check include/net/dst.h:470 [inline]
BUG: KASAN: use-after-free in tcp_v4_early_demux+0x95b/0x960 net/ipv4/tcp_ipv4.c:1792
Read of size 2 at addr ffff88807f1cb73a by task syz-executor.5/9204

CPU: 0 PID: 9204 Comm: syz-executor.5 Not tainted 5.16.0-rc5-syzkaller #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:88 [inline]
 dump_stack_lvl+0xcd/0x134 lib/dump_stack.c:106
 print_address_description.constprop.0.cold+0x8d/0x320 mm/kasan/report.c:247
 __kasan_report mm/kasan/report.c:433 [inline]
 kasan_report.cold+0x83/0xdf mm/kasan/report.c:450
 dst_check include/net/dst.h:470 [inline]
 tcp_v4_early_demux+0x95b/0x960 net/ipv4/tcp_ipv4.c:1792
 ip_rcv_finish_core.constprop.0+0x15de/0x1e80 net/ipv4/ip_input.c:340
 ip_list_rcv_finish.constprop.0+0x1b2/0x6e0 net/ipv4/ip_input.c:583
 ip_sublist_rcv net/ipv4/ip_input.c:609 [inline]
 ip_list_rcv+0x34e/0x490 net/ipv4/ip_input.c:644
 __netif_receive_skb_list_ptype net/core/dev.c:5508 [inline]
 __netif_receive_skb_list_core+0x549/0x8e0 net/core/dev.c:5556
 __netif_receive_skb_list net/core/dev.c:5608 [inline]
 netif_receive_skb_list_internal+0x75e/0xd80 net/core/dev.c:5699
 gro_normal_list net/core/dev.c:5853 [inline]
 gro_normal_list net/core/dev.c:5849 [inline]
 napi_complete_done+0x1f1/0x880 net/core/dev.c:6590
 virtqueue_napi_complete drivers/net/virtio_net.c:339 [inline]
 virtnet_poll+0xca2/0x11b0 drivers/net/virtio_net.c:1557
 __napi_poll+0xaf/0x440 net/core/dev.c:7023
 napi_poll net/core/dev.c:7090 [inline]
 net_rx_action+0x801/0xb40 net/core/dev.c:7177
 __do_softirq+0x29b/0x9c2 kernel/softirq.c:558
 invoke_softirq kernel/softirq.c:432 [inline]
 __irq_exit_rcu+0x123/0x180 kernel/softirq.c:637
 irq_exit_rcu+0x5/0x20 kernel/softirq.c:649
 common_interrupt+0x52/0xc0 arch/x86/kernel/irq.c:240
 asm_common_interrupt+0x1e/0x40 arch/x86/include/asm/idtentry.h:629
RIP: 0033:0x7f5e972bfd57
Code: 39 d1 73 14 0f 1f 80 00 00 00 00 48 8b 50 f8 48 83 e8 08 48 39 ca 77 f3 48 39 c3 73 3e 48 89 13 48 8b 50 f8 48 89 38 49 8b 0e <48> 8b 3e 48 83 c3 08 48 83 c6 08 eb bc 48 39 d1 72 9e 48 39 d0 73
RSP: 002b:00007fff8a413210 EFLAGS: 00000283
RAX: 00007f5e97108990 RBX: 00007f5e97108338 RCX: ffffffff81d3aa45
RDX: ffffffff81d3aa45 RSI: 00007f5e97108340 RDI: ffffffff81d3aa45
RBP: 00007f5e97107eb8 R08: 00007f5e97108d88 R09: 0000000093c2e8d9
R10: 0000000000000000 R11: 0000000000000000 R12: 00007f5e97107eb0
R13: 00007f5e97108338 R14: 00007f5e97107ea8 R15: 0000000000000019
 </TASK>

Allocated by task 13:
 kasan_save_stack+0x1e/0x50 mm/kasan/common.c:38
 kasan_set_track mm/kasan/common.c:46 [inline]
 set_alloc_info mm/kasan/common.c:434 [inline]
 __kasan_slab_alloc+0x90/0xc0 mm/kasan/common.c:467
 kasan_slab_alloc include/linux/kasan.h:259 [inline]
 slab_post_alloc_hook mm/slab.h:519 [inline]
 slab_alloc_node mm/slub.c:3234 [inline]
 slab_alloc mm/slub.c:3242 [inline]
 kmem_cache_alloc+0x202/0x3a0 mm/slub.c:3247
 dst_alloc+0x146/0x1f0 net/core/dst.c:92
 rt_dst_alloc+0x73/0x430 net/ipv4/route.c:1613
 ip_route_input_slow+0x1817/0x3a20 net/ipv4/route.c:2340
 ip_route_input_rcu net/ipv4/route.c:2470 [inline]
 ip_route_input_noref+0x116/0x2a0 net/ipv4/route.c:2415
 ip_rcv_finish_core.constprop.0+0x288/0x1e80 net/ipv4/ip_input.c:354
 ip_list_rcv_finish.constprop.0+0x1b2/0x6e0 net/ipv4/ip_input.c:583
 ip_sublist_rcv net/ipv4/ip_input.c:609 [inline]
 ip_list_rcv+0x34e/0x490 net/ipv4/ip_input.c:644
 __netif_receive_skb_list_ptype net/core/dev.c:5508 [inline]
 __netif_receive_skb_list_core+0x549/0x8e0 net/core/dev.c:5556
 __netif_receive_skb_list net/core/dev.c:5608 [inline]
 netif_receive_skb_list_internal+0x75e/0xd80 net/core/dev.c:5699
 gro_normal_list net/core/dev.c:5853 [inline]
 gro_normal_list net/core/dev.c:5849 [inline]
 napi_complete_done+0x1f1/0x880 net/core/dev.c:6590
 virtqueue_napi_complete drivers/net/virtio_net.c:339 [inline]
 virtnet_poll+0xca2/0x11b0 drivers/net/virtio_net.c:1557
 __napi_poll+0xaf/0x440 net/core/dev.c:7023
 napi_poll net/core/dev.c:7090 [inline]
 net_rx_action+0x801/0xb40 net/core/dev.c:7177
 __do_softirq+0x29b/0x9c2 kernel/softirq.c:558

Freed by task 13:
 kasan_save_stack+0x1e/0x50 mm/kasan/common.c:38
 kasan_set_track+0x21/0x30 mm/kasan/common.c:46
 kasan_set_free_info+0x20/0x30 mm/kasan/generic.c:370
 ____kasan_slab_free mm/kasan/common.c:366 [inline]
 ____kasan_slab_free mm/kasan/common.c:328 [inline]
 __kasan_slab_free+0xff/0x130 mm/kasan/common.c:374
 kasan_slab_free include/linux/kasan.h:235 [inline]
 slab_free_hook mm/slub.c:1723 [inline]
 slab_free_freelist_hook+0x8b/0x1c0 mm/slub.c:1749
 slab_free mm/slub.c:3513 [inline]
 kmem_cache_free+0xbd/0x5d0 mm/slub.c:3530
 dst_destroy+0x2d6/0x3f0 net/core/dst.c:127
 rcu_do_batch kernel/rcu/tree.c:2506 [inline]
 rcu_core+0x7ab/0x1470 kernel/rcu/tree.c:2741
 __do_softirq+0x29b/0x9c2 kernel/softirq.c:558

Last potentially related work creation:
 kasan_save_stack+0x1e/0x50 mm/kasan/common.c:38
 __kasan_record_aux_stack+0xf5/0x120 mm/kasan/generic.c:348
 __call_rcu kernel/rcu/tree.c:2985 [inline]
 call_rcu+0xb1/0x740 kernel/rcu/tree.c:3065
 dst_release net/core/dst.c:177 [inline]
 dst_release+0x79/0xe0 net/core/dst.c:167
 tcp_v4_do_rcv+0x612/0x8d0 net/ipv4/tcp_ipv4.c:1712
 sk_backlog_rcv include/net/sock.h:1030 [inline]
 __release_sock+0x134/0x3b0 net/core/sock.c:2768
 release_sock+0x54/0x1b0 net/core/sock.c:3300
 tcp_sendmsg+0x36/0x40 net/ipv4/tcp.c:1441
 inet_sendmsg+0x99/0xe0 net/ipv4/af_inet.c:819
 sock_sendmsg_nosec net/socket.c:704 [inline]
 sock_sendmsg+0xcf/0x120 net/socket.c:724
 sock_write_iter+0x289/0x3c0 net/socket.c:1057
 call_write_iter include/linux/fs.h:2162 [inline]
 new_sync_write+0x429/0x660 fs/read_write.c:503
 vfs_write+0x7cd/0xae0 fs/read_write.c:590
 ksys_write+0x1ee/0x250 fs/read_write.c:643
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x44/0xae

The buggy address belongs to the object at ffff88807f1cb700
 which belongs to the cache ip_dst_cache of size 176
The buggy address is located 58 bytes inside of
 176-byte region [ffff88807f1cb700, ffff88807f1cb7b0)
The buggy address belongs to the page:
page:ffffea0001fc72c0 refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x7f1cb
flags: 0xfff00000000200(slab|node=0|zone=1|lastcpupid=0x7ff)
raw: 00fff00000000200 dead000000000100 dead000000000122 ffff8881413bb780
raw: 0000000000000000 0000000000100010 00000001ffffffff 0000000000000000
page dumped because: kasan: bad access detected
page_owner tracks the page as allocated
page last allocated via order 0, migratetype Unmovable, gfp_mask 0x112a20(GFP_ATOMIC|__GFP_NOWARN|__GFP_NORETRY|__GFP_HARDWALL), pid 5, ts 108466983062, free_ts 108048976062
 prep_new_page mm/page_alloc.c:2418 [inline]
 get_page_from_freelist+0xa72/0x2f50 mm/page_alloc.c:4149
 __alloc_pages+0x1b2/0x500 mm/page_alloc.c:5369
 alloc_pages+0x1a7/0x300 mm/mempolicy.c:2191
 alloc_slab_page mm/slub.c:1793 [inline]
 allocate_slab mm/slub.c:1930 [inline]
 new_slab+0x32d/0x4a0 mm/slub.c:1993
 ___slab_alloc+0x918/0xfe0 mm/slub.c:3022
 __slab_alloc.constprop.0+0x4d/0xa0 mm/slub.c:3109
 slab_alloc_node mm/slub.c:3200 [inline]
 slab_alloc mm/slub.c:3242 [inline]
 kmem_cache_alloc+0x35c/0x3a0 mm/slub.c:3247
 dst_alloc+0x146/0x1f0 net/core/dst.c:92
 rt_dst_alloc+0x73/0x430 net/ipv4/route.c:1613
 __mkroute_output net/ipv4/route.c:2564 [inline]
 ip_route_output_key_hash_rcu+0x921/0x2d00 net/ipv4/route.c:2791
 ip_route_output_key_hash+0x18b/0x300 net/ipv4/route.c:2619
 __ip_route_output_key include/net/route.h:126 [inline]
 ip_route_output_flow+0x23/0x150 net/ipv4/route.c:2850
 ip_route_output_key include/net/route.h:142 [inline]
 geneve_get_v4_rt+0x3a6/0x830 drivers/net/geneve.c:809
 geneve_xmit_skb drivers/net/geneve.c:899 [inline]
 geneve_xmit+0xc4a/0x3540 drivers/net/geneve.c:1082
 __netdev_start_xmit include/linux/netdevice.h:4994 [inline]
 netdev_start_xmit include/linux/netdevice.h:5008 [inline]
 xmit_one net/core/dev.c:3590 [inline]
 dev_hard_start_xmit+0x1eb/0x920 net/core/dev.c:3606
 __dev_queue_xmit+0x299a/0x3650 net/core/dev.c:4229
page last free stack trace:
 reset_page_owner include/linux/page_owner.h:24 [inline]
 free_pages_prepare mm/page_alloc.c:1338 [inline]
 free_pcp_prepare+0x374/0x870 mm/page_alloc.c:1389
 free_unref_page_prepare mm/page_alloc.c:3309 [inline]
 free_unref_page+0x19/0x690 mm/page_alloc.c:3388
 qlink_free mm/kasan/quarantine.c:146 [inline]
 qlist_free_all+0x5a/0xc0 mm/kasan/quarantine.c:165
 kasan_quarantine_reduce+0x180/0x200 mm/kasan/quarantine.c:272
 __kasan_slab_alloc+0xa2/0xc0 mm/kasan/common.c:444
 kasan_slab_alloc include/linux/kasan.h:259 [inline]
 slab_post_alloc_hook mm/slab.h:519 [inline]
 slab_alloc_node mm/slub.c:3234 [inline]
 kmem_cache_alloc_node+0x255/0x3f0 mm/slub.c:3270
 __alloc_skb+0x215/0x340 net/core/skbuff.c:414
 alloc_skb include/linux/skbuff.h:1126 [inline]
 alloc_skb_with_frags+0x93/0x620 net/core/skbuff.c:6078
 sock_alloc_send_pskb+0x783/0x910 net/core/sock.c:2575
 mld_newpack+0x1df/0x770 net/ipv6/mcast.c:1754
 add_grhead+0x265/0x330 net/ipv6/mcast.c:1857
 add_grec+0x1053/0x14e0 net/ipv6/mcast.c:1995
 mld_send_initial_cr.part.0+0xf6/0x230 net/ipv6/mcast.c:2242
 mld_send_initial_cr net/ipv6/mcast.c:1232 [inline]
 mld_dad_work+0x1d3/0x690 net/ipv6/mcast.c:2268
 process_one_work+0x9b2/0x1690 kernel/workqueue.c:2298
 worker_thread+0x658/0x11f0 kernel/workqueue.c:2445

Memory state around the buggy address:
 ffff88807f1cb600: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
 ffff88807f1cb680: fb fb fb fb fb fb fc fc fc fc fc fc fc fc fc fc
>ffff88807f1cb700: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
                                        ^
 ffff88807f1cb780: fb fb fb fb fb fb fc fc fc fc fc fc fc fc fc fc
 ffff88807f1cb800: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb

Fixes: 41063e9dd119 ("ipv4: Early TCP socket demux.")
Signed-off-by: Eric Dumazet <edumazet@google.com>
---
 include/net/sock.h   |  2 +-
 net/ipv4/af_inet.c   |  2 +-
 net/ipv4/tcp.c       |  3 +--
 net/ipv4/tcp_input.c |  2 +-
 net/ipv4/tcp_ipv4.c  | 11 +++++++----
 net/ipv4/udp.c       |  6 +++---
 net/ipv6/tcp_ipv6.c  | 11 +++++++----
 net/ipv6/udp.c       |  4 ++--
 8 files changed, 23 insertions(+), 18 deletions(-)

Comments

patchwork-bot+netdevbpf@kernel.org Dec. 21, 2021, 3:10 a.m. UTC | #1
Hello:

This patch was applied to netdev/net.git (master)
by Jakub Kicinski <kuba@kernel.org>:

On Mon, 20 Dec 2021 06:33:30 -0800 you wrote:
> From: Eric Dumazet <edumazet@google.com>
> 
> syzbot reported various issues around early demux,
> one being included in this changelog [1]
> 
> sk->sk_rx_dst is using RCU protection without clearly
> documenting it.
> 
> [...]

Here is the summary with links:
  - [net] inet: fully convert sk->sk_rx_dst to RCU rules
    https://git.kernel.org/netdev/net/c/8f905c0e7354

You are awesome, thank you!
Carlos Llamas Oct. 20, 2022, 9:05 p.m. UTC | #2
On Mon, Dec 20, 2021 at 06:33:30AM -0800, Eric Dumazet wrote:
> From: Eric Dumazet <edumazet@google.com>
> 
> syzbot reported various issues around early demux,
> one being included in this changelog [1]
> 
> sk->sk_rx_dst is using RCU protection without clearly
> documenting it.
> 
> And following sequences in tcp_v4_do_rcv()/tcp_v6_do_rcv()
> are not following standard RCU rules.
> 
> [a]    dst_release(dst);
> [b]    sk->sk_rx_dst = NULL;
> 
> They look wrong because a delete operation of RCU protected
> pointer is supposed to clear the pointer before
> the call_rcu()/synchronize_rcu() guarding actual memory freeing.
> 
> In some cases indeed, dst could be freed before [b] is done.
> 
> We could cheat by clearing sk_rx_dst before calling
> dst_release(), but this seems the right time to stick
> to standard RCU annotations and debugging facilities.
> 
> [1]
> BUG: KASAN: use-after-free in dst_check include/net/dst.h:470 [inline]
> BUG: KASAN: use-after-free in tcp_v4_early_demux+0x95b/0x960 net/ipv4/tcp_ipv4.c:1792
> Read of size 2 at addr ffff88807f1cb73a by task syz-executor.5/9204
> 
> CPU: 0 PID: 9204 Comm: syz-executor.5 Not tainted 5.16.0-rc5-syzkaller #0
> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
> Call Trace:
>  <TASK>
>  __dump_stack lib/dump_stack.c:88 [inline]
>  dump_stack_lvl+0xcd/0x134 lib/dump_stack.c:106
>  print_address_description.constprop.0.cold+0x8d/0x320 mm/kasan/report.c:247
>  __kasan_report mm/kasan/report.c:433 [inline]
>  kasan_report.cold+0x83/0xdf mm/kasan/report.c:450
>  dst_check include/net/dst.h:470 [inline]
>  tcp_v4_early_demux+0x95b/0x960 net/ipv4/tcp_ipv4.c:1792
>  ip_rcv_finish_core.constprop.0+0x15de/0x1e80 net/ipv4/ip_input.c:340
>  ip_list_rcv_finish.constprop.0+0x1b2/0x6e0 net/ipv4/ip_input.c:583
>  ip_sublist_rcv net/ipv4/ip_input.c:609 [inline]
>  ip_list_rcv+0x34e/0x490 net/ipv4/ip_input.c:644
>  __netif_receive_skb_list_ptype net/core/dev.c:5508 [inline]
>  __netif_receive_skb_list_core+0x549/0x8e0 net/core/dev.c:5556
>  __netif_receive_skb_list net/core/dev.c:5608 [inline]
>  netif_receive_skb_list_internal+0x75e/0xd80 net/core/dev.c:5699
>  gro_normal_list net/core/dev.c:5853 [inline]
>  gro_normal_list net/core/dev.c:5849 [inline]
>  napi_complete_done+0x1f1/0x880 net/core/dev.c:6590
>  virtqueue_napi_complete drivers/net/virtio_net.c:339 [inline]
>  virtnet_poll+0xca2/0x11b0 drivers/net/virtio_net.c:1557
>  __napi_poll+0xaf/0x440 net/core/dev.c:7023
>  napi_poll net/core/dev.c:7090 [inline]
>  net_rx_action+0x801/0xb40 net/core/dev.c:7177
>  __do_softirq+0x29b/0x9c2 kernel/softirq.c:558
>  invoke_softirq kernel/softirq.c:432 [inline]
>  __irq_exit_rcu+0x123/0x180 kernel/softirq.c:637
>  irq_exit_rcu+0x5/0x20 kernel/softirq.c:649
>  common_interrupt+0x52/0xc0 arch/x86/kernel/irq.c:240
>  asm_common_interrupt+0x1e/0x40 arch/x86/include/asm/idtentry.h:629
> RIP: 0033:0x7f5e972bfd57
> Code: 39 d1 73 14 0f 1f 80 00 00 00 00 48 8b 50 f8 48 83 e8 08 48 39 ca 77 f3 48 39 c3 73 3e 48 89 13 48 8b 50 f8 48 89 38 49 8b 0e <48> 8b 3e 48 83 c3 08 48 83 c6 08 eb bc 48 39 d1 72 9e 48 39 d0 73
> RSP: 002b:00007fff8a413210 EFLAGS: 00000283
> RAX: 00007f5e97108990 RBX: 00007f5e97108338 RCX: ffffffff81d3aa45
> RDX: ffffffff81d3aa45 RSI: 00007f5e97108340 RDI: ffffffff81d3aa45
> RBP: 00007f5e97107eb8 R08: 00007f5e97108d88 R09: 0000000093c2e8d9
> R10: 0000000000000000 R11: 0000000000000000 R12: 00007f5e97107eb0
> R13: 00007f5e97108338 R14: 00007f5e97107ea8 R15: 0000000000000019
>  </TASK>
> 
> Allocated by task 13:
>  kasan_save_stack+0x1e/0x50 mm/kasan/common.c:38
>  kasan_set_track mm/kasan/common.c:46 [inline]
>  set_alloc_info mm/kasan/common.c:434 [inline]
>  __kasan_slab_alloc+0x90/0xc0 mm/kasan/common.c:467
>  kasan_slab_alloc include/linux/kasan.h:259 [inline]
>  slab_post_alloc_hook mm/slab.h:519 [inline]
>  slab_alloc_node mm/slub.c:3234 [inline]
>  slab_alloc mm/slub.c:3242 [inline]
>  kmem_cache_alloc+0x202/0x3a0 mm/slub.c:3247
>  dst_alloc+0x146/0x1f0 net/core/dst.c:92
>  rt_dst_alloc+0x73/0x430 net/ipv4/route.c:1613
>  ip_route_input_slow+0x1817/0x3a20 net/ipv4/route.c:2340
>  ip_route_input_rcu net/ipv4/route.c:2470 [inline]
>  ip_route_input_noref+0x116/0x2a0 net/ipv4/route.c:2415
>  ip_rcv_finish_core.constprop.0+0x288/0x1e80 net/ipv4/ip_input.c:354
>  ip_list_rcv_finish.constprop.0+0x1b2/0x6e0 net/ipv4/ip_input.c:583
>  ip_sublist_rcv net/ipv4/ip_input.c:609 [inline]
>  ip_list_rcv+0x34e/0x490 net/ipv4/ip_input.c:644
>  __netif_receive_skb_list_ptype net/core/dev.c:5508 [inline]
>  __netif_receive_skb_list_core+0x549/0x8e0 net/core/dev.c:5556
>  __netif_receive_skb_list net/core/dev.c:5608 [inline]
>  netif_receive_skb_list_internal+0x75e/0xd80 net/core/dev.c:5699
>  gro_normal_list net/core/dev.c:5853 [inline]
>  gro_normal_list net/core/dev.c:5849 [inline]
>  napi_complete_done+0x1f1/0x880 net/core/dev.c:6590
>  virtqueue_napi_complete drivers/net/virtio_net.c:339 [inline]
>  virtnet_poll+0xca2/0x11b0 drivers/net/virtio_net.c:1557
>  __napi_poll+0xaf/0x440 net/core/dev.c:7023
>  napi_poll net/core/dev.c:7090 [inline]
>  net_rx_action+0x801/0xb40 net/core/dev.c:7177
>  __do_softirq+0x29b/0x9c2 kernel/softirq.c:558
> 
> Freed by task 13:
>  kasan_save_stack+0x1e/0x50 mm/kasan/common.c:38
>  kasan_set_track+0x21/0x30 mm/kasan/common.c:46
>  kasan_set_free_info+0x20/0x30 mm/kasan/generic.c:370
>  ____kasan_slab_free mm/kasan/common.c:366 [inline]
>  ____kasan_slab_free mm/kasan/common.c:328 [inline]
>  __kasan_slab_free+0xff/0x130 mm/kasan/common.c:374
>  kasan_slab_free include/linux/kasan.h:235 [inline]
>  slab_free_hook mm/slub.c:1723 [inline]
>  slab_free_freelist_hook+0x8b/0x1c0 mm/slub.c:1749
>  slab_free mm/slub.c:3513 [inline]
>  kmem_cache_free+0xbd/0x5d0 mm/slub.c:3530
>  dst_destroy+0x2d6/0x3f0 net/core/dst.c:127
>  rcu_do_batch kernel/rcu/tree.c:2506 [inline]
>  rcu_core+0x7ab/0x1470 kernel/rcu/tree.c:2741
>  __do_softirq+0x29b/0x9c2 kernel/softirq.c:558
> 
> Last potentially related work creation:
>  kasan_save_stack+0x1e/0x50 mm/kasan/common.c:38
>  __kasan_record_aux_stack+0xf5/0x120 mm/kasan/generic.c:348
>  __call_rcu kernel/rcu/tree.c:2985 [inline]
>  call_rcu+0xb1/0x740 kernel/rcu/tree.c:3065
>  dst_release net/core/dst.c:177 [inline]
>  dst_release+0x79/0xe0 net/core/dst.c:167
>  tcp_v4_do_rcv+0x612/0x8d0 net/ipv4/tcp_ipv4.c:1712
>  sk_backlog_rcv include/net/sock.h:1030 [inline]
>  __release_sock+0x134/0x3b0 net/core/sock.c:2768
>  release_sock+0x54/0x1b0 net/core/sock.c:3300
>  tcp_sendmsg+0x36/0x40 net/ipv4/tcp.c:1441
>  inet_sendmsg+0x99/0xe0 net/ipv4/af_inet.c:819
>  sock_sendmsg_nosec net/socket.c:704 [inline]
>  sock_sendmsg+0xcf/0x120 net/socket.c:724
>  sock_write_iter+0x289/0x3c0 net/socket.c:1057
>  call_write_iter include/linux/fs.h:2162 [inline]
>  new_sync_write+0x429/0x660 fs/read_write.c:503
>  vfs_write+0x7cd/0xae0 fs/read_write.c:590
>  ksys_write+0x1ee/0x250 fs/read_write.c:643
>  do_syscall_x64 arch/x86/entry/common.c:50 [inline]
>  do_syscall_64+0x35/0xb0 arch/x86/entry/common.c:80
>  entry_SYSCALL_64_after_hwframe+0x44/0xae
> 
> The buggy address belongs to the object at ffff88807f1cb700
>  which belongs to the cache ip_dst_cache of size 176
> The buggy address is located 58 bytes inside of
>  176-byte region [ffff88807f1cb700, ffff88807f1cb7b0)
> The buggy address belongs to the page:
> page:ffffea0001fc72c0 refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x7f1cb
> flags: 0xfff00000000200(slab|node=0|zone=1|lastcpupid=0x7ff)
> raw: 00fff00000000200 dead000000000100 dead000000000122 ffff8881413bb780
> raw: 0000000000000000 0000000000100010 00000001ffffffff 0000000000000000
> page dumped because: kasan: bad access detected
> page_owner tracks the page as allocated
> page last allocated via order 0, migratetype Unmovable, gfp_mask 0x112a20(GFP_ATOMIC|__GFP_NOWARN|__GFP_NORETRY|__GFP_HARDWALL), pid 5, ts 108466983062, free_ts 108048976062
>  prep_new_page mm/page_alloc.c:2418 [inline]
>  get_page_from_freelist+0xa72/0x2f50 mm/page_alloc.c:4149
>  __alloc_pages+0x1b2/0x500 mm/page_alloc.c:5369
>  alloc_pages+0x1a7/0x300 mm/mempolicy.c:2191
>  alloc_slab_page mm/slub.c:1793 [inline]
>  allocate_slab mm/slub.c:1930 [inline]
>  new_slab+0x32d/0x4a0 mm/slub.c:1993
>  ___slab_alloc+0x918/0xfe0 mm/slub.c:3022
>  __slab_alloc.constprop.0+0x4d/0xa0 mm/slub.c:3109
>  slab_alloc_node mm/slub.c:3200 [inline]
>  slab_alloc mm/slub.c:3242 [inline]
>  kmem_cache_alloc+0x35c/0x3a0 mm/slub.c:3247
>  dst_alloc+0x146/0x1f0 net/core/dst.c:92
>  rt_dst_alloc+0x73/0x430 net/ipv4/route.c:1613
>  __mkroute_output net/ipv4/route.c:2564 [inline]
>  ip_route_output_key_hash_rcu+0x921/0x2d00 net/ipv4/route.c:2791
>  ip_route_output_key_hash+0x18b/0x300 net/ipv4/route.c:2619
>  __ip_route_output_key include/net/route.h:126 [inline]
>  ip_route_output_flow+0x23/0x150 net/ipv4/route.c:2850
>  ip_route_output_key include/net/route.h:142 [inline]
>  geneve_get_v4_rt+0x3a6/0x830 drivers/net/geneve.c:809
>  geneve_xmit_skb drivers/net/geneve.c:899 [inline]
>  geneve_xmit+0xc4a/0x3540 drivers/net/geneve.c:1082
>  __netdev_start_xmit include/linux/netdevice.h:4994 [inline]
>  netdev_start_xmit include/linux/netdevice.h:5008 [inline]
>  xmit_one net/core/dev.c:3590 [inline]
>  dev_hard_start_xmit+0x1eb/0x920 net/core/dev.c:3606
>  __dev_queue_xmit+0x299a/0x3650 net/core/dev.c:4229
> page last free stack trace:
>  reset_page_owner include/linux/page_owner.h:24 [inline]
>  free_pages_prepare mm/page_alloc.c:1338 [inline]
>  free_pcp_prepare+0x374/0x870 mm/page_alloc.c:1389
>  free_unref_page_prepare mm/page_alloc.c:3309 [inline]
>  free_unref_page+0x19/0x690 mm/page_alloc.c:3388
>  qlink_free mm/kasan/quarantine.c:146 [inline]
>  qlist_free_all+0x5a/0xc0 mm/kasan/quarantine.c:165
>  kasan_quarantine_reduce+0x180/0x200 mm/kasan/quarantine.c:272
>  __kasan_slab_alloc+0xa2/0xc0 mm/kasan/common.c:444
>  kasan_slab_alloc include/linux/kasan.h:259 [inline]
>  slab_post_alloc_hook mm/slab.h:519 [inline]
>  slab_alloc_node mm/slub.c:3234 [inline]
>  kmem_cache_alloc_node+0x255/0x3f0 mm/slub.c:3270
>  __alloc_skb+0x215/0x340 net/core/skbuff.c:414
>  alloc_skb include/linux/skbuff.h:1126 [inline]
>  alloc_skb_with_frags+0x93/0x620 net/core/skbuff.c:6078
>  sock_alloc_send_pskb+0x783/0x910 net/core/sock.c:2575
>  mld_newpack+0x1df/0x770 net/ipv6/mcast.c:1754
>  add_grhead+0x265/0x330 net/ipv6/mcast.c:1857
>  add_grec+0x1053/0x14e0 net/ipv6/mcast.c:1995
>  mld_send_initial_cr.part.0+0xf6/0x230 net/ipv6/mcast.c:2242
>  mld_send_initial_cr net/ipv6/mcast.c:1232 [inline]
>  mld_dad_work+0x1d3/0x690 net/ipv6/mcast.c:2268
>  process_one_work+0x9b2/0x1690 kernel/workqueue.c:2298
>  worker_thread+0x658/0x11f0 kernel/workqueue.c:2445
> 
> Memory state around the buggy address:
>  ffff88807f1cb600: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>  ffff88807f1cb680: fb fb fb fb fb fb fc fc fc fc fc fc fc fc fc fc
> >ffff88807f1cb700: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>                                         ^
>  ffff88807f1cb780: fb fb fb fb fb fb fc fc fc fc fc fc fc fc fc fc
>  ffff88807f1cb800: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
> 
> Fixes: 41063e9dd119 ("ipv4: Early TCP socket demux.")
> Signed-off-by: Eric Dumazet <edumazet@google.com>
> ---
>  include/net/sock.h   |  2 +-
>  net/ipv4/af_inet.c   |  2 +-
>  net/ipv4/tcp.c       |  3 +--
>  net/ipv4/tcp_input.c |  2 +-
>  net/ipv4/tcp_ipv4.c  | 11 +++++++----
>  net/ipv4/udp.c       |  6 +++---
>  net/ipv6/tcp_ipv6.c  | 11 +++++++----
>  net/ipv6/udp.c       |  4 ++--
>  8 files changed, 23 insertions(+), 18 deletions(-)
> 
> diff --git a/include/net/sock.h b/include/net/sock.h
> index bea21ff70e74d906216f4eaa2d5a712d12551216..d47e9658da28545c1f6afd9db0cf136b3e13d7b6 100644
> --- a/include/net/sock.h
> +++ b/include/net/sock.h
> @@ -431,7 +431,7 @@ struct sock {
>  #ifdef CONFIG_XFRM
>  	struct xfrm_policy __rcu *sk_policy[2];
>  #endif
> -	struct dst_entry	*sk_rx_dst;
> +	struct dst_entry __rcu	*sk_rx_dst;
>  	int			sk_rx_dst_ifindex;
>  	u32			sk_rx_dst_cookie;
>  
> diff --git a/net/ipv4/af_inet.c b/net/ipv4/af_inet.c
> index 0189e3cd4a7df2dc2ea7121182ee290e0164df90..6b5956500436187d5e9b801ebb6f8f52ba0db090 100644
> --- a/net/ipv4/af_inet.c
> +++ b/net/ipv4/af_inet.c
> @@ -154,7 +154,7 @@ void inet_sock_destruct(struct sock *sk)
>  
>  	kfree(rcu_dereference_protected(inet->inet_opt, 1));
>  	dst_release(rcu_dereference_protected(sk->sk_dst_cache, 1));
> -	dst_release(sk->sk_rx_dst);
> +	dst_release(rcu_dereference_protected(sk->sk_rx_dst, 1));
>  	sk_refcnt_debug_dec(sk);
>  }
>  EXPORT_SYMBOL(inet_sock_destruct);
> diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
> index bbb3d39c69afc2d5a42c6ace8d473657861da61f..2bb28bfd83bf621b5d9f3fb7ce5695195697b43a 100644
> --- a/net/ipv4/tcp.c
> +++ b/net/ipv4/tcp.c
> @@ -3012,8 +3012,7 @@ int tcp_disconnect(struct sock *sk, int flags)
>  	icsk->icsk_ack.rcv_mss = TCP_MIN_MSS;
>  	memset(&tp->rx_opt, 0, sizeof(tp->rx_opt));
>  	__sk_dst_reset(sk);
> -	dst_release(sk->sk_rx_dst);
> -	sk->sk_rx_dst = NULL;
> +	dst_release(xchg((__force struct dst_entry **)&sk->sk_rx_dst, NULL));
>  	tcp_saved_syn_free(tp);
>  	tp->compressed_ack = 0;
>  	tp->segs_in = 0;
> diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
> index 246ab7b5e857eb9e802c4805075e89c98cf00636..0ce46849ec3d4595699dd54229919b2d66b70257 100644
> --- a/net/ipv4/tcp_input.c
> +++ b/net/ipv4/tcp_input.c
> @@ -5787,7 +5787,7 @@ void tcp_rcv_established(struct sock *sk, struct sk_buff *skb)
>  	trace_tcp_probe(sk, skb);
>  
>  	tcp_mstamp_refresh(tp);
> -	if (unlikely(!sk->sk_rx_dst))
> +	if (unlikely(!rcu_access_pointer(sk->sk_rx_dst)))
>  		inet_csk(sk)->icsk_af_ops->sk_rx_dst_set(sk, skb);
>  	/*
>  	 *	Header prediction.
> diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c
> index 13d868c43284584ee0c58ddfd411bb52c8b0c830..084df223b5dff8089a615a4a8d392b620fc0a28a 100644
> --- a/net/ipv4/tcp_ipv4.c
> +++ b/net/ipv4/tcp_ipv4.c
> @@ -1701,7 +1701,10 @@ int tcp_v4_do_rcv(struct sock *sk, struct sk_buff *skb)
>  	struct sock *rsk;
>  
>  	if (sk->sk_state == TCP_ESTABLISHED) { /* Fast path */
> -		struct dst_entry *dst = sk->sk_rx_dst;
> +		struct dst_entry *dst;
> +
> +		dst = rcu_dereference_protected(sk->sk_rx_dst,
> +						lockdep_sock_is_held(sk));
>  
>  		sock_rps_save_rxhash(sk, skb);
>  		sk_mark_napi_id(sk, skb);
> @@ -1709,8 +1712,8 @@ int tcp_v4_do_rcv(struct sock *sk, struct sk_buff *skb)
>  			if (sk->sk_rx_dst_ifindex != skb->skb_iif ||
>  			    !INDIRECT_CALL_1(dst->ops->check, ipv4_dst_check,
>  					     dst, 0)) {
> +				RCU_INIT_POINTER(sk->sk_rx_dst, NULL);
>  				dst_release(dst);
> -				sk->sk_rx_dst = NULL;
>  			}
>  		}
>  		tcp_rcv_established(sk, skb);
> @@ -1786,7 +1789,7 @@ int tcp_v4_early_demux(struct sk_buff *skb)
>  		skb->sk = sk;
>  		skb->destructor = sock_edemux;
>  		if (sk_fullsock(sk)) {
> -			struct dst_entry *dst = READ_ONCE(sk->sk_rx_dst);
> +			struct dst_entry *dst = rcu_dereference(sk->sk_rx_dst);
>  
>  			if (dst)
>  				dst = dst_check(dst, 0);
> @@ -2201,7 +2204,7 @@ void inet_sk_rx_dst_set(struct sock *sk, const struct sk_buff *skb)
>  	struct dst_entry *dst = skb_dst(skb);
>  
>  	if (dst && dst_hold_safe(dst)) {
> -		sk->sk_rx_dst = dst;
> +		rcu_assign_pointer(sk->sk_rx_dst, dst);
>  		sk->sk_rx_dst_ifindex = skb->skb_iif;
>  	}
>  }
> diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c
> index 23b05e28490b0a0a690a837027f26167e353f8ce..15c6b450b8dba44d5f344a554eea6b991c1ea5f1 100644
> --- a/net/ipv4/udp.c
> +++ b/net/ipv4/udp.c
> @@ -2250,7 +2250,7 @@ bool udp_sk_rx_dst_set(struct sock *sk, struct dst_entry *dst)
>  	struct dst_entry *old;
>  
>  	if (dst_hold_safe(dst)) {
> -		old = xchg(&sk->sk_rx_dst, dst);
> +		old = xchg((__force struct dst_entry **)&sk->sk_rx_dst, dst);
>  		dst_release(old);
>  		return old != dst;
>  	}
> @@ -2440,7 +2440,7 @@ int __udp4_lib_rcv(struct sk_buff *skb, struct udp_table *udptable,
>  		struct dst_entry *dst = skb_dst(skb);
>  		int ret;
>  
> -		if (unlikely(sk->sk_rx_dst != dst))
> +		if (unlikely(rcu_dereference(sk->sk_rx_dst) != dst))
>  			udp_sk_rx_dst_set(sk, dst);
>  
>  		ret = udp_unicast_rcv_skb(sk, skb, uh);
> @@ -2599,7 +2599,7 @@ int udp_v4_early_demux(struct sk_buff *skb)
>  
>  	skb->sk = sk;
>  	skb->destructor = sock_efree;
> -	dst = READ_ONCE(sk->sk_rx_dst);
> +	dst = rcu_dereference(sk->sk_rx_dst);
>  
>  	if (dst)
>  		dst = dst_check(dst, 0);
> diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c
> index 551fce49841d7f53a111b0435855634cece2b40a..680e6481b9672040ccb41fd08ab80166575bef50 100644
> --- a/net/ipv6/tcp_ipv6.c
> +++ b/net/ipv6/tcp_ipv6.c
> @@ -107,7 +107,7 @@ static void inet6_sk_rx_dst_set(struct sock *sk, const struct sk_buff *skb)
>  	if (dst && dst_hold_safe(dst)) {
>  		const struct rt6_info *rt = (const struct rt6_info *)dst;
>  
> -		sk->sk_rx_dst = dst;
> +		rcu_assign_pointer(sk->sk_rx_dst, dst);
>  		sk->sk_rx_dst_ifindex = skb->skb_iif;
>  		sk->sk_rx_dst_cookie = rt6_get_cookie(rt);
>  	}
> @@ -1505,7 +1505,10 @@ static int tcp_v6_do_rcv(struct sock *sk, struct sk_buff *skb)
>  		opt_skb = skb_clone(skb, sk_gfp_mask(sk, GFP_ATOMIC));
>  
>  	if (sk->sk_state == TCP_ESTABLISHED) { /* Fast path */
> -		struct dst_entry *dst = sk->sk_rx_dst;
> +		struct dst_entry *dst;
> +
> +		dst = rcu_dereference_protected(sk->sk_rx_dst,
> +						lockdep_sock_is_held(sk));
>  
>  		sock_rps_save_rxhash(sk, skb);
>  		sk_mark_napi_id(sk, skb);
> @@ -1513,8 +1516,8 @@ static int tcp_v6_do_rcv(struct sock *sk, struct sk_buff *skb)
>  			if (sk->sk_rx_dst_ifindex != skb->skb_iif ||
>  			    INDIRECT_CALL_1(dst->ops->check, ip6_dst_check,
>  					    dst, sk->sk_rx_dst_cookie) == NULL) {
> +				RCU_INIT_POINTER(sk->sk_rx_dst, NULL);
>  				dst_release(dst);
> -				sk->sk_rx_dst = NULL;
>  			}
>  		}
>  
> @@ -1874,7 +1877,7 @@ INDIRECT_CALLABLE_SCOPE void tcp_v6_early_demux(struct sk_buff *skb)
>  		skb->sk = sk;
>  		skb->destructor = sock_edemux;
>  		if (sk_fullsock(sk)) {
> -			struct dst_entry *dst = READ_ONCE(sk->sk_rx_dst);
> +			struct dst_entry *dst = rcu_dereference(sk->sk_rx_dst);
>  
>  			if (dst)
>  				dst = dst_check(dst, sk->sk_rx_dst_cookie);
> diff --git a/net/ipv6/udp.c b/net/ipv6/udp.c
> index e43b31d25fb61c7875f3bb8a93eb74da244d912a..a2caca6ccf114546f9e4ea854ad67208b2f3873e 100644
> --- a/net/ipv6/udp.c
> +++ b/net/ipv6/udp.c
> @@ -956,7 +956,7 @@ int __udp6_lib_rcv(struct sk_buff *skb, struct udp_table *udptable,
>  		struct dst_entry *dst = skb_dst(skb);
>  		int ret;
>  
> -		if (unlikely(sk->sk_rx_dst != dst))
> +		if (unlikely(rcu_dereference(sk->sk_rx_dst) != dst))
>  			udp6_sk_rx_dst_set(sk, dst);
>  
>  		if (!uh->check && !udp_sk(sk)->no_check6_rx) {
> @@ -1070,7 +1070,7 @@ INDIRECT_CALLABLE_SCOPE void udp_v6_early_demux(struct sk_buff *skb)
>  
>  	skb->sk = sk;
>  	skb->destructor = sock_efree;
> -	dst = READ_ONCE(sk->sk_rx_dst);
> +	dst = rcu_dereference(sk->sk_rx_dst);
>  
>  	if (dst)
>  		dst = dst_check(dst, sk->sk_rx_dst_cookie);
> -- 
> 2.34.1.173.g76aa8bc2d0-goog
> 
> 

Eric, this patch was picked for v5.15 stable and I wonder whether this
needs to be backported to older branches too. The Fixes commit quoted
here seems to go back all the way to v3.5. Would you know?

--
Carlos Llamas
Eric Dumazet Oct. 20, 2022, 9:53 p.m. UTC | #3
On Thu, Oct 20, 2022 at 2:05 PM Carlos Llamas <cmllamas@google.com> wrote:
>
> On Mon, Dec 20, 2021 at 06:33:30AM -0800, Eric Dumazet wrote:

>
> Eric, this patch was picked for v5.15 stable and I wonder whether this
> needs to be backported to older branches too. The Fixes commit quoted
> here seems to go back all the way to v3.5. Would you know?

I guess nobody cared to address some merge conflicts on older kernel versions.

If you want, you could handle the backport, this patch can help some
rare race windows
on preemptable kernels.

>
> --
> Carlos Llamas
Carlos Llamas Oct. 20, 2022, 10:55 p.m. UTC | #4
On Thu, Oct 20, 2022 at 02:53:06PM -0700, Eric Dumazet wrote:
> On Thu, Oct 20, 2022 at 2:05 PM Carlos Llamas <cmllamas@google.com> wrote:
> >
> > On Mon, Dec 20, 2021 at 06:33:30AM -0800, Eric Dumazet wrote:
> 
> >
> > Eric, this patch was picked for v5.15 stable and I wonder whether this
> > needs to be backported to older branches too. The Fixes commit quoted
> > here seems to go back all the way to v3.5. Would you know?
> 
> I guess nobody cared to address some merge conflicts on older kernel versions.
> 
> If you want, you could handle the backport, this patch can help some
> rare race windows
> on preemptable kernels.

Sounds good, I'll have a look at backporting this patch.
diff mbox series

Patch

diff --git a/include/net/sock.h b/include/net/sock.h
index bea21ff70e74d906216f4eaa2d5a712d12551216..d47e9658da28545c1f6afd9db0cf136b3e13d7b6 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -431,7 +431,7 @@  struct sock {
 #ifdef CONFIG_XFRM
 	struct xfrm_policy __rcu *sk_policy[2];
 #endif
-	struct dst_entry	*sk_rx_dst;
+	struct dst_entry __rcu	*sk_rx_dst;
 	int			sk_rx_dst_ifindex;
 	u32			sk_rx_dst_cookie;
 
diff --git a/net/ipv4/af_inet.c b/net/ipv4/af_inet.c
index 0189e3cd4a7df2dc2ea7121182ee290e0164df90..6b5956500436187d5e9b801ebb6f8f52ba0db090 100644
--- a/net/ipv4/af_inet.c
+++ b/net/ipv4/af_inet.c
@@ -154,7 +154,7 @@  void inet_sock_destruct(struct sock *sk)
 
 	kfree(rcu_dereference_protected(inet->inet_opt, 1));
 	dst_release(rcu_dereference_protected(sk->sk_dst_cache, 1));
-	dst_release(sk->sk_rx_dst);
+	dst_release(rcu_dereference_protected(sk->sk_rx_dst, 1));
 	sk_refcnt_debug_dec(sk);
 }
 EXPORT_SYMBOL(inet_sock_destruct);
diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index bbb3d39c69afc2d5a42c6ace8d473657861da61f..2bb28bfd83bf621b5d9f3fb7ce5695195697b43a 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -3012,8 +3012,7 @@  int tcp_disconnect(struct sock *sk, int flags)
 	icsk->icsk_ack.rcv_mss = TCP_MIN_MSS;
 	memset(&tp->rx_opt, 0, sizeof(tp->rx_opt));
 	__sk_dst_reset(sk);
-	dst_release(sk->sk_rx_dst);
-	sk->sk_rx_dst = NULL;
+	dst_release(xchg((__force struct dst_entry **)&sk->sk_rx_dst, NULL));
 	tcp_saved_syn_free(tp);
 	tp->compressed_ack = 0;
 	tp->segs_in = 0;
diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index 246ab7b5e857eb9e802c4805075e89c98cf00636..0ce46849ec3d4595699dd54229919b2d66b70257 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -5787,7 +5787,7 @@  void tcp_rcv_established(struct sock *sk, struct sk_buff *skb)
 	trace_tcp_probe(sk, skb);
 
 	tcp_mstamp_refresh(tp);
-	if (unlikely(!sk->sk_rx_dst))
+	if (unlikely(!rcu_access_pointer(sk->sk_rx_dst)))
 		inet_csk(sk)->icsk_af_ops->sk_rx_dst_set(sk, skb);
 	/*
 	 *	Header prediction.
diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c
index 13d868c43284584ee0c58ddfd411bb52c8b0c830..084df223b5dff8089a615a4a8d392b620fc0a28a 100644
--- a/net/ipv4/tcp_ipv4.c
+++ b/net/ipv4/tcp_ipv4.c
@@ -1701,7 +1701,10 @@  int tcp_v4_do_rcv(struct sock *sk, struct sk_buff *skb)
 	struct sock *rsk;
 
 	if (sk->sk_state == TCP_ESTABLISHED) { /* Fast path */
-		struct dst_entry *dst = sk->sk_rx_dst;
+		struct dst_entry *dst;
+
+		dst = rcu_dereference_protected(sk->sk_rx_dst,
+						lockdep_sock_is_held(sk));
 
 		sock_rps_save_rxhash(sk, skb);
 		sk_mark_napi_id(sk, skb);
@@ -1709,8 +1712,8 @@  int tcp_v4_do_rcv(struct sock *sk, struct sk_buff *skb)
 			if (sk->sk_rx_dst_ifindex != skb->skb_iif ||
 			    !INDIRECT_CALL_1(dst->ops->check, ipv4_dst_check,
 					     dst, 0)) {
+				RCU_INIT_POINTER(sk->sk_rx_dst, NULL);
 				dst_release(dst);
-				sk->sk_rx_dst = NULL;
 			}
 		}
 		tcp_rcv_established(sk, skb);
@@ -1786,7 +1789,7 @@  int tcp_v4_early_demux(struct sk_buff *skb)
 		skb->sk = sk;
 		skb->destructor = sock_edemux;
 		if (sk_fullsock(sk)) {
-			struct dst_entry *dst = READ_ONCE(sk->sk_rx_dst);
+			struct dst_entry *dst = rcu_dereference(sk->sk_rx_dst);
 
 			if (dst)
 				dst = dst_check(dst, 0);
@@ -2201,7 +2204,7 @@  void inet_sk_rx_dst_set(struct sock *sk, const struct sk_buff *skb)
 	struct dst_entry *dst = skb_dst(skb);
 
 	if (dst && dst_hold_safe(dst)) {
-		sk->sk_rx_dst = dst;
+		rcu_assign_pointer(sk->sk_rx_dst, dst);
 		sk->sk_rx_dst_ifindex = skb->skb_iif;
 	}
 }
diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c
index 23b05e28490b0a0a690a837027f26167e353f8ce..15c6b450b8dba44d5f344a554eea6b991c1ea5f1 100644
--- a/net/ipv4/udp.c
+++ b/net/ipv4/udp.c
@@ -2250,7 +2250,7 @@  bool udp_sk_rx_dst_set(struct sock *sk, struct dst_entry *dst)
 	struct dst_entry *old;
 
 	if (dst_hold_safe(dst)) {
-		old = xchg(&sk->sk_rx_dst, dst);
+		old = xchg((__force struct dst_entry **)&sk->sk_rx_dst, dst);
 		dst_release(old);
 		return old != dst;
 	}
@@ -2440,7 +2440,7 @@  int __udp4_lib_rcv(struct sk_buff *skb, struct udp_table *udptable,
 		struct dst_entry *dst = skb_dst(skb);
 		int ret;
 
-		if (unlikely(sk->sk_rx_dst != dst))
+		if (unlikely(rcu_dereference(sk->sk_rx_dst) != dst))
 			udp_sk_rx_dst_set(sk, dst);
 
 		ret = udp_unicast_rcv_skb(sk, skb, uh);
@@ -2599,7 +2599,7 @@  int udp_v4_early_demux(struct sk_buff *skb)
 
 	skb->sk = sk;
 	skb->destructor = sock_efree;
-	dst = READ_ONCE(sk->sk_rx_dst);
+	dst = rcu_dereference(sk->sk_rx_dst);
 
 	if (dst)
 		dst = dst_check(dst, 0);
diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c
index 551fce49841d7f53a111b0435855634cece2b40a..680e6481b9672040ccb41fd08ab80166575bef50 100644
--- a/net/ipv6/tcp_ipv6.c
+++ b/net/ipv6/tcp_ipv6.c
@@ -107,7 +107,7 @@  static void inet6_sk_rx_dst_set(struct sock *sk, const struct sk_buff *skb)
 	if (dst && dst_hold_safe(dst)) {
 		const struct rt6_info *rt = (const struct rt6_info *)dst;
 
-		sk->sk_rx_dst = dst;
+		rcu_assign_pointer(sk->sk_rx_dst, dst);
 		sk->sk_rx_dst_ifindex = skb->skb_iif;
 		sk->sk_rx_dst_cookie = rt6_get_cookie(rt);
 	}
@@ -1505,7 +1505,10 @@  static int tcp_v6_do_rcv(struct sock *sk, struct sk_buff *skb)
 		opt_skb = skb_clone(skb, sk_gfp_mask(sk, GFP_ATOMIC));
 
 	if (sk->sk_state == TCP_ESTABLISHED) { /* Fast path */
-		struct dst_entry *dst = sk->sk_rx_dst;
+		struct dst_entry *dst;
+
+		dst = rcu_dereference_protected(sk->sk_rx_dst,
+						lockdep_sock_is_held(sk));
 
 		sock_rps_save_rxhash(sk, skb);
 		sk_mark_napi_id(sk, skb);
@@ -1513,8 +1516,8 @@  static int tcp_v6_do_rcv(struct sock *sk, struct sk_buff *skb)
 			if (sk->sk_rx_dst_ifindex != skb->skb_iif ||
 			    INDIRECT_CALL_1(dst->ops->check, ip6_dst_check,
 					    dst, sk->sk_rx_dst_cookie) == NULL) {
+				RCU_INIT_POINTER(sk->sk_rx_dst, NULL);
 				dst_release(dst);
-				sk->sk_rx_dst = NULL;
 			}
 		}
 
@@ -1874,7 +1877,7 @@  INDIRECT_CALLABLE_SCOPE void tcp_v6_early_demux(struct sk_buff *skb)
 		skb->sk = sk;
 		skb->destructor = sock_edemux;
 		if (sk_fullsock(sk)) {
-			struct dst_entry *dst = READ_ONCE(sk->sk_rx_dst);
+			struct dst_entry *dst = rcu_dereference(sk->sk_rx_dst);
 
 			if (dst)
 				dst = dst_check(dst, sk->sk_rx_dst_cookie);
diff --git a/net/ipv6/udp.c b/net/ipv6/udp.c
index e43b31d25fb61c7875f3bb8a93eb74da244d912a..a2caca6ccf114546f9e4ea854ad67208b2f3873e 100644
--- a/net/ipv6/udp.c
+++ b/net/ipv6/udp.c
@@ -956,7 +956,7 @@  int __udp6_lib_rcv(struct sk_buff *skb, struct udp_table *udptable,
 		struct dst_entry *dst = skb_dst(skb);
 		int ret;
 
-		if (unlikely(sk->sk_rx_dst != dst))
+		if (unlikely(rcu_dereference(sk->sk_rx_dst) != dst))
 			udp6_sk_rx_dst_set(sk, dst);
 
 		if (!uh->check && !udp_sk(sk)->no_check6_rx) {
@@ -1070,7 +1070,7 @@  INDIRECT_CALLABLE_SCOPE void udp_v6_early_demux(struct sk_buff *skb)
 
 	skb->sk = sk;
 	skb->destructor = sock_efree;
-	dst = READ_ONCE(sk->sk_rx_dst);
+	dst = rcu_dereference(sk->sk_rx_dst);
 
 	if (dst)
 		dst = dst_check(dst, sk->sk_rx_dst_cookie);