| Message ID | 604c45c188c609b732286b47ac2a451a40f6cf6d.1730828007.git.pabeni@redhat.com (mailing list archive) |
|---|---|
| State | New |
| Headers | show |
| Series | ipv6: fix hangup on device removal | expand |
On Tue, Nov 5, 2024 at 7:24 PM Paolo Abeni <pabeni@redhat.com> wrote: > > The CI is hitting some aperiodic hangup at device removal time in the > pmtu.sh self-test: > > unregister_netdevice: waiting for veth_A-R1 to become free. Usage count = 6 > ref_tracker: veth_A-R1@ffff888013df15d8 has 1/5 users at > dst_init+0x84/0x4a0 > dst_alloc+0x97/0x150 > ip6_dst_alloc+0x23/0x90 > ip6_rt_pcpu_alloc+0x1e6/0x520 > ip6_pol_route+0x56f/0x840 > fib6_rule_lookup+0x334/0x630 > ip6_route_output_flags+0x259/0x480 > ip6_dst_lookup_tail.constprop.0+0x5c2/0x940 > ip6_dst_lookup_flow+0x88/0x190 > udp_tunnel6_dst_lookup+0x2a7/0x4c0 > vxlan_xmit_one+0xbde/0x4a50 [vxlan] > vxlan_xmit+0x9ad/0xf20 [vxlan] > dev_hard_start_xmit+0x10e/0x360 > __dev_queue_xmit+0xf95/0x18c0 > arp_solicit+0x4a2/0xe00 > neigh_probe+0xaa/0xf0 > > While the first suspect is the dst_cache, explicitly tracking the dst > owing the last device reference via probes proved such dst is held by > the nexthop in the originating fib6_info. > > Similar to commit f5b51fe804ec ("ipv6: route: purge exception on > removal"), we need to explicitly release the originating fib info when > disconnecting a to-be-removed device from a live ipv6 dst: move the > fib6_info cleanup into ip6_dst_ifdown(). > > Tested running: > > ./pmtu.sh cleanup_ipv6_exception > > in a tight loop for more than 400 iterations with no spat, running an > unpatched kernel I observed a splat every ~10 iterations. > > Fixes: f88d8ea67fbd ("ipv6: Plumb support for nexthop object in a fib6_info") > Signed-off-by: Paolo Abeni <pabeni@redhat.com> Thanks a lot Paolo Reviewed-by: Eric Dumazet <edumazet@google.com>
On 11/5/24 11:23 AM, Paolo Abeni wrote: > The CI is hitting some aperiodic hangup at device removal time in the > pmtu.sh self-test: > > unregister_netdevice: waiting for veth_A-R1 to become free. Usage count = 6 > ref_tracker: veth_A-R1@ffff888013df15d8 has 1/5 users at > dst_init+0x84/0x4a0 > dst_alloc+0x97/0x150 > ip6_dst_alloc+0x23/0x90 > ip6_rt_pcpu_alloc+0x1e6/0x520 > ip6_pol_route+0x56f/0x840 > fib6_rule_lookup+0x334/0x630 > ip6_route_output_flags+0x259/0x480 > ip6_dst_lookup_tail.constprop.0+0x5c2/0x940 > ip6_dst_lookup_flow+0x88/0x190 > udp_tunnel6_dst_lookup+0x2a7/0x4c0 > vxlan_xmit_one+0xbde/0x4a50 [vxlan] > vxlan_xmit+0x9ad/0xf20 [vxlan] > dev_hard_start_xmit+0x10e/0x360 > __dev_queue_xmit+0xf95/0x18c0 > arp_solicit+0x4a2/0xe00 > neigh_probe+0xaa/0xf0 > > While the first suspect is the dst_cache, explicitly tracking the dst > owing the last device reference via probes proved such dst is held by > the nexthop in the originating fib6_info. > > Similar to commit f5b51fe804ec ("ipv6: route: purge exception on > removal"), we need to explicitly release the originating fib info when > disconnecting a to-be-removed device from a live ipv6 dst: move the > fib6_info cleanup into ip6_dst_ifdown(). > > Tested running: > > ./pmtu.sh cleanup_ipv6_exception > > in a tight loop for more than 400 iterations with no spat, running an > unpatched kernel I observed a splat every ~10 iterations. > > Fixes: f88d8ea67fbd ("ipv6: Plumb support for nexthop object in a fib6_info") are you sure that is the correct Fixes? That commit is June 2019 and there have been stable periods since then without netdev release problems. > Signed-off-by: Paolo Abeni <pabeni@redhat.com> > --- > v1 -> v2: > - dropped unintended whitespace change > --- > net/ipv6/route.c | 6 +++--- > 1 file changed, 3 insertions(+), 3 deletions(-) > Reviewed-by: David Ahern <dsahern@kernel.org>
On 11/5/24 22:40, David Ahern wrote: > On 11/5/24 11:23 AM, Paolo Abeni wrote: >> The CI is hitting some aperiodic hangup at device removal time in the >> pmtu.sh self-test: >> >> unregister_netdevice: waiting for veth_A-R1 to become free. Usage count = 6 >> ref_tracker: veth_A-R1@ffff888013df15d8 has 1/5 users at >> dst_init+0x84/0x4a0 >> dst_alloc+0x97/0x150 >> ip6_dst_alloc+0x23/0x90 >> ip6_rt_pcpu_alloc+0x1e6/0x520 >> ip6_pol_route+0x56f/0x840 >> fib6_rule_lookup+0x334/0x630 >> ip6_route_output_flags+0x259/0x480 >> ip6_dst_lookup_tail.constprop.0+0x5c2/0x940 >> ip6_dst_lookup_flow+0x88/0x190 >> udp_tunnel6_dst_lookup+0x2a7/0x4c0 >> vxlan_xmit_one+0xbde/0x4a50 [vxlan] >> vxlan_xmit+0x9ad/0xf20 [vxlan] >> dev_hard_start_xmit+0x10e/0x360 >> __dev_queue_xmit+0xf95/0x18c0 >> arp_solicit+0x4a2/0xe00 >> neigh_probe+0xaa/0xf0 >> >> While the first suspect is the dst_cache, explicitly tracking the dst >> owing the last device reference via probes proved such dst is held by >> the nexthop in the originating fib6_info. >> >> Similar to commit f5b51fe804ec ("ipv6: route: purge exception on >> removal"), we need to explicitly release the originating fib info when >> disconnecting a to-be-removed device from a live ipv6 dst: move the >> fib6_info cleanup into ip6_dst_ifdown(). >> >> Tested running: >> >> ./pmtu.sh cleanup_ipv6_exception >> >> in a tight loop for more than 400 iterations with no spat, running an >> unpatched kernel I observed a splat every ~10 iterations. >> >> Fixes: f88d8ea67fbd ("ipv6: Plumb support for nexthop object in a fib6_info") > > are you sure that is the correct Fixes? That commit is June 2019 and > there have been stable periods since then without netdev release problems. "Sure" is a big word ;) AFAICS the mentioned commit let fib6_info store indirectly the extra dev reference via nexthop and does not clean it at device removal time. Note that the issue is not deterministic - I needed ~30 mptu.sh iterations in a row to see it, so it could go unnoticed for a long time. >> Signed-off-by: Paolo Abeni <pabeni@redhat.com> >> --- >> v1 -> v2: >> - dropped unintended whitespace change >> --- >> net/ipv6/route.c | 6 +++--- >> 1 file changed, 3 insertions(+), 3 deletions(-) >> > > Reviewed-by: David Ahern <dsahern@kernel.org> Thanks! Paolo
diff --git a/net/ipv6/route.c b/net/ipv6/route.c index d7ce5cf2017a..038c1eeef0be 100644 --- a/net/ipv6/route.c +++ b/net/ipv6/route.c @@ -374,6 +374,7 @@ static void ip6_dst_ifdown(struct dst_entry *dst, struct net_device *dev) { struct rt6_info *rt = dst_rt6_info(dst); struct inet6_dev *idev = rt->rt6i_idev; + struct fib6_info *from; if (idev && idev->dev != blackhole_netdev) { struct inet6_dev *blackhole_idev = in6_dev_get(blackhole_netdev); @@ -383,6 +384,8 @@ static void ip6_dst_ifdown(struct dst_entry *dst, struct net_device *dev) in6_dev_put(idev); } } + from = unrcu_pointer(xchg(&rt->from, NULL)); + fib6_info_release(from); } static bool __rt6_check_expired(const struct rt6_info *rt) @@ -1455,7 +1458,6 @@ static DEFINE_SPINLOCK(rt6_exception_lock); static void rt6_remove_exception(struct rt6_exception_bucket *bucket, struct rt6_exception *rt6_ex) { - struct fib6_info *from; struct net *net; if (!bucket || !rt6_ex) @@ -1467,8 +1469,6 @@ static void rt6_remove_exception(struct rt6_exception_bucket *bucket, /* purge completely the exception to allow releasing the held resources: * some [sk] cache may keep the dst around for unlimited time */ - from = unrcu_pointer(xchg(&rt6_ex->rt6i->from, NULL)); - fib6_info_release(from); dst_dev_put(&rt6_ex->rt6i->dst); hlist_del_rcu(&rt6_ex->hlist);
The CI is hitting some aperiodic hangup at device removal time in the pmtu.sh self-test: unregister_netdevice: waiting for veth_A-R1 to become free. Usage count = 6 ref_tracker: veth_A-R1@ffff888013df15d8 has 1/5 users at dst_init+0x84/0x4a0 dst_alloc+0x97/0x150 ip6_dst_alloc+0x23/0x90 ip6_rt_pcpu_alloc+0x1e6/0x520 ip6_pol_route+0x56f/0x840 fib6_rule_lookup+0x334/0x630 ip6_route_output_flags+0x259/0x480 ip6_dst_lookup_tail.constprop.0+0x5c2/0x940 ip6_dst_lookup_flow+0x88/0x190 udp_tunnel6_dst_lookup+0x2a7/0x4c0 vxlan_xmit_one+0xbde/0x4a50 [vxlan] vxlan_xmit+0x9ad/0xf20 [vxlan] dev_hard_start_xmit+0x10e/0x360 __dev_queue_xmit+0xf95/0x18c0 arp_solicit+0x4a2/0xe00 neigh_probe+0xaa/0xf0 While the first suspect is the dst_cache, explicitly tracking the dst owing the last device reference via probes proved such dst is held by the nexthop in the originating fib6_info. Similar to commit f5b51fe804ec ("ipv6: route: purge exception on removal"), we need to explicitly release the originating fib info when disconnecting a to-be-removed device from a live ipv6 dst: move the fib6_info cleanup into ip6_dst_ifdown(). Tested running: ./pmtu.sh cleanup_ipv6_exception in a tight loop for more than 400 iterations with no spat, running an unpatched kernel I observed a splat every ~10 iterations. Fixes: f88d8ea67fbd ("ipv6: Plumb support for nexthop object in a fib6_info") Signed-off-by: Paolo Abeni <pabeni@redhat.com> --- v1 -> v2: - dropped unintended whitespace change --- net/ipv6/route.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)