| Message ID | 20210114115506.1713330-1-ivecera@redhat.com (mailing list archive) |
|---|---|
| State | Changes Requested |
| Delegated to: | Netdev Maintainers |
| Headers | show |
| Series | [net] team: fix deadlock during setting of MTU | expand |
| Context | Check | Description |
|---|---|---|
| netdev/cover_letter | success | Link |
| netdev/fixes_present | success | Link |
| netdev/patch_count | success | Link |
| netdev/tree_selection | success | Clearly marked for net |
| netdev/subject_prefix | success | Link |
| netdev/cc_maintainers | warning | 2 maintainers not CCed: kuba@kernel.org davem@davemloft.net |
| netdev/source_inline | success | Was 0 now: 0 |
| netdev/verify_signedoff | success | Link |
| netdev/module_param | success | Was 0 now: 0 |
| netdev/build_32bit | success | Errors and warnings before: 0 this patch: 0 |
| netdev/kdoc | success | Errors and warnings before: 0 this patch: 0 |
| netdev/verify_fixes | success | Link |
| netdev/checkpatch | success | total: 0 errors, 0 warnings, 0 checks, 45 lines checked |
| netdev/build_allmodconfig_warn | success | Errors and warnings before: 0 this patch: 0 |
| netdev/header_inline | success | Link |
| netdev/stable | success | Stable not CCed |
On Thu, 2021-01-14 at 12:55 +0100, Ivan Vecera wrote: > Team driver protects port list traversal in team_change_mtu() > by its team->lock mutex. This causes a deadlock with certain > devices that calls netdev_update_features() during their > .ndo_change_mtu() callback. In this case netdev_update_features() > calls team's netdevice notifier team_device_event() that in case > of NETDEV_FEAT_CHANGE tries lock team->lock mutex again. > > Example (r8169 case): > ... > [ 6391.348202] __mutex_lock.isra.6+0x2d0/0x4a0 > [ 6391.358602] team_device_event+0x9d/0x160 [team] > [ 6391.363756] notifier_call_chain+0x47/0x70 > [ 6391.368329] netdev_update_features+0x56/0x60 > [ 6391.373207] rtl8169_change_mtu+0x14/0x50 [r8169] > [ 6391.378457] dev_set_mtu_ext+0xe1/0x1d0 > [ 6391.387022] dev_set_mtu+0x52/0x90 > [ 6391.390820] team_change_mtu+0x64/0xf0 [team] > [ 6391.395683] dev_set_mtu_ext+0xe1/0x1d0 > [ 6391.399963] do_setlink+0x231/0xf50 > ... > > To fix the problem the port list traversal in team_change_mtu() can > be protected by RCU read lock. In case of failure the failing port > is marked and unwind code-path is done also under RCU read lock > protection (but not in reverse order). > > Fixes: 3d249d4ca7d0 ("net: introduce ethernet teaming device") > Cc: Jiri Pirko <jiri@resnulli.us> > Signed-off-by: Ivan Vecera <ivecera@redhat.com> > --- > drivers/net/team/team.c | 20 ++++++++++---------- > 1 file changed, 10 insertions(+), 10 deletions(-) > > diff --git a/drivers/net/team/team.c b/drivers/net/team/team.c > index c19dac21c468..69d4b28beb17 100644 > --- a/drivers/net/team/team.c > +++ b/drivers/net/team/team.c > @@ -1802,35 +1802,35 @@ static int team_set_mac_address(struct > net_device *dev, void *p) > static int team_change_mtu(struct net_device *dev, int new_mtu) > { > struct team *team = netdev_priv(dev); > - struct team_port *port; > + struct team_port *port, *fail_port; > int err; > > - /* > - * Alhough this is reader, it's guarded by team lock. It's not > possible > - * to traverse list in reverse under rcu_read_lock > - */ > - mutex_lock(&team->lock); > + rcu_read_lock(); > team->port_mtu_change_allowed = true; > - list_for_each_entry(port, &team->port_list, list) { > + list_for_each_entry_rcu(port, &team->port_list, list) { > err = dev_set_mtu(port->dev, new_mtu); some netdevs will need to sleep in their set_mtu, and you can't sleep under rcu! according to your explanation in the commit message the team->lock mutex will be also taken under this rcu lock, so this is bad even if dev_set_mtu does not sleep.
On Thu, 14 Jan 2021 16:34:24 -0800 Saeed Mahameed <saeed@kernel.org> wrote: > On Thu, 2021-01-14 at 12:55 +0100, Ivan Vecera wrote: > > Team driver protects port list traversal in team_change_mtu() > > by its team->lock mutex. This causes a deadlock with certain > > devices that calls netdev_update_features() during their > > .ndo_change_mtu() callback. In this case netdev_update_features() > > calls team's netdevice notifier team_device_event() that in case > > of NETDEV_FEAT_CHANGE tries lock team->lock mutex again. > > > > Example (r8169 case): > > ... > > [ 6391.348202] __mutex_lock.isra.6+0x2d0/0x4a0 > > [ 6391.358602] team_device_event+0x9d/0x160 [team] > > [ 6391.363756] notifier_call_chain+0x47/0x70 > > [ 6391.368329] netdev_update_features+0x56/0x60 > > [ 6391.373207] rtl8169_change_mtu+0x14/0x50 [r8169] > > [ 6391.378457] dev_set_mtu_ext+0xe1/0x1d0 > > [ 6391.387022] dev_set_mtu+0x52/0x90 > > [ 6391.390820] team_change_mtu+0x64/0xf0 [team] > > [ 6391.395683] dev_set_mtu_ext+0xe1/0x1d0 > > [ 6391.399963] do_setlink+0x231/0xf50 > > ... > > > > To fix the problem the port list traversal in team_change_mtu() can > > be protected by RCU read lock. In case of failure the failing port > > is marked and unwind code-path is done also under RCU read lock > > protection (but not in reverse order). > > > > Fixes: 3d249d4ca7d0 ("net: introduce ethernet teaming device") > > Cc: Jiri Pirko <jiri@resnulli.us> > > Signed-off-by: Ivan Vecera <ivecera@redhat.com> > > --- > > drivers/net/team/team.c | 20 ++++++++++---------- > > 1 file changed, 10 insertions(+), 10 deletions(-) > > > > diff --git a/drivers/net/team/team.c b/drivers/net/team/team.c > > index c19dac21c468..69d4b28beb17 100644 > > --- a/drivers/net/team/team.c > > +++ b/drivers/net/team/team.c > > @@ -1802,35 +1802,35 @@ static int team_set_mac_address(struct > > net_device *dev, void *p) > > static int team_change_mtu(struct net_device *dev, int new_mtu) > > { > > struct team *team = netdev_priv(dev); > > - struct team_port *port; > > + struct team_port *port, *fail_port; > > int err; > > > > - /* > > - * Alhough this is reader, it's guarded by team lock. It's not > > possible > > - * to traverse list in reverse under rcu_read_lock > > - */ > > - mutex_lock(&team->lock); > > + rcu_read_lock(); > > team->port_mtu_change_allowed = true; > > - list_for_each_entry(port, &team->port_list, list) { > > + list_for_each_entry_rcu(port, &team->port_list, list) { > > err = dev_set_mtu(port->dev, new_mtu); > > some netdevs will need to sleep in their set_mtu, and you can't sleep > under rcu! > > according to your explanation in the commit message the team->lock > mutex will be also taken under this rcu lock, so this is bad even > if dev_set_mtu does not sleep. > Hmm, you are right... btw do we need to take this mutex at this place? team_change_mtu() is protected by RTNL, team_device_event() as a netdevice notifier as well... and team_{add,del}_slave() that modify port list also. Thoughts? Thanks, Ivan
On Fri, 15 Jan 2021 09:24:05 +0100 Ivan Vecera <ivecera@redhat.com> wrote: > > according to your explanation in the commit message the team->lock > > mutex will be also taken under this rcu lock, so this is bad even > > if dev_set_mtu does not sleep. > > > Hmm, you are right... btw do we need to take this mutex at this place? > > team_change_mtu() is protected by RTNL, team_device_event() as a netdevice > notifier as well... and team_{add,del}_slave() that modify port list > also. > > Thoughts? After private discussion with Jiri, this is no-go as he would not like to introduce a dependency on RTNL into team driver. The other solution is to postpone features calculation in certain cases... the patch follows. Ivan
diff --git a/drivers/net/team/team.c b/drivers/net/team/team.c index c19dac21c468..69d4b28beb17 100644 --- a/drivers/net/team/team.c +++ b/drivers/net/team/team.c @@ -1802,35 +1802,35 @@ static int team_set_mac_address(struct net_device *dev, void *p) static int team_change_mtu(struct net_device *dev, int new_mtu) { struct team *team = netdev_priv(dev); - struct team_port *port; + struct team_port *port, *fail_port; int err; - /* - * Alhough this is reader, it's guarded by team lock. It's not possible - * to traverse list in reverse under rcu_read_lock - */ - mutex_lock(&team->lock); + rcu_read_lock(); team->port_mtu_change_allowed = true; - list_for_each_entry(port, &team->port_list, list) { + list_for_each_entry_rcu(port, &team->port_list, list) { err = dev_set_mtu(port->dev, new_mtu); if (err) { netdev_err(dev, "Device %s failed to change mtu", port->dev->name); + fail_port = port; goto unwind; } } team->port_mtu_change_allowed = false; - mutex_unlock(&team->lock); + rcu_read_unlock(); dev->mtu = new_mtu; return 0; unwind: - list_for_each_entry_continue_reverse(port, &team->port_list, list) + list_for_each_entry_rcu(port, &team->port_list, list) { + if (port == fail_port) + break; dev_set_mtu(port->dev, dev->mtu); + } team->port_mtu_change_allowed = false; - mutex_unlock(&team->lock); + rcu_read_unlock(); return err; }
Team driver protects port list traversal in team_change_mtu() by its team->lock mutex. This causes a deadlock with certain devices that calls netdev_update_features() during their .ndo_change_mtu() callback. In this case netdev_update_features() calls team's netdevice notifier team_device_event() that in case of NETDEV_FEAT_CHANGE tries lock team->lock mutex again. Example (r8169 case): ... [ 6391.348202] __mutex_lock.isra.6+0x2d0/0x4a0 [ 6391.358602] team_device_event+0x9d/0x160 [team] [ 6391.363756] notifier_call_chain+0x47/0x70 [ 6391.368329] netdev_update_features+0x56/0x60 [ 6391.373207] rtl8169_change_mtu+0x14/0x50 [r8169] [ 6391.378457] dev_set_mtu_ext+0xe1/0x1d0 [ 6391.387022] dev_set_mtu+0x52/0x90 [ 6391.390820] team_change_mtu+0x64/0xf0 [team] [ 6391.395683] dev_set_mtu_ext+0xe1/0x1d0 [ 6391.399963] do_setlink+0x231/0xf50 ... To fix the problem the port list traversal in team_change_mtu() can be protected by RCU read lock. In case of failure the failing port is marked and unwind code-path is done also under RCU read lock protection (but not in reverse order). Fixes: 3d249d4ca7d0 ("net: introduce ethernet teaming device") Cc: Jiri Pirko <jiri@resnulli.us> Signed-off-by: Ivan Vecera <ivecera@redhat.com> --- drivers/net/team/team.c | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)