| Message ID | 20230414083526.1984362-1-liuhangbin@gmail.com (mailing list archive) |
|---|---|
| State | Superseded |
| Delegated to: | Netdev Maintainers |
| Headers | show |
| Series | [PATCHv4,net-next] bonding: add software tx timestamping support | expand |
Hangbin Liu <liuhangbin@gmail.com> wrote: >Currently, bonding only obtain the timestamp (ts) information of >the active slave, which is available only for modes 1, 5, and 6. >For other modes, bonding only has software rx timestamping support. > >However, some users who use modes such as LACP also want tx timestamp >support. To address this issue, let's check the ts information of each >slave. If all slaves support tx timestamping, we can enable tx >timestamping support for the bond. > >Suggested-by: Miroslav Lichvar <mlichvar@redhat.com> >Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> Acked-by: Jay Vosburgh <jay.vosburgh@canonical.com> -J >--- >v4: add ASSERT_RTNL to make sure bond_ethtool_get_ts_info() called via > RTNL. Only check _TX_SOFTWARE for the slaves. >v3: remove dev_hold/dev_put. remove the '\' for line continuation. >v2: check each slave's ts info to make sure bond support sw tx > timestamping >--- > drivers/net/bonding/bond_main.c | 32 ++++++++++++++++++++++++++++++++ > 1 file changed, 32 insertions(+) > >diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c >index 00646aa315c3..9cf49b61f4b3 100644 >--- a/drivers/net/bonding/bond_main.c >+++ b/drivers/net/bonding/bond_main.c >@@ -5686,11 +5686,17 @@ static int bond_ethtool_get_ts_info(struct net_device *bond_dev, > struct ethtool_ts_info *info) > { > struct bonding *bond = netdev_priv(bond_dev); >+ struct ethtool_ts_info ts_info; > const struct ethtool_ops *ops; > struct net_device *real_dev; >+ bool sw_tx_support = false; > struct phy_device *phydev; >+ struct list_head *iter; >+ struct slave *slave; > int ret = 0; > >+ ASSERT_RTNL(); >+ > rcu_read_lock(); > real_dev = bond_option_active_slave_get_rcu(bond); > dev_hold(real_dev); >@@ -5707,10 +5713,36 @@ static int bond_ethtool_get_ts_info(struct net_device *bond_dev, > ret = ops->get_ts_info(real_dev, info); > goto out; > } >+ } else { >+ /* Check if all slaves support software tx timestamping */ >+ rcu_read_lock(); >+ bond_for_each_slave_rcu(bond, slave, iter) { >+ ret = -1; >+ ops = slave->dev->ethtool_ops; >+ phydev = slave->dev->phydev; >+ >+ if (phy_has_tsinfo(phydev)) >+ ret = phy_ts_info(phydev, &ts_info); >+ else if (ops->get_ts_info) >+ ret = ops->get_ts_info(slave->dev, &ts_info); >+ >+ if (!ret && (ts_info.so_timestamping & SOF_TIMESTAMPING_TX_SOFTWARE)) { >+ sw_tx_support = true; >+ continue; >+ } >+ >+ sw_tx_support = false; >+ break; >+ } >+ rcu_read_unlock(); > } > >+ ret = 0; > info->so_timestamping = SOF_TIMESTAMPING_RX_SOFTWARE | > SOF_TIMESTAMPING_SOFTWARE; >+ if (sw_tx_support) >+ info->so_timestamping |= SOF_TIMESTAMPING_TX_SOFTWARE; >+ > info->phc_index = -1; > > out: >-- >2.38.1 >
On Fri, 14 Apr 2023 16:35:26 +0800 Hangbin Liu wrote: > v4: add ASSERT_RTNL to make sure bond_ethtool_get_ts_info() called via > RTNL. Only check _TX_SOFTWARE for the slaves. > + ASSERT_RTNL(); > + > rcu_read_lock(); > real_dev = bond_option_active_slave_get_rcu(bond); > dev_hold(real_dev); > @@ -5707,10 +5713,36 @@ static int bond_ethtool_get_ts_info(struct net_device *bond_dev, > ret = ops->get_ts_info(real_dev, info); > goto out; > } > + } else { > + /* Check if all slaves support software tx timestamping */ > + rcu_read_lock(); > + bond_for_each_slave_rcu(bond, slave, iter) { > + ret = -1; > + ops = slave->dev->ethtool_ops; > + phydev = slave->dev->phydev; > + > + if (phy_has_tsinfo(phydev)) > + ret = phy_ts_info(phydev, &ts_info); > + else if (ops->get_ts_info) > + ret = ops->get_ts_info(slave->dev, &ts_info); My comment about this path being under rtnl was to point out that we don't need the RCU protection to iterate over the slaves. This is a bit of a guess, I don't know bonding, but can we not use bond_for_each_slave() ? As a general rule we should let all driver callbacks sleep. Drivers may need to consult the FW or read something over a slow asynchronous bus which requires process / non-atomic context. RCU lock puts us in an atomic context. And ->get_ts_info() is a driver callback. It's not a deal breaker if we can't avoid RCU, but if we can - we should let the drivers sleep. Sorry if I wasn't very clear previously.
Jakub Kicinski <kuba@kernel.org> wrote: >On Fri, 14 Apr 2023 16:35:26 +0800 Hangbin Liu wrote: >> v4: add ASSERT_RTNL to make sure bond_ethtool_get_ts_info() called via >> RTNL. Only check _TX_SOFTWARE for the slaves. > >> + ASSERT_RTNL(); >> + >> rcu_read_lock(); >> real_dev = bond_option_active_slave_get_rcu(bond); >> dev_hold(real_dev); >> @@ -5707,10 +5713,36 @@ static int bond_ethtool_get_ts_info(struct net_device *bond_dev, >> ret = ops->get_ts_info(real_dev, info); >> goto out; >> } >> + } else { >> + /* Check if all slaves support software tx timestamping */ >> + rcu_read_lock(); >> + bond_for_each_slave_rcu(bond, slave, iter) { > >> + ret = -1; >> + ops = slave->dev->ethtool_ops; >> + phydev = slave->dev->phydev; >> + >> + if (phy_has_tsinfo(phydev)) >> + ret = phy_ts_info(phydev, &ts_info); >> + else if (ops->get_ts_info) >> + ret = ops->get_ts_info(slave->dev, &ts_info); > >My comment about this path being under rtnl was to point out that we >don't need the RCU protection to iterate over the slaves. This is >a bit of a guess, I don't know bonding, but can we not use >bond_for_each_slave() ? Ah, I missed that nuance. And, yes, you're correct, bond_for_each_slave() works with RTNL and we don't need RCU here if RTNL is held. >As a general rule we should let all driver callbacks sleep. Drivers >may need to consult the FW or read something over a slow asynchronous >bus which requires process / non-atomic context. RCU lock puts us in >an atomic context. And ->get_ts_info() is a driver callback. Agreed. >It's not a deal breaker if we can't avoid RCU, but if we can - we should >let the drivers sleep. Sorry if I wasn't very clear previously. Understood; I should have remembered this, as it's been an issue arising from the "in the middle" aspect of bonding in the past. -J --- -Jay Vosburgh, jay.vosburgh@canonical.com
On Fri, Apr 14, 2023 at 06:02:05PM -0700, Jakub Kicinski wrote: > On Fri, 14 Apr 2023 16:35:26 +0800 Hangbin Liu wrote: > > v4: add ASSERT_RTNL to make sure bond_ethtool_get_ts_info() called via > > RTNL. Only check _TX_SOFTWARE for the slaves. > > > + ASSERT_RTNL(); > > + > > rcu_read_lock(); > > real_dev = bond_option_active_slave_get_rcu(bond); > > dev_hold(real_dev); > > @@ -5707,10 +5713,36 @@ static int bond_ethtool_get_ts_info(struct net_device *bond_dev, > > ret = ops->get_ts_info(real_dev, info); > > goto out; > > } > > + } else { > > + /* Check if all slaves support software tx timestamping */ > > + rcu_read_lock(); > > + bond_for_each_slave_rcu(bond, slave, iter) { > > > + ret = -1; > > + ops = slave->dev->ethtool_ops; > > + phydev = slave->dev->phydev; > > + > > + if (phy_has_tsinfo(phydev)) > > + ret = phy_ts_info(phydev, &ts_info); > > + else if (ops->get_ts_info) > > + ret = ops->get_ts_info(slave->dev, &ts_info); > > My comment about this path being under rtnl was to point out that we > don't need the RCU protection to iterate over the slaves. This is > a bit of a guess, I don't know bonding, but can we not use > bond_for_each_slave() ? Sorry, I misunderstood your comment in patchv3. I thought you agreed to use the same use case like rlb_next_rx_slave(). I will post a new fix version. > > As a general rule we should let all driver callbacks sleep. Drivers > may need to consult the FW or read something over a slow asynchronous > bus which requires process / non-atomic context. RCU lock puts us in > an atomic context. And ->get_ts_info() is a driver callback. > > It's not a deal breaker if we can't avoid RCU, but if we can - we should > let the drivers sleep. Sorry if I wasn't very clear previously. Thanks for the explanation. Regards Hangbin
On Fri, Apr 14, 2023 at 08:43:14PM -0700, Jay Vosburgh wrote: > Jakub Kicinski <kuba@kernel.org> wrote: > > >On Fri, 14 Apr 2023 16:35:26 +0800 Hangbin Liu wrote: > >> v4: add ASSERT_RTNL to make sure bond_ethtool_get_ts_info() called via > >> RTNL. Only check _TX_SOFTWARE for the slaves. > > > >> + ASSERT_RTNL(); > >> + > >> rcu_read_lock(); > >> real_dev = bond_option_active_slave_get_rcu(bond); > >> dev_hold(real_dev); > >> @@ -5707,10 +5713,36 @@ static int bond_ethtool_get_ts_info(struct net_device *bond_dev, > >> ret = ops->get_ts_info(real_dev, info); > >> goto out; > >> } > >> + } else { > >> + /* Check if all slaves support software tx timestamping */ > >> + rcu_read_lock(); > >> + bond_for_each_slave_rcu(bond, slave, iter) { > > > >> + ret = -1; > >> + ops = slave->dev->ethtool_ops; > >> + phydev = slave->dev->phydev; > >> + > >> + if (phy_has_tsinfo(phydev)) > >> + ret = phy_ts_info(phydev, &ts_info); > >> + else if (ops->get_ts_info) > >> + ret = ops->get_ts_info(slave->dev, &ts_info); > > > >My comment about this path being under rtnl was to point out that we > >don't need the RCU protection to iterate over the slaves. This is > >a bit of a guess, I don't know bonding, but can we not use > >bond_for_each_slave() ? > > Ah, I missed that nuance. And, yes, you're correct, > bond_for_each_slave() works with RTNL and we don't need RCU here if RTNL > is held. Hi Jay, Jakub, I remember why I use bond_for_each_slave_rcu() here now. In commit 9b80ccda233f ("bonding: fix missed rcu protection"), I added the rcu_read_lock() as syzbot reported[1] the following path doesn't hold rtnl lock. - sock_setsockopt - sock_set_timestamping - sock_timestamping_bind_phc - ethtool_get_phc_vclocks - __ethtool_get_ts_info - bond_ethtool_get_ts_info [1] https://lore.kernel.org/netdev/20220513084819.zrg4ssnw667rhndt@skbuf/T/ Thanks Hangbin
On Mon, 17 Apr 2023 08:17:39 +0800 Hangbin Liu wrote: > I remember why I use bond_for_each_slave_rcu() here now. In commit > 9b80ccda233f ("bonding: fix missed rcu protection"), I added the > rcu_read_lock() as syzbot reported[1] the following path doesn't hold > rtnl lock. > - sock_setsockopt > - sock_set_timestamping > - sock_timestamping_bind_phc > - ethtool_get_phc_vclocks > - __ethtool_get_ts_info > - bond_ethtool_get_ts_info Well spotted, okay :( Could you respin with this info added to the commit message and an update to the kdoc in include/linux/ethtool.h for @get_ts_info that it may be called with RCU, or rtnl or reference on the device?
diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c index 00646aa315c3..9cf49b61f4b3 100644 --- a/drivers/net/bonding/bond_main.c +++ b/drivers/net/bonding/bond_main.c @@ -5686,11 +5686,17 @@ static int bond_ethtool_get_ts_info(struct net_device *bond_dev, struct ethtool_ts_info *info) { struct bonding *bond = netdev_priv(bond_dev); + struct ethtool_ts_info ts_info; const struct ethtool_ops *ops; struct net_device *real_dev; + bool sw_tx_support = false; struct phy_device *phydev; + struct list_head *iter; + struct slave *slave; int ret = 0; + ASSERT_RTNL(); + rcu_read_lock(); real_dev = bond_option_active_slave_get_rcu(bond); dev_hold(real_dev); @@ -5707,10 +5713,36 @@ static int bond_ethtool_get_ts_info(struct net_device *bond_dev, ret = ops->get_ts_info(real_dev, info); goto out; } + } else { + /* Check if all slaves support software tx timestamping */ + rcu_read_lock(); + bond_for_each_slave_rcu(bond, slave, iter) { + ret = -1; + ops = slave->dev->ethtool_ops; + phydev = slave->dev->phydev; + + if (phy_has_tsinfo(phydev)) + ret = phy_ts_info(phydev, &ts_info); + else if (ops->get_ts_info) + ret = ops->get_ts_info(slave->dev, &ts_info); + + if (!ret && (ts_info.so_timestamping & SOF_TIMESTAMPING_TX_SOFTWARE)) { + sw_tx_support = true; + continue; + } + + sw_tx_support = false; + break; + } + rcu_read_unlock(); } + ret = 0; info->so_timestamping = SOF_TIMESTAMPING_RX_SOFTWARE | SOF_TIMESTAMPING_SOFTWARE; + if (sw_tx_support) + info->so_timestamping |= SOF_TIMESTAMPING_TX_SOFTWARE; + info->phc_index = -1; out:
Currently, bonding only obtain the timestamp (ts) information of the active slave, which is available only for modes 1, 5, and 6. For other modes, bonding only has software rx timestamping support. However, some users who use modes such as LACP also want tx timestamp support. To address this issue, let's check the ts information of each slave. If all slaves support tx timestamping, we can enable tx timestamping support for the bond. Suggested-by: Miroslav Lichvar <mlichvar@redhat.com> Signed-off-by: Hangbin Liu <liuhangbin@gmail.com> --- v4: add ASSERT_RTNL to make sure bond_ethtool_get_ts_info() called via RTNL. Only check _TX_SOFTWARE for the slaves. v3: remove dev_hold/dev_put. remove the '\' for line continuation. v2: check each slave's ts info to make sure bond support sw tx timestamping --- drivers/net/bonding/bond_main.c | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+)