Message ID | 20211123132116.913520-1-olteanv@gmail.com (mailing list archive) |
---|---|
State | Changes Requested |
Delegated to: | Netdev Maintainers |
Headers | show |
Series | [v2,net-next] net: dsa: felix: enable cut-through forwarding between ports by default | expand |
On Tue, 23 Nov 2021 15:21:16 +0200 Vladimir Oltean wrote: > +static void vsc9959_cut_through_fwd(struct ocelot *ocelot) > +{ > + struct felix *felix = ocelot_to_felix(ocelot); > + struct dsa_switch *ds = felix->ds; > + int port, other_port; > + > + for (port = 0; port < ocelot->num_phys_ports; port++) { > + struct ocelot_port *ocelot_port = ocelot->ports[port]; > + unsigned long mask; > + int min_speed; > + u32 val = 0; > + > + if (ocelot_port->speed <= 0) > + continue; Would it not be safer to disable cut-thru for ports which are down? goto set; > + min_speed = ocelot_port->speed; > + if (port == ocelot->npi) { > + /* Ocelot switches forward from the NPI port towards > + * any port, regardless of it being in the NPI port's > + * forwarding domain or not. > + */ > + mask = dsa_user_ports(ds); > + } else { > + mask = ocelot_read_rix(ocelot, ANA_PGID_PGID, > + PGID_SRC + port); > + /* Ocelot switches forward to the NPI port despite it > + * not being in the source ports' forwarding domain. > + */ > + if (ocelot->npi >= 0) > + mask |= BIT(ocelot->npi); > + } > + > + for_each_set_bit(other_port, &mask, ocelot->num_phys_ports) { > + struct ocelot_port *other_ocelot_port; > + > + other_ocelot_port = ocelot->ports[other_port]; > + if (other_ocelot_port->speed <= 0) > + continue; > + > + if (min_speed > other_ocelot_port->speed) > + min_speed = other_ocelot_port->speed; break; ? > + } > + > + /* Enable cut-through forwarding for all traffic classes. */ > + if (ocelot_port->speed == min_speed) Any particular reason this is not <= ? > + val = GENMASK(7, 0); set: ? > + ocelot_write_rix(ocelot, val, ANA_CUT_THRU_CFG, port); > + } > +} > static const struct felix_info felix_info_vsc9959 = { > diff --git a/drivers/net/ethernet/mscc/ocelot.c b/drivers/net/ethernet/mscc/ocelot.c > index 95920668feb0..30c790f401be 100644 > --- a/drivers/net/ethernet/mscc/ocelot.c > +++ b/drivers/net/ethernet/mscc/ocelot.c > @@ -697,6 +702,8 @@ void ocelot_phylink_mac_link_up(struct ocelot *ocelot, int port, > int mac_speed, mode = 0; > u32 mac_fc_cfg; > > + ocelot_port->speed = speed; > + > /* The MAC might be integrated in systems where the MAC speed is fixed > * and it's the PCS who is performing the rate adaptation, so we have > * to write "1000Mbps" into the LINK_SPEED field of DEV_CLOCK_CFG > @@ -772,6 +779,9 @@ void ocelot_phylink_mac_link_up(struct ocelot *ocelot, int port, > /* Core: Enable port for frame transfer */ > ocelot_fields_write(ocelot, port, > QSYS_SWITCH_PORT_MODE_PORT_ENA, 1); Does this enable forwarding? Is there a window here with forwarding enabled and old cut-thru masks if we don't clear cut-thru when port goes down? > + if (ocelot->ops->cut_through_fwd) > + ocelot->ops->cut_through_fwd(ocelot); > } > EXPORT_SYMBOL_GPL(ocelot_phylink_mac_link_up); > > @@ -1637,6 +1647,9 @@ void ocelot_apply_bridge_fwd_mask(struct ocelot *ocelot) > > ocelot_write_rix(ocelot, mask, ANA_PGID_PGID, PGID_SRC + port); > } Obviously shooting from the hip here, but I was expecting the cut-thru update to be before the bridge reconfig if port is joining, and after if port is leaving. Do you know what I'm getting at? > + if (ocelot->ops->cut_through_fwd) > + ocelot->ops->cut_through_fwd(ocelot); > } > EXPORT_SYMBOL(ocelot_apply_bridge_fwd_mask);
On Wed, 24 Nov 2021 18:39:00 -0800 Jakub Kicinski wrote: > > + /* Enable cut-through forwarding for all traffic classes. */ > > + if (ocelot_port->speed == min_speed) > > Any particular reason this is not <= ? Because it can't be...
On Wed, Nov 24, 2021 at 06:39:00PM -0800, Jakub Kicinski wrote: > On Tue, 23 Nov 2021 15:21:16 +0200 Vladimir Oltean wrote: > > +static void vsc9959_cut_through_fwd(struct ocelot *ocelot) > > +{ > > + struct felix *felix = ocelot_to_felix(ocelot); > > + struct dsa_switch *ds = felix->ds; > > + int port, other_port; > > + > > + for (port = 0; port < ocelot->num_phys_ports; port++) { > > + struct ocelot_port *ocelot_port = ocelot->ports[port]; > > + unsigned long mask; > > + int min_speed; > > + u32 val = 0; > > + > > + if (ocelot_port->speed <= 0) > > + continue; > > Would it not be safer to disable cut-thru for ports which are down? > > goto set; I don't know, I suppose we can do that. > > + min_speed = ocelot_port->speed; > > + if (port == ocelot->npi) { > > + /* Ocelot switches forward from the NPI port towards > > + * any port, regardless of it being in the NPI port's > > + * forwarding domain or not. > > + */ > > + mask = dsa_user_ports(ds); > > + } else { > > + mask = ocelot_read_rix(ocelot, ANA_PGID_PGID, > > + PGID_SRC + port); > > + /* Ocelot switches forward to the NPI port despite it > > + * not being in the source ports' forwarding domain. > > + */ > > + if (ocelot->npi >= 0) > > + mask |= BIT(ocelot->npi); > > + } > > + > > + for_each_set_bit(other_port, &mask, ocelot->num_phys_ports) { > > + struct ocelot_port *other_ocelot_port; > > + > > + other_ocelot_port = ocelot->ports[other_port]; > > + if (other_ocelot_port->speed <= 0) > > + continue; > > + > > + if (min_speed > other_ocelot_port->speed) > > + min_speed = other_ocelot_port->speed; > > break; ? Break where and why? Breaking in the "if" block means "stop at the first @other_port in @port's forwarding domain which has a lower speed than @port". But that isn't necessarily the minimum... And breaking below the "if" block means stopping at the first @other_port in @port's forwarding domain, which doesn't make sense. This is the simple calculation of the minimum value of an array, no special sauce here. > > + } > > + > > + /* Enable cut-through forwarding for all traffic classes. */ > > + if (ocelot_port->speed == min_speed) > > Any particular reason this is not <= ? So min_speed is by construction always a valid speed: SPEED_10, SPEED_100 etc (never SPEED_UNKNOWN). What this statement is saying is that cut-through forwarding can be enabled only for the ports operating at the lowest link speed within a forwarding domain. If I change "==" into "<=", I would also be enabling cut-through for the ports with SPEED_UNKNOWN (were it not for this condition right at the beginning): if (ocelot_port->speed <= 0) continue; So practically speaking, since there is never any port with a speed smaller than the min_speed, it doesn't make sense to check for <= min_speed. > > + val = GENMASK(7, 0); > > set: ? This I can do. > > + ocelot_write_rix(ocelot, val, ANA_CUT_THRU_CFG, port); > > + } > > +} > > > static const struct felix_info felix_info_vsc9959 = { > > diff --git a/drivers/net/ethernet/mscc/ocelot.c b/drivers/net/ethernet/mscc/ocelot.c > > index 95920668feb0..30c790f401be 100644 > > --- a/drivers/net/ethernet/mscc/ocelot.c > > +++ b/drivers/net/ethernet/mscc/ocelot.c > > > @@ -697,6 +702,8 @@ void ocelot_phylink_mac_link_up(struct ocelot *ocelot, int port, > > int mac_speed, mode = 0; > > u32 mac_fc_cfg; > > > > + ocelot_port->speed = speed; > > + > > /* The MAC might be integrated in systems where the MAC speed is fixed > > * and it's the PCS who is performing the rate adaptation, so we have > > * to write "1000Mbps" into the LINK_SPEED field of DEV_CLOCK_CFG > > @@ -772,6 +779,9 @@ void ocelot_phylink_mac_link_up(struct ocelot *ocelot, int port, > > /* Core: Enable port for frame transfer */ > > ocelot_fields_write(ocelot, port, > > QSYS_SWITCH_PORT_MODE_PORT_ENA, 1); > > Does this enable forwarding? Is there a window here with forwarding > enabled and old cut-thru masks if we don't clear cut-thru when port > goes down? Correct, I should be updating the cut-through masks before this, thanks. > > + if (ocelot->ops->cut_through_fwd) > > + ocelot->ops->cut_through_fwd(ocelot); > > } > > EXPORT_SYMBOL_GPL(ocelot_phylink_mac_link_up); > > > > @@ -1637,6 +1647,9 @@ void ocelot_apply_bridge_fwd_mask(struct ocelot *ocelot) > > > > ocelot_write_rix(ocelot, mask, ANA_PGID_PGID, PGID_SRC + port); > > } > > Obviously shooting from the hip here, but I was expecting the cut-thru > update to be before the bridge reconfig if port is joining, and after > if port is leaving. Do you know what I'm getting at? Yes, I know what you're getting at. But it's a bit complicated to do, given the layering constraints and that cut-through forwarding is an optional feature which isn't present on all devices, so I am trying to keep its footprint minimal on the ocelot library. What I can do is I can disable cut-through forwarding for ports that are standalone (not in a bridge). I don't have a use case for that anyway: the store-and-forward latency is indistinguishable from network stack latency. This will guarantee that when a port joins a bridge, it has cut-through forwarding disabled. So there are no issues if it happens to join a bridge and its link speed is higher than anybody else: there will be no packet underruns. > > + if (ocelot->ops->cut_through_fwd) > > + ocelot->ops->cut_through_fwd(ocelot); > > } > > EXPORT_SYMBOL(ocelot_apply_bridge_fwd_mask); Thanks for taking a look at the patch!
On Thu, 25 Nov 2021 12:16:52 +0200 Vladimir Oltean wrote: > > > + if (min_speed > other_ocelot_port->speed) > > > + min_speed = other_ocelot_port->speed; > > > > break; ? > > Break where and why? > Breaking in the "if" block means "stop at the first @other_port in > @port's forwarding domain which has a lower speed than @port". But that > isn't necessarily the minimum... > And breaking below the "if" block means stopping at the first > @other_port in @port's forwarding domain, which doesn't make sense. > This is the simple calculation of the minimum value of an array, no > special sauce here. A single slower port is enough to disable cut through, this is can be read as a proof of nonexistence rather than min calculation. But really just a nit pick, don't think any bot will bother us about it. > > > /* Core: Enable port for frame transfer */ > > > ocelot_fields_write(ocelot, port, > > > QSYS_SWITCH_PORT_MODE_PORT_ENA, 1); > > > > Does this enable forwarding? Is there a window here with forwarding > > enabled and old cut-thru masks if we don't clear cut-thru when port > > goes down? > > Correct, I should be updating the cut-through masks before this, thanks. > > > > + if (ocelot->ops->cut_through_fwd) > > > + ocelot->ops->cut_through_fwd(ocelot); > > > } > > > EXPORT_SYMBOL_GPL(ocelot_phylink_mac_link_up); > > > > > > @@ -1637,6 +1647,9 @@ void ocelot_apply_bridge_fwd_mask(struct ocelot *ocelot) > > > > > > ocelot_write_rix(ocelot, mask, ANA_PGID_PGID, PGID_SRC + port); > > > } > > > > Obviously shooting from the hip here, but I was expecting the cut-thru > > update to be before the bridge reconfig if port is joining, and after > > if port is leaving. Do you know what I'm getting at? > > Yes, I know what you're getting at. But it's a bit complicated to do, > given the layering constraints and that cut-through forwarding is an > optional feature which isn't present on all devices, so I am trying to > keep its footprint minimal on the ocelot library. > > What I can do is I can disable cut-through forwarding for ports that are > standalone (not in a bridge). I don't have a use case for that anyway: > the store-and-forward latency is indistinguishable from network stack > latency. This will guarantee that when a port joins a bridge, it has > cut-through forwarding disabled. So there are no issues if it happens to > join a bridge and its link speed is higher than anybody else: there will > be no packet underruns. Hm, to make sure I understand - fixing standalone ports doesn't necessary address the issue of a slow standalone port joining, right?
On Thu, Nov 25, 2021 at 07:29:09AM -0800, Jakub Kicinski wrote: > > > Obviously shooting from the hip here, but I was expecting the cut-thru > > > update to be before the bridge reconfig if port is joining, and after > > > if port is leaving. Do you know what I'm getting at? > > > > Yes, I know what you're getting at. But it's a bit complicated to do, > > given the layering constraints and that cut-through forwarding is an > > optional feature which isn't present on all devices, so I am trying to > > keep its footprint minimal on the ocelot library. > > > > What I can do is I can disable cut-through forwarding for ports that are > > standalone (not in a bridge). I don't have a use case for that anyway: > > the store-and-forward latency is indistinguishable from network stack > > latency. This will guarantee that when a port joins a bridge, it has > > cut-through forwarding disabled. So there are no issues if it happens to > > join a bridge and its link speed is higher than anybody else: there will > > be no packet underruns. > > Hm, to make sure I understand - fixing standalone ports doesn't > necessary address the issue of a slow standalone port joining, right? Yeah, I think I handled it better in v3 than my initial idea here.
diff --git a/drivers/net/dsa/ocelot/felix_vsc9959.c b/drivers/net/dsa/ocelot/felix_vsc9959.c index 42ac1952b39a..b7941d4dbfc6 100644 --- a/drivers/net/dsa/ocelot/felix_vsc9959.c +++ b/drivers/net/dsa/ocelot/felix_vsc9959.c @@ -2125,6 +2125,66 @@ static void vsc9959_psfp_init(struct ocelot *ocelot) INIT_LIST_HEAD(&psfp->sgi_list); } +/* When using cut-through forwarding and the egress port runs at a higher data + * rate than the ingress port, the packet currently under transmission would + * suffer an underrun since it would be transmitted faster than it is received. + * The Felix switch implementation of cut-through forwarding does not check in + * hardware whether this condition is satisfied or not, so we must restrict the + * list of ports that have cut-through forwarding enabled on egress to only be + * the ports operating at the lowest link speed within their respective + * forwarding domain. + */ +static void vsc9959_cut_through_fwd(struct ocelot *ocelot) +{ + struct felix *felix = ocelot_to_felix(ocelot); + struct dsa_switch *ds = felix->ds; + int port, other_port; + + for (port = 0; port < ocelot->num_phys_ports; port++) { + struct ocelot_port *ocelot_port = ocelot->ports[port]; + unsigned long mask; + int min_speed; + u32 val = 0; + + if (ocelot_port->speed <= 0) + continue; + + min_speed = ocelot_port->speed; + if (port == ocelot->npi) { + /* Ocelot switches forward from the NPI port towards + * any port, regardless of it being in the NPI port's + * forwarding domain or not. + */ + mask = dsa_user_ports(ds); + } else { + mask = ocelot_read_rix(ocelot, ANA_PGID_PGID, + PGID_SRC + port); + /* Ocelot switches forward to the NPI port despite it + * not being in the source ports' forwarding domain. + */ + if (ocelot->npi >= 0) + mask |= BIT(ocelot->npi); + } + + for_each_set_bit(other_port, &mask, ocelot->num_phys_ports) { + struct ocelot_port *other_ocelot_port; + + other_ocelot_port = ocelot->ports[other_port]; + if (other_ocelot_port->speed <= 0) + continue; + + if (min_speed > other_ocelot_port->speed) + min_speed = other_ocelot_port->speed; + } + + /* Enable cut-through forwarding for all traffic classes. */ + if (ocelot_port->speed == min_speed) + val = GENMASK(7, 0); + + ocelot_write_rix(ocelot, val, ANA_CUT_THRU_CFG, port); + } +} + static const struct ocelot_ops vsc9959_ops = { .reset = vsc9959_reset, .wm_enc = vsc9959_wm_enc, @@ -2136,6 +2196,7 @@ static const struct ocelot_ops vsc9959_ops = { .psfp_filter_add = vsc9959_psfp_filter_add, .psfp_filter_del = vsc9959_psfp_filter_del, .psfp_stats_get = vsc9959_psfp_stats_get, + .cut_through_fwd = vsc9959_cut_through_fwd, }; static const struct felix_info felix_info_vsc9959 = { diff --git a/drivers/net/ethernet/mscc/ocelot.c b/drivers/net/ethernet/mscc/ocelot.c index 95920668feb0..30c790f401be 100644 --- a/drivers/net/ethernet/mscc/ocelot.c +++ b/drivers/net/ethernet/mscc/ocelot.c @@ -682,6 +682,11 @@ void ocelot_phylink_mac_link_down(struct ocelot *ocelot, int port, DEV_CLOCK_CFG_MAC_TX_RST | DEV_CLOCK_CFG_MAC_RX_RST, DEV_CLOCK_CFG); + + ocelot_port->speed = SPEED_UNKNOWN; + + if (ocelot->ops->cut_through_fwd) + ocelot->ops->cut_through_fwd(ocelot); } EXPORT_SYMBOL_GPL(ocelot_phylink_mac_link_down); @@ -697,6 +702,8 @@ void ocelot_phylink_mac_link_up(struct ocelot *ocelot, int port, int mac_speed, mode = 0; u32 mac_fc_cfg; + ocelot_port->speed = speed; + /* The MAC might be integrated in systems where the MAC speed is fixed * and it's the PCS who is performing the rate adaptation, so we have * to write "1000Mbps" into the LINK_SPEED field of DEV_CLOCK_CFG @@ -772,6 +779,9 @@ void ocelot_phylink_mac_link_up(struct ocelot *ocelot, int port, /* Core: Enable port for frame transfer */ ocelot_fields_write(ocelot, port, QSYS_SWITCH_PORT_MODE_PORT_ENA, 1); + + if (ocelot->ops->cut_through_fwd) + ocelot->ops->cut_through_fwd(ocelot); } EXPORT_SYMBOL_GPL(ocelot_phylink_mac_link_up); @@ -1637,6 +1647,9 @@ void ocelot_apply_bridge_fwd_mask(struct ocelot *ocelot) ocelot_write_rix(ocelot, mask, ANA_PGID_PGID, PGID_SRC + port); } + + if (ocelot->ops->cut_through_fwd) + ocelot->ops->cut_through_fwd(ocelot); } EXPORT_SYMBOL(ocelot_apply_bridge_fwd_mask); diff --git a/include/soc/mscc/ocelot.h b/include/soc/mscc/ocelot.h index 89d17629efe5..611847ee5faf 100644 --- a/include/soc/mscc/ocelot.h +++ b/include/soc/mscc/ocelot.h @@ -561,6 +561,7 @@ struct ocelot_ops { int (*psfp_filter_del)(struct ocelot *ocelot, struct flow_cls_offload *f); int (*psfp_stats_get)(struct ocelot *ocelot, struct flow_cls_offload *f, struct flow_stats *stats); + void (*cut_through_fwd)(struct ocelot *ocelot); }; struct ocelot_vcap_policer { @@ -655,6 +656,8 @@ struct ocelot_port { struct net_device *bridge; u8 stp_state; + + int speed; }; struct ocelot {