| Message ID | 20220125165410.252903-1-robert.hancock@calian.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | at803x fiber/SFP support | expand |
On Tue, Jan 25, 2022 at 10:54:07AM -0600, Robert Hancock wrote: > Add support for 1000Base-X fiber modes to the at803x PHY driver, as > well as support for connecting a downstream SFP cage. > > Changes since v3: > -Renamed some constants with OHM suffix for clarity > > Changes since v2: > -fixed tabs/spaces issue in one patch > > Changes since v1: > -moved page selection to config_init so it is handled properly > after suspend/resume > -added explicit check for empty sfp_support bitmask > > Robert Hancock (3): > net: phy: at803x: move page selection fix to config_init > net: phy: at803x: add fiber support > net: phy: at803x: Support downstream SFP cage > Backported this series to 5.15.16 and tested on Ubiquiti EdgeRouter X SFP hardware. Optical SFP modules working without problems, but cooper SFP with Marvell 88E1111 not work - link is up but no packets sent/recieved via interface. # dmesg | grep -iE '(dsa|eth|mdio|sfp)' [ 0.000000] MIPS: machine is Ubiquiti EdgeRouter X SFP [ 1.349162] gpio-7 (sfp_i2c_clk_gate): hogged as output/high [ 4.685535] libphy: Fixed MDIO Bus: probed [ 4.732207] libphy: mdio: probed [ 4.739151] mt7530 mdio-bus:00: MT7530 adapts as multi-chip module [ 4.768955] mtk_soc_eth 1e100000.ethernet dsa: mediatek frame engine at 0xbe100000, irq 20 [ 4.852254] mt7530 mdio-bus:00: MT7530 adapts as multi-chip module [ 4.973393] mt7530 mdio-bus:00 eth0 (uninitialized): PHY [mt7530-0:00] driver [Generic PHY] (irq=POLL) [ 4.993676] mt7530 mdio-bus:00 eth1 (uninitialized): PHY [mt7530-0:01] driver [Generic PHY] (irq=POLL) [ 5.014054] mt7530 mdio-bus:00 eth2 (uninitialized): PHY [mt7530-0:02] driver [Generic PHY] (irq=POLL) [ 5.034405] mt7530 mdio-bus:00 eth3 (uninitialized): PHY [mt7530-0:03] driver [Generic PHY] (irq=POLL) [ 5.054704] mt7530 mdio-bus:00 eth4 (uninitialized): PHY [mt7530-0:04] driver [Generic PHY] (irq=POLL) [ 5.131850] mt7530 mdio-bus:00 eth5 (uninitialized): PHY [mdio-bus:07] driver [Qualcomm Atheros AR8031/AR8033] (irq=POLL) [ 5.155828] mt7530 mdio-bus:00: configuring for fixed/rgmii link mode [ 5.172290] DSA: tree 0 setup [ 5.179302] mt7530 mdio-bus:00: Link is Up - 1Gbps/Full - flow control off [ 9.103291] mtk_soc_eth 1e100000.ethernet dsa: configuring for fixed/rgmii link mode [ 9.119230] device dsa entered promiscuous mode [ 9.128654] mtk_soc_eth 1e100000.ethernet dsa: Link is Up - 1Gbps/Full - flow control rx/tx [ 9.131563] mt7530 mdio-bus:00 eth1: configuring for phy/gmii link mode [ 9.158926] IPv6: ADDRCONF(NETDEV_CHANGE): dsa: link becomes ready [ 13.306590] device dsa left promiscuous mode [ 14.587953] libphy: SFP I2C Bus: probed [ 14.751420] sfp sfp_eth5: Host maximum power 1.0W [ 14.821296] sfp sfp_eth5: No tx_disable pin: SFP modules will always be emitting. [ 15.691261] sfp sfp_eth5: module Gateray GR-S1-RJ rev A sn X201604162293 dc 160422 [ 15.710686] Qualcomm Atheros AR8031/AR8033 mdio-bus:07: module may not function if 1000Base-X not supported [ 36.440072] mtk_soc_eth 1e100000.ethernet dsa: Link is Down [ 36.461374] mtk_soc_eth 1e100000.ethernet dsa: configuring for fixed/rgmii link mode [ 36.477856] mtk_soc_eth 1e100000.ethernet dsa: Link is Up - 1Gbps/Full - flow control rx/tx [ 36.495142] IPv6: ADDRCONF(NETDEV_CHANGE): dsa: link becomes ready [ 36.508246] device dsa entered promiscuous mode [ 36.517976] mt7530 mdio-bus:00 eth1: configuring for phy/gmii link mode [ 36.532794] br-lan: port 1(eth1) entered blocking state [ 36.543322] br-lan: port 1(eth1) entered disabled state [ 36.555768] device eth1 entered promiscuous mode [ 36.592332] mt7530 mdio-bus:00 eth2: configuring for phy/gmii link mode [ 36.607113] br-lan: port 2(eth2) entered blocking state [ 36.617651] br-lan: port 2(eth2) entered disabled state [ 36.630268] device eth2 entered promiscuous mode [ 36.655233] mt7530 mdio-bus:00 eth3: configuring for phy/gmii link mode [ 36.669957] br-lan: port 3(eth3) entered blocking state [ 36.680454] br-lan: port 3(eth3) entered disabled state [ 36.693351] device eth3 entered promiscuous mode [ 36.718596] mt7530 mdio-bus:00 eth4: configuring for phy/gmii link mode [ 36.733435] br-lan: port 4(eth4) entered blocking state [ 36.743977] br-lan: port 4(eth4) entered disabled state [ 36.756966] device eth4 entered promiscuous mode [ 36.779983] mt7530 mdio-bus:00 eth5: configuring for phy/rgmii-rxid link mode [ 36.795073] mt7530 mdio-bus:00 eth5: No phy led trigger registered for speed(-1) [ 36.810051] br-lan: port 5(eth5) entered blocking state [ 36.810277] mt7530 mdio-bus:00 eth5: Link is Up - Unknown/Unknown - flow control off Link is up - but what mode is selected ?? No packets received from interface, remote side report 1000/Full speed negotiated, TX counters increased. [ 36.820606] br-lan: port 5(eth5) entered disabled state [ 36.849515] device eth5 entered promiscuous mode [ 36.862142] br-lan: port 5(eth5) entered blocking state [ 36.872673] br-lan: port 5(eth5) entered forwarding state [ 37.260595] mt7530 mdio-bus:00 eth0: configuring for phy/gmii link mode [ 40.362029] mt7530 mdio-bus:00 eth0: Link is Up - 1Gbps/Full - flow control off [ 40.376656] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 670.201487] mt7530 mdio-bus:00 eth5: Link is Down [ 670.211423] br-lan: port 5(eth5) entered disabled state [ 671.241339] mt7530 mdio-bus:00 eth5: No phy led trigger registered for speed(-1) [ 671.256789] mt7530 mdio-bus:00 eth5: Link is Up - Unknown/Unknown - flow control off [ 671.272343] br-lan: port 5(eth5) entered blocking state [ 671.282821] br-lan: port 5(eth5) entered forwarding state [ 671.471210] sfp sfp_eth5: module removed Now, I'm removed copper SFP, and install optilcal SFP: [ 672.281451] mt7530 mdio-bus:00 eth5: Link is Down [ 672.290943] br-lan: port 5(eth5) entered disabled state [ 688.921868] mt7530 mdio-bus:00 eth5: Link is Up - 1Gbps/Full - flow control off [ 688.936512] br-lan: port 5(eth5) entered blocking state [ 688.946944] br-lan: port 5(eth5) entered forwarding state [ 689.591192] sfp sfp_eth5: module Gateray GR-S1-X3120L-D rev A sn X201602220961 dc 160229 Link is up, 1000/Full, packets traverse in both directions. Now, insert back copper SFP: [ 731.561446] mt7530 mdio-bus:00 eth5: Link is Down [ 731.570947] br-lan: port 5(eth5) entered disabled state [ 733.841609] sfp sfp_eth5: module removed [ 751.321576] mt7530 mdio-bus:00 eth5: Link is Up - 1Gbps/Unknown - flow control off Whoa, link up with 1Gbps speed. [ 751.336733] br-lan: port 5(eth5) entered blocking state [ 751.347167] br-lan: port 5(eth5) entered forwarding state [ 751.961193] sfp sfp_eth5: module Gateray GR-S1-RJ rev A sn X201604162293 dc 160422 [ 751.980667] Qualcomm Atheros AR8031/AR8033 mdio-bus:07: module may not function if 1000Base-X not supported [ 753.401483] mt7530 mdio-bus:00 eth5: Link is Down [ 753.410984] br-lan: port 5(eth5) entered disabled state [ 754.441627] mt7530 mdio-bus:00 eth5: Link is Up - 1Gbps/Unknown - flow control off [ 754.457144] br-lan: port 5(eth5) entered blocking state [ 754.467619] br-lan: port 5(eth5) entered forwarding state Now packets traverse in both directions for copper module too. Can someone explain - why copper module not work from boot? And how controll 88E1111 inside SFP ?
On Wed, Jan 26, 2022 at 04:56:40PM +0300, Andrey Jr. Melnikov wrote: > On Tue, Jan 25, 2022 at 10:54:07AM -0600, Robert Hancock wrote: > > Add support for 1000Base-X fiber modes to the at803x PHY driver, as > > well as support for connecting a downstream SFP cage. > > > > Changes since v3: > > -Renamed some constants with OHM suffix for clarity > > > > Changes since v2: > > -fixed tabs/spaces issue in one patch > > > > Changes since v1: > > -moved page selection to config_init so it is handled properly > > after suspend/resume > > -added explicit check for empty sfp_support bitmask > > > > Robert Hancock (3): > > net: phy: at803x: move page selection fix to config_init > > net: phy: at803x: add fiber support > > net: phy: at803x: Support downstream SFP cage > > > Backported this series to 5.15.16 and tested on Ubiquiti EdgeRouter X SFP > hardware. Optical SFP modules working without problems, but cooper SFP with > Marvell 88E1111 not work - link is up but no packets sent/recieved via > interface. Could be that the 88E1111 is in SGMII mode, and with the 803x in 1000base-x mode, they just don't want to talk... and having a link up event with an optical SFP (in 1000base-x mode) changes the state in the 803x so it somehow works. SGMII and 1000base-x are similar enough that it can appear to work at gigabit speeds if everything is just right. > Can someone explain - why copper module not work from boot? And how controll 88E1111 > inside SFP ? The Linux networking layer only permits one PHY per network interface, so in the case of a SFP with a PHY connected to another PHY, the only PHY we can expose is the PHY closest to the network interface. There is no way e.g. via ethtool to be able to direct the PHY specific ethtool calls to a specific PHY.
On Wed, 2022-01-26 at 16:56 +0300, Andrey Jr. Melnikov wrote: > On Tue, Jan 25, 2022 at 10:54:07AM -0600, Robert Hancock wrote: > > Add support for 1000Base-X fiber modes to the at803x PHY driver, as > > well as support for connecting a downstream SFP cage. > > > > Changes since v3: > > -Renamed some constants with OHM suffix for clarity > > > > Changes since v2: > > -fixed tabs/spaces issue in one patch > > > > Changes since v1: > > -moved page selection to config_init so it is handled properly > > after suspend/resume > > -added explicit check for empty sfp_support bitmask > > > > Robert Hancock (3): > > net: phy: at803x: move page selection fix to config_init > > net: phy: at803x: add fiber support > > net: phy: at803x: Support downstream SFP cage > > > Backported this series to 5.15.16 and tested on Ubiquiti EdgeRouter X SFP > hardware. Optical SFP modules working without problems, but cooper SFP with > Marvell 88E1111 not work - link is up but no packets sent/recieved via > interface. Interesting, apparently we weren't the only ones weird enough to use an AR8031 for an SFP module.. > > # dmesg | grep -iE '(dsa|eth|mdio|sfp)' > [ 0.000000] MIPS: machine is Ubiquiti EdgeRouter X SFP > [ 1.349162] gpio-7 (sfp_i2c_clk_gate): hogged as output/high > [ 4.685535] libphy: Fixed MDIO Bus: probed > [ 4.732207] libphy: mdio: probed > [ 4.739151] mt7530 mdio-bus:00: MT7530 adapts as multi-chip module > [ 4.768955] mtk_soc_eth 1e100000.ethernet dsa: mediatek frame engine at > 0xbe100000, irq 20 > [ 4.852254] mt7530 mdio-bus:00: MT7530 adapts as multi-chip module > [ 4.973393] mt7530 mdio-bus:00 eth0 (uninitialized): PHY [mt7530-0:00] > driver [Generic PHY] (irq=POLL) > [ 4.993676] mt7530 mdio-bus:00 eth1 (uninitialized): PHY [mt7530-0:01] > driver [Generic PHY] (irq=POLL) > [ 5.014054] mt7530 mdio-bus:00 eth2 (uninitialized): PHY [mt7530-0:02] > driver [Generic PHY] (irq=POLL) > [ 5.034405] mt7530 mdio-bus:00 eth3 (uninitialized): PHY [mt7530-0:03] > driver [Generic PHY] (irq=POLL) > [ 5.054704] mt7530 mdio-bus:00 eth4 (uninitialized): PHY [mt7530-0:04] > driver [Generic PHY] (irq=POLL) > [ 5.131850] mt7530 mdio-bus:00 eth5 (uninitialized): PHY [mdio-bus:07] > driver [Qualcomm Atheros AR8031/AR8033] (irq=POLL) > [ 5.155828] mt7530 mdio-bus:00: configuring for fixed/rgmii link mode > [ 5.172290] DSA: tree 0 setup > [ 5.179302] mt7530 mdio-bus:00: Link is Up - 1Gbps/Full - flow control off > [ 9.103291] mtk_soc_eth 1e100000.ethernet dsa: configuring for fixed/rgmii > link mode > [ 9.119230] device dsa entered promiscuous mode > [ 9.128654] mtk_soc_eth 1e100000.ethernet dsa: Link is Up - 1Gbps/Full - > flow control rx/tx > [ 9.131563] mt7530 mdio-bus:00 eth1: configuring for phy/gmii link mode > [ 9.158926] IPv6: ADDRCONF(NETDEV_CHANGE): dsa: link becomes ready > [ 13.306590] device dsa left promiscuous mode > [ 14.587953] libphy: SFP I2C Bus: probed > [ 14.751420] sfp sfp_eth5: Host maximum power 1.0W > [ 14.821296] sfp sfp_eth5: No tx_disable pin: SFP modules will always be > emitting. > [ 15.691261] sfp sfp_eth5: module Gateray GR-S1-RJ rev > A sn X201604162293 dc 160422 > [ 15.710686] Qualcomm Atheros AR8031/AR8033 mdio-bus:07: module may not > function if 1000Base-X not supported > [ 36.440072] mtk_soc_eth 1e100000.ethernet dsa: Link is Down > [ 36.461374] mtk_soc_eth 1e100000.ethernet dsa: configuring for fixed/rgmii > link mode > [ 36.477856] mtk_soc_eth 1e100000.ethernet dsa: Link is Up - 1Gbps/Full - > flow control rx/tx > [ 36.495142] IPv6: ADDRCONF(NETDEV_CHANGE): dsa: link becomes ready > [ 36.508246] device dsa entered promiscuous mode > [ 36.517976] mt7530 mdio-bus:00 eth1: configuring for phy/gmii link mode > [ 36.532794] br-lan: port 1(eth1) entered blocking state > [ 36.543322] br-lan: port 1(eth1) entered disabled state > [ 36.555768] device eth1 entered promiscuous mode > [ 36.592332] mt7530 mdio-bus:00 eth2: configuring for phy/gmii link mode > [ 36.607113] br-lan: port 2(eth2) entered blocking state > [ 36.617651] br-lan: port 2(eth2) entered disabled state > [ 36.630268] device eth2 entered promiscuous mode > [ 36.655233] mt7530 mdio-bus:00 eth3: configuring for phy/gmii link mode > [ 36.669957] br-lan: port 3(eth3) entered blocking state > [ 36.680454] br-lan: port 3(eth3) entered disabled state > [ 36.693351] device eth3 entered promiscuous mode > [ 36.718596] mt7530 mdio-bus:00 eth4: configuring for phy/gmii link mode > [ 36.733435] br-lan: port 4(eth4) entered blocking state > [ 36.743977] br-lan: port 4(eth4) entered disabled state > [ 36.756966] device eth4 entered promiscuous mode > [ 36.779983] mt7530 mdio-bus:00 eth5: configuring for phy/rgmii-rxid link > mode > [ 36.795073] mt7530 mdio-bus:00 eth5: No phy led trigger registered for > speed(-1) > [ 36.810051] br-lan: port 5(eth5) entered blocking state > [ 36.810277] mt7530 mdio-bus:00 eth5: Link is Up - Unknown/Unknown - flow > control off > > Link is up - but what mode is selected ?? No packets received from > interface, remote side report 1000/Full speed negotiated, TX counters > increased. I think you're running into the inherent limitation of this setup (which the "module may not function if 1000Base-X not supported" message is trying to tell you) - the AR8031 only supports 1000Base-X on this interface, not SGMII. Many (perhaps most) copper SFP modules use or default to SGMII mode because it allows the link to work at 100 or 10 Mbps speeds as well as 1 Gbps. The difference between 1000Base-X and SGMII, at least when at 1000 Mbps speeds, is mostly in the auto-negotiation, so we've found that in many cases you can get it to work by using ethtool to disable auto-negotiation and forcing 1000 Mbps full duplex mode on the interface. It seems that (at least from what we have seen) the SFP module side will often decide to give up on auto-negotiation when it sees a link and just run at the proper rate - that doesn't generally affect the separate negotiation that happens on the copper link to whatever is connected on the other end. Obviously if the link partner connected on the copper side doesn't support 1 Gbps it won't work. I'm not entirely sure why it worked after the module was unplugged and plugged back in however. But my guess it's related to this issue somehow. > > [ 36.820606] br-lan: port 5(eth5) entered disabled state > [ 36.849515] device eth5 entered promiscuous mode > [ 36.862142] br-lan: port 5(eth5) entered blocking state > [ 36.872673] br-lan: port 5(eth5) entered forwarding state > [ 37.260595] mt7530 mdio-bus:00 eth0: configuring for phy/gmii link mode > [ 40.362029] mt7530 mdio-bus:00 eth0: Link is Up - 1Gbps/Full - flow > control off > [ 40.376656] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready > [ 670.201487] mt7530 mdio-bus:00 eth5: Link is Down > [ 670.211423] br-lan: port 5(eth5) entered disabled state > [ 671.241339] mt7530 mdio-bus:00 eth5: No phy led trigger registered for > speed(-1) > [ 671.256789] mt7530 mdio-bus:00 eth5: Link is Up - Unknown/Unknown - flow > control off > [ 671.272343] br-lan: port 5(eth5) entered blocking state > [ 671.282821] br-lan: port 5(eth5) entered forwarding state > [ 671.471210] sfp sfp_eth5: module removed > > Now, I'm removed copper SFP, and install optilcal SFP: > > [ 672.281451] mt7530 mdio-bus:00 eth5: Link is Down > [ 672.290943] br-lan: port 5(eth5) entered disabled state > [ 688.921868] mt7530 mdio-bus:00 eth5: Link is Up - 1Gbps/Full - flow > control off > [ 688.936512] br-lan: port 5(eth5) entered blocking state > [ 688.946944] br-lan: port 5(eth5) entered forwarding state > [ 689.591192] sfp sfp_eth5: module Gateray GR-S1-X3120L-D rev > A sn X201602220961 dc 160229 > > Link is up, 1000/Full, packets traverse in both directions. Now, insert back > copper SFP: > > [ 731.561446] mt7530 mdio-bus:00 eth5: Link is Down > [ 731.570947] br-lan: port 5(eth5) entered disabled state > [ 733.841609] sfp sfp_eth5: module removed > [ 751.321576] mt7530 mdio-bus:00 eth5: Link is Up - 1Gbps/Unknown - flow > control off > > Whoa, link up with 1Gbps speed. > > [ 751.336733] br-lan: port 5(eth5) entered blocking state > [ 751.347167] br-lan: port 5(eth5) entered forwarding state > [ 751.961193] sfp sfp_eth5: module Gateray GR-S1-RJ rev > A sn X201604162293 dc 160422 > [ 751.980667] Qualcomm Atheros AR8031/AR8033 mdio-bus:07: module may not > function if 1000Base-X not supported > [ 753.401483] mt7530 mdio-bus:00 eth5: Link is Down > [ 753.410984] br-lan: port 5(eth5) entered disabled state > [ 754.441627] mt7530 mdio-bus:00 eth5: Link is Up - 1Gbps/Unknown - flow > control off > [ 754.457144] br-lan: port 5(eth5) entered blocking state > [ 754.467619] br-lan: port 5(eth5) entered forwarding state > > Now packets traverse in both directions for copper module too. > > Can someone explain - why copper module not work from boot? And how controll > 88E1111 > inside SFP ? That's the other issue with this setup with copper modules - you end up with the AR8031 PHY device, which is basically acting as an RGMII etc. to 1000Base-X converter, and the PHY device inside the module which is the one talking to the outside world. As Russell mentioned, this sort of dual-PHY setup isn't handled well in Linux right now - in this AR8031 setup, the kernel talks to the SFP module enough to identify it, but the driver for the module PHY is not probed. Even if it was, it can't really be hooked up to the network interface because the network layer only supports binding one PHY to a device, so there's not really a way to control it. There's support for MAC drivers which have an integrated or semi-integrated PCS layer which handles 1000Base-X or SGMII - some of which look like a PHY in that they support standard PHY registers and/or have an MDIO interface - but that essentially is handled internally to phylink and the MAC driver and the "PHY"- ness isn't really exposed to the rest of the kernel. But in this case, the MAC driver has no real knowledge of the downstream SFP contraption that's been attached on its PHY interface, and the only PHY the kernel associates with the interface is the AR8031. In the AR8031 case however, this isn't really a big issue because the inherent limitation of only supporting 1000Base-X is more significant than the limitations the kernel side is imposing..
Hello: This series was applied to netdev/net-next.git (master) by David S. Miller <davem@davemloft.net>: On Tue, 25 Jan 2022 10:54:07 -0600 you wrote: > Add support for 1000Base-X fiber modes to the at803x PHY driver, as > well as support for connecting a downstream SFP cage. > > Changes since v3: > -Renamed some constants with OHM suffix for clarity > > Changes since v2: > -fixed tabs/spaces issue in one patch > > [...] Here is the summary with links: - [net-next,v4,1/3] net: phy: at803x: move page selection fix to config_init https://git.kernel.org/netdev/net-next/c/4f3a00c7f5b2 - [net-next,v4,2/3] net: phy: at803x: add fiber support https://git.kernel.org/netdev/net-next/c/3265f4218878 - [net-next,v4,3/3] net: phy: at803x: Support downstream SFP cage https://git.kernel.org/netdev/net-next/c/dc4d5fcc5d36 You are awesome, thank you!