[net,0/3] Fix trainwreck with Ocelot switch statistics counters

Message

Vladimir Oltean March 21, 2023, 1:03 a.m. UTC

While testing the patch set for preemptible traffic classes with some
controlled traffic and measuring counter deltas:
https://lore.kernel.org/netdev/[e-mail]/

I noticed that in the output of "ethtool -S swp0 --groups eth-mac
eth-phy eth-ctrl rmon -- --src emac | grep -v ': 0'", the TX counters
were off. Quickly I realized that their values were permutated by 1
compared to their names, and that for example
tx-rmon-etherStatsPkts64to64Octets was incrementing when
tx-rmon-etherStatsPkts65to127Octets should have.

Initially I suspected something having to do with the bulk reading
logic, and indeed I found a bug there (fixed as 1/3), but that was not
the source of the problems. Instead it revealed other problems.

While dumping the regions created by the driver on my switch, I figured
out that it sees a discontinuity which shouldn't have existed between
reg 0x278 and reg 0x280.

Discontinuity between last reg 0x0 and new reg 0x0, creating new region
Discontinuity between last reg 0x108 and new reg 0x200, creating new region
Discontinuity between last reg 0x278 and new reg 0x280, creating new region
Discontinuity between last reg 0x2b0 and new reg 0x400, creating new region
region of 67 contiguous counters starting with SYS:STAT:CNT[0x000]
region of 31 contiguous counters starting with SYS:STAT:CNT[0x080]
region of 13 contiguous counters starting with SYS:STAT:CNT[0x0a0]
region of 18 contiguous counters starting with SYS:STAT:CNT[0x100]

That is where TX_MM_HOLD should have been, and that was the bug, since
it was missing. After adding it, the regions look like this and the
off-by-one issue is resolved:

Discontinuity between last reg 0x000000 and new reg 0x000000, creating new region
Discontinuity between last reg 0x000108 and new reg 0x000200, creating new region
Discontinuity between last reg 0x0002b0 and new reg 0x000400, creating new region
region of 67 contiguous counters starting with SYS:STAT:CNT[0x000]
region of 45 contiguous counters starting with SYS:STAT:CNT[0x080]
region of 18 contiguous counters starting with SYS:STAT:CNT[0x100]

However, as I am thinking out loud, it should have not reported the
other counters as off by one even when skipping TX_MM_HOLD... after all,
on Ocelot/Seville, there are more counters which need to be skipped.

Which is when I investigated and noticed the bug solved in 2/3.
I've validated that both on native VSC9959 (which uses
ocelot_mm_stats_layout) as well as by faking the other switches by
making VSC9959 use the plain ocelot_stats_layout.

To summarize: on all Ocelot switches, the TX counters and drop counters
are completely broken. The RX counters are mostly fine.

With this occasion, I have collected more cleanup patches in this area,
which I'm going to submit after the net -> net-next merge.

Vladimir Oltean (3):
  net: mscc: ocelot: fix stats region batching
  net: mscc: ocelot: fix transfer from region->buf to ocelot->stats
  net: mscc: ocelot: add TX_MM_HOLD to ocelot_mm_stats_layout

 drivers/net/ethernet/mscc/ocelot_stats.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

Comments

[e-mail] March 22, 2023, 4:40 a.m. UTC | #1

Hello:

This series was applied to netdev/net.git (main)
by Jakub Kicinski <[e-mail]>:

On Tue, 21 Mar 2023 03:03:22 +0200 you wrote:
> While testing the patch set for preemptible traffic classes with some
> controlled traffic and measuring counter deltas:
> https://lore.kernel.org/netdev/[e-mail]/
> 
> I noticed that in the output of "ethtool -S swp0 --groups eth-mac
> eth-phy eth-ctrl rmon -- --src emac | grep -v ': 0'", the TX counters
> were off. Quickly I realized that their values were permutated by 1
> compared to their names, and that for example
> tx-rmon-etherStatsPkts64to64Octets was incrementing when
> tx-rmon-etherStatsPkts65to127Octets should have.
> 
> [...]

Here is the summary with links:
  - [net,1/3] net: mscc: ocelot: fix stats region batching
    https://git.kernel.org/netdev/net/c/6acc72a43eac
  - [net,2/3] net: mscc: ocelot: fix transfer from region->buf to ocelot->stats
    https://git.kernel.org/netdev/net/c/17dfd2104598
  - [net,3/3] net: mscc: ocelot: add TX_MM_HOLD to ocelot_mm_stats_layout
    https://git.kernel.org/netdev/net/c/5291099e0f61

You are awesome, thank you!