| Message ID | 20220302163128.218798-1-idosch@nvidia.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | HW counters for soft devices | expand |
Hello: This series was applied to netdev/net-next.git (master) by David S. Miller <davem@davemloft.net>: On Wed, 2 Mar 2022 18:31:14 +0200 you wrote: > Petr says: > > Offloading switch device drivers may be able to collect statistics of the > traffic taking place in the HW datapath that pertains to a certain soft > netdevice, such as a VLAN. In this patch set, add the necessary > infrastructure to allow exposing these statistics to the offloaded > netdevice in question, and add mlxsw offload. > > [...] Here is the summary with links: - [net-next,v2,01/14] net: rtnetlink: Namespace functions related to IFLA_OFFLOAD_XSTATS_* https://git.kernel.org/netdev/net-next/c/6b524a1d012b - [net-next,v2,02/14] net: rtnetlink: Stop assuming that IFLA_OFFLOAD_XSTATS_* are dev-backed https://git.kernel.org/netdev/net-next/c/f6e0fb812988 - [net-next,v2,03/14] net: rtnetlink: RTM_GETSTATS: Allow filtering inside nests https://git.kernel.org/netdev/net-next/c/46efc97b7306 - [net-next,v2,04/14] net: rtnetlink: Propagate extack to rtnl_offload_xstats_fill() https://git.kernel.org/netdev/net-next/c/05415bccbb09 - [net-next,v2,05/14] net: rtnetlink: rtnl_fill_statsinfo(): Permit non-EMSGSIZE error returns https://git.kernel.org/netdev/net-next/c/216e690631f5 - [net-next,v2,06/14] net: dev: Add hardware stats support https://git.kernel.org/netdev/net-next/c/9309f97aef6d - [net-next,v2,07/14] net: rtnetlink: Add UAPI for obtaining L3 offload xstats https://git.kernel.org/netdev/net-next/c/0e7788fd7622 - [net-next,v2,08/14] net: rtnetlink: Add RTM_SETSTATS https://git.kernel.org/netdev/net-next/c/03ba35667091 - [net-next,v2,09/14] net: rtnetlink: Add UAPI toggle for IFLA_OFFLOAD_XSTATS_L3_STATS https://git.kernel.org/netdev/net-next/c/5fd0b838efac - [net-next,v2,10/14] mlxsw: reg: Fix packing of router interface counters https://git.kernel.org/netdev/net-next/c/8fe96f586b83 - [net-next,v2,11/14] mlxsw: spectrum_router: Drop mlxsw_sp arg from counter alloc/free functions https://git.kernel.org/netdev/net-next/c/9834e2467c86 - [net-next,v2,12/14] mlxsw: Extract classification of router-related events to a helper https://git.kernel.org/netdev/net-next/c/c1de13f91ee5 - [net-next,v2,13/14] mlxsw: Add support for IFLA_OFFLOAD_XSTATS_L3_STATS https://git.kernel.org/netdev/net-next/c/8d0f7d3ac647 - [net-next,v2,14/14] selftests: forwarding: hw_stats_l3: Add a new test https://git.kernel.org/netdev/net-next/c/ba95e7930957 You are awesome, thank you!
Petr says: Offloading switch device drivers may be able to collect statistics of the traffic taking place in the HW datapath that pertains to a certain soft netdevice, such as a VLAN. In this patch set, add the necessary infrastructure to allow exposing these statistics to the offloaded netdevice in question, and add mlxsw offload. Across HW platforms, the counter itself very likely constitutes a limited resource, and the act of counting may have a performance impact. Therefore this patch set makes the HW statistics collection opt-in and togglable from userspace on a per-netdevice basis. Additionally, HW devices may have various limiting conditions under which they can realize the counter. Therefore it is also possible to query whether the requested counter is realized by any driver. In TC parlance, which is to a degree reused in this patch set, two values are recognized: "request" tracks whether the user enabled collecting HW statistics, and "used" tracks whether any HW statistics are actually collected. In the past, this author has expressed the opinion that `a typical user doing "ip -s l sh", including various scripts, wants to see the full picture and not worry what's going on where'. While that would be nice, unfortunately it cannot work: - Packets that trap from the HW datapath to the SW datapath would be double counted. For a given netdevice, some traffic can be purely a SW artifact, and some may flow through the HW object corresponding to the netdevice. But some traffic can also get trapped to the SW datapath after bumping the HW counter. It is not clear how to make sure double-counting does not occur in the SW datapath in that case, while still making sure that possibly divergent SW forwarding path gets bumped as appropriate. So simply adding HW and SW stats may work roughly, most of the time, but there are scenarios where the result is nonsensical. - HW devices will have limitations as to what type of traffic they can count. In case of mlxsw, which is part of this patch set, there is no reasonable way to count all traffic going through a certain netdevice, such as a VLAN netdevice enslaved to a bridge. It is however very simple to count traffic flowing through an L3 object, such as a VLAN netdevice with an IP address. Similarly for physical netdevices, the L3 object at which the counter is installed is the subport carrying untagged traffic. These are not "just counters". It is important that the user understands what is being counted. It would be incorrect to conflate these statistics with another existing statistics suite. To that end, this patch set introduces a statistics suite called "L3 stats". This label should make it easy to understand what is being counted, and to decide whether a given device can or cannot implement this suite for some type of netdevice. At the same time, the code is written to make future extensions easy, should a device pop up that can implement a different flavor of statistics suite (say L2, or an address-family-specific suite). For example, using a work-in-progress iproute2[1], to turn on and then list the counters on a VLAN netdevice: # ip stats set dev swp1.200 l3_stats on # ip stats show dev swp1.200 group offload subgroup l3_stats 56: swp1.200: group offload subgroup l3_stats on used on RX: bytes packets errors dropped missed mcast 0 0 0 0 0 0 TX: bytes packets errors dropped carrier collsns 0 0 0 0 0 0 The patchset progresses as follows: - Patch #1 is a cleanup. - In patch #2, remove the assumption that all LINK_OFFLOAD_XSTATS are dev-backed. The only attribute defined under the nest is currently IFLA_OFFLOAD_XSTATS_CPU_HIT. L3_STATS differs from CPU_HIT in that the driver that supplies the statistics is not the same as the driver that implements the netdevice. Make the code compatible with this in patch #2. - In patch #3, add the possibility to filter inside nests. The filter_mask field of RTM_GETSTATS header determines which top-level attributes should be included in the netlink response. This saves processing time by only including the bits that the user cares about instead of always dumping everything. This is doubly important for HW-backed statistics that would typically require a trip to the device to fetch the stats. In this patch, the UAPI is extended to allow filtering inside IFLA_STATS_LINK_OFFLOAD_XSTATS in particular, but the scheme is easily extensible to other nests as well. - In patch #4, propagate extack where we need it. In patch #5, make it possible to propagate errors from drivers to the user. - In patch #6, add the in-kernel APIs for keeping track of the new stats suite, and the notifiers that the core uses to communicate with the drivers. - In patch #7, add UAPI for obtaining the new stats suite. - In patch #8, add a new UAPI message, RTM_SETSTATS, which will carry the message to toggle the newly-added stats suite. In patch #9, add the toggle itself. At this point the core is ready for drivers to add support for the new stats suite. - In patches #10, #11 and #12, apply small tweaks to mlxsw code. - In patch #13, add support for L3 stats, which are realized as RIF counters. - Finally in patch #14, a selftest is added to the net/forwarding directory. Technically this is a HW-specific test, in that without a HW implementing the counters, it just will not pass. But devices that support L3 statistics at all are likely to be able to reuse this selftest, so it seems appropriate to put it in the general forwarding directory. We also have a netdevsim implementation, and a corresponding selftest that verifies specifically some of the core code. We intend to contribute these later. Interested parties can take a look at the raw code at [2]. [1] https://github.com/pmachata/iproute2/commits/soft_counters [2] https://github.com/pmachata/linux_mlxsw/commits/petrm_soft_counters_2 v2: - Patch #3: - Do not declare strict_start_type at the new policies, since they are used with nla_parse_nested() (sans _deprecated). - Use NLA_POLICY_NESTED to declare what the nest contents should be - Use NLA_POLICY_MASK instead of BITFIELD32 for the filtering attribute. - Patch #6: - s/monotonous/monotonic/ in commit message - Use a newly-added struct rtnl_hw_stats64 for stats transfer - Patch #7: - Use a newly-added struct rtnl_hw_stats64 for stats transfer - Patch #8: - Do not declare strict_start_type at the new policies, since they are used with nla_parse_nested() (sans _deprecated). - Patch #13: - Use a newly-added struct rtnl_hw_stats64 for stats transfer Petr Machata (14): net: rtnetlink: Namespace functions related to IFLA_OFFLOAD_XSTATS_* net: rtnetlink: Stop assuming that IFLA_OFFLOAD_XSTATS_* are dev-backed net: rtnetlink: RTM_GETSTATS: Allow filtering inside nests net: rtnetlink: Propagate extack to rtnl_offload_xstats_fill() net: rtnetlink: rtnl_fill_statsinfo(): Permit non-EMSGSIZE error returns net: dev: Add hardware stats support net: rtnetlink: Add UAPI for obtaining L3 offload xstats net: rtnetlink: Add RTM_SETSTATS net: rtnetlink: Add UAPI toggle for IFLA_OFFLOAD_XSTATS_L3_STATS mlxsw: reg: Fix packing of router interface counters mlxsw: spectrum_router: Drop mlxsw_sp arg from counter alloc/free functions mlxsw: Extract classification of router-related events to a helper mlxsw: Add support for IFLA_OFFLOAD_XSTATS_L3_STATS selftests: forwarding: hw_stats_l3: Add a new test drivers/net/ethernet/mellanox/mlxsw/reg.h | 8 +- .../net/ethernet/mellanox/mlxsw/spectrum.c | 20 +- .../ethernet/mellanox/mlxsw/spectrum_dpipe.c | 4 +- .../ethernet/mellanox/mlxsw/spectrum_router.c | 305 +++++++++- .../ethernet/mellanox/mlxsw/spectrum_router.h | 6 +- include/linux/netdevice.h | 42 ++ include/linux/rtnetlink.h | 3 + include/uapi/linux/if_link.h | 37 ++ include/uapi/linux/rtnetlink.h | 4 + net/core/dev.c | 267 ++++++++- net/core/rtnetlink.c | 522 +++++++++++++++--- security/selinux/nlmsgtab.c | 1 + .../selftests/net/forwarding/hw_stats_l3.sh | 332 +++++++++++ 13 files changed, 1458 insertions(+), 93 deletions(-) create mode 100755 tools/testing/selftests/net/forwarding/hw_stats_l3.sh