| Message ID | 20250206141538.549-4-darinzon@amazon.com (mailing list archive) |
|---|---|
| State | Superseded |
| Delegated to: | Netdev Maintainers |
| Headers | show |
| Series | PHC support in ENA driver | expand |
On Thu, 6 Feb 2025 16:15:37 +0200 David Arinzon wrote: > +PHC can be monitored using :code:`ethtool -S` counters: > + > +================= ====================================================== > +**phc_cnt** Number of successful retrieved timestamps (below expire timeout). > +**phc_exp** Number of expired retrieved timestamps (above expire timeout). > +**phc_skp** Number of skipped get time attempts (during block period). > +**phc_err** Number of failed get time attempts (entering into block state). > +================= ====================================================== ethtool -S is for networking counters.
> > +PHC can be monitored using :code:`ethtool -S` counters: > > + > > +================= > ====================================================== > > +**phc_cnt** Number of successful retrieved timestamps (below > expire timeout). > > +**phc_exp** Number of expired retrieved timestamps (above > expire timeout). > > +**phc_skp** Number of skipped get time attempts (during block > period). > > +**phc_err** Number of failed get time attempts (entering into block > state). > > +================= > ====================================================== > > ethtool -S is for networking counters. > -- > pw-bot: cr Hi Jakub, You are right in the regard that it is not a network specific functionality. Having said that, PHC is a network card capability, making it a network-related component rather than purely a timekeeping feature. Moreover we failed to find an existing tool which would allow users to get valuable feedback of the system's overall health. Researching its existing support in the kernel we noted that: - PHC is embedded in network NIC and is supported by multiple NIC vendors in the kernel - PHC information is visible through ethtool -T - The Linux networking stack uses PHC for timekeeping as well as for packet timestamping (via SO_TIMESTAMPING). Packet timestamping statistics are available through ethtool get_ts_stats hook We have found `ethtool -S` as a suitable location for exposing these statistics, which are unique to the ENA NIC. We'd appreciate your thoughts on the matter, is there an alternative tool you can recommend?
On Mon, 10 Feb 2025 15:28:19 +0000 Arinzon, David wrote: > You are right in the regard that it is not a network specific functionality. > Having said that, PHC is a network card capability, making it a network-related component rather than purely a timekeeping feature. > Moreover we failed to find an existing tool which would allow users to get valuable feedback of the system's overall health. > > Researching its existing support in the kernel we noted that: > - PHC is embedded in network NIC and is supported by multiple NIC vendors in the kernel > - PHC information is visible through ethtool -T > - The Linux networking stack uses PHC for timekeeping as well as for packet timestamping (via SO_TIMESTAMPING). > Packet timestamping statistics are available through ethtool get_ts_stats hook > > We have found `ethtool -S` as a suitable location for exposing these statistics, which are unique to the ENA NIC. > > We'd appreciate your thoughts on the matter, is there an alternative tool you can recommend? We try to steer folks towards read-only debugfs files for stuff that's not strictly networking related. You also add a custom sysfs file in patch 4, I reckon adding stats there may also be a natural place for the user? Patch 4 FWIW is lacking slightly in the justification, would be good to clarify why it's disabled by default. Single sentence of "why" would be great.
> > You are right in the regard that it is not a network specific functionality. > > Having said that, PHC is a network card capability, making it a network- > related component rather than purely a timekeeping feature. > > Moreover we failed to find an existing tool which would allow users to get > valuable feedback of the system's overall health. > > > > Researching its existing support in the kernel we noted that: > > - PHC is embedded in network NIC and is supported by multiple NIC > > vendors in the kernel > > - PHC information is visible through ethtool -T > > - The Linux networking stack uses PHC for timekeeping as well as for packet > timestamping (via SO_TIMESTAMPING). > > Packet timestamping statistics are available through ethtool > > get_ts_stats hook > > > > We have found `ethtool -S` as a suitable location for exposing these > statistics, which are unique to the ENA NIC. > > > > We'd appreciate your thoughts on the matter, is there an alternative tool > you can recommend? > > We try to steer folks towards read-only debugfs files for stuff that's not > strictly networking related. You also add a custom sysfs file in patch 4, I > reckon adding stats there may also be a natural place for the user? > > Patch 4 FWIW is lacking slightly in the justification, would be good to clarify > why it's disabled by default. Single sentence of "why" would be great. Hi Jakub, Thank you for the feedback and the recommendations, we will incorporate them in v7.
diff --git a/Documentation/networking/device_drivers/ethernet/amazon/ena.rst b/Documentation/networking/device_drivers/ethernet/amazon/ena.rst index 4561e8ab..12b13da0 100644 --- a/Documentation/networking/device_drivers/ethernet/amazon/ena.rst +++ b/Documentation/networking/device_drivers/ethernet/amazon/ena.rst @@ -56,6 +56,7 @@ ena_netdev.[ch] Main Linux kernel driver. ena_ethtool.c ethtool callbacks. ena_xdp.[ch] XDP files ena_pci_id_tbl.h Supported device IDs. +ena_phc.[ch] PTP hardware clock infrastructure (see `PHC`_ for more info) ================= ====================================================== Management Interface: @@ -221,6 +222,83 @@ descriptor it was received on would be recycled. When a packet smaller than RX copybreak bytes is received, it is copied into a new memory buffer and the RX descriptor is returned to HW. +.. _`PHC`: + +PTP Hardware Clock (PHC) +======================== +.. _`ptp-userspace-api`: https://docs.kernel.org/driver-api/ptp.html#ptp-hardware-clock-user-space-api +.. _`testptp`: https://elixir.bootlin.com/linux/latest/source/tools/testing/selftests/ptp/testptp.c + +ENA Linux driver supports PTP hardware clock providing timestamp reference to achieve nanosecond resolution. + +**PHC support** + +PHC depends on the PTP module, which needs to be either loaded as a module or compiled into the kernel. + +Verify if the PTP module is present: + +.. code-block:: shell + + grep -w '^CONFIG_PTP_1588_CLOCK=[ym]' /boot/config-`uname -r` + +- If no output is provided, the ENA driver cannot be loaded with PHC support. + +- ``CONFIG_PTP_1588_CLOCK=y``: the PTP module is already compiled and loaded inside the kernel binary file. + +- ``CONFIG_PTP_1588_CLOCK=m``: the PTP module needs to be loaded prior to loading the ENA driver: + +Load PTP module: + +.. code-block:: shell + + sudo modprobe ptp + +All available PTP clock sources can be tracked here: + +.. code-block:: shell + + ls /sys/class/ptp + +PHC support and capabilities can be verified using ethtool: + +.. code-block:: shell + + ethtool -T <interface> + +**PHC timestamp** + +To retrieve PHC timestamp, use `ptp-userspace-api`_, usage example using `testptp`_: + +.. code-block:: shell + + testptp -d /dev/ptp$(ethtool -T <interface> | awk '/PTP Hardware Clock:/ {print $NF}') -k 1 + +PHC get time requests should be within reasonable bounds, +avoid excessive utilization to ensure optimal performance and efficiency. +The ENA device restricts the frequency of PHC get time requests to a maximum +of 125 requests per second. If this limit is surpassed, the get time request +will fail, leading to an increment in the phc_err statistic. + +**PHC statistics** + +PHC can be monitored using :code:`ethtool -S` counters: + +================= ====================================================== +**phc_cnt** Number of successful retrieved timestamps (below expire timeout). +**phc_exp** Number of expired retrieved timestamps (above expire timeout). +**phc_skp** Number of skipped get time attempts (during block period). +**phc_err** Number of failed get time attempts (entering into block state). +================= ====================================================== + +PHC timeouts: + +================= ====================================================== +**expire** Max time for a valid timestamp retrieval, passing this threshold will fail + the get time request and block new requests until block timeout. +**block** Blocking period starts once get time request expires or fails, all get time + requests during block period will be skipped. +================= ====================================================== + Statistics ==========