| Message ID | 20241021052011.591-4-darinzon@amazon.com (mailing list archive) |
|---|---|
| State | Changes Requested |
| Delegated to: | Netdev Maintainers |
| Headers | show |
| Series | PHC support in ENA driver | expand |
On 10/21, David Arinzon wrote: > Provide the relevant information and guidelines > about the feature support in the ENA driver. > > Signed-off-by: Amit Bernstein <amitbern@amazon.com> > Signed-off-by: David Arinzon <darinzon@amazon.com> > --- > .../device_drivers/ethernet/amazon/ena.rst | 78 +++++++++++++++++++ > 1 file changed, 78 insertions(+) > > diff --git a/Documentation/networking/device_drivers/ethernet/amazon/ena.rst b/Documentation/networking/device_drivers/ethernet/amazon/ena.rst > index 4561e8ab..9f490bb8 100644 > --- a/Documentation/networking/device_drivers/ethernet/amazon/ena.rst > +++ b/Documentation/networking/device_drivers/ethernet/amazon/ena.rst > @@ -56,6 +56,7 @@ ena_netdev.[ch] Main Linux kernel driver. > ena_ethtool.c ethtool callbacks. > ena_xdp.[ch] XDP files > ena_pci_id_tbl.h Supported device IDs. > +ena_phc.[ch] PTP hardware clock infrastructure (see `PHC`_ for more info) > ================= ====================================================== > > Management Interface: > @@ -221,6 +222,83 @@ descriptor it was received on would be recycled. When a packet smaller > than RX copybreak bytes is received, it is copied into a new memory > buffer and the RX descriptor is returned to HW. > > +.. _`PHC`: > + > +PTP Hardware Clock (PHC) > +====================== > +.. _`ptp-userspace-api`: https://docs.kernel.org/driver-api/ptp.html#ptp-hardware-clock-user-space-api > +.. _`testptp`: https://elixir.bootlin.com/linux/latest/source/tools/testing/selftests/ptp/testptp.c nit: Documentation/networking/device_drivers/ethernet/amazon/ena.rst:228: WARNING: Title underline too short. --- pw-bot: cr
> > Provide the relevant information and guidelines about the feature > > support in the ENA driver. > > > > Signed-off-by: Amit Bernstein <amitbern@amazon.com> > > Signed-off-by: David Arinzon <darinzon@amazon.com> > > --- > > .../device_drivers/ethernet/amazon/ena.rst | 78 > +++++++++++++++++++ > > 1 file changed, 78 insertions(+) > > > > diff --git > > a/Documentation/networking/device_drivers/ethernet/amazon/ena.rst > > b/Documentation/networking/device_drivers/ethernet/amazon/ena.rst > > index 4561e8ab..9f490bb8 100644 > > --- > a/Documentation/networking/device_drivers/ethernet/amazon/ena.rst > > +++ > b/Documentation/networking/device_drivers/ethernet/amazon/ena.rst > > @@ -56,6 +56,7 @@ ena_netdev.[ch] Main Linux kernel driver. > > ena_ethtool.c ethtool callbacks. > > ena_xdp.[ch] XDP files > > ena_pci_id_tbl.h Supported device IDs. > > +ena_phc.[ch] PTP hardware clock infrastructure (see `PHC`_ for more > info) > > ================= > ====================================================== > > > > Management Interface: > > @@ -221,6 +222,83 @@ descriptor it was received on would be recycled. > > When a packet smaller than RX copybreak bytes is received, it is > > copied into a new memory buffer and the RX descriptor is returned to HW. > > > > +.. _`PHC`: > > + > > +PTP Hardware Clock (PHC) > > +====================== > > +.. _`ptp-userspace-api`: > > +https://docs.kernel.org/driver-api/ptp.html#ptp-hardware-clock-user-s > > +pace-api .. _`testptp`: > > +https://elixir.bootlin.com/linux/latest/source/tools/testing/selftest > > +s/ptp/testptp.c > > nit: > Documentation/networking/device_drivers/ethernet/amazon/ena.rst:228: > WARNING: Title underline too short. > > --- > pw-bot: cr Thanks for raising this, will fix in the next revision
On Mon, 2024-10-21 at 08:20 +0300, David Arinzon wrote: > > + testptp -d /dev/ptp$(ethtool -T <interface> | awk '/PTP Hardware Clock:/ {print $NF}') -k 1 > + Does udev create a stable symlink for this, like it does for e.g. /dev/ptp_kvm ? I note the EC2 documentation *still* tells users to use /dev/ptp0 without even checking which device that is, so they could get something *very* different to what they expect, if they accidentally start using the KVM PTP clock instead!
> > + testptp -d /dev/ptp$(ethtool -T <interface> | awk '/PTP Hardware > Clock:/ {print $NF}') -k 1 > > + > > Does udev create a stable symlink for this, like it does for e.g. > /dev/ptp_kvm ? > Yes, you can add a specific rule for the ena ptp device SUBSYSTEM=="ptp", ATTR{clock_name}=="ena-ptp-*", SYMLINK += "ena-ptp" > I note the EC2 documentation *still* tells users to use /dev/ptp0 > without even checking which device that is, so they could get something > *very* different to what they expect, if they accidentally start using > the KVM PTP clock instead!
On Tue, 2024-10-22 at 14:17 +0000, Arinzon, David wrote: > > > + testptp -d /dev/ptp$(ethtool -T <interface> | awk '/PTP > > > Hardware > > Clock:/ {print $NF}') -k 1 > > > + > > > > Does udev create a stable symlink for this, like it does for e.g. > > /dev/ptp_kvm ? > > > > Yes, you can add a specific rule for the ena ptp device > > SUBSYSTEM=="ptp", ATTR{clock_name}=="ena-ptp-*", SYMLINK += "ena-ptp" I don't see it here yet though: https://github.com/systemd/systemd/blob/main/rules.d/50-udev-default.rules.in#L33
> > > > + testptp -d /dev/ptp$(ethtool -T <interface> | awk '/PTP > > > > Hardware > > > Clock:/ {print $NF}') -k 1 > > > > + > > > > > > Does udev create a stable symlink for this, like it does for e.g. > > > /dev/ptp_kvm ? > > > > > > > Yes, you can add a specific rule for the ena ptp device > > > > SUBSYSTEM=="ptp", ATTR{clock_name}=="ena-ptp-*", SYMLINK += "ena- > ptp" > > I don't see it here yet though: > > https://github.com/systemd/systemd/blob/main/rules.d/50-udev- > default.rules.in#L33 We will add a pull request to systemd which adds the above suggested rule.
diff --git a/Documentation/networking/device_drivers/ethernet/amazon/ena.rst b/Documentation/networking/device_drivers/ethernet/amazon/ena.rst index 4561e8ab..9f490bb8 100644 --- a/Documentation/networking/device_drivers/ethernet/amazon/ena.rst +++ b/Documentation/networking/device_drivers/ethernet/amazon/ena.rst @@ -56,6 +56,7 @@ ena_netdev.[ch] Main Linux kernel driver. ena_ethtool.c ethtool callbacks. ena_xdp.[ch] XDP files ena_pci_id_tbl.h Supported device IDs. +ena_phc.[ch] PTP hardware clock infrastructure (see `PHC`_ for more info) ================= ====================================================== Management Interface: @@ -221,6 +222,83 @@ descriptor it was received on would be recycled. When a packet smaller than RX copybreak bytes is received, it is copied into a new memory buffer and the RX descriptor is returned to HW. +.. _`PHC`: + +PTP Hardware Clock (PHC) +====================== +.. _`ptp-userspace-api`: https://docs.kernel.org/driver-api/ptp.html#ptp-hardware-clock-user-space-api +.. _`testptp`: https://elixir.bootlin.com/linux/latest/source/tools/testing/selftests/ptp/testptp.c + +ENA Linux driver supports PTP hardware clock providing timestamp reference to achieve nanosecond accuracy. + +**PHC support** + +PHC depends on the PTP module, which needs to be either loaded as a module or compiled into the kernel. + +Verify if the PTP module is present: + +.. code-block:: shell + + grep -w '^CONFIG_PTP_1588_CLOCK=[ym]' /boot/config-`uname -r` + +- If no output is provided, the ENA driver cannot be loaded with PHC support. + +- ``CONFIG_PTP_1588_CLOCK=y``: the PTP module is already compiled and loaded inside the kernel binary file. + +- ``CONFIG_PTP_1588_CLOCK=m``: the PTP module needs to be loaded prior to loading the ENA driver: + +Load PTP module: + +.. code-block:: shell + + sudo modprobe ptp + +All available PTP clock sources can be tracked here: + +.. code-block:: shell + + ls /sys/class/ptp + +PHC support and capabilities can be verified using ethtool: + +.. code-block:: shell + + ethtool -T <interface> + +**PHC timestamp** + +To retrieve PHC timestamp, use `ptp-userspace-api`_, usage example using `testptp`_: + +.. code-block:: shell + + testptp -d /dev/ptp$(ethtool -T <interface> | awk '/PTP Hardware Clock:/ {print $NF}') -k 1 + +PHC get time requests should be within reasonable bounds, +avoid excessive utilization to ensure optimal performance and efficiency. +The ENA device restricts the frequency of PHC get time requests to a maximum +of 125 requests per second. If this limit is surpassed, the get time request +will fail, leading to an increment in the phc_err statistic. + +**PHC statistics** + +PHC can be monitored using :code:`ethtool -S` counters: + +================= ====================================================== +**phc_cnt** Number of successful retrieved timestamps (below expire timeout). +**phc_exp** Number of expired retrieved timestamps (above expire timeout). +**phc_skp** Number of skipped get time attempts (during block period). +**phc_err** Number of failed get time attempts (entering into block state). +================= ====================================================== + +PHC timeouts: + +================= ====================================================== +**expire** Max time for a valid timestamp retrieval, passing this threshold will fail + the get time request and block new requests until block timeout. +**block** Blocking period starts once get time request expires or fails, all get time + requests during block period will be skipped. +================= ====================================================== + Statistics ==========