| Message ID | 20221018010733.4765-1-muhammad.husaini.zulkifli@intel.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Add support for DMA timestamp for non-PTP packets | expand |
On Tue, Oct 18, 2022 at 09:07:28AM +0800, Muhammad Husaini Zulkifli wrote: > With these additional socket options, users can continue to utilise > HW timestamps for PTP while specifying non-PTP packets to use DMA > timestamps if the NIC can support them. What is the use case for reporting DMA transmit time? How is this better than using SOF_TIMESTAMPING_TX_SOFTWARE ? Thanks, Richard
Hi Richard, Thanks for your review. Replied inline. > -----Original Message----- > From: Richard Cochran <richardcochran@gmail.com> > Sent: Tuesday, 18 October, 2022 7:45 PM > To: Zulkifli, Muhammad Husaini <muhammad.husaini.zulkifli@intel.com> > Cc: intel-wired-lan@osuosl.org; netdev@vger.kernel.org; kuba@kernel.org; > davem@davemloft.net; edumazet@google.com; Gunasekaran, Aravindhan > <aravindhan.gunasekaran@intel.com>; gal@nvidia.com; saeed@kernel.org; > leon@kernel.org; michael.chan@broadcom.com; andy@greyhouse.net; > Gomes, Vinicius <vinicius.gomes@intel.com> > Subject: Re: [PATCH v2 0/5] Add support for DMA timestamp for non-PTP > packets > > On Tue, Oct 18, 2022 at 09:07:28AM +0800, Muhammad Husaini Zulkifli > wrote: > > > With these additional socket options, users can continue to utilise HW > > timestamps for PTP while specifying non-PTP packets to use DMA > > timestamps if the NIC can support them. > > What is the use case for reporting DMA transmit time? > > How is this better than using SOF_TIMESTAMPING_TX_SOFTWARE ? The ideal situation is to use SOF TIMESTAMPING TX SOFTWARE. Application side will undoubtedly want to utilize PHY timestamps as much as possible. But if every application requested the HW Tx timestamp, especially during periods of high load like TSN, in some cases when the controller only supports 1 HW timestamp, it would be a disaster. We will observe some missing timestamp reported later. Using the DMA transmit time for the tx timestamp, if supported, is one of the solutions that we propose here. > > > Thanks, > Richard
Hello Muhammad, On 18/10/2022 04:07, Muhammad Husaini Zulkifli wrote: > The problem is that, when there is a lot of traffic, there is a high chance > that the timestamps for a PTP packet will be lost if both PTP and Non-PTP > packets use the same SOF TIMESTAMPING TX HARDWARE causing the tx timeout. Why would the timestamp be affected by the amount of traffic? What tx timeout? > DMA timestamps through socket options are not currently available to > the user. Because if the user wants to, they can configure the hwtstamp > config option to use the new introduced DMA Time Stamp flag through the > setsockopt(). > > With these additional socket options, users can continue to utilise > HW timestamps for PTP while specifying non-PTP packets to use DMA > timestamps if the NIC can support them. Is it per socket? Will there be a way to know which types of timestamps are going to be used on queues setup time? > This patch series also add a new HWTSTAMP_FILTER_DMA_TIMESTAMP receive > filters. This filter can be configured for devices that support/allow the > DMA timestamp retrieval on receive side. So if I understand correctly, to solve the problem you described at the beginning, you'll disable port timestamp for all incoming packets? ptp included?
On Tue, 18 Oct 2022 09:07:28 +0800 Muhammad Husaini Zulkifli wrote: > v1 -> v2: > - Move to the end for the new enum. > - Add new HWTSTAMP_FILTER_DMA_TIMESTAMP receive filters. Did you address my feedback? How do we know if existing HWTSTAMP_FILTER_ALL is PHC quality of DMA?
On Tue, Oct 18, 2022 at 02:12:37PM +0000, Zulkifli, Muhammad Husaini wrote: > > What is the use case for reporting DMA transmit time? So the use case (mentioned below) is TSN? > > How is this better than using SOF_TIMESTAMPING_TX_SOFTWARE ? > > The ideal situation is to use SOF TIMESTAMPING TX SOFTWARE. > Application side will undoubtedly want to utilize PHY timestamps as much as possible. I'm asking about SOFTWARE not hardware time stamps. > But if every application requested the HW Tx timestamp, especially during periods of high load like TSN, > in some cases when the controller only supports 1 HW timestamp, it would be a disaster. But can't you offer 1 HW time stamp with many SW time stamps? Thanks, Rihcard
Hi Gal, > -----Original Message----- > From: Gal Pressman <gal@nvidia.com> > Sent: Tuesday, 18 October, 2022 10:23 PM > To: Zulkifli, Muhammad Husaini <muhammad.husaini.zulkifli@intel.com>; > intel-wired-lan@osuosl.org > Cc: netdev@vger.kernel.org; kuba@kernel.org; davem@davemloft.net; > edumazet@google.com; Gunasekaran, Aravindhan > <aravindhan.gunasekaran@intel.com>; richardcochran@gmail.com; > saeed@kernel.org; leon@kernel.org; michael.chan@broadcom.com; > andy@greyhouse.net; Gomes, Vinicius <vinicius.gomes@intel.com> > Subject: Re: [PATCH v2 0/5] Add support for DMA timestamp for non-PTP > packets > > Hello Muhammad, > > On 18/10/2022 04:07, Muhammad Husaini Zulkifli wrote: > > The problem is that, when there is a lot of traffic, there is a high > > chance that the timestamps for a PTP packet will be lost if both PTP > > and Non-PTP packets use the same SOF TIMESTAMPING TX HARDWARE > causing the tx timeout. > > Why would the timestamp be affected by the amount of traffic? > What tx timeout? Basically, the original timestamp register collect the timestamp at the PHY level and interrupt is triggered for the driver to read it. We observed timestamp missed issues when the traffic is high. DMA Timestamp guarantees the timestamp for every packets as it is delivered through descriptor write-back mechanism. When there is a timestamp missed, ptp4l application will complain there is a tx_timeout. > > > DMA timestamps through socket options are not currently available to > > the user. Because if the user wants to, they can configure the > > hwtstamp config option to use the new introduced DMA Time Stamp flag > > through the setsockopt(). > > > > With these additional socket options, users can continue to utilise HW > > timestamps for PTP while specifying non-PTP packets to use DMA > > timestamps if the NIC can support them. > > Is it per socket? Yes it is per socket. > Will there be a way to know which types of timestamps are going to be used > on queues setup time? We can get the which timestamp that is supported through "ethtool -T" command. May I know why you want to know which timestamp need to configure during queue setup? > > > This patch series also add a new HWTSTAMP_FILTER_DMA_TIMESTAMP > receive > > filters. This filter can be configured for devices that support/allow > > the DMA timestamp retrieval on receive side. > > So if I understand correctly, to solve the problem you described at the > beginning, you'll disable port timestamp for all incoming packets? ptp > included? For ptp, it will always use Port Timestamp. Other application that want to use DMA Timestamp can request for the same.
Hi Jakub, > -----Original Message----- > From: Jakub Kicinski <kuba@kernel.org> > Sent: Wednesday, 19 October, 2022 3:03 AM > To: Zulkifli, Muhammad Husaini <muhammad.husaini.zulkifli@intel.com> > Cc: intel-wired-lan@osuosl.org; netdev@vger.kernel.org; > davem@davemloft.net; edumazet@google.com; Gunasekaran, Aravindhan > <aravindhan.gunasekaran@intel.com>; richardcochran@gmail.com; > gal@nvidia.com; saeed@kernel.org; leon@kernel.org; > michael.chan@broadcom.com; andy@greyhouse.net; Gomes, Vinicius > <vinicius.gomes@intel.com> > Subject: Re: [PATCH v2 0/5] Add support for DMA timestamp for non-PTP > packets > > On Tue, 18 Oct 2022 09:07:28 +0800 Muhammad Husaini Zulkifli wrote: > > v1 -> v2: > > - Move to the end for the new enum. > > - Add new HWTSTAMP_FILTER_DMA_TIMESTAMP receive filters. > > Did you address my feedback? How do we know if existing > HWTSTAMP_FILTER_ALL is PHC quality of DMA? I apologize if I didn't respond to your feedback. If I recall properly, you agreed to only use the flag rather than creating a new tx type as a result of below conversation. https://lore.kernel.org/all/20220930074016.295cbfab@kernel.org/
On Thu, 20 Oct 2022 02:16:25 +0000 Zulkifli, Muhammad Husaini wrote: > > On Tue, 18 Oct 2022 09:07:28 +0800 Muhammad Husaini Zulkifli wrote: > > > v1 -> v2: > > > - Move to the end for the new enum. > > > - Add new HWTSTAMP_FILTER_DMA_TIMESTAMP receive filters. > > > > Did you address my feedback? How do we know if existing > > HWTSTAMP_FILTER_ALL is PHC quality of DMA? > > I apologize if I didn't respond to your feedback. If I recall properly, you agreed to only > use the flag rather than creating a new tx type as a result of below conversation. > https://lore.kernel.org/all/20220930074016.295cbfab@kernel.org/ My bad, I wasn't very clear. I meant to agree that if you prefer we can forgo adding a new tx_type, and depend on SOF_TIMESTAMPING_TX_* to advertise the capabilities and request particular type. But SOF_TIMESTAMPING_TX_* should still not assume that SOF_TIMESTAMPING_TX_HARDWARE is guaranteed to be measured close to the wire. I was just looking at some mlx5 patches from last night: https://lore.kernel.org/all/20221019063813.802772-5-saeed@kernel.org/ and if you look around you'll see that they advertise hardware timestamps but AFAIU only packets they match to be PTP will go on the special queue that gets real close-to-wire time: https://elixir.bootlin.com/linux/v6.1-rc1/source/drivers/net/ethernet/mellanox/mlx5/core/en/selq.c#L247 If application requests a HW stamp on a non-PTP packet (NTP? custom proto?), or PTP is running on a custom UDP port, or simply the FW/HW does not support this feature (mlx5 supports at least a decade's worth of HW generations) - the application will get a DMA stamp. Same for Rx - close-to-wire stamps may be more expensive for the NIC and some modern TCP congestion control algos (BBRv2, Swift) need HW timestamps of all packets but are okay with DMA stamps. So we need: 1) clear disambiguation when time stamps are *really* taken close-to-the-wire (CTW); 2) ability to configure Rx to provide CTW stamps for PTP/NTP frames and DMA stamps for all the rest for congestion control. I think extending the documentation in: Documentation/networking/timestamping.rst to explain the API you're adding and how it will allow the above cases to be satisfied will be a good start.
Hi Richard, > -----Original Message----- > From: Richard Cochran <richardcochran@gmail.com> > Sent: Wednesday, 19 October, 2022 10:37 AM > To: Zulkifli, Muhammad Husaini <muhammad.husaini.zulkifli@intel.com> > Cc: intel-wired-lan@osuosl.org; netdev@vger.kernel.org; kuba@kernel.org; > davem@davemloft.net; edumazet@google.com; Gunasekaran, Aravindhan > <aravindhan.gunasekaran@intel.com>; gal@nvidia.com; saeed@kernel.org; > leon@kernel.org; michael.chan@broadcom.com; andy@greyhouse.net; > Gomes, Vinicius <vinicius.gomes@intel.com> > Subject: Re: [PATCH v2 0/5] Add support for DMA timestamp for non-PTP > packets > > On Tue, Oct 18, 2022 at 02:12:37PM +0000, Zulkifli, Muhammad Husaini > wrote: > > > What is the use case for reporting DMA transmit time? > > So the use case (mentioned below) is TSN? Yes. TSN/Real Time applications. The most desirable is reporting the HW Timestamp. But if all real time applications in a heavy traffic load trying to use same HW Timestamp (ex. 1 HW Timestamp only available) there will be a failure event Like missing a timestamp. The purpose of DMA transmit time is to give the application a different option for using DMA Timestamp when this event occur. > > > > How is this better than using SOF_TIMESTAMPING_TX_SOFTWARE ? > > > > The ideal situation is to use SOF TIMESTAMPING TX SOFTWARE. > > Application side will undoubtedly want to utilize PHY timestamps as much > as possible. > > I'm asking about SOFTWARE not hardware time stamps. Sorry for misinterpreting SOFTWARE as HARDWARE in my previous reply. DMA Timestamping is definitely better than SOFTWARE timestamp because we sample the time at the controller MAC level. > > > But if every application requested the HW Tx timestamp, especially > > during periods of high load like TSN, in some cases when the controller only > supports 1 HW timestamp, it would be a disaster. > > But can't you offer 1 HW time stamp with many SW time stamps? Could you please provide additional details about this? What do you meant by offering 1 HW Timestamp with many SW timestamps? > > Thanks, > Rihcard
On Fri, Oct 21, 2022 at 12:25:13AM +0000, Zulkifli, Muhammad Husaini wrote: > Sorry for misinterpreting SOFTWARE as HARDWARE in my previous reply. > DMA Timestamping is definitely better than SOFTWARE timestamp because > we sample the time at the controller MAC level. Do you have numbers to back up this claim? > Could you please provide additional details about this? What do you meant by > offering 1 HW Timestamp with many SW timestamps? - Let the PTP stack use hardware time stamps. - Let all other applications use software time stamps. Hm? Thanks, Richard
On Fri, 21 Oct 2022 05:48:44 -0700 Richard Cochran wrote: > > Could you please provide additional details about this? What do you meant by > > offering 1 HW Timestamp with many SW timestamps? > > - Let the PTP stack use hardware time stamps. > > - Let all other applications use software time stamps. We do need HW stamps for congestion control, the Rx ring queuing (as well as Tx ring scheduling jitter) is very often in 10s of msec. Comparing the SW stamp to the HW stamp is where we derive the signal that the system is under excessive load.
On 19/10/2022 17:23, Zulkifli, Muhammad Husaini wrote: >>> DMA timestamps through socket options are not currently available to >>> the user. Because if the user wants to, they can configure the >>> hwtstamp config option to use the new introduced DMA Time Stamp flag >>> through the setsockopt(). >>> >>> With these additional socket options, users can continue to utilise HW >>> timestamps for PTP while specifying non-PTP packets to use DMA >>> timestamps if the NIC can support them. >> Is it per socket? > Yes it is per socket. > >> Will there be a way to know which types of timestamps are going to be used >> on queues setup time? > We can get the which timestamp that is supported through "ethtool -T" command. > May I know why you want to know which timestamp need to configure during queue setup? In mlx5 we need dedicated queues which support port timestamp. In today's implementation we have a setup time priv-flag which dictates whether these queues should be opened or not. > >>> This patch series also add a new HWTSTAMP_FILTER_DMA_TIMESTAMP >> receive >>> filters. This filter can be configured for devices that support/allow >>> the DMA timestamp retrieval on receive side. >> So if I understand correctly, to solve the problem you described at the >> beginning, you'll disable port timestamp for all incoming packets? ptp >> included? > For ptp, it will always use Port Timestamp. > Other application that want to use DMA Timestamp can request for the same. > How? When a packet arrives from the wire the hardware doesn't know if it's ptp or not. Does your hardware inspect the packet headers to decide when to timestamp it?
The HW TX timestamps created by the NIC via socket options can be requested using the current network timestamps generation capability of SOF_TIMESTAMPING_TX_HARDWARE. The most common users of this socket flag is PTP, however other packet applications that require tx timestamps might also ask for it. The problem is that, when there is a lot of traffic, there is a high chance that the timestamps for a PTP packet will be lost if both PTP and Non-PTP packets use the same SOF TIMESTAMPING TX HARDWARE causing the tx timeout. DMA timestamps through socket options are not currently available to the user. Because if the user wants to, they can configure the hwtstamp config option to use the new introduced DMA Time Stamp flag through the setsockopt(). With these additional socket options, users can continue to utilise HW timestamps for PTP while specifying non-PTP packets to use DMA timestamps if the NIC can support them. This patch series also add a new HWTSTAMP_FILTER_DMA_TIMESTAMP receive filters. This filter can be configured for devices that support/allow the DMA timestamp retrieval on receive side. Any socket application can be use to verify this. TSN Ref SW application is been used for testing by changing as below: int timestamping_flags = SOF_TIMESTAMPING_TX_HARDWARE_DMA_FETCH; strncpy(hwtstamp.ifr_name, opt->ifname, sizeof(hwtstamp.ifr_name)-1); hwtstamp.ifr_data = (void *)&hwconfig; hwconfig.tx_type = HWTSTAMP_TX_ON; hwconfig.flags = HWTSTAMP_FLAG_DMA_TIMESTAMP; hwconfig.rx_filter = HWTSTAMP_FILTER_ALL; if (ioctl(sock, SIOCSHWTSTAMP, &hwtstamp) < 0) { fprintf(stderr, "%s: %s\n", "ioctl", strerror(errno)); exit(1); } if (setsockopt(sock, SOL_SOCKET, SO_TIMESTAMPING, ×tamping_flags, sizeof(timestamping_flags)) < 0) exit_with_error("setsockopt SO_TIMESTAMPING"); v1 -> v2: - Move to the end for the new enum. - Add new HWTSTAMP_FILTER_DMA_TIMESTAMP receive filters. Muhammad Husaini Zulkifli (4): ethtool: Add new hwtstamp flag net-timestamp: Increase the size of tsflags net: sock: extend SO_TIMESTAMPING for DMA Fetch ethtool: Add support for HWTSTAMP_FILTER_DMA_TIMESTAMP Vinicius Costa Gomes (1): igc: Add support for DMA timestamp for non-PTP packets drivers/net/ethernet/intel/igc/igc.h | 10 +++ drivers/net/ethernet/intel/igc/igc_base.h | 2 +- drivers/net/ethernet/intel/igc/igc_defines.h | 2 + drivers/net/ethernet/intel/igc/igc_ethtool.c | 5 +- drivers/net/ethernet/intel/igc/igc_main.c | 24 ++++-- drivers/net/ethernet/intel/igc/igc_ptp.c | 84 ++++++++++++++++++++ include/linux/skbuff.h | 3 + include/net/sock.h | 12 +-- include/uapi/linux/ethtool.h | 3 + include/uapi/linux/ethtool_netlink.h | 1 + include/uapi/linux/net_tstamp.h | 14 +++- net/core/dev_ioctl.c | 1 + net/ethtool/common.c | 8 ++ net/ethtool/common.h | 2 + net/ethtool/strset.c | 5 ++ net/ethtool/tsinfo.c | 17 ++++ net/socket.c | 5 +- 17 files changed, 181 insertions(+), 17 deletions(-) -- 2.17.1