| Message ID | 20220927130656.32567-1-muhammad.husaini.zulkifli@intel.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Add support for DMA timestamp for non-PTP packets | expand |
On Tue, Sep 27, 2022 at 09:06:52PM +0800, Muhammad Husaini Zulkifli wrote: > The HW TX timestamps created by the NIC via socket options can be > requested using the current network timestamps generation capability of > SOF_TIMESTAMPING_TX_HARDWARE. The most common users of this socket flag > is PTP, however other packet applications that require tx timestamps might > also ask for it. > > The problem is that, when there is a lot of traffic, there is a high chance > that the timestamps for a PTP packet will be lost if both PTP and Non-PTP > packets use the same SOF TIMESTAMPING TX HARDWARE causing the tx timeout. > > DMA timestamps through socket options are not currently available to > the user. Because if the user wants to, they can configure the hwtstamp > config option to use the new introduced DMA Time Stamp flag through the > setsockopt(). > > With these additional socket options, users can continue to utilise > HW timestamps for PTP while specifying non-PTP packets to use DMA > timestamps if the NIC can support them. Although this is not actually for PTP, you probably should Cc: the PTP maintainer for patches like this. Andrew
On Tue, 27 Sep 2022 21:06:52 +0800 Muhammad Husaini Zulkifli wrote: > The HW TX timestamps created by the NIC via socket options can be > requested using the current network timestamps generation capability of > SOF_TIMESTAMPING_TX_HARDWARE. The most common users of this socket flag > is PTP, however other packet applications that require tx timestamps might > also ask for it. > > The problem is that, when there is a lot of traffic, there is a high chance > that the timestamps for a PTP packet will be lost if both PTP and Non-PTP > packets use the same SOF TIMESTAMPING TX HARDWARE causing the tx timeout. > > DMA timestamps through socket options are not currently available to > the user. Because if the user wants to, they can configure the hwtstamp > config option to use the new introduced DMA Time Stamp flag through the > setsockopt(). > > With these additional socket options, users can continue to utilise > HW timestamps for PTP while specifying non-PTP packets to use DMA > timestamps if the NIC can support them. > > Any socket application can be use to verify this. > TSN Ref SW application is been used for testing by changing as below: Very glad to see this being worked on! High level tho, are we assuming that the existing HW timestamps are always PTP-quality, i.e. captured when SFD crosses the RS layer, or whatnot? I'm afraid some NICs already report PCI stamps as the HW ones. So the existing HW stamps are conceptually of "any" type, if we want to be 100% sure NIC actually stamps at the PHY we'd need another tx_type to express that. Same story on the Rx - what do you plan to do there? We'll need to configure the filters per type, but that's likely to mean two new filters, because the current one gives no guarantee.
Hi Andrew, Thanks for reviewing. Please see my reply in-line. > -----Original Message----- > From: Andrew Lunn <andrew@lunn.ch> > Sent: Tuesday, 27 September, 2022 11:27 PM > To: Zulkifli, Muhammad Husaini <muhammad.husaini.zulkifli@intel.com> > Cc: intel-wired-lan@osuosl.org; netdev@vger.kernel.org; > davem@davemloft.net; kuba@kernel.org; edumazet@google.com; Gomes, > Vinicius <vinicius.gomes@intel.com>; Gunasekaran, Aravindhan > <aravindhan.gunasekaran@intel.com>; Ahmad Tarmizi, Noor Azura > <noor.azura.ahmad.tarmizi@intel.com> > Subject: Re: [PATCH v1 0/4] Add support for DMA timestamp for non-PTP > packets > > On Tue, Sep 27, 2022 at 09:06:52PM +0800, Muhammad Husaini Zulkifli > wrote: > > The HW TX timestamps created by the NIC via socket options can be > > requested using the current network timestamps generation capability > > of SOF_TIMESTAMPING_TX_HARDWARE. The most common users of this > socket > > flag is PTP, however other packet applications that require tx > > timestamps might also ask for it. > > > > The problem is that, when there is a lot of traffic, there is a high > > chance that the timestamps for a PTP packet will be lost if both PTP > > and Non-PTP packets use the same SOF TIMESTAMPING TX HARDWARE > causing the tx timeout. > > > > DMA timestamps through socket options are not currently available to > > the user. Because if the user wants to, they can configure the > > hwtstamp config option to use the new introduced DMA Time Stamp flag > > through the setsockopt(). > > > > With these additional socket options, users can continue to utilise HW > > timestamps for PTP while specifying non-PTP packets to use DMA > > timestamps if the NIC can support them. > > Although this is not actually for PTP, you probably should Cc: the PTP > maintainer for patches like this. Thanks. Added Richard in the CC list. > > Andrew
Hi Jakub, Thanks for reviewing. Please see my reply in-line. > -----Original Message----- > From: Jakub Kicinski <kuba@kernel.org> > Sent: Wednesday, 28 September, 2022 8:09 AM > To: Zulkifli, Muhammad Husaini <muhammad.husaini.zulkifli@intel.com> > Cc: intel-wired-lan@osuosl.org; netdev@vger.kernel.org; > davem@davemloft.net; edumazet@google.com; Gomes, Vinicius > <vinicius.gomes@intel.com>; Gunasekaran, Aravindhan > <aravindhan.gunasekaran@intel.com>; Ahmad Tarmizi, Noor Azura > <noor.azura.ahmad.tarmizi@intel.com>; Richard Cochran > <richardcochran@gmail.com> > Subject: Re: [PATCH v1 0/4] Add support for DMA timestamp for non-PTP > packets > > On Tue, 27 Sep 2022 21:06:52 +0800 Muhammad Husaini Zulkifli wrote: > > The HW TX timestamps created by the NIC via socket options can be > > requested using the current network timestamps generation capability > > of SOF_TIMESTAMPING_TX_HARDWARE. The most common users of this > socket > > flag is PTP, however other packet applications that require tx > > timestamps might also ask for it. > > > > The problem is that, when there is a lot of traffic, there is a high > > chance that the timestamps for a PTP packet will be lost if both PTP > > and Non-PTP packets use the same SOF TIMESTAMPING TX HARDWARE > causing the tx timeout. > > > > DMA timestamps through socket options are not currently available to > > the user. Because if the user wants to, they can configure the > > hwtstamp config option to use the new introduced DMA Time Stamp flag > > through the setsockopt(). > > > > With these additional socket options, users can continue to utilise HW > > timestamps for PTP while specifying non-PTP packets to use DMA > > timestamps if the NIC can support them. > > > > Any socket application can be use to verify this. > > TSN Ref SW application is been used for testing by changing as below: > > Very glad to see this being worked on! > > High level tho, are we assuming that the existing HW timestamps are always > PTP-quality, i.e. captured when SFD crosses the RS layer, or whatnot? I'm > afraid some NICs already report PCI stamps as the HW ones. Yes. HW timestamps always can be assume equivalent to PTP quality. Could you provide additional information regarding SFD crosses the RS layer? According to what I observed, The HW Timestamps will be requested if the application side specifies tx type = HWTSTAMP TX ON and timestamping flags = SOF TIMESTAMPING TX HARDWARE. So it depends on how the application used it. > So the existing HW stamps are conceptually of "any" type, if we want to be > 100% sure NIC actually stamps at the PHY we'd need another tx_type to > express that. Yes, you're right. Are you suggesting that we add a new tx_type to specify Only MAC/PHY timestamp ? Ex. HWTSTAMP_TX_PHY/MAC_ON. Sorry about the naming here. Just so you know, the DMA timestamp does not quite match the PTP's level timestamping. The DMA timestamp will be capture when DMA request to fetch the data from the memory. > > Same story on the Rx - what do you plan to do there? We'll need to configure > the filters per type, but that's likely to mean two new filters, because the > current one gives no guarantee. Current I225 HW only allow to retrieve the dma time for TX packets only. So as of now based on our HW, on RX side we just requesting rx filter to timestamps any incoming packets. We always allocating additional bytes in the packet buffer for the receive packets for timestamp. It is a 1588 PTP level kind of timestamping accuracy here.
On Thu, 29 Sep 2022 02:35:29 +0000 Zulkifli, Muhammad Husaini wrote: > > High level tho, are we assuming that the existing HW timestamps are always > > PTP-quality, i.e. captured when SFD crosses the RS layer, or whatnot? I'm > > afraid some NICs already report PCI stamps as the HW ones. > > Yes. HW timestamps always can be assume equivalent to PTP quality. > Could you provide additional information regarding SFD crosses the RS layer? I mean true PTP timestamps, rather than captured somewhere in the NIC pipeline or at the DMA engine. > According to what I observed, The HW Timestamps will be requested if the application side > specifies tx type = HWTSTAMP TX ON and timestamping flags = SOF TIMESTAMPING TX HARDWARE. > So it depends on how the application used it. > > > So the existing HW stamps are conceptually of "any" type, if we want to be > > 100% sure NIC actually stamps at the PHY we'd need another tx_type to > > express that. > > Yes, you're right. Are you suggesting that we add a new tx_type to specify > Only MAC/PHY timestamp ? Ex. HWTSTAMP_TX_PHY/MAC_ON. Perhaps we can call them HWTSTAMP_TX_PTP_* ? Was the general time stamping requirement specified in IEEE 1588 or 802.1 (AS?)? Both MAC and PHY can provide the time stamps IIUC, so picking one of those will not be entirely fortunate. In fact perhaps embedded folks will use this opportunity to disambiguate the two.. > Sorry about the naming here. Just so you know, the DMA timestamp does not > quite match the PTP's level timestamping. The DMA timestamp will be capture when > DMA request to fetch the data from the memory. > > > > > Same story on the Rx - what do you plan to do there? We'll need to configure > > the filters per type, but that's likely to mean two new filters, because the > > current one gives no guarantee. > > Current I225 HW only allow to retrieve the dma time for TX packets only. > So as of now based on our HW, on RX side we just requesting rx filter to timestamps any incoming packets. > We always allocating additional bytes in the packet buffer for the receive packets for timestamp. > It is a 1588 PTP level kind of timestamping accuracy here. I see. I think datacenter NICs can provide DMA stamps for Rx as well. Intel, Mellanox, Broadcom folks, could you confirm if your NIC can do Rx DMA stamps?
On 29/09/2022 16:56, Jakub Kicinski wrote: > On Thu, 29 Sep 2022 02:35:29 +0000 Zulkifli, Muhammad Husaini wrote: >>> High level tho, are we assuming that the existing HW timestamps are always >>> PTP-quality, i.e. captured when SFD crosses the RS layer, or whatnot? I'm >>> afraid some NICs already report PCI stamps as the HW ones. >> Yes. HW timestamps always can be assume equivalent to PTP quality. >> Could you provide additional information regarding SFD crosses the RS layer? > I mean true PTP timestamps, rather than captured somewhere in the NIC > pipeline or at the DMA engine. > >> According to what I observed, The HW Timestamps will be requested if the application side >> specifies tx type = HWTSTAMP TX ON and timestamping flags = SOF TIMESTAMPING TX HARDWARE. >> So it depends on how the application used it. >> >>> So the existing HW stamps are conceptually of "any" type, if we want to be >>> 100% sure NIC actually stamps at the PHY we'd need another tx_type to >>> express that. >> Yes, you're right. Are you suggesting that we add a new tx_type to specify >> Only MAC/PHY timestamp ? Ex. HWTSTAMP_TX_PHY/MAC_ON. > Perhaps we can call them HWTSTAMP_TX_PTP_* ? Was the general time > stamping requirement specified in IEEE 1588 or 802.1 (AS?)? > > Both MAC and PHY can provide the time stamps IIUC, so picking one of > those will not be entirely fortunate. In fact perhaps embedded folks > will use this opportunity to disambiguate the two.. > >> Sorry about the naming here. Just so you know, the DMA timestamp does not >> quite match the PTP's level timestamping. The DMA timestamp will be capture when >> DMA request to fetch the data from the memory. >> >>> Same story on the Rx - what do you plan to do there? We'll need to configure >>> the filters per type, but that's likely to mean two new filters, because the >>> current one gives no guarantee. >> Current I225 HW only allow to retrieve the dma time for TX packets only. >> So as of now based on our HW, on RX side we just requesting rx filter to timestamps any incoming packets. >> We always allocating additional bytes in the packet buffer for the receive packets for timestamp. >> It is a 1588 PTP level kind of timestamping accuracy here. > I see. I think datacenter NICs can provide DMA stamps for Rx as well. > Intel, Mellanox, Broadcom folks, could you confirm if your NIC can do Rx > DMA stamps? What exactly do you mean by DMA stamps? Our NIC supports two modes of operation (both TX/RX): - CQE timestamp (I think that's what you call DMA timestamp), where the timestamp is written when the completion is being written/generated. - Port timestamp (MAC timestamp), where the timstamp is written when the packet is being sent to the wire, or received from the wire. This doesn't account for the time the packet spent inside the NIC pipeline. So I believe the answer to your question is yes :).
On Thu, 29 Sep 2022 17:46:04 +0300 Gal Pressman wrote: > What exactly do you mean by DMA stamps? > > Our NIC supports two modes of operation (both TX/RX): > - CQE timestamp (I think that's what you call DMA timestamp), where the > timestamp is written when the completion is being written/generated. > - Port timestamp (MAC timestamp), where the timstamp is written when the > packet is being sent to the wire, or received from the wire. This > doesn't account for the time the packet spent inside the NIC pipeline. > > So I believe the answer to your question is yes :). Thanks! I think we should provide the config API for both Tx and Rx, then.
> -----Original Message----- > From: Jakub Kicinski <kuba@kernel.org> > Sent: Thursday, 29 September, 2022 9:56 PM > To: Zulkifli, Muhammad Husaini <muhammad.husaini.zulkifli@intel.com> > Cc: intel-wired-lan@osuosl.org; netdev@vger.kernel.org; > davem@davemloft.net; edumazet@google.com; Gomes, Vinicius > <vinicius.gomes@intel.com>; Gunasekaran, Aravindhan > <aravindhan.gunasekaran@intel.com>; Ahmad Tarmizi, Noor Azura > <noor.azura.ahmad.tarmizi@intel.com>; Richard Cochran > <richardcochran@gmail.com>; Gal Pressman <gal@nvidia.com>; Saeed > Mahameed <saeed@kernel.org>; leon@kernel.org; Michael Chan > <michael.chan@broadcom.com>; Andy Gospodarek <andy@greyhouse.net> > Subject: Re: [PATCH v1 0/4] Add support for DMA timestamp for non-PTP > packets > > On Thu, 29 Sep 2022 02:35:29 +0000 Zulkifli, Muhammad Husaini wrote: > > > High level tho, are we assuming that the existing HW timestamps are > > > always PTP-quality, i.e. captured when SFD crosses the RS layer, or > > > whatnot? I'm afraid some NICs already report PCI stamps as the HW > ones. > > > > Yes. HW timestamps always can be assume equivalent to PTP quality. > > Could you provide additional information regarding SFD crosses the RS > layer? > > I mean true PTP timestamps, rather than captured somewhere in the NIC > pipeline or at the DMA engine. When SOF_TIMESTAMPING_TX_HARDWARE is been used, we guaranteed a PTP quality Timestamps (timestamp capture when packet leave the wire upon sensing the SFD). As of SOF_TIMESTAMPING_TX_HARDWARE_DMA_FETCH, it is not a PTP quality because the HW timestamp reported in this case, is a time when the data is DMA'ed into the NIC packet buffer. > > > According to what I observed, The HW Timestamps will be requested if > > the application side specifies tx type = HWTSTAMP TX ON and > timestamping flags = SOF TIMESTAMPING TX HARDWARE. > > So it depends on how the application used it. > > > > > So the existing HW stamps are conceptually of "any" type, if we want > > > to be 100% sure NIC actually stamps at the PHY we'd need another > > > tx_type to express that. > > > > Yes, you're right. Are you suggesting that we add a new tx_type to > > specify Only MAC/PHY timestamp ? Ex. HWTSTAMP_TX_PHY/MAC_ON. > > Perhaps we can call them HWTSTAMP_TX_PTP_* ? Was the general time > stamping requirement specified in IEEE 1588 or 802.1 (AS?)? > > Both MAC and PHY can provide the time stamps IIUC, so picking one of those > will not be entirely fortunate. In fact perhaps embedded folks will use this > opportunity to disambiguate the two.. With the help of SOF_TIMESTAMPING_TX_HARDWARE, we will get the PHY level timestamp(PTP quality) while using SOF_TIMESTAMPING_TX_HARDWARE_DMA_FETCH, we will get the timestamp at a point in the NIC pipeline. Linuxptp application uses SOF_TIMESTAMPING_TX_HARDWARE for their socket option. And this can guarantee a PTP quality timestamp. Can we just use a SOF_TIMESTAMPING to identify which timestamp that we want rather than creating a new tx_type? > > > Sorry about the naming here. Just so you know, the DMA timestamp does > > not quite match the PTP's level timestamping. The DMA timestamp will > > be capture when DMA request to fetch the data from the memory. > > > > > > > > Same story on the Rx - what do you plan to do there? We'll need to > > > configure the filters per type, but that's likely to mean two new > > > filters, because the current one gives no guarantee. > > > > Current I225 HW only allow to retrieve the dma time for TX packets only. > > So as of now based on our HW, on RX side we just requesting rx filter to > timestamps any incoming packets. > > We always allocating additional bytes in the packet buffer for the receive > packets for timestamp. > > It is a 1588 PTP level kind of timestamping accuracy here. > > I see. I think datacenter NICs can provide DMA stamps for Rx as well. > Intel, Mellanox, Broadcom folks, could you confirm if your NIC can do Rx > DMA stamps?
On Fri, 30 Sep 2022 08:52:38 +0000 Zulkifli, Muhammad Husaini wrote: > > > Yes. HW timestamps always can be assume equivalent to PTP quality. > > > Could you provide additional information regarding SFD crosses the RS > > > layer? > > > > I mean true PTP timestamps, rather than captured somewhere in the NIC > > pipeline or at the DMA engine. > > When SOF_TIMESTAMPING_TX_HARDWARE is been used, we guaranteed a PTP quality > Timestamps (timestamp capture when packet leave the wire upon sensing the SFD). > As of SOF_TIMESTAMPING_TX_HARDWARE_DMA_FETCH, it is not a PTP quality because > the HW timestamp reported in this case, is a time when the data is > DMA'ed into the NIC packet buffer. I understand that _your_ device does it right. But there are vendors out there who treat SOF_TIMESTAMPING_TX_HARDWARE as your new SOF_TIMESTAMPING_TX_HARDWARE_DMA_FETCH. > > > Yes, you're right. Are you suggesting that we add a new tx_type to > > > specify Only MAC/PHY timestamp ? Ex. HWTSTAMP_TX_PHY/MAC_ON. > > > > Perhaps we can call them HWTSTAMP_TX_PTP_* ? Was the general time > > stamping requirement specified in IEEE 1588 or 802.1 (AS?)? > > > > Both MAC and PHY can provide the time stamps IIUC, so picking one of those > > will not be entirely fortunate. In fact perhaps embedded folks will use this > > opportunity to disambiguate the two.. > > With the help of SOF_TIMESTAMPING_TX_HARDWARE, we will get the > PHY level timestamp(PTP quality) while using SOF_TIMESTAMPING_TX_HARDWARE_DMA_FETCH, > we will get the timestamp at a point in the NIC pipeline. > > Linuxptp application uses SOF_TIMESTAMPING_TX_HARDWARE for their socket option. > And this can guarantee a PTP quality timestamp. > > Can we just use a SOF_TIMESTAMPING to identify which timestamp that we want rather > than creating a new tx_type? Hm, perhaps, yes, we can stick to that.
The HW TX timestamps created by the NIC via socket options can be requested using the current network timestamps generation capability of SOF_TIMESTAMPING_TX_HARDWARE. The most common users of this socket flag is PTP, however other packet applications that require tx timestamps might also ask for it. The problem is that, when there is a lot of traffic, there is a high chance that the timestamps for a PTP packet will be lost if both PTP and Non-PTP packets use the same SOF TIMESTAMPING TX HARDWARE causing the tx timeout. DMA timestamps through socket options are not currently available to the user. Because if the user wants to, they can configure the hwtstamp config option to use the new introduced DMA Time Stamp flag through the setsockopt(). With these additional socket options, users can continue to utilise HW timestamps for PTP while specifying non-PTP packets to use DMA timestamps if the NIC can support them. Any socket application can be use to verify this. TSN Ref SW application is been used for testing by changing as below: int timestamping_flags = SOF_TIMESTAMPING_TX_HARDWARE_DMA_FETCH; strncpy(hwtstamp.ifr_name, opt->ifname, sizeof(hwtstamp.ifr_name)-1); hwtstamp.ifr_data = (void *)&hwconfig; hwconfig.tx_type = HWTSTAMP_TX_ON; hwconfig.flags = HWTSTAMP_FLAG_DMA_TIMESTAMP; hwconfig.rx_filter = HWTSTAMP_FILTER_ALL; if (ioctl(sock, SIOCSHWTSTAMP, &hwtstamp) < 0) { fprintf(stderr, "%s: %s\n", "ioctl", strerror(errno)); exit(1); } if (setsockopt(sock, SOL_SOCKET, SO_TIMESTAMPING, ×tamping_flags, sizeof(timestamping_flags)) < 0) exit_with_error("setsockopt SO_TIMESTAMPING"); Muhammad Husaini Zulkifli (3): ethtool: Add new hwtstamp flag net-timestamp: Increase the size of tsflags net: sock: extend SO_TIMESTAMPING for DMA Fetch Vinicius Costa Gomes (1): igc: Add support for DMA timestamp for non-PTP packets drivers/net/ethernet/intel/igc/igc.h | 10 +++ drivers/net/ethernet/intel/igc/igc_base.h | 2 +- drivers/net/ethernet/intel/igc/igc_defines.h | 2 + drivers/net/ethernet/intel/igc/igc_ethtool.c | 5 +- drivers/net/ethernet/intel/igc/igc_main.c | 24 ++++-- drivers/net/ethernet/intel/igc/igc_ptp.c | 83 ++++++++++++++++++++ include/linux/skbuff.h | 3 + include/net/sock.h | 12 +-- include/uapi/linux/ethtool.h | 3 + include/uapi/linux/ethtool_netlink.h | 1 + include/uapi/linux/net_tstamp.h | 11 ++- net/ethtool/common.c | 7 ++ net/ethtool/common.h | 2 + net/ethtool/strset.c | 5 ++ net/ethtool/tsinfo.c | 17 ++++ net/socket.c | 5 +- 16 files changed, 175 insertions(+), 17 deletions(-) -- 2.17.1