| Message ID | 20230414154902.2950535-1-yoong.siang.song@intel.com (mailing list archive) |
|---|---|
| State | Awaiting Upstream |
| Delegated to: | Netdev Maintainers |
| Headers | show |
| Series | [net,v3,1/1] igc: read before write to SRRCTL register | expand |
On 14/04/2023 17.49, Song Yoong Siang wrote: > igc_configure_rx_ring() function will be called as part of XDP program > setup. If Rx hardware timestamp is enabled prio to XDP program setup, > this timestamp enablement will be overwritten when buffer size is > written into SRRCTL register. > > Thus, this commit read the register value before write to SRRCTL > register. This commit is tested by using xdp_hw_metadata bpf selftest > tool. The tool enables Rx hardware timestamp and then attach XDP program > to igc driver. It will display hardware timestamp of UDP packet with > port number 9092. Below are detail of test steps and results. > > Command on DUT: > sudo ./xdp_hw_metadata <interface name> > > Command on Link Partner: > echo -n skb | nc -u -q1 <destination IPv4 addr> 9092 > > Result before this patch: > skb hwtstamp is not found! > > Result after this patch: > found skb hwtstamp = 1677800973.642836757 > > Optionally, read PHC to confirm the values obtained are almost the same: > Command: > sudo ./testptp -d /dev/ptp0 -g > Result: > clock time: 1677800973.913598978 or Fri Mar 3 07:49:33 2023 > > Fixes: fc9df2a0b520 ("igc: Enable RX via AF_XDP zero-copy") > Cc: <stable@vger.kernel.org> # 5.14+ > Signed-off-by: Song Yoong Siang <yoong.siang.song@intel.com> > Reviewed-by: Jacob Keller <jacob.e.keller@intel.com> > Reviewed-by: Jesper Dangaard Brouer <brouer@redhat.com> > --- LGTM, thank for the adjustments :-) Acked-by: Jesper Dangaard Brouer <brouer@redhat.com> > v2 -> v3: Refactor SRRCTL definitions to more human readable definitions > v1 -> v2: Fix indention > --- > drivers/net/ethernet/intel/igc/igc_base.h | 11 ++++++++--- > drivers/net/ethernet/intel/igc/igc_main.c | 7 +++++-- > 2 files changed, 13 insertions(+), 5 deletions(-) > > diff --git a/drivers/net/ethernet/intel/igc/igc_base.h b/drivers/net/ethernet/intel/igc/igc_base.h > index 7a992befca24..9f3827eda157 100644 > --- a/drivers/net/ethernet/intel/igc/igc_base.h > +++ b/drivers/net/ethernet/intel/igc/igc_base.h > @@ -87,8 +87,13 @@ union igc_adv_rx_desc { > #define IGC_RXDCTL_SWFLUSH 0x04000000 /* Receive Software Flush */ > > /* SRRCTL bit definitions */ > -#define IGC_SRRCTL_BSIZEPKT_SHIFT 10 /* Shift _right_ */ > -#define IGC_SRRCTL_BSIZEHDRSIZE_SHIFT 2 /* Shift _left_ */ > -#define IGC_SRRCTL_DESCTYPE_ADV_ONEBUF 0x02000000 > +#define IGC_SRRCTL_BSIZEPKT_MASK GENMASK(6, 0) > +#define IGC_SRRCTL_BSIZEPKT(x) FIELD_PREP(IGC_SRRCTL_BSIZEPKT_MASK, \ > + (x) / 1024) /* in 1 KB resolution */ > +#define IGC_SRRCTL_BSIZEHDR_MASK GENMASK(13, 8) > +#define IGC_SRRCTL_BSIZEHDR(x) FIELD_PREP(IGC_SRRCTL_BSIZEHDR_MASK, \ > + (x) / 64) /* in 64 bytes resolution */ > +#define IGC_SRRCTL_DESCTYPE_MASK GENMASK(27, 25) > +#define IGC_SRRCTL_DESCTYPE_ADV_ONEBUF FIELD_PREP(IGC_SRRCTL_DESCTYPE_MASK, 1) > > #endif /* _IGC_BASE_H */ > diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c > index 25fc6c65209b..a2d823e64609 100644 > --- a/drivers/net/ethernet/intel/igc/igc_main.c > +++ b/drivers/net/ethernet/intel/igc/igc_main.c > @@ -641,8 +641,11 @@ static void igc_configure_rx_ring(struct igc_adapter *adapter, > else > buf_size = IGC_RXBUFFER_2048; > > - srrctl = IGC_RX_HDR_LEN << IGC_SRRCTL_BSIZEHDRSIZE_SHIFT; > - srrctl |= buf_size >> IGC_SRRCTL_BSIZEPKT_SHIFT; > + srrctl = rd32(IGC_SRRCTL(reg_idx)); > + srrctl &= ~(IGC_SRRCTL_BSIZEPKT_MASK | IGC_SRRCTL_BSIZEHDR_MASK | > + IGC_SRRCTL_DESCTYPE_MASK); > + srrctl |= IGC_SRRCTL_BSIZEHDR(IGC_RX_HDR_LEN); > + srrctl |= IGC_SRRCTL_BSIZEPKT(buf_size); > srrctl |= IGC_SRRCTL_DESCTYPE_ADV_ONEBUF; > > wr32(IGC_SRRCTL(reg_idx), srrctl);
On 14.04.23 17:49, Song Yoong Siang wrote: > igc_configure_rx_ring() function will be called as part of XDP program > setup. If Rx hardware timestamp is enabled prio to XDP program setup, > this timestamp enablement will be overwritten when buffer size is > written into SRRCTL register. > Hi all, I'm actually searching for the root cause of a similar problem (RX timestamp lost) that I can reproduce here, but the setup is slightly different. The setup: - igc driver - i225/226 network card - When starting to transmit frames using XDP with zero copy enabled another application (ptp4l) complains about missing RX timestamps - Loading the XDP program has already been removed (not needed for TX only) - ptp4l is using the traditional socket API - The RX timestamps seem to come back once we stop sending frames using XDP The "zero copy support" enabled part is important. If ZC support is not requested everything works fine. Any ideas? Best regards, Florian
On Saturday, April 15, 2023 5:20 PM, Florian Bezdeka <florian.bezdeka@siemens.com> wrote: >On 14.04.23 17:49, Song Yoong Siang wrote: >> igc_configure_rx_ring() function will be called as part of XDP program >> setup. If Rx hardware timestamp is enabled prio to XDP program setup, >> this timestamp enablement will be overwritten when buffer size is >> written into SRRCTL register. >> > >Hi all, > >I'm actually searching for the root cause of a similar problem (RX timestamp lost) >that I can reproduce here, but the setup is slightly different. The setup: > >- igc driver >- i225/226 network card >- When starting to transmit frames using XDP with zero copy enabled > another application (ptp4l) complains about missing RX timestamps >- Loading the XDP program has already been removed (not needed for > TX only) >- ptp4l is using the traditional socket API >- The RX timestamps seem to come back once we stop sending frames > using XDP > >The "zero copy support" enabled part is important. If ZC support is not requested >everything works fine. > >Any ideas? > >Best regards, >Florian > Hi Florian, You means this patch does not help on your issue? Need to understand more on the setup and behavior to tell. Are ptp4l and XDP ZC Tx apps running on same queue or separate queue? I suggest you to run " sudo hwstamp_ctl -i eth0 -r 1 " multiple times in the middle to check the behavior. Thanks & Regards Siang
Hi Florian, > -----Original Message----- > From: Intel-wired-lan <intel-wired-lan-bounces@osuosl.org> On Behalf Of > Song, Yoong Siang > Sent: Sunday, 16 April, 2023 10:19 AM > To: Bezdeka, Florian <florian.bezdeka@siemens.com>; Brandeburg, Jesse > <jesse.brandeburg@intel.com>; Nguyen, Anthony L > <anthony.l.nguyen@intel.com>; David S . Miller <davem@davemloft.net>; > Eric Dumazet <edumazet@google.com>; Jakub Kicinski <kuba@kernel.org>; > Paolo Abeni <pabeni@redhat.com>; Alexei Starovoitov <ast@kernel.org>; > Daniel Borkmann <daniel@iogearbox.net>; Jesper Dangaard Brouer > <hawk@kernel.org>; John Fastabend <john.fastabend@gmail.com>; > Fijalkowski, Maciej <maciej.fijalkowski@intel.com>; Vedang Patel > <vedang.patel@intel.com>; Joseph, Jithu <jithu.joseph@intel.com>; Andre > Guedes <andre.guedes@intel.com>; Brouer, Jesper <brouer@redhat.com>; > Stanislav Fomichev <sdf@google.com>; Keller, Jacob E > <jacob.e.keller@intel.com>; David Laight <David.Laight@ACULAB.COM> > Cc: xdp-hints@xdp-project.net; netdev@vger.kernel.org; linux- > kernel@vger.kernel.org; stable@vger.kernel.org; intel-wired- > lan@lists.osuosl.org; bpf@vger.kernel.org > Subject: Re: [Intel-wired-lan] [PATCH net v3 1/1] igc: read before write to > SRRCTL register > > On Saturday, April 15, 2023 5:20 PM, Florian Bezdeka > <florian.bezdeka@siemens.com> wrote: > >On 14.04.23 17:49, Song Yoong Siang wrote: > >> igc_configure_rx_ring() function will be called as part of XDP > >> program setup. If Rx hardware timestamp is enabled prio to XDP > >> program setup, this timestamp enablement will be overwritten when > >> buffer size is written into SRRCTL register. > >> > > > >Hi all, > > > >I'm actually searching for the root cause of a similar problem (RX > >timestamp lost) that I can reproduce here, but the setup is slightly > different. The setup: > > > >- igc driver > >- i225/226 network card > >- When starting to transmit frames using XDP with zero copy enabled > > another application (ptp4l) complains about missing RX timestamps > >- Loading the XDP program has already been removed (not needed for > > TX only) > >- ptp4l is using the traditional socket API > >- The RX timestamps seem to come back once we stop sending frames > > using XDP > > > >The "zero copy support" enabled part is important. If ZC support is not > >requested everything works fine. > > > >Any ideas? Are you observing similar issue like below? ptp4l: timed out while polling for tx timestamp ptp4l: increasing tx_timestamp_timeout may correct this issue If yes, only TXSTAMPO register is used for both PTP and non-PTP packets in the current driver code. There is a possibility that the time stamp for a PTP packet will be lost when there is a lot of traffic when multiple applications request for hardware transmission timestamps. Few months back, I submitted a patch series to enable the DMA Timestamp for non-ptp packet which can resolve the above issue. https://lore.kernel.org/netdev/20221018010733.4765-1-muhammad.husaini.zulkifli@intel.com/T/ Will continuing back the activity soon. But you might want to try with series submitted by Vinicius as well. This patch series add support for four sets of timestamping register. https://patchwork.ozlabs.org/project/intel-wired-lan/patch/20230228054534.1093483-2-vinicius.gomes@intel.com/ https://patchwork.ozlabs.org/project/intel-wired-lan/patch/20230228054534.1093483-3-vinicius.gomes@intel.com/ https://patchwork.ozlabs.org/project/intel-wired-lan/patch/20230228054534.1093483-4-vinicius.gomes@intel.com/ Would it be possible for you to try any of the above patch series and see if that fixes your problem? Thanks, Husaini > > > >Best regards, > >Florian > > > > Hi Florian, > > You means this patch does not help on your issue? > Need to understand more on the setup and behavior to tell. > Are ptp4l and XDP ZC Tx apps running on same queue or separate queue? > I suggest you to run " sudo hwstamp_ctl -i eth0 -r 1 " multiple times in the > middle to check the behavior. > > Thanks & Regards > Siang > _______________________________________________ > Intel-wired-lan mailing list > Intel-wired-lan@osuosl.org > https://lists.osuosl.org/mailman/listinfo/intel-wired-lan
On 14/04/2023 22.05, Jesper Dangaard Brouer wrote: > > On 14/04/2023 17.49, Song Yoong Siang wrote: >> igc_configure_rx_ring() function will be called as part of XDP program >> setup. If Rx hardware timestamp is enabled prio to XDP program setup, >> this timestamp enablement will be overwritten when buffer size is >> written into SRRCTL register. >> >> Thus, this commit read the register value before write to SRRCTL >> register. This commit is tested by using xdp_hw_metadata bpf selftest >> tool. The tool enables Rx hardware timestamp and then attach XDP program >> to igc driver. It will display hardware timestamp of UDP packet with >> port number 9092. Below are detail of test steps and results. >> [...] >> >> Fixes: fc9df2a0b520 ("igc: Enable RX via AF_XDP zero-copy") >> Cc: <stable@vger.kernel.org> # 5.14+ >> Signed-off-by: Song Yoong Siang <yoong.siang.song@intel.com> >> Reviewed-by: Jacob Keller <jacob.e.keller@intel.com> >> Reviewed-by: Jesper Dangaard Brouer <brouer@redhat.com> >> --- > > LGTM, thank for the adjustments :-) > > Acked-by: Jesper Dangaard Brouer <brouer@redhat.com> > Tested-by: Jesper Dangaard Brouer <brouer@redhat.com> I can confirm that this patch fix the issue I experienced with igc. This patch clearly fixes a bug in igc when writing the SRRCTL register. (as bit 30 in register is "Timestamp Received Packet" which got cleared before). Florian might have found another bug around RX timestamps, but this patch should be safe and sane to apply as is. >> v2 -> v3: Refactor SRRCTL definitions to more human readable definitions >> v1 -> v2: Fix indention >> --- >> drivers/net/ethernet/intel/igc/igc_base.h | 11 ++++++++--- >> drivers/net/ethernet/intel/igc/igc_main.c | 7 +++++-- >> 2 files changed, 13 insertions(+), 5 deletions(-)
On Mon, 17 Apr 2023 02:53:13 +0000 Zulkifli, Muhammad Husaini wrote: > Are you observing similar issue like below? > ptp4l: timed out while polling for tx timestamp > ptp4l: increasing tx_timestamp_timeout may correct this issue > > If yes, only TXSTAMPO register is used for both PTP and non-PTP packets in > the current driver code. There is a possibility that the time stamp > for a PTP packet will be lost when there is a lot of traffic when multiple > applications request for hardware transmission timestamps. > Few months back, I submitted a patch series to enable the DMA > Timestamp for non-ptp packet which can resolve the above issue. > https://lore.kernel.org/netdev/20221018010733.4765-1-muhammad.husaini.zulkifli@intel.com/T/ > Will continuing back the activity soon. FWIW the work on selecting the source of the timestamp is progressing slowly: https://lore.kernel.org/all/20230406173308.401924-1-kory.maincent@bootlin.com/
On Mon, 2023-04-17 at 16:24 +0200, Jesper Dangaard Brouer wrote: > On 14/04/2023 22.05, Jesper Dangaard Brouer wrote: > > > > On 14/04/2023 17.49, Song Yoong Siang wrote: > > > igc_configure_rx_ring() function will be called as part of XDP program > > > setup. If Rx hardware timestamp is enabled prio to XDP program setup, > > > this timestamp enablement will be overwritten when buffer size is > > > written into SRRCTL register. > > > > > > Thus, this commit read the register value before write to SRRCTL > > > register. This commit is tested by using xdp_hw_metadata bpf selftest > > > tool. The tool enables Rx hardware timestamp and then attach XDP program > > > to igc driver. It will display hardware timestamp of UDP packet with > > > port number 9092. Below are detail of test steps and results. > > > > [...] > > > > > > Fixes: fc9df2a0b520 ("igc: Enable RX via AF_XDP zero-copy") > > > Cc: <stable@vger.kernel.org> # 5.14+ > > > Signed-off-by: Song Yoong Siang <yoong.siang.song@intel.com> > > > Reviewed-by: Jacob Keller <jacob.e.keller@intel.com> > > > Reviewed-by: Jesper Dangaard Brouer <brouer@redhat.com> > > > --- > > > > LGTM, thank for the adjustments :-) > > > > Acked-by: Jesper Dangaard Brouer <brouer@redhat.com> > > > > Tested-by: Jesper Dangaard Brouer <brouer@redhat.com> > > I can confirm that this patch fix the issue I experienced with igc. > > This patch clearly fixes a bug in igc when writing the SRRCTL register. > (as bit 30 in register is "Timestamp Received Packet" which got cleared > before). > > Florian might have found another bug around RX timestamps, but this > patch should be safe and sane to apply as is. After a closer look I'm quite sure now that this patch should fix my issue as well. The register will be overwritten when setting up a XSK_POOL as well: igc_bpf igc_xdp_setup_pool igc_enable_rx_ring igc_configure_rx_ring wr32(IGC_SRRCTL) I already removed the BPF loading (which is the use case that the patch description mentions) from my setup to limit the search scope. If you like you could extend the patch description, but I'm fine with it. Thanks a lot for all the support / ideas! Highly appreciated! Florian > > > > v2 -> v3: Refactor SRRCTL definitions to more human readable definitions > > > v1 -> v2: Fix indention > > > --- > > > drivers/net/ethernet/intel/igc/igc_base.h | 11 ++++++++--- > > > drivers/net/ethernet/intel/igc/igc_main.c | 7 +++++-- > > > 2 files changed, 13 insertions(+), 5 deletions(-) >
On 4/14/2023 18:49, Song Yoong Siang wrote: > igc_configure_rx_ring() function will be called as part of XDP program > setup. If Rx hardware timestamp is enabled prio to XDP program setup, > this timestamp enablement will be overwritten when buffer size is > written into SRRCTL register. > > Thus, this commit read the register value before write to SRRCTL > register. This commit is tested by using xdp_hw_metadata bpf selftest > tool. The tool enables Rx hardware timestamp and then attach XDP program > to igc driver. It will display hardware timestamp of UDP packet with > port number 9092. Below are detail of test steps and results. > > Command on DUT: > sudo ./xdp_hw_metadata <interface name> > > Command on Link Partner: > echo -n skb | nc -u -q1 <destination IPv4 addr> 9092 > > Result before this patch: > skb hwtstamp is not found! > > Result after this patch: > found skb hwtstamp = 1677800973.642836757 > > Optionally, read PHC to confirm the values obtained are almost the same: > Command: > sudo ./testptp -d /dev/ptp0 -g > Result: > clock time: 1677800973.913598978 or Fri Mar 3 07:49:33 2023 > > Fixes: fc9df2a0b520 ("igc: Enable RX via AF_XDP zero-copy") > Cc: <stable@vger.kernel.org> # 5.14+ > Signed-off-by: Song Yoong Siang <yoong.siang.song@intel.com> > Reviewed-by: Jacob Keller <jacob.e.keller@intel.com> > Reviewed-by: Jesper Dangaard Brouer <brouer@redhat.com> > --- > v2 -> v3: Refactor SRRCTL definitions to more human readable definitions > v1 -> v2: Fix indention > --- > drivers/net/ethernet/intel/igc/igc_base.h | 11 ++++++++--- > drivers/net/ethernet/intel/igc/igc_main.c | 7 +++++-- > 2 files changed, 13 insertions(+), 5 deletions(-) Tested-by: Naama Meir <naamax.meir@linux.intel.com>
diff --git a/drivers/net/ethernet/intel/igc/igc_base.h b/drivers/net/ethernet/intel/igc/igc_base.h index 7a992befca24..9f3827eda157 100644 --- a/drivers/net/ethernet/intel/igc/igc_base.h +++ b/drivers/net/ethernet/intel/igc/igc_base.h @@ -87,8 +87,13 @@ union igc_adv_rx_desc { #define IGC_RXDCTL_SWFLUSH 0x04000000 /* Receive Software Flush */ /* SRRCTL bit definitions */ -#define IGC_SRRCTL_BSIZEPKT_SHIFT 10 /* Shift _right_ */ -#define IGC_SRRCTL_BSIZEHDRSIZE_SHIFT 2 /* Shift _left_ */ -#define IGC_SRRCTL_DESCTYPE_ADV_ONEBUF 0x02000000 +#define IGC_SRRCTL_BSIZEPKT_MASK GENMASK(6, 0) +#define IGC_SRRCTL_BSIZEPKT(x) FIELD_PREP(IGC_SRRCTL_BSIZEPKT_MASK, \ + (x) / 1024) /* in 1 KB resolution */ +#define IGC_SRRCTL_BSIZEHDR_MASK GENMASK(13, 8) +#define IGC_SRRCTL_BSIZEHDR(x) FIELD_PREP(IGC_SRRCTL_BSIZEHDR_MASK, \ + (x) / 64) /* in 64 bytes resolution */ +#define IGC_SRRCTL_DESCTYPE_MASK GENMASK(27, 25) +#define IGC_SRRCTL_DESCTYPE_ADV_ONEBUF FIELD_PREP(IGC_SRRCTL_DESCTYPE_MASK, 1) #endif /* _IGC_BASE_H */ diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c index 25fc6c65209b..a2d823e64609 100644 --- a/drivers/net/ethernet/intel/igc/igc_main.c +++ b/drivers/net/ethernet/intel/igc/igc_main.c @@ -641,8 +641,11 @@ static void igc_configure_rx_ring(struct igc_adapter *adapter, else buf_size = IGC_RXBUFFER_2048; - srrctl = IGC_RX_HDR_LEN << IGC_SRRCTL_BSIZEHDRSIZE_SHIFT; - srrctl |= buf_size >> IGC_SRRCTL_BSIZEPKT_SHIFT; + srrctl = rd32(IGC_SRRCTL(reg_idx)); + srrctl &= ~(IGC_SRRCTL_BSIZEPKT_MASK | IGC_SRRCTL_BSIZEHDR_MASK | + IGC_SRRCTL_DESCTYPE_MASK); + srrctl |= IGC_SRRCTL_BSIZEHDR(IGC_RX_HDR_LEN); + srrctl |= IGC_SRRCTL_BSIZEPKT(buf_size); srrctl |= IGC_SRRCTL_DESCTYPE_ADV_ONEBUF; wr32(IGC_SRRCTL(reg_idx), srrctl);
igc_configure_rx_ring() function will be called as part of XDP program setup. If Rx hardware timestamp is enabled prio to XDP program setup, this timestamp enablement will be overwritten when buffer size is written into SRRCTL register. Thus, this commit read the register value before write to SRRCTL register. This commit is tested by using xdp_hw_metadata bpf selftest tool. The tool enables Rx hardware timestamp and then attach XDP program to igc driver. It will display hardware timestamp of UDP packet with port number 9092. Below are detail of test steps and results. Command on DUT: sudo ./xdp_hw_metadata <interface name> Command on Link Partner: echo -n skb | nc -u -q1 <destination IPv4 addr> 9092 Result before this patch: skb hwtstamp is not found! Result after this patch: found skb hwtstamp = 1677800973.642836757 Optionally, read PHC to confirm the values obtained are almost the same: Command: sudo ./testptp -d /dev/ptp0 -g Result: clock time: 1677800973.913598978 or Fri Mar 3 07:49:33 2023 Fixes: fc9df2a0b520 ("igc: Enable RX via AF_XDP zero-copy") Cc: <stable@vger.kernel.org> # 5.14+ Signed-off-by: Song Yoong Siang <yoong.siang.song@intel.com> Reviewed-by: Jacob Keller <jacob.e.keller@intel.com> Reviewed-by: Jesper Dangaard Brouer <brouer@redhat.com> --- v2 -> v3: Refactor SRRCTL definitions to more human readable definitions v1 -> v2: Fix indention --- drivers/net/ethernet/intel/igc/igc_base.h | 11 ++++++++--- drivers/net/ethernet/intel/igc/igc_main.c | 7 +++++-- 2 files changed, 13 insertions(+), 5 deletions(-)