| Message ID | 20190306223454.2959-1-papadakospan@gmail.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | [v2] arm64: dts: rockchip: fix rk3328-roc-cc gmac2io tx/rx_delay | expand |
I think this should be included in the 5.1 release cycle
if possible.
It goes hand-in-hand with the previous fix
("Enable thresh dma mode to disable checksuming.")
when it comes to network stability
On 3/6/2019 5:34 PM, Leonidas P. Papadakos wrote: > The rk3328-roc-cc board exhibits tx stability issues with large packets, > as does the rock64 board, which was fixed with this patch > https://patchwork.kernel.org/patch/10178969/ > > A similar patch was merged for the rk3328-roc-cc here > https://patchwork.kernel.org/patch/10804863/ > but it doesn't include the tx/rx_delay tweaks, and I find that they > help with an issue where large transfers would bring the ethernet > link down, causing a link reset regularly. > > These values are better for the Renegade specifically, borrowed from > the armbian rk3328-roc-cc.dts: > https://github.com/armbian/build/blob/master/patch/kernel/rockchip64-default/Add_dts_rk3328-roc-cc.patch > > Signed-off-by: Leonidas P. Papadakos <papadakospan@gmail.com> > --- > arch/arm64/boot/dts/rockchip/rk3328-roc-cc.dts | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/arch/arm64/boot/dts/rockchip/rk3328-roc-cc.dts b/arch/arm64/boot/dts/rockchip/rk3328-roc-cc.dts > index 33c44e857..ec5454028 100644 > --- a/arch/arm64/boot/dts/rockchip/rk3328-roc-cc.dts > +++ b/arch/arm64/boot/dts/rockchip/rk3328-roc-cc.dts > @@ -108,8 +108,8 @@ > snps,reset-gpio = <&gpio1 RK_PC2 GPIO_ACTIVE_LOW>; > snps,reset-active-low; > snps,reset-delays-us = <0 10000 50000>; > - tx_delay = <0x25>; > - rx_delay = <0x11>; > + tx_delay = <0x28>; > + rx_delay = <0x16>; > status = "okay"; > }; > > When I submitted the patch to fix the TX stability issue, I intended to follow up with another patch for delay updates. Unfortunately, I was unable to locate a delay that was acceptable, as I was never able to get below ~700 retry attempts on the TX side in testing. I am unsure if this is due to manufacturing defects or if I'm missing something else. It may be worth considering implementing some sort of delay auto tuning, similar to the mmc driver. Thoughts?
> When I submitted the patch to fix the TX stability issue, I intended > to follow up with another patch for delay updates. > Unfortunately, I was unable to locate a delay that was acceptable, as > I was never able to get below ~700 retry attempts on the TX side in > testing. > > I am unsure if this is due to manufacturing defects or if I'm missing > something else. > It may be worth considering implementing some sort of delay auto > tuning, similar to the mmc driver. > Thoughts? I feared as much. The delays on the tx side do seem to be more of a mitigation. I you have the know-how for it, it would be pretty cool
>> When I submitted the patch to fix the TX stability issue, I intended >> to follow up with another patch for delay updates. >> Unfortunately, I was unable to locate a delay that was acceptable, >> as I was never able to get below ~700 retry attempts on the TX side >> in testing. >> >> I am unsure if this is due to manufacturing defects or if I'm >> missing something else. >> It may be worth considering implementing some sort of delay auto >> tuning, similar to the mmc driver. >> Thoughts? > > I feared as much. The delays on the tx side do seem to be more of a > mitigation. > I you have the know-how for it, it would be pretty cool Interesting approach to this by ayufan: https://github.com/ayufan-rock64/linux-mainline-kernel/commit/41eeb7cd789ea7cac0cbf4b35b53055f354da757 I hope he sends it upstream as well
On 3/8/2019 6:26 PM, Leonidas P. Papadakos wrote: > > >>> When I submitted the patch to fix the TX stability issue, I >>> intended to follow up with another patch for delay updates. >>> Unfortunately, I was unable to locate a delay that was acceptable, >>> as I was never able to get below ~700 retry attempts on the TX >>> side in testing. >>> >>> I am unsure if this is due to manufacturing defects or if I'm >>> missing something else. >>> It may be worth considering implementing some sort of delay auto >>> tuning, similar to the mmc driver. >>> Thoughts? >> >> I feared as much. The delays on the tx side do seem to be more of a >> mitigation. >> I you have the know-how for it, it would be pretty cool > Interesting approach to this by ayufan: > https://github.com/ayufan-rock64/linux-mainline-kernel/commit/41eeb7cd789ea7cac0cbf4b35b53055f354da757 > > > I hope he sends it upstream as well > So I discovered the cause of my TX issues, at least on my rk3328-roc-cc. The rk3328.dtsi is missing several rgmii tx pull strength values. This is causing them to default to 0ma. I've pushed them to 12ma, same as the other tx values, and I'm getting 600 mbps tx, with no TX failures. I am also going to test his adaptive patch once it's done building. Thoughts? diff --git a/arch/arm65/boot/dts/rockchip/rk3328-roc-cc.dts b/arch/arm64/boot/dts/rockchip/rk3328-roc-cc.dts index c0519bc5884e..696627bf3331 100644 --- a/arch/arm64/boot/dts/rockchip/rk3328-roc-cc.dts +++ b/arch/arm64/boot/dts/rockchip/rk3328-roc-cc.dts @@ -280,6 +280,26 @@ rockchip,pins = <0 RK_PA0 RK_FUNC_GPIO &pcfg_pull_none>; }; }; + + gmac-1 { + rgmiim1_pins: rgmiim1-pins { + rockchip,pins = + /* mac_txclk */ + <0 RK_PB0 1 &pcfg_pull_none_12ma>, + /* mac_txen */ + <0 RK_PB4 1 &pcfg_pull_none_12ma>, + /* mac_clk */ + <0 RK_PD0 1 &pcfg_pull_none_12ma>, + /* mac_txd1 */ + <0 RK_PC0 1 &pcfg_pull_none_12ma>, + /* mac_txd0 */ + <0 RK_PC1 1 &pcfg_pull_none_12ma>, + /* mac_txd3 */ + <0 RK_PC7 1 &pcfg_pull_none_12ma>, + /* mac_txd2 */ + <0 RK_PC6 1 &pcfg_pull_none_12ma>; + }; + }; }; &sdmmc {
> So I discovered the cause of my TX issues, at least on my > rk3328-roc-cc. > The rk3328.dtsi is missing several rgmii tx pull strength values. > This is causing them to default to 0ma. > > I've pushed them to 12ma, same as the other tx values, and I'm > getting 600 mbps tx, with no TX failures. > > I am also going to test his adaptive patch once it's done building. > > Thoughts? > > > diff --git a/arch/arm65/boot/dts/rockchip/rk3328-roc-cc.dts > b/arch/arm64/boot/dts/rockchip/rk3328-roc-cc.dts > index c0519bc5884e..696627bf3331 100644 > --- a/arch/arm64/boot/dts/rockchip/rk3328-roc-cc.dts > +++ b/arch/arm64/boot/dts/rockchip/rk3328-roc-cc.dts > @@ -280,6 +280,26 @@ > rockchip,pins = <0 RK_PA0 RK_FUNC_GPIO &pcfg_pull_none>; > }; > }; > + > + gmac-1 { > + rgmiim1_pins: rgmiim1-pins { > + rockchip,pins = > + /* mac_txclk */ > + <0 RK_PB0 1 &pcfg_pull_none_12ma>, > + /* mac_txen */ > + <0 RK_PB4 1 &pcfg_pull_none_12ma>, > + /* mac_clk */ > + <0 RK_PD0 1 &pcfg_pull_none_12ma>, > + /* mac_txd1 */ > + <0 RK_PC0 1 &pcfg_pull_none_12ma>, > + /* mac_txd0 */ > + <0 RK_PC1 1 &pcfg_pull_none_12ma>, > + /* mac_txd3 */ > + <0 RK_PC7 1 &pcfg_pull_none_12ma>, > + /* mac_txd2 */ > + <0 RK_PC6 1 &pcfg_pull_none_12ma>; > + }; > + }; > }; > > &sdmmc { Good find! It seems like they're there in the dtsi, but configured as none when it comes to ma. I'll test it
On 3/8/2019 10:26 PM, Leonidas P. Papadakos wrote: > >> So I discovered the cause of my TX issues, at least on my rk3328-roc-cc. >> The rk3328.dtsi is missing several rgmii tx pull strength values. >> This is causing them to default to 0ma. >> >> I've pushed them to 12ma, same as the other tx values, and I'm getting >> 600 mbps tx, with no TX failures. >> >> I am also going to test his adaptive patch once it's done building. >> >> Thoughts? >> >> >> diff --git a/arch/arm65/boot/dts/rockchip/rk3328-roc-cc.dts >> b/arch/arm64/boot/dts/rockchip/rk3328-roc-cc.dts >> index c0519bc5884e..696627bf3331 100644 >> --- a/arch/arm64/boot/dts/rockchip/rk3328-roc-cc.dts >> +++ b/arch/arm64/boot/dts/rockchip/rk3328-roc-cc.dts >> @@ -280,6 +280,26 @@ >> rockchip,pins = <0 RK_PA0 RK_FUNC_GPIO &pcfg_pull_none>; >> }; >> }; >> + >> + gmac-1 { >> + rgmiim1_pins: rgmiim1-pins { >> + rockchip,pins = >> + /* mac_txclk */ >> + <0 RK_PB0 1 &pcfg_pull_none_12ma>, >> + /* mac_txen */ >> + <0 RK_PB4 1 &pcfg_pull_none_12ma>, >> + /* mac_clk */ >> + <0 RK_PD0 1 &pcfg_pull_none_12ma>, >> + /* mac_txd1 */ >> + <0 RK_PC0 1 &pcfg_pull_none_12ma>, >> + /* mac_txd0 */ >> + <0 RK_PC1 1 &pcfg_pull_none_12ma>, >> + /* mac_txd3 */ >> + <0 RK_PC7 1 &pcfg_pull_none_12ma>, >> + /* mac_txd2 */ >> + <0 RK_PC6 1 &pcfg_pull_none_12ma>; >> + }; >> + }; >> }; >> >> &sdmmc { > Good find! It seems like they're there in the dtsi, but configured as > none when it comes to ma. I'll test it > > With Ayufan's patch offload patch and my pull patch, I now have the following results: [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.01 sec 1022 MBytes 857 Mbits/sec 885 sender [ 5] 0.00-10.01 sec 1022 MBytes 856 Mbits/sec receiver eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet6 fe80::e8b5:5eff:fe49:d353 prefixlen 64 scopeid 0x20<link> ether ea:b5:5e:49:d3:53 txqueuelen 1000 (Ethernet) RX packets 39822 bytes 2668618 (2.6 MB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 1005649 bytes 1516698158 (1.5 GB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 29 Much better overall, still not perfect, but I think it might be worth re-running the delay tests now.
Am Samstag, 9. März 2019, 13:45:21 CET schrieb Peter Geis: > On 3/8/2019 10:26 PM, Leonidas P. Papadakos wrote: > > > >> So I discovered the cause of my TX issues, at least on my rk3328-roc-cc. > >> The rk3328.dtsi is missing several rgmii tx pull strength values. > >> This is causing them to default to 0ma. > >> > >> I've pushed them to 12ma, same as the other tx values, and I'm getting > >> 600 mbps tx, with no TX failures. > >> > >> I am also going to test his adaptive patch once it's done building. > >> > >> Thoughts? > >> > >> > >> diff --git a/arch/arm65/boot/dts/rockchip/rk3328-roc-cc.dts > >> b/arch/arm64/boot/dts/rockchip/rk3328-roc-cc.dts > >> index c0519bc5884e..696627bf3331 100644 > >> --- a/arch/arm64/boot/dts/rockchip/rk3328-roc-cc.dts > >> +++ b/arch/arm64/boot/dts/rockchip/rk3328-roc-cc.dts > >> @@ -280,6 +280,26 @@ > >> rockchip,pins = <0 RK_PA0 RK_FUNC_GPIO &pcfg_pull_none>; > >> }; > >> }; > >> + > >> + gmac-1 { > >> + rgmiim1_pins: rgmiim1-pins { > >> + rockchip,pins = > >> + /* mac_txclk */ > >> + <0 RK_PB0 1 &pcfg_pull_none_12ma>, > >> + /* mac_txen */ > >> + <0 RK_PB4 1 &pcfg_pull_none_12ma>, > >> + /* mac_clk */ > >> + <0 RK_PD0 1 &pcfg_pull_none_12ma>, > >> + /* mac_txd1 */ > >> + <0 RK_PC0 1 &pcfg_pull_none_12ma>, > >> + /* mac_txd0 */ > >> + <0 RK_PC1 1 &pcfg_pull_none_12ma>, > >> + /* mac_txd3 */ > >> + <0 RK_PC7 1 &pcfg_pull_none_12ma>, > >> + /* mac_txd2 */ > >> + <0 RK_PC6 1 &pcfg_pull_none_12ma>; > >> + }; > >> + }; > >> }; > >> > >> &sdmmc { > > Good find! It seems like they're there in the dtsi, but configured as > > none when it comes to ma. I'll test it > > > > > > With Ayufan's patch offload patch and my pull patch, I now have the > following results: > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-10.01 sec 1022 MBytes 857 Mbits/sec 885 sender > [ 5] 0.00-10.01 sec 1022 MBytes 856 Mbits/sec > receiver > > eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 > inet6 fe80::e8b5:5eff:fe49:d353 prefixlen 64 scopeid 0x20<link> > ether ea:b5:5e:49:d3:53 txqueuelen 1000 (Ethernet) > RX packets 39822 bytes 2668618 (2.6 MB) > RX errors 0 dropped 0 overruns 0 frame 0 > TX packets 1005649 bytes 1516698158 (1.5 GB) > TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 > device interrupt 29 > > Much better overall, still not perfect, but I think it might be worth > re-running the delay tests now. also you could check if the thres-dma-flag is still necessary or can be removed - might be interesting to know if fixing the drive strength magically also fixes that issue. Heiko
On 3/9/2019 9:16 AM, Heiko Stuebner wrote: > Am Samstag, 9. März 2019, 13:45:21 CET schrieb Peter Geis: >> On 3/8/2019 10:26 PM, Leonidas P. Papadakos wrote: >>> >>>> So I discovered the cause of my TX issues, at least on my rk3328-roc-cc. >>>> The rk3328.dtsi is missing several rgmii tx pull strength values. >>>> This is causing them to default to 0ma. >>>> >>>> I've pushed them to 12ma, same as the other tx values, and I'm getting >>>> 600 mbps tx, with no TX failures. >>>> >>>> I am also going to test his adaptive patch once it's done building. >>>> >>>> Thoughts? >>>> >>>> >>>> diff --git a/arch/arm65/boot/dts/rockchip/rk3328-roc-cc.dts >>>> b/arch/arm64/boot/dts/rockchip/rk3328-roc-cc.dts >>>> index c0519bc5884e..696627bf3331 100644 >>>> --- a/arch/arm64/boot/dts/rockchip/rk3328-roc-cc.dts >>>> +++ b/arch/arm64/boot/dts/rockchip/rk3328-roc-cc.dts >>>> @@ -280,6 +280,26 @@ >>>> rockchip,pins = <0 RK_PA0 RK_FUNC_GPIO &pcfg_pull_none>; >>>> }; >>>> }; >>>> + >>>> + gmac-1 { >>>> + rgmiim1_pins: rgmiim1-pins { >>>> + rockchip,pins = >>>> + /* mac_txclk */ >>>> + <0 RK_PB0 1 &pcfg_pull_none_12ma>, >>>> + /* mac_txen */ >>>> + <0 RK_PB4 1 &pcfg_pull_none_12ma>, >>>> + /* mac_clk */ >>>> + <0 RK_PD0 1 &pcfg_pull_none_12ma>, >>>> + /* mac_txd1 */ >>>> + <0 RK_PC0 1 &pcfg_pull_none_12ma>, >>>> + /* mac_txd0 */ >>>> + <0 RK_PC1 1 &pcfg_pull_none_12ma>, >>>> + /* mac_txd3 */ >>>> + <0 RK_PC7 1 &pcfg_pull_none_12ma>, >>>> + /* mac_txd2 */ >>>> + <0 RK_PC6 1 &pcfg_pull_none_12ma>; >>>> + }; >>>> + }; >>>> }; >>>> >>>> &sdmmc { >>> Good find! It seems like they're there in the dtsi, but configured as >>> none when it comes to ma. I'll test it >>> >>> >> >> With Ayufan's patch offload patch and my pull patch, I now have the >> following results: >> [ ID] Interval Transfer Bitrate Retr >> [ 5] 0.00-10.01 sec 1022 MBytes 857 Mbits/sec 885 sender >> [ 5] 0.00-10.01 sec 1022 MBytes 856 Mbits/sec >> receiver >> >> eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 >> inet6 fe80::e8b5:5eff:fe49:d353 prefixlen 64 scopeid 0x20<link> >> ether ea:b5:5e:49:d3:53 txqueuelen 1000 (Ethernet) >> RX packets 39822 bytes 2668618 (2.6 MB) >> RX errors 0 dropped 0 overruns 0 frame 0 >> TX packets 1005649 bytes 1516698158 (1.5 GB) >> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >> device interrupt 29 >> >> Much better overall, still not perfect, but I think it might be worth >> re-running the delay tests now. > > also you could check if the thres-dma-flag is still necessary or can > be removed - might be interesting to know if fixing the drive strength > magically also fixes that issue. > > > Heiko > Part of my test was removing that flag. I actually haven't run with it for a while, since I chose to disable offload in userspace instead. With threshold DMA I was having serious performance issues, but it was better than networking being completely broken. I'll write up a patch for the .dtsi and submit it today. If we can contact ayufan about his patch, that would be great!
Στις Σαβ, 9 Μαρ, 2019 at 4:21 ΜΜ, ο/η Peter Geis <pgwipeout@gmail.com> έγραψε: > > > On 3/9/2019 9:16 AM, Heiko Stuebner wrote: >> Am Samstag, 9. März 2019, 13:45:21 CET schrieb Peter Geis: >>> On 3/8/2019 10:26 PM, Leonidas P. Papadakos wrote: >>>> >>>>> So I discovered the cause of my TX issues, at least on my >>>>> rk3328-roc-cc. >>>>> The rk3328.dtsi is missing several rgmii tx pull strength values. >>>>> This is causing them to default to 0ma. >>>>> >>>>> I've pushed them to 12ma, same as the other tx values, and I'm >>>>> getting >>>>> 600 mbps tx, with no TX failures. >>>>> >>>>> I am also going to test his adaptive patch once it's done >>>>> building. >>>>> >>>>> Thoughts? >>>>> >>>>> >>>>> diff --git a/arch/arm65/boot/dts/rockchip/rk3328-roc-cc.dts >>>>> b/arch/arm64/boot/dts/rockchip/rk3328-roc-cc.dts >>>>> index c0519bc5884e..696627bf3331 100644 >>>>> --- a/arch/arm64/boot/dts/rockchip/rk3328-roc-cc.dts >>>>> +++ b/arch/arm64/boot/dts/rockchip/rk3328-roc-cc.dts >>>>> @@ -280,6 +280,26 @@ >>>>> rockchip,pins = <0 RK_PA0 RK_FUNC_GPIO >>>>> &pcfg_pull_none>; >>>>> }; >>>>> }; >>>>> + >>>>> + gmac-1 { >>>>> + rgmiim1_pins: rgmiim1-pins { >>>>> + rockchip,pins = >>>>> + /* mac_txclk */ >>>>> + <0 RK_PB0 1 &pcfg_pull_none_12ma>, >>>>> + /* mac_txen */ >>>>> + <0 RK_PB4 1 &pcfg_pull_none_12ma>, >>>>> + /* mac_clk */ >>>>> + <0 RK_PD0 1 &pcfg_pull_none_12ma>, >>>>> + /* mac_txd1 */ >>>>> + <0 RK_PC0 1 &pcfg_pull_none_12ma>, >>>>> + /* mac_txd0 */ >>>>> + <0 RK_PC1 1 &pcfg_pull_none_12ma>, >>>>> + /* mac_txd3 */ >>>>> + <0 RK_PC7 1 &pcfg_pull_none_12ma>, >>>>> + /* mac_txd2 */ >>>>> + <0 RK_PC6 1 &pcfg_pull_none_12ma>; >>>>> + }; >>>>> + }; >>>>> }; >>>>> >>>>> &sdmmc { >>>> Good find! It seems like they're there in the dtsi, but configured >>>> as >>>> none when it comes to ma. I'll test it >>>> >>>> >>> >>> With Ayufan's patch offload patch and my pull patch, I now have the >>> following results: >>> [ ID] Interval Transfer Bitrate Retr >>> [ 5] 0.00-10.01 sec 1022 MBytes 857 Mbits/sec 885 >>> sender >>> [ 5] 0.00-10.01 sec 1022 MBytes 856 Mbits/sec >>> receiver >>> >>> eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 >>> inet6 fe80::e8b5:5eff:fe49:d353 prefixlen 64 scopeid >>> 0x20<link> >>> ether ea:b5:5e:49:d3:53 txqueuelen 1000 (Ethernet) >>> RX packets 39822 bytes 2668618 (2.6 MB) >>> RX errors 0 dropped 0 overruns 0 frame 0 >>> TX packets 1005649 bytes 1516698158 (1.5 GB) >>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >>> device interrupt 29 >>> >>> Much better overall, still not perfect, but I think it might be >>> worth >>> re-running the delay tests now. >> >> also you could check if the thres-dma-flag is still necessary or can >> be removed - might be interesting to know if fixing the drive >> strength >> magically also fixes that issue. >> >> >> Heiko >> > > Part of my test was removing that flag. > I actually haven't run with it for a while, since I chose to disable > offload in userspace instead. > With threshold DMA I was having serious performance issues, but it > was better than networking being completely broken. > > I'll write up a patch for the .dtsi and submit it today. > If we can contact ayufan about his patch, that would be great! For the test I use iperf3 -s on my laptop (Gigabit Ethernet) and on the renegade: iperf3 -c <laptop_ip> -t 30. I tried using: 1) the force-thresh-dma-mode patch by itself with the changed tx/rx_delay. (Maybe the default one is better? It needs testing nayway) No ssh lag but I do get the Link reset on tx activity 2) just this patch by itself the ssh lag hit me, but it does seem better. The link reset still happens 3) the force-thresh-dma-mode patch with this one: No shs lag, and as before, the tx activity seems to hit less? (Maybe placebo) but the link reset does happen regularly. As far as I understand ayufan's offload patch removes the need for the force-thresh-dma-mode flag. I'll try ayufan's patch (adding the option to the rk3328-roc-cc as well) together with this patch. Maybe it'll be better
On 3/9/2019 6:05 PM, Leonidas P. Papadakos wrote: > > > Στις Σαβ, 9 Μαρ, 2019 at 4:21 ΜΜ, ο/η Peter Geis <pgwipeout@gmail.com> > έγραψε: >> >> >> On 3/9/2019 9:16 AM, Heiko Stuebner wrote: >>> Am Samstag, 9. März 2019, 13:45:21 CET schrieb Peter Geis: >>>> On 3/8/2019 10:26 PM, Leonidas P. Papadakos wrote: >>>>> >>>>>> So I discovered the cause of my TX issues, at least on my >>>>>> rk3328-roc-cc. >>>>>> The rk3328.dtsi is missing several rgmii tx pull strength values. >>>>>> This is causing them to default to 0ma. >>>>>> >>>>>> I've pushed them to 12ma, same as the other tx values, and I'm >>>>>> getting >>>>>> 600 mbps tx, with no TX failures. >>>>>> >>>>>> I am also going to test his adaptive patch once it's done building. >>>>>> >>>>>> Thoughts? >>>>>> >>>>>> >>>>>> diff --git a/arch/arm65/boot/dts/rockchip/rk3328-roc-cc.dts >>>>>> b/arch/arm64/boot/dts/rockchip/rk3328-roc-cc.dts >>>>>> index c0519bc5884e..696627bf3331 100644 >>>>>> --- a/arch/arm64/boot/dts/rockchip/rk3328-roc-cc.dts >>>>>> +++ b/arch/arm64/boot/dts/rockchip/rk3328-roc-cc.dts >>>>>> @@ -280,6 +280,26 @@ >>>>>> rockchip,pins = <0 RK_PA0 RK_FUNC_GPIO >>>>>> &pcfg_pull_none>; >>>>>> }; >>>>>> }; >>>>>> + >>>>>> + gmac-1 { >>>>>> + rgmiim1_pins: rgmiim1-pins { >>>>>> + rockchip,pins = >>>>>> + /* mac_txclk */ >>>>>> + <0 RK_PB0 1 &pcfg_pull_none_12ma>, >>>>>> + /* mac_txen */ >>>>>> + <0 RK_PB4 1 &pcfg_pull_none_12ma>, >>>>>> + /* mac_clk */ >>>>>> + <0 RK_PD0 1 &pcfg_pull_none_12ma>, >>>>>> + /* mac_txd1 */ >>>>>> + <0 RK_PC0 1 &pcfg_pull_none_12ma>, >>>>>> + /* mac_txd0 */ >>>>>> + <0 RK_PC1 1 &pcfg_pull_none_12ma>, >>>>>> + /* mac_txd3 */ >>>>>> + <0 RK_PC7 1 &pcfg_pull_none_12ma>, >>>>>> + /* mac_txd2 */ >>>>>> + <0 RK_PC6 1 &pcfg_pull_none_12ma>; >>>>>> + }; >>>>>> + }; >>>>>> }; >>>>>> >>>>>> &sdmmc { >>>>> Good find! It seems like they're there in the dtsi, but configured as >>>>> none when it comes to ma. I'll test it >>>>> >>>>> >>>> >>>> With Ayufan's patch offload patch and my pull patch, I now have the >>>> following results: >>>> [ ID] Interval Transfer Bitrate Retr >>>> [ 5] 0.00-10.01 sec 1022 MBytes 857 Mbits/sec 885 sender >>>> [ 5] 0.00-10.01 sec 1022 MBytes 856 Mbits/sec >>>> receiver >>>> >>>> eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 >>>> inet6 fe80::e8b5:5eff:fe49:d353 prefixlen 64 scopeid >>>> 0x20<link> >>>> ether ea:b5:5e:49:d3:53 txqueuelen 1000 (Ethernet) >>>> RX packets 39822 bytes 2668618 (2.6 MB) >>>> RX errors 0 dropped 0 overruns 0 frame 0 >>>> TX packets 1005649 bytes 1516698158 (1.5 GB) >>>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 >>>> device interrupt 29 >>>> >>>> Much better overall, still not perfect, but I think it might be worth >>>> re-running the delay tests now. >>> >>> also you could check if the thres-dma-flag is still necessary or can >>> be removed - might be interesting to know if fixing the drive strength >>> magically also fixes that issue. >>> >>> >>> Heiko >>> >> >> Part of my test was removing that flag. >> I actually haven't run with it for a while, since I chose to disable >> offload in userspace instead. >> With threshold DMA I was having serious performance issues, but it was >> better than networking being completely broken. >> >> I'll write up a patch for the .dtsi and submit it today. >> If we can contact ayufan about his patch, that would be great! > > For the test I use iperf3 -s on my laptop (Gigabit Ethernet) > and on the renegade: iperf3 -c <laptop_ip> -t 30. > > I tried using: > 1) the force-thresh-dma-mode patch by itself with the changed > tx/rx_delay. (Maybe the default one is better? It needs testing nayway) > No ssh lag but I do get the Link reset on tx activity > > 2) just this patch by itself > the ssh lag hit me, but it does seem better. The link reset still > happens > > 3) the force-thresh-dma-mode patch with this one: > No shs lag, and as before, the tx activity seems to hit less? (Maybe > placebo) but the link reset does happen regularly. > > As far as I understand ayufan's offload patch removes the need for the > force-thresh-dma-mode flag. > I'll try ayufan's patch (adding the option to the rk3328-roc-cc as well) > together with this patch. > Maybe it'll be better > We have several issues on the rk3328. The first issue, which you describe as the ssh lag, is the tx-offload issue. Packets that are larger than the MTU and will be fragmented break if tx-offload is enabled. To fix this, you must somehow disable tx-offload. force-thresh-dma-mode disabled tx-offload, but had other side effects. Ayufan's patch disables tx-offload only if the packet is larger than 1498. You can also disable tx-offload from userspace with ethtool. The second issue is the high rate of tx packets being corrupted, which my patch to set the tx pull values fixes. The third issue is the tx and rx delays need to be tuned now that the tx packet corruption issue has been fixed. I hope this clears things up a little for you.
Στις Κυρ, 10 Μαρ, 2019 at 1:16 ΠΜ, ο/η Peter Geis <pgwipeout@gmail.com> έγραψε: > > > On 3/9/2019 6:05 PM, Leonidas P. Papadakos wrote: >> >> >> Στις Σαβ, 9 Μαρ, 2019 at 4:21 ΜΜ, ο/η Peter Geis >> <pgwipeout@gmail.com> έγραψε: >>> >>> >>> On 3/9/2019 9:16 AM, Heiko Stuebner wrote: >>>> Am Samstag, 9. März 2019, 13:45:21 CET schrieb Peter Geis: >>>>> On 3/8/2019 10:26 PM, Leonidas P. Papadakos wrote: >>>>>> >>>>>>> So I discovered the cause of my TX issues, at least on my >>>>>>> rk3328-roc-cc. >>>>>>> The rk3328.dtsi is missing several rgmii tx pull strength >>>>>>> values. >>>>>>> This is causing them to default to 0ma. >>>>>>> >>>>>>> I've pushed them to 12ma, same as the other tx values, and I'm >>>>>>> getting >>>>>>> 600 mbps tx, with no TX failures. >>>>>>> >>>>>>> I am also going to test his adaptive patch once it's done >>>>>>> building. >>>>>>> >>>>>>> Thoughts? >>>>>>> >>>>>>> >>>>>>> diff --git a/arch/arm65/boot/dts/rockchip/rk3328-roc-cc.dts >>>>>>> b/arch/arm64/boot/dts/rockchip/rk3328-roc-cc.dts >>>>>>> index c0519bc5884e..696627bf3331 100644 >>>>>>> --- a/arch/arm64/boot/dts/rockchip/rk3328-roc-cc.dts >>>>>>> +++ b/arch/arm64/boot/dts/rockchip/rk3328-roc-cc.dts >>>>>>> @@ -280,6 +280,26 @@ >>>>>>> rockchip,pins = <0 RK_PA0 RK_FUNC_GPIO >>>>>>> &pcfg_pull_none>; >>>>>>> }; >>>>>>> }; >>>>>>> + >>>>>>> + gmac-1 { >>>>>>> + rgmiim1_pins: rgmiim1-pins { >>>>>>> + rockchip,pins = >>>>>>> + /* mac_txclk */ >>>>>>> + <0 RK_PB0 1 &pcfg_pull_none_12ma>, >>>>>>> + /* mac_txen */ >>>>>>> + <0 RK_PB4 1 &pcfg_pull_none_12ma>, >>>>>>> + /* mac_clk */ >>>>>>> + <0 RK_PD0 1 &pcfg_pull_none_12ma>, >>>>>>> + /* mac_txd1 */ >>>>>>> + <0 RK_PC0 1 &pcfg_pull_none_12ma>, >>>>>>> + /* mac_txd0 */ >>>>>>> + <0 RK_PC1 1 &pcfg_pull_none_12ma>, >>>>>>> + /* mac_txd3 */ >>>>>>> + <0 RK_PC7 1 &pcfg_pull_none_12ma>, >>>>>>> + /* mac_txd2 */ >>>>>>> + <0 RK_PC6 1 &pcfg_pull_none_12ma>; >>>>>>> + }; >>>>>>> + }; >>>>>>> }; >>>>>>> >>>>>>> &sdmmc { >>>>>> Good find! It seems like they're there in the dtsi, but >>>>>> configured as >>>>>> none when it comes to ma. I'll test it >>>>>> >>>>>> >>>>> >>>>> With Ayufan's patch offload patch and my pull patch, I now have >>>>> the >>>>> following results: >>>>> [ ID] Interval Transfer Bitrate Retr >>>>> [ 5] 0.00-10.01 sec 1022 MBytes 857 Mbits/sec 885 sender >>>>> [ 5] 0.00-10.01 sec 1022 MBytes 856 Mbits/sec >>>>> receiver >>>>> >>>>> eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 >>>>> inet6 fe80::e8b5:5eff:fe49:d353 prefixlen 64 scopeid >>>>> 0x20<link> >>>>> ether ea:b5:5e:49:d3:53 txqueuelen 1000 (Ethernet) >>>>> RX packets 39822 bytes 2668618 (2.6 MB) >>>>> RX errors 0 dropped 0 overruns 0 frame 0 >>>>> TX packets 1005649 bytes 1516698158 (1.5 GB) >>>>> TX errors 0 dropped 0 overruns 0 carrier 0 >>>>> collisions 0 >>>>> device interrupt 29 >>>>> >>>>> Much better overall, still not perfect, but I think it might be >>>>> worth >>>>> re-running the delay tests now. >>>> >>>> also you could check if the thres-dma-flag is still necessary or >>>> can >>>> be removed - might be interesting to know if fixing the drive >>>> strength >>>> magically also fixes that issue. >>>> >>>> >>>> Heiko >>>> >>> >>> Part of my test was removing that flag. >>> I actually haven't run with it for a while, since I chose to >>> disable offload in userspace instead. >>> With threshold DMA I was having serious performance issues, but it >>> was better than networking being completely broken. >>> >>> I'll write up a patch for the .dtsi and submit it today. >>> If we can contact ayufan about his patch, that would be great! >> >> For the test I use iperf3 -s on my laptop (Gigabit Ethernet) >> and on the renegade: iperf3 -c <laptop_ip> -t 30. >> >> I tried using: >> 1) the force-thresh-dma-mode patch by itself with the changed >> tx/rx_delay. (Maybe the default one is better? It needs testing >> nayway) >> No ssh lag but I do get the Link reset on tx activity >> >> 2) just this patch by itself >> the ssh lag hit me, but it does seem better. The link reset still >> happens >> >> 3) the force-thresh-dma-mode patch with this one: >> No shs lag, and as before, the tx activity seems to hit less? >> (Maybe placebo) but the link reset does happen regularly. >> >> As far as I understand ayufan's offload patch removes the need for >> the force-thresh-dma-mode flag. >> I'll try ayufan's patch (adding the option to the rk3328-roc-cc as >> well) together with this patch. >> Maybe it'll be better >> > > We have several issues on the rk3328. > > The first issue, which you describe as the ssh lag, is the tx-offload > issue. > Packets that are larger than the MTU and will be fragmented break if > tx-offload is enabled. > To fix this, you must somehow disable tx-offload. > force-thresh-dma-mode disabled tx-offload, but had other side effects. > Ayufan's patch disables tx-offload only if the packet is larger than > 1498. > You can also disable tx-offload from userspace with ethtool. > > The second issue is the high rate of tx packets being corrupted, > which my patch to set the tx pull values fixes. > > The third issue is the tx and rx delays need to be tuned now that the > tx packet corruption issue has been fixed. > > I hope this clears things up a little for you. Yup, that's what I have in my mind already. My previous comment wasn't clear, I know. I pieced it together after your dma-mode patch, because that one fixed the ssh lag. My previous email was just what I saw after those changes. In my case the link reset still happens with your patch. You're thinking that the combination of your patch, ayufan's "conditional" patch and a specific tweak in the delays would work for the network instability problem?
[clip] >>> >>> For the test I use iperf3 -s on my laptop (Gigabit Ethernet) >>> and on the renegade: iperf3 -c <laptop_ip> -t 30. >>> >>> I tried using: >>> 1) the force-thresh-dma-mode patch by itself with the changed >>> tx/rx_delay. (Maybe the default one is better? It needs testing nayway) >>> No ssh lag but I do get the Link reset on tx activity >>> >>> 2) just this patch by itself >>> the ssh lag hit me, but it does seem better. The link reset >>> still happens >>> >>> 3) the force-thresh-dma-mode patch with this one: >>> No shs lag, and as before, the tx activity seems to hit less? >>> (Maybe placebo) but the link reset does happen regularly. >>> >>> As far as I understand ayufan's offload patch removes the need for >>> the force-thresh-dma-mode flag. >>> I'll try ayufan's patch (adding the option to the rk3328-roc-cc as >>> well) together with this patch. >>> Maybe it'll be better >>> >> >> We have several issues on the rk3328. >> >> The first issue, which you describe as the ssh lag, is the tx-offload >> issue. >> Packets that are larger than the MTU and will be fragmented break if >> tx-offload is enabled. >> To fix this, you must somehow disable tx-offload. >> force-thresh-dma-mode disabled tx-offload, but had other side effects. >> Ayufan's patch disables tx-offload only if the packet is larger than >> 1498. >> You can also disable tx-offload from userspace with ethtool. >> >> The second issue is the high rate of tx packets being corrupted, which >> my patch to set the tx pull values fixes. >> >> The third issue is the tx and rx delays need to be tuned now that the >> tx packet corruption issue has been fixed. >> >> I hope this clears things up a little for you. > > Yup, that's what I have in my mind already. My previous comment wasn't > clear, I know. > I pieced it together after your dma-mode patch, because that one fixed > the ssh lag. > My previous email was just what I saw after those changes. > > In my case the link reset still happens with your patch. > > You're thinking that the combination of your patch, ayufan's > "conditional" patch and a specific tweak in the delays would work for > the network instability problem? > Correct, the link-reset tends to happen when the delays are at the edge of working. There is also the possibility that board level manufacturing tolerances are causing issues where the delay can change between boards. If this is the case, we will have to make an auto-tune function, probably based off the mmc version.
> > > We have several issues on the rk3328. > > > > > > The first issue, which you describe as the ssh lag, is the > > > tx-offload issue. > > > Packets that are larger than the MTU and will be fragmented break if > > > tx-offload is enabled. > > > To fix this, you must somehow disable tx-offload. > > > force-thresh-dma-mode disabled tx-offload, but had other side effects. > > > Ayufan's patch disables tx-offload only if the packet is larger than > > > 1498. > > > You can also disable tx-offload from userspace with ethtool. > > > > > > The second issue is the high rate of tx packets being corrupted, > > > which my patch to set the tx pull values fixes. > > > > > > The third issue is the tx and rx delays need to be tuned now that > > > the tx packet corruption issue has been fixed. > > > > > > I hope this clears things up a little for you. > > > > Yup, that's what I have in my mind already. My previous comment wasn't > > clear, I know. > > I pieced it together after your dma-mode patch, because that one fixed > > the ssh lag. > > My previous email was just what I saw after those changes. > > > > In my case the link reset still happens with your patch. > > > > You're thinking that the combination of your patch, ayufan's > > "conditional" patch and a specific tweak in the delays would work for > > the network instability problem? > > > > Correct, the link-reset tends to happen when the delays are at the edge of > working. > There is also the possibility that board level manufacturing tolerances are > causing issues where the delay can change between boards. > If this is the case, we will have to make an auto-tune function, probably > based off the mmc version. So, I tried just your patch with archlinuxARM linux 5.0 (meaning the tx/rx_delays are 0x25/0x11 respectively if I'm not mistaken) and disabled tx offload through ethtool. After 3 30-second iperf3 tests, no link resets! I get about the same speed you mentioned before, 800-something to 700-something lowest. Seems like the delays that are already there in the dts are better than the ones I picked up from the Armbian dts. It's worth noting that I've never noticed a link reset in my ~3 months of using the 4.4 Rockchip kernel that Armbian packages using ayufan's linux-kernel repo. Maybe something there? All in all, it does seem to be much better with your patch.
Στις Κυρ, 10 Μαρ, 2019 at 2:26 ΠΜ, ο/η "Leonidas P. Papadakos" <papadakospan@gmail.com> έγραψε: >> > > We have several issues on the rk3328. >> > > >> > > The first issue, which you describe as the ssh lag, is the >> > > tx-offload issue. >> > > Packets that are larger than the MTU and will be fragmented >> break if >> > > tx-offload is enabled. >> > > To fix this, you must somehow disable tx-offload. >> > > force-thresh-dma-mode disabled tx-offload, but had other side >> effects. >> > > Ayufan's patch disables tx-offload only if the packet is larger >> than >> > > 1498. >> > > You can also disable tx-offload from userspace with ethtool. >> > > >> > > The second issue is the high rate of tx packets being corrupted, >> > > which my patch to set the tx pull values fixes. >> > > >> > > The third issue is the tx and rx delays need to be tuned now >> that >> > > the tx packet corruption issue has been fixed. >> > > >> > > I hope this clears things up a little for you. >> > >> > Yup, that's what I have in my mind already. My previous comment >> wasn't >> > clear, I know. >> > I pieced it together after your dma-mode patch, because that one >> fixed >> > the ssh lag. >> > My previous email was just what I saw after those changes. >> > >> > In my case the link reset still happens with your patch. >> > >> > You're thinking that the combination of your patch, ayufan's >> > "conditional" patch and a specific tweak in the delays would work >> for >> > the network instability problem? >> > >> >> Correct, the link-reset tends to happen when the delays are at the >> edge of >> working. >> There is also the possibility that board level manufacturing >> tolerances are >> causing issues where the delay can change between boards. >> If this is the case, we will have to make an auto-tune function, >> probably >> based off the mmc version. > > So, I tried just your patch with archlinuxARM linux 5.0 > (meaning the tx/rx_delays are 0x25/0x11 respectively if I'm not > mistaken) > and disabled tx offload through ethtool. > > After 3 30-second iperf3 tests, no link resets! I get about the same > speed you mentioned before, > 800-something to 700-something lowest. > > Seems like the delays that are already there in the dts are better > than the ones I picked up from the Armbian dts. > > It's worth noting that I've never noticed a link reset in my ~3 > months of using the 4.4 Rockchip kernel > that Armbian packages using ayufan's linux-kernel repo. Maybe > something there? > > All in all, it does seem to be much better with your patch. I've been trying to reach ayufan about his patch, getting is into mainline: https://github.com/ayufan-rock64/linux-mainline-kernel/commit/41eeb7cd789ea7cac0cbf4b35b53055f354da757 but it seems he's not responding. I'll cc him. His patch would be a really cool fix if it could be in 5.1, or even 5.2, as a replacement to force_thresh_dma_mode
diff --git a/arch/arm64/boot/dts/rockchip/rk3328-roc-cc.dts b/arch/arm64/boot/dts/rockchip/rk3328-roc-cc.dts index 33c44e857..ec5454028 100644 --- a/arch/arm64/boot/dts/rockchip/rk3328-roc-cc.dts +++ b/arch/arm64/boot/dts/rockchip/rk3328-roc-cc.dts @@ -108,8 +108,8 @@ snps,reset-gpio = <&gpio1 RK_PC2 GPIO_ACTIVE_LOW>; snps,reset-active-low; snps,reset-delays-us = <0 10000 50000>; - tx_delay = <0x25>; - rx_delay = <0x11>; + tx_delay = <0x28>; + rx_delay = <0x16>; status = "okay"; };
The rk3328-roc-cc board exhibits tx stability issues with large packets, as does the rock64 board, which was fixed with this patch https://patchwork.kernel.org/patch/10178969/ A similar patch was merged for the rk3328-roc-cc here https://patchwork.kernel.org/patch/10804863/ but it doesn't include the tx/rx_delay tweaks, and I find that they help with an issue where large transfers would bring the ethernet link down, causing a link reset regularly. These values are better for the Renegade specifically, borrowed from the armbian rk3328-roc-cc.dts: https://github.com/armbian/build/blob/master/patch/kernel/rockchip64-default/Add_dts_rk3328-roc-cc.patch Signed-off-by: Leonidas P. Papadakos <papadakospan@gmail.com> --- arch/arm64/boot/dts/rockchip/rk3328-roc-cc.dts | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)