| Message ID | ef808c4d8447ee8cf832821a985ba68939455461.1623239847.git.lorenzo@kernel.org (mailing list archive) |
|---|---|
| State | Superseded |
| Delegated to: | Netdev Maintainers |
| Headers | show |
| Series | [RFT,net-next] net: ti: add pp skb recycling support | expand |
| Context | Check | Description |
|---|---|---|
| netdev/cover_letter | success | Link |
| netdev/fixes_present | success | Link |
| netdev/patch_count | success | Link |
| netdev/tree_selection | success | Clearly marked for net-next |
| netdev/subject_prefix | success | Link |
| netdev/cc_maintainers | warning | 9 maintainers not CCed: zhangchangzhong@huawei.com michael@walle.cc daniel@iogearbox.net camelia.groza@nxp.com shayagr@amazon.com m-karicheri2@ti.com yanaijie@huawei.com linux-omap@vger.kernel.org jesse.brandeburg@intel.com |
| netdev/source_inline | success | Was 0 now: 0 |
| netdev/verify_signedoff | success | Link |
| netdev/module_param | success | Was 0 now: 0 |
| netdev/build_32bit | success | Errors and warnings before: 0 this patch: 0 |
| netdev/kdoc | success | Errors and warnings before: 0 this patch: 0 |
| netdev/verify_fixes | success | Link |
| netdev/checkpatch | success | total: 0 errors, 0 warnings, 0 checks, 20 lines checked |
| netdev/build_allmodconfig_warn | success | Errors and warnings before: 0 this patch: 0 |
| netdev/header_inline | success | Link |
On Wed, Jun 9, 2021 at 2:01 PM Lorenzo Bianconi <lorenzo@kernel.org> wrote: > > As already done for mvneta and mvpp2, enable skb recycling for ti > ethernet drivers > > Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org> Looks good! If someone with the HW could provide a with and without the patch, that would be nice!
hi On 09/06/2021 15:20, Matteo Croce wrote: > On Wed, Jun 9, 2021 at 2:01 PM Lorenzo Bianconi <lorenzo@kernel.org> wrote: >> >> As already done for mvneta and mvpp2, enable skb recycling for ti >> ethernet drivers >> >> Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org> > > Looks good! If someone with the HW could provide a with and without > the patch, that would be nice! > What test would you recommend to run?
> hi > > On 09/06/2021 15:20, Matteo Croce wrote: > > On Wed, Jun 9, 2021 at 2:01 PM Lorenzo Bianconi <lorenzo@kernel.org> wrote: > > > > > > As already done for mvneta and mvpp2, enable skb recycling for ti > > > ethernet drivers > > > > > > Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org> > > > > Looks good! If someone with the HW could provide a with and without > > the patch, that would be nice! > > > > What test would you recommend to run? Hi Grygorii, Just pass the skb to the stack, so I think regular XDP_PASS use case is enough. Regards, Lorenzo > > -- > Best regards, > grygorii >
On Wed, Jun 9, 2021 at 5:03 PM Grygorii Strashko <grygorii.strashko@ti.com> wrote: > > hi > > On 09/06/2021 15:20, Matteo Croce wrote: > > On Wed, Jun 9, 2021 at 2:01 PM Lorenzo Bianconi <lorenzo@kernel.org> wrote: > >> > >> As already done for mvneta and mvpp2, enable skb recycling for ti > >> ethernet drivers > >> > >> Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org> > > > > Looks good! If someone with the HW could provide a with and without > > the patch, that would be nice! > > > > What test would you recommend to run? > > -- > Best regards, > grygorii Hi Grygorii, A test which benefits most from this kind of change is one in which the frames are freed early. One option would be to use mausezahn which by default sends frames with an invalid ethertype, that are dropped very early from the stack (I think in __netif_receive_skb_core() or near there). Then, on the device I just watch the device statistics to count packets per second: mausezahn eth0 -c 0 -b $board_mac_address This test should be precise enough for a gigabit link, on faster links I usually use a DPDK or AF_XDP based one. Regards,
On Wed, 9 Jun 2021 17:43:57 +0200 Matteo Croce <mcroce@linux.microsoft.com> wrote: > On Wed, Jun 9, 2021 at 5:03 PM Grygorii Strashko > <grygorii.strashko@ti.com> wrote: > > > > On 09/06/2021 15:20, Matteo Croce wrote: > > > On Wed, Jun 9, 2021 at 2:01 PM Lorenzo Bianconi <lorenzo@kernel.org> wrote: > > >> > > >> As already done for mvneta and mvpp2, enable skb recycling for ti > > >> ethernet drivers > > >> > > >> Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org> > > > > > > Looks good! If someone with the HW could provide a with and without > > > the patch, that would be nice! > > > > > > > What test would you recommend to run? > > [...] > > A test which benefits most from this kind of change is one in which > the frames are freed early. I would also recommend running an XDP_PASS program, and then running something that let the packets travel as deep as possible into netstack. Not to test performance, but to make sure we didn't break something! I've hacked up bnxt driver (it's not as straight forward as this driver to convert) and is running some TCP tests. Not problems so-far :-0 I wanted to ask if someone knows howto setup the zero-copy TCP stuff that google did? (but is that TX only?)
On 6/9/21 5:55 PM, Jesper Dangaard Brouer wrote: > On Wed, 9 Jun 2021 17:43:57 +0200 > Matteo Croce <mcroce@linux.microsoft.com> wrote: > >> On Wed, Jun 9, 2021 at 5:03 PM Grygorii Strashko >> <grygorii.strashko@ti.com> wrote: >>> >>> On 09/06/2021 15:20, Matteo Croce wrote: >>>> On Wed, Jun 9, 2021 at 2:01 PM Lorenzo Bianconi <lorenzo@kernel.org> wrote: >>>>> >>>>> As already done for mvneta and mvpp2, enable skb recycling for ti >>>>> ethernet drivers >>>>> >>>>> Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org> >>>> >>>> Looks good! If someone with the HW could provide a with and without >>>> the patch, that would be nice! >>>> >>> >>> What test would you recommend to run? >>> > [...] >> >> A test which benefits most from this kind of change is one in which >> the frames are freed early. > > I would also recommend running an XDP_PASS program, and then running > something that let the packets travel as deep as possible into netstack. > Not to test performance, but to make sure we didn't break something! > > I've hacked up bnxt driver (it's not as straight forward as this driver > to convert) and is running some TCP tests. Not problems so-far :-0 > > I wanted to ask if someone knows howto setup the zero-copy TCP stuff > that google did? (but is that TX only?) > TX does not need anything fancy really. For RX part, you can look at https://netdevconf.info/0x14/pub/slides/62/Implementing%20TCP%20RX%20zero%20copy.pdf
On 09/06/2021 15:20, Matteo Croce wrote: > On Wed, Jun 9, 2021 at 2:01 PM Lorenzo Bianconi <lorenzo@kernel.org> wrote: >> >> As already done for mvneta and mvpp2, enable skb recycling for ti >> ethernet drivers >> >> Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org> > > Looks good! If someone with the HW could provide a with and without > the patch, that would be nice! > Not sure to which mail to answer, so answering here - thanks all 1) I've simulated packet drop using iperf Host: - arp -s <some IP> <unknown MAC> - iperf -c <some IP> -u -l60 -b700M -t60 -i1 DUP: - place interface in promisc mode - check rx_packets stats I see big improvement ~47Kpps vs ~64Kpps 2) I've run iperf3 tests - see no regressions, but also not too much improvements 3) I've applied 2 patches - this one - and [1] Results are below, Thank you Tested-by: Grygorii Strashko <grygorii.strashko@ti.com> Reviewed-by: Grygorii Strashko <grygorii.strashko@ti.com> [1] https://patchwork.kernel.org/project/netdevbpf/patch/20210609103326.278782-18-toke@redhat.com/ =========== Before: [perf top] 47.15% [kernel] [k] _raw_spin_unlock_irqrestore 11.77% [kernel] [k] __cpdma_chan_free 3.16% [kernel] [k] ___bpf_prog_run 2.52% [kernel] [k] cpsw_rx_vlan_encap 2.34% [kernel] [k] __netif_receive_skb_core 2.27% [kernel] [k] free_unref_page 2.26% [kernel] [k] kmem_cache_free 2.24% [kernel] [k] kmem_cache_alloc 1.69% [kernel] [k] __softirqentry_text_start 1.61% [kernel] [k] cpsw_rx_handler 1.19% [kernel] [k] page_pool_release_page 1.19% [kernel] [k] clear_bits_ll 1.15% [kernel] [k] page_frag_free 1.06% [kernel] [k] __dma_page_dev_to_cpu 0.99% [kernel] [k] memset 0.94% [kernel] [k] __alloc_pages_bulk 0.92% [kernel] [k] kfree_skb 0.85% [kernel] [k] packet_rcv 0.78% [kernel] [k] page_address 0.75% [kernel] [k] v7_dma_inv_range 0.71% [kernel] [k] __lock_text_start [rx packets - with packet drop] rxdiff 48004 rxdiff 47630 rxdiff 47538 [iperf3 TCP] iperf3 -c 192.168.1.1 -i1 [ 5] 0.00-10.00 sec 873 MBytes 732 Mbits/sec 0 sender [ 5] 0.00-10.01 sec 866 MBytes 726 Mbits/sec receiver =========== After: [perf top - with packet drop] 40.58% [kernel] [k] _raw_spin_unlock_irqrestore 16.18% [kernel] [k] __softirqentry_text_start 10.33% [kernel] [k] __cpdma_chan_free 2.62% [kernel] [k] ___bpf_prog_run 2.05% [kernel] [k] cpsw_rx_vlan_encap 2.00% [kernel] [k] kmem_cache_alloc 1.86% [kernel] [k] __netif_receive_skb_core 1.80% [kernel] [k] kmem_cache_free 1.63% [kernel] [k] cpsw_rx_handler 1.12% [kernel] [k] cpsw_rx_mq_poll 1.11% [kernel] [k] page_pool_put_page 1.04% [kernel] [k] _raw_spin_unlock 0.97% [kernel] [k] clear_bits_ll 0.90% [kernel] [k] packet_rcv 0.88% [kernel] [k] __dma_page_dev_to_cpu 0.85% [kernel] [k] kfree_skb 0.80% [kernel] [k] memset 0.71% [kernel] [k] __lock_text_start 0.66% [kernel] [k] v7_dma_inv_range 0.64% [kernel] [k] gen_pool_free_owner 0.58% [kernel] [k] __rcu_read_unlock [rx packets - with packet drop] rxdiff 65843 rxdiff 66722 rxdiff 65264 [iperf3 TCP] iperf3 -c 192.168.1.1 -i1 [ 5] 0.00-10.00 sec 884 MBytes 742 Mbits/sec 0 sender [ 5] 0.00-10.01 sec 878 MBytes 735 Mbits/sec receiver
diff --git a/drivers/net/ethernet/ti/cpsw.c b/drivers/net/ethernet/ti/cpsw.c index c0cd7de88316..049508667a6d 100644 --- a/drivers/net/ethernet/ti/cpsw.c +++ b/drivers/net/ethernet/ti/cpsw.c @@ -430,8 +430,8 @@ static void cpsw_rx_handler(void *token, int len, int status) cpts_rx_timestamp(cpsw->cpts, skb); skb->protocol = eth_type_trans(skb, ndev); - /* unmap page as no netstack skb page recycling */ - page_pool_release_page(pool, page); + /* mark skb for recycling */ + skb_mark_for_recycle(skb, page, pool); netif_receive_skb(skb); ndev->stats.rx_bytes += len; diff --git a/drivers/net/ethernet/ti/cpsw_new.c b/drivers/net/ethernet/ti/cpsw_new.c index 69b7a4e0220a..11f536138495 100644 --- a/drivers/net/ethernet/ti/cpsw_new.c +++ b/drivers/net/ethernet/ti/cpsw_new.c @@ -373,8 +373,8 @@ static void cpsw_rx_handler(void *token, int len, int status) cpts_rx_timestamp(cpsw->cpts, skb); skb->protocol = eth_type_trans(skb, ndev); - /* unmap page as no netstack skb page recycling */ - page_pool_release_page(pool, page); + /* mark skb for recycling */ + skb_mark_for_recycle(skb, page, pool); netif_receive_skb(skb); ndev->stats.rx_bytes += len;
As already done for mvneta and mvpp2, enable skb recycling for ti ethernet drivers Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org> --- This patch has just compile-tested --- drivers/net/ethernet/ti/cpsw.c | 4 ++-- drivers/net/ethernet/ti/cpsw_new.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)