| Message ID | 1629257542-36145-1-git-send-email-linyunsheng@huawei.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | add socket to netdev page frag recycling support | expand |
On Wed, Aug 18, 2021 at 5:33 AM Yunsheng Lin <linyunsheng@huawei.com> wrote: > > This patchset adds the socket to netdev page frag recycling > support based on the busy polling and page pool infrastructure. I really do not see how this can scale to thousands of sockets. tcp_mem[] defaults to ~ 9 % of physical memory. If you now run tests with thousands of sockets, their skbs will consume Gigabytes of memory on typical servers, now backed by order-0 pages (instead of current order-3 pages) So IOMMU costs will actually be much bigger. Are we planning to use Gigabyte sized page pools for NIC ? Have you tried instead to make TCP frags twice bigger ? This would require less IOMMU mappings. (Note: This could require some mm help, since PAGE_ALLOC_COSTLY_ORDER is currently 3, not 4) diff --git a/net/core/sock.c b/net/core/sock.c index a3eea6e0b30a7d43793f567ffa526092c03e3546..6b66b51b61be9f198f6f1c4a3d81b57fa327986a 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -2560,7 +2560,7 @@ static void sk_leave_memory_pressure(struct sock *sk) } } -#define SKB_FRAG_PAGE_ORDER get_order(32768) +#define SKB_FRAG_PAGE_ORDER get_order(65536) DEFINE_STATIC_KEY_FALSE(net_high_order_alloc_disable_key); /** > > The profermance improve from 30Gbit to 41Gbit for one thread iperf > tcp flow, and the CPU usages decreases about 20% for four threads > iperf flow with 100Gb line speed in IOMMU strict mode. > > The profermance improve about 2.5% for one thread iperf tcp flow > in IOMMU passthrough mode. > > Yunsheng Lin (7): > page_pool: refactor the page pool to support multi alloc context > skbuff: add interface to manipulate frag count for tx recycling > net: add NAPI api to register and retrieve the page pool ptr > net: pfrag_pool: add pfrag pool support based on page pool > sock: support refilling pfrag from pfrag_pool > net: hns3: support tx recycling in the hns3 driver > sysctl_tcp_use_pfrag_pool > > drivers/net/ethernet/hisilicon/hns3/hns3_enet.c | 32 +++++---- > include/linux/netdevice.h | 9 +++ > include/linux/skbuff.h | 43 +++++++++++- > include/net/netns/ipv4.h | 1 + > include/net/page_pool.h | 15 ++++ > include/net/pfrag_pool.h | 24 +++++++ > include/net/sock.h | 1 + > net/core/Makefile | 1 + > net/core/dev.c | 34 ++++++++- > net/core/page_pool.c | 86 ++++++++++++----------- > net/core/pfrag_pool.c | 92 +++++++++++++++++++++++++ > net/core/sock.c | 12 ++++ > net/ipv4/sysctl_net_ipv4.c | 7 ++ > net/ipv4/tcp.c | 34 ++++++--- > 14 files changed, 325 insertions(+), 66 deletions(-) > create mode 100644 include/net/pfrag_pool.h > create mode 100644 net/core/pfrag_pool.c > > -- > 2.7.4 >
On 2021/8/18 16:57, Eric Dumazet wrote: > On Wed, Aug 18, 2021 at 5:33 AM Yunsheng Lin <linyunsheng@huawei.com> wrote: >> >> This patchset adds the socket to netdev page frag recycling >> support based on the busy polling and page pool infrastructure. > > I really do not see how this can scale to thousands of sockets. > > tcp_mem[] defaults to ~ 9 % of physical memory. > > If you now run tests with thousands of sockets, their skbs will > consume Gigabytes > of memory on typical servers, now backed by order-0 pages (instead of > current order-3 pages) > So IOMMU costs will actually be much bigger. As the page allocator support bulk allocating now, see: https://elixir.bootlin.com/linux/latest/source/net/core/page_pool.c#L252 if the DMA also support batch mapping/unmapping, maybe having a small-sized page pool for thousands of sockets may not be a problem? Christoph Hellwig mentioned the batch DMA operation support in below thread: https://www.spinics.net/lists/netdev/msg666715.html if the batched DMA operation is supported, maybe having the page pool is mainly benefit the case of small number of socket? > > Are we planning to use Gigabyte sized page pools for NIC ? > > Have you tried instead to make TCP frags twice bigger ? Not yet. > This would require less IOMMU mappings. > (Note: This could require some mm help, since PAGE_ALLOC_COSTLY_ORDER > is currently 3, not 4) I am not familiar with mm yet, but I will take a look about that:) > > diff --git a/net/core/sock.c b/net/core/sock.c > index a3eea6e0b30a7d43793f567ffa526092c03e3546..6b66b51b61be9f198f6f1c4a3d81b57fa327986a > 100644 > --- a/net/core/sock.c > +++ b/net/core/sock.c > @@ -2560,7 +2560,7 @@ static void sk_leave_memory_pressure(struct sock *sk) > } > } > > -#define SKB_FRAG_PAGE_ORDER get_order(32768) > +#define SKB_FRAG_PAGE_ORDER get_order(65536) > DEFINE_STATIC_KEY_FALSE(net_high_order_alloc_disable_key); > > /** > > > >>
On 8/17/21 9:32 PM, Yunsheng Lin wrote: > This patchset adds the socket to netdev page frag recycling > support based on the busy polling and page pool infrastructure. > > The profermance improve from 30Gbit to 41Gbit for one thread iperf > tcp flow, and the CPU usages decreases about 20% for four threads > iperf flow with 100Gb line speed in IOMMU strict mode. > > The profermance improve about 2.5% for one thread iperf tcp flow > in IOMMU passthrough mode. > Details about the test setup? cpu model, mtu, any other relevant changes / settings. How does that performance improvement compare with using the Tx ZC API? At 1500 MTU I see a CPU drop on the Tx side from 80% to 20% with the ZC API and ~10% increase in throughput. Bumping the MTU to 3300 and performance with the ZC API is 2x the current model with 1/2 the cpu. Epyc 7502, ConnectX-6, IOMMU off. In short, it seems like improving the Tx ZC API is the better path forward than per-socket page pools.
On 2021/8/19 6:05, David Ahern wrote: > On 8/17/21 9:32 PM, Yunsheng Lin wrote: >> This patchset adds the socket to netdev page frag recycling >> support based on the busy polling and page pool infrastructure. >> >> The profermance improve from 30Gbit to 41Gbit for one thread iperf >> tcp flow, and the CPU usages decreases about 20% for four threads >> iperf flow with 100Gb line speed in IOMMU strict mode. >> >> The profermance improve about 2.5% for one thread iperf tcp flow >> in IOMMU passthrough mode. >> > > Details about the test setup? cpu model, mtu, any other relevant changes > / settings. CPU is arm64 Kunpeng 920, see: https://www.hisilicon.com/en/products/Kunpeng/Huawei-Kunpeng-920 mtu is 1500, the relevant changes/settings I can think of the iperf client runs on the same numa as the nic hw exists(which has one 100Gbit port), and the driver has the XPS enabled too. > > How does that performance improvement compare with using the Tx ZC API? > At 1500 MTU I see a CPU drop on the Tx side from 80% to 20% with the ZC > API and ~10% increase in throughput. Bumping the MTU to 3300 and > performance with the ZC API is 2x the current model with 1/2 the cpu. I added a sysctl node to decide whether pfrag pool is used: net.ipv4.tcp_use_pfrag_pool and use msg_zerocopy to compare the result: Server uses cmd "./msg_zerocopy -4 -i eth4 -C 32 -S 192.168.100.2 -r tcp" Client uses cmd "./msg_zerocopy -4 -i eth4 -C 0 -S 192.168.100.1 -D 192.168.100.2 tcp -" The zc does seem to improve the CPU usages significantly, but not for throughput with mtu 1500. And the result seems to be similar with mtu 3300. the detail result is below: (1) IOMMU strict mode + net.ipv4.tcp_use_pfrag_pool = 0: root@(none):/# perf stat -e cycles ./msg_zerocopy -4 -i eth4 -C 0 -S 192.168.100.1 -D 192.168.100.2 tcp tx=115317 (7196 MB) txc=0 zc=n Performance counter stats for './msg_zerocopy -4 -i eth4 -C 0 -S 192.168.100.1 -D 192.168.100.2 tcp': 4315472244 cycles 4.199890190 seconds time elapsed 0.084328000 seconds user 1.528714000 seconds sys root@(none):/# perf stat -e cycles ./msg_zerocopy -4 -i eth4 -C 0 -S 192.168.100.1 -D 192.168.100.2 tcp -z tx=90121 (5623 MB) txc=90121 zc=y Performance counter stats for './msg_zerocopy -4 -i eth4 -C 0 -S 192.168.100.1 -D 192.168.100.2 tcp -z': 1715892155 cycles 4.243329050 seconds time elapsed 0.083275000 seconds user 0.755355000 seconds sys (2)IOMMU strict mode + net.ipv4.tcp_use_pfrag_pool = 1: root@(none):/# perf stat -e cycles ./msg_zerocopy -4 -i eth4 -C 0 -S 192.168.100.1 -D 192.168.100.2 tcp tx=138932 (8669 MB) txc=0 zc=n Performance counter stats for './msg_zerocopy -4 -i eth4 -C 0 -S 192.168.100.1 -D 192.168.100.2 tcp': 4034016168 cycles 4.199877510 seconds time elapsed 0.058143000 seconds user 1.644480000 seconds sys root@(none):/# perf stat -e cycles ./msg_zerocopy -4 -i eth4 -C 0 -S 192.168.100.1 -D 192.168.100.2 tcp -z tx=93369 (5826 MB) txc=93369 zc=y Performance counter stats for './msg_zerocopy -4 -i eth4 -C 0 -S 192.168.100.1 -D 192.168.100.2 tcp -z': 1815300491 cycles 4.243259530 seconds time elapsed 0.051767000 seconds user 0.796610000 seconds sys (3)IOMMU passthrough + net.ipv4.tcp_use_pfrag_pool=0 root@(none):/# perf stat -e cycles ./msg_zerocopy -4 -i eth4 -C 0 -S 192.168.100.1 -D 192.168.100.2 tcp tx=129927 (8107 MB) txc=0 zc=n Performance counter stats for './msg_zerocopy -4 -i eth4 -C 0 -S 192.168.100.1 -D 192.168.100.2 tcp': 3720131007 cycles 4.200651840 seconds time elapsed 0.038604000 seconds user 1.455521000 seconds sys root@(none):/# perf stat -e cycles ./msg_zerocopy -4 -i eth4 -C 0 -S 192.168.100.1 -D 192.168.100.2 tcp -z tx=135285 (8442 MB) txc=135285 zc=y Performance counter stats for './msg_zerocopy -4 -i eth4 -C 0 -S 192.168.100.1 -D 192.168.100.2 tcp -z': 1721949875 cycles 4.242596800 seconds time elapsed 0.024963000 seconds user 0.779391000 seconds sys (4)IOMMU passthrough + net.ipv4.tcp_use_pfrag_pool=1 root@(none):/# perf stat -e cycles ./msg_zerocopy -4 -i eth4 -C 0 -S 192.168.100.1 -D 192.168.100.2 tcp tx=151844 (9475 MB) txc=0 zc=n Performance counter stats for './msg_zerocopy -4 -i eth4 -C 0 -S 192.168.100.1 -D 192.168.100.2 tcp': 3786216097 cycles 4.200606520 seconds time elapsed 0.028633000 seconds user 1.569736000 seconds sys > > Epyc 7502, ConnectX-6, IOMMU off. > > In short, it seems like improving the Tx ZC API is the better path > forward than per-socket page pools. The main goal is to optimize the SMMU mapping/unmaping, if the cost of memcpy it higher than the SMMU mapping/unmaping + page pinning, then Tx ZC may be a better path, at leas it is not sure for small packet? > . >
On 8/19/21 2:18 AM, Yunsheng Lin wrote: > On 2021/8/19 6:05, David Ahern wrote: >> On 8/17/21 9:32 PM, Yunsheng Lin wrote: >>> This patchset adds the socket to netdev page frag recycling >>> support based on the busy polling and page pool infrastructure. >>> >>> The profermance improve from 30Gbit to 41Gbit for one thread iperf >>> tcp flow, and the CPU usages decreases about 20% for four threads >>> iperf flow with 100Gb line speed in IOMMU strict mode. >>> >>> The profermance improve about 2.5% for one thread iperf tcp flow >>> in IOMMU passthrough mode. >>> >> >> Details about the test setup? cpu model, mtu, any other relevant changes >> / settings. > > CPU is arm64 Kunpeng 920, see: > https://www.hisilicon.com/en/products/Kunpeng/Huawei-Kunpeng-920 > > mtu is 1500, the relevant changes/settings I can think of the iperf > client runs on the same numa as the nic hw exists(which has one 100Gbit > port), and the driver has the XPS enabled too. > >> >> How does that performance improvement compare with using the Tx ZC API? >> At 1500 MTU I see a CPU drop on the Tx side from 80% to 20% with the ZC >> API and ~10% increase in throughput. Bumping the MTU to 3300 and >> performance with the ZC API is 2x the current model with 1/2 the cpu. > > I added a sysctl node to decide whether pfrag pool is used: > net.ipv4.tcp_use_pfrag_pool > > and use msg_zerocopy to compare the result: > Server uses cmd "./msg_zerocopy -4 -i eth4 -C 32 -S 192.168.100.2 -r tcp" > Client uses cmd "./msg_zerocopy -4 -i eth4 -C 0 -S 192.168.100.1 -D 192.168.100.2 tcp -" > > The zc does seem to improve the CPU usages significantly, but not for throughput > with mtu 1500. And the result seems to be similar with mtu 3300. > > the detail result is below: > > (1) IOMMU strict mode + net.ipv4.tcp_use_pfrag_pool = 0: > root@(none):/# perf stat -e cycles ./msg_zerocopy -4 -i eth4 -C 0 -S 192.168.100.1 -D 192.168.100.2 tcp > tx=115317 (7196 MB) txc=0 zc=n > > Performance counter stats for './msg_zerocopy -4 -i eth4 -C 0 -S 192.168.100.1 -D 192.168.100.2 tcp': > > 4315472244 cycles > > 4.199890190 seconds time elapsed > > 0.084328000 seconds user > 1.528714000 seconds sys > root@(none):/# perf stat -e cycles ./msg_zerocopy -4 -i eth4 -C 0 -S 192.168.100.1 -D 192.168.100.2 tcp -z > tx=90121 (5623 MB) txc=90121 zc=y > > Performance counter stats for './msg_zerocopy -4 -i eth4 -C 0 -S 192.168.100.1 -D 192.168.100.2 tcp -z': > > 1715892155 cycles > > 4.243329050 seconds time elapsed > > 0.083275000 seconds user > 0.755355000 seconds sys > > > (2)IOMMU strict mode + net.ipv4.tcp_use_pfrag_pool = 1: > root@(none):/# perf stat -e cycles ./msg_zerocopy -4 -i eth4 -C 0 -S 192.168.100.1 -D 192.168.100.2 tcp > tx=138932 (8669 MB) txc=0 zc=n > > Performance counter stats for './msg_zerocopy -4 -i eth4 -C 0 -S 192.168.100.1 -D 192.168.100.2 tcp': > > 4034016168 cycles > > 4.199877510 seconds time elapsed > > 0.058143000 seconds user > 1.644480000 seconds sys > root@(none):/# perf stat -e cycles ./msg_zerocopy -4 -i eth4 -C 0 -S 192.168.100.1 -D 192.168.100.2 tcp -z > tx=93369 (5826 MB) txc=93369 zc=y > > Performance counter stats for './msg_zerocopy -4 -i eth4 -C 0 -S 192.168.100.1 -D 192.168.100.2 tcp -z': > > 1815300491 cycles > > 4.243259530 seconds time elapsed > > 0.051767000 seconds user > 0.796610000 seconds sys > > > (3)IOMMU passthrough + net.ipv4.tcp_use_pfrag_pool=0 > root@(none):/# perf stat -e cycles ./msg_zerocopy -4 -i eth4 -C 0 -S 192.168.100.1 -D 192.168.100.2 tcp > tx=129927 (8107 MB) txc=0 zc=n > > Performance counter stats for './msg_zerocopy -4 -i eth4 -C 0 -S 192.168.100.1 -D 192.168.100.2 tcp': > > 3720131007 cycles > > 4.200651840 seconds time elapsed > > 0.038604000 seconds user > 1.455521000 seconds sys > root@(none):/# perf stat -e cycles ./msg_zerocopy -4 -i eth4 -C 0 -S 192.168.100.1 -D 192.168.100.2 tcp -z > tx=135285 (8442 MB) txc=135285 zc=y > > Performance counter stats for './msg_zerocopy -4 -i eth4 -C 0 -S 192.168.100.1 -D 192.168.100.2 tcp -z': > > 1721949875 cycles > > 4.242596800 seconds time elapsed > > 0.024963000 seconds user > 0.779391000 seconds sys > > (4)IOMMU passthrough + net.ipv4.tcp_use_pfrag_pool=1 > root@(none):/# perf stat -e cycles ./msg_zerocopy -4 -i eth4 -C 0 -S 192.168.100.1 -D 192.168.100.2 tcp > tx=151844 (9475 MB) txc=0 zc=n > > Performance counter stats for './msg_zerocopy -4 -i eth4 -C 0 -S 192.168.100.1 -D 192.168.100.2 tcp': > > 3786216097 cycles > > 4.200606520 seconds time elapsed > > 0.028633000 seconds user > 1.569736000 seconds sys > > >> >> Epyc 7502, ConnectX-6, IOMMU off. >> >> In short, it seems like improving the Tx ZC API is the better path >> forward than per-socket page pools. > > The main goal is to optimize the SMMU mapping/unmaping, if the cost of memcpy > it higher than the SMMU mapping/unmaping + page pinning, then Tx ZC may be a > better path, at leas it is not sure for small packet? > It's a CPU bound problem - either Rx or Tx is cpu bound depending on the test configuration. In my tests 3.3 to 3.5M pps is the limit (not using LRO in the NIC - that's a different solution with its own problems). At 1500 MTU lowering CPU usage on the Tx side does not accomplish much on throughput since the Rx is 100% cpu. At 3300 MTU you have ~47% the pps for the same throughput. Lower pps reduces Rx processing and lower CPU to process the incoming stream. Then using the Tx ZC API you lower the Tx overehad allowing a single stream to faster - sending more data which in the end results in much higher pps and throughput. At the limit you are CPU bound (both ends in my testing as Rx side approaches the max pps, and Tx side as it continually tries to send data). Lowering CPU usage on Tx the side is a win regardless of whether there is a big increase on the throughput at 1500 MTU since that configuration is an Rx CPU bound problem. Hence, my point that we have a good start point for lowering CPU usage on the Tx side; we should improve it rather than add per-socket page pools. You can stress the Tx side and emphasize its overhead by modifying the receiver to drop the data on Rx rather than copy to userspace which is a huge bottleneck (e.g., MSG_TRUNC on recv). This allows the single flow stream to go faster and emphasize Tx bottlenecks as the pps at 3300 approaches the top pps at 1500. e.g., doing this with iperf3 shows the spinlock overhead with tcp_sendmsg, overhead related to 'select' and then gup_pgd_range.
On 2021/8/20 22:35, David Ahern wrote: > On 8/19/21 2:18 AM, Yunsheng Lin wrote: >> On 2021/8/19 6:05, David Ahern wrote: >>> On 8/17/21 9:32 PM, Yunsheng Lin wrote: >>>> This patchset adds the socket to netdev page frag recycling >>>> support based on the busy polling and page pool infrastructure. >>>> >>>> The profermance improve from 30Gbit to 41Gbit for one thread iperf >>>> tcp flow, and the CPU usages decreases about 20% for four threads >>>> iperf flow with 100Gb line speed in IOMMU strict mode. >>>> >>>> The profermance improve about 2.5% for one thread iperf tcp flow >>>> in IOMMU passthrough mode. >>>> >>> >>> Details about the test setup? cpu model, mtu, any other relevant changes >>> / settings. >> >> CPU is arm64 Kunpeng 920, see: >> https://www.hisilicon.com/en/products/Kunpeng/Huawei-Kunpeng-920 >> >> mtu is 1500, the relevant changes/settings I can think of the iperf >> client runs on the same numa as the nic hw exists(which has one 100Gbit >> port), and the driver has the XPS enabled too. >> >>> >>> How does that performance improvement compare with using the Tx ZC API? >>> At 1500 MTU I see a CPU drop on the Tx side from 80% to 20% with the ZC >>> API and ~10% increase in throughput. Bumping the MTU to 3300 and >>> performance with the ZC API is 2x the current model with 1/2 the cpu. >> >> I added a sysctl node to decide whether pfrag pool is used: >> net.ipv4.tcp_use_pfrag_pool >> [..] >> >> >>> >>> Epyc 7502, ConnectX-6, IOMMU off. >>> >>> In short, it seems like improving the Tx ZC API is the better path >>> forward than per-socket page pools. >> >> The main goal is to optimize the SMMU mapping/unmaping, if the cost of memcpy >> it higher than the SMMU mapping/unmaping + page pinning, then Tx ZC may be a >> better path, at leas it is not sure for small packet? >> > > It's a CPU bound problem - either Rx or Tx is cpu bound depending on the > test configuration. In my tests 3.3 to 3.5M pps is the limit (not using > LRO in the NIC - that's a different solution with its own problems). I assumed the "either Rx or Tx is cpu bound" meant either Rx or Tx is the bottleneck? It seems iperf3 support the Tx ZC, I retested using the iperf3, Rx settings is not changed when testing, MTU is 1500: IOMMU in strict mode: 1. Tx ZC case: 22Gbit with Tx being bottleneck(cpu bound) 2. Tx non-ZC case with pfrag pool enabled: 40Git with Rx being bottleneck(cpu bound) 3. Tx non-ZC case with pfrag pool disabled: 30Git, the bottleneck seems not to be cpu bound, as the Rx and Tx does not have a single CPU reaching about 100% usage. > > At 1500 MTU lowering CPU usage on the Tx side does not accomplish much > on throughput since the Rx is 100% cpu. As above performance data, enabling ZC does not seems to help when IOMMU is involved, which has about 30% performance degrade when pfrag pool is disabled and 50% performance degrade when pfrag pool is enabled. > > At 3300 MTU you have ~47% the pps for the same throughput. Lower pps > reduces Rx processing and lower CPU to process the incoming stream. Then > using the Tx ZC API you lower the Tx overehad allowing a single stream > to faster - sending more data which in the end results in much higher > pps and throughput. At the limit you are CPU bound (both ends in my > testing as Rx side approaches the max pps, and Tx side as it continually > tries to send data). > > Lowering CPU usage on Tx the side is a win regardless of whether there > is a big increase on the throughput at 1500 MTU since that configuration > is an Rx CPU bound problem. Hence, my point that we have a good start > point for lowering CPU usage on the Tx side; we should improve it rather > than add per-socket page pools. Acctually it is not a per-socket page pools, the page pool is still per NAPI, this patchset adds multi allocation context to the page pool, so that the tx can reuse the same page pool with rx, which is quite usefully if the ARFS is enabled. > > You can stress the Tx side and emphasize its overhead by modifying the > receiver to drop the data on Rx rather than copy to userspace which is a > huge bottleneck (e.g., MSG_TRUNC on recv). This allows the single flow As the frag page is supported in page pool for Rx, the Rx probably is not a bottleneck any more, at least not for IOMMU in strict mode. It seems iperf3 does not support MSG_TRUNC yet, any testing tool supporting MSG_TRUNC? Or do I have to hack the kernel or iperf3 tool to do that? > stream to go faster and emphasize Tx bottlenecks as the pps at 3300 > approaches the top pps at 1500. e.g., doing this with iperf3 shows the > spinlock overhead with tcp_sendmsg, overhead related to 'select' and > then gup_pgd_range. When IOMMU is in strict mode, the overhead with IOMMU seems to be much bigger than spinlock(23% to 10%). Anyway, I still think ZC mostly benefit to packet which is bigger than a specific size and IOMMU disabling case. > . >
On 2021/8/18 17:36, Yunsheng Lin wrote: > On 2021/8/18 16:57, Eric Dumazet wrote: >> On Wed, Aug 18, 2021 at 5:33 AM Yunsheng Lin <linyunsheng@huawei.com> wrote: >>> >>> This patchset adds the socket to netdev page frag recycling >>> support based on the busy polling and page pool infrastructure. >> >> I really do not see how this can scale to thousands of sockets. >> >> tcp_mem[] defaults to ~ 9 % of physical memory. >> >> If you now run tests with thousands of sockets, their skbs will >> consume Gigabytes >> of memory on typical servers, now backed by order-0 pages (instead of >> current order-3 pages) >> So IOMMU costs will actually be much bigger. > > As the page allocator support bulk allocating now, see: > https://elixir.bootlin.com/linux/latest/source/net/core/page_pool.c#L252 > > if the DMA also support batch mapping/unmapping, maybe having a > small-sized page pool for thousands of sockets may not be a problem? > Christoph Hellwig mentioned the batch DMA operation support in below > thread: > https://www.spinics.net/lists/netdev/msg666715.html > > if the batched DMA operation is supported, maybe having the > page pool is mainly benefit the case of small number of socket? > >> >> Are we planning to use Gigabyte sized page pools for NIC ? >> >> Have you tried instead to make TCP frags twice bigger ? > > Not yet. > >> This would require less IOMMU mappings. >> (Note: This could require some mm help, since PAGE_ALLOC_COSTLY_ORDER >> is currently 3, not 4) > > I am not familiar with mm yet, but I will take a look about that:) It seems PAGE_ALLOC_COSTLY_ORDER is mostly related to pcp page, OOM, memory compact and memory isolation, as the test system has a lot of memory installed (about 500G, only 3-4G is used), so I used the below patch to test the max possible performance improvement when making TCP frags twice bigger, and the performance improvement went from about 30Gbit to 32Gbit for one thread iperf tcp flow in IOMMU strict mode, and using the pfrag pool, the improvement went from about 30Gbit to 40Gbit for the same testing configuation: diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index fcb5355..dda20f9 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -37,7 +37,7 @@ * coalesce naturally under reasonable reclaim pressure and those which * will not. */ -#define PAGE_ALLOC_COSTLY_ORDER 3 +#define PAGE_ALLOC_COSTLY_ORDER 4 enum migratetype { MIGRATE_UNMOVABLE, diff --git a/net/core/sock.c b/net/core/sock.c index 870a3b7..b1e0dfc 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -2580,7 +2580,7 @@ static void sk_leave_memory_pressure(struct sock *sk) } } -#define SKB_FRAG_PAGE_ORDER get_order(32768) +#define SKB_FRAG_PAGE_ORDER get_order(65536) DEFINE_STATIC_KEY_FALSE(net_high_order_alloc_disable_key); /** > >> >> diff --git a/net/core/sock.c b/net/core/sock.c >> index a3eea6e0b30a7d43793f567ffa526092c03e3546..6b66b51b61be9f198f6f1c4a3d81b57fa327986a >> 100644 >> --- a/net/core/sock.c >> +++ b/net/core/sock.c >> @@ -2560,7 +2560,7 @@ static void sk_leave_memory_pressure(struct sock *sk) >> } >> } >> >> -#define SKB_FRAG_PAGE_ORDER get_order(32768) >> +#define SKB_FRAG_PAGE_ORDER get_order(65536) >> DEFINE_STATIC_KEY_FALSE(net_high_order_alloc_disable_key); >> >> /** >> >> >> >>> > _______________________________________________ > Linuxarm mailing list -- linuxarm@openeuler.org > To unsubscribe send an email to linuxarm-leave@openeuler.org >
On Mon, Aug 23, 2021 at 2:25 AM Yunsheng Lin <linyunsheng@huawei.com> wrote: > > On 2021/8/18 17:36, Yunsheng Lin wrote: > > On 2021/8/18 16:57, Eric Dumazet wrote: > >> On Wed, Aug 18, 2021 at 5:33 AM Yunsheng Lin <linyunsheng@huawei.com> wrote: > >>> > >>> This patchset adds the socket to netdev page frag recycling > >>> support based on the busy polling and page pool infrastructure. > >> > >> I really do not see how this can scale to thousands of sockets. > >> > >> tcp_mem[] defaults to ~ 9 % of physical memory. > >> > >> If you now run tests with thousands of sockets, their skbs will > >> consume Gigabytes > >> of memory on typical servers, now backed by order-0 pages (instead of > >> current order-3 pages) > >> So IOMMU costs will actually be much bigger. > > > > As the page allocator support bulk allocating now, see: > > https://elixir.bootlin.com/linux/latest/source/net/core/page_pool.c#L252 > > > > if the DMA also support batch mapping/unmapping, maybe having a > > small-sized page pool for thousands of sockets may not be a problem? > > Christoph Hellwig mentioned the batch DMA operation support in below > > thread: > > https://www.spinics.net/lists/netdev/msg666715.html > > > > if the batched DMA operation is supported, maybe having the > > page pool is mainly benefit the case of small number of socket? > > > >> > >> Are we planning to use Gigabyte sized page pools for NIC ? > >> > >> Have you tried instead to make TCP frags twice bigger ? > > > > Not yet. > > > >> This would require less IOMMU mappings. > >> (Note: This could require some mm help, since PAGE_ALLOC_COSTLY_ORDER > >> is currently 3, not 4) > > > > I am not familiar with mm yet, but I will take a look about that:) > > > It seems PAGE_ALLOC_COSTLY_ORDER is mostly related to pcp page, OOM, memory > compact and memory isolation, as the test system has a lot of memory installed > (about 500G, only 3-4G is used), so I used the below patch to test the max > possible performance improvement when making TCP frags twice bigger, and > the performance improvement went from about 30Gbit to 32Gbit for one thread > iperf tcp flow in IOMMU strict mode, This is encouraging, and means we can do much better. Even with SKB_FRAG_PAGE_ORDER set to 4, typical skbs will need 3 mappings 1) One for the headers (in skb->head) 2) Two page frags, because one TSO packet payload is not a nice power-of-two. The first issue can be addressed using a piece of coherent memory (128 or 256 bytes per entry in TX ring). Copying the headers can avoid one IOMMU mapping, and improve IOTLB hits, because all slots of the TX ring buffer will use one single IOTLB slot. The second issue can be solved by tweaking a bit skb_page_frag_refill() to accept an additional parameter so that the whole skb payload fits in a single order-4 page. and using the pfrag pool, the improvement > went from about 30Gbit to 40Gbit for the same testing configuation: Yes, but you have not provided performance number when 200 (or 1000+) concurrent flows are running. Optimizing singe flow TCP performance while killing performance for the more common case is not an option. > > diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h > index fcb5355..dda20f9 100644 > --- a/include/linux/mmzone.h > +++ b/include/linux/mmzone.h > @@ -37,7 +37,7 @@ > * coalesce naturally under reasonable reclaim pressure and those which > * will not. > */ > -#define PAGE_ALLOC_COSTLY_ORDER 3 > +#define PAGE_ALLOC_COSTLY_ORDER 4 > > enum migratetype { > MIGRATE_UNMOVABLE, > diff --git a/net/core/sock.c b/net/core/sock.c > index 870a3b7..b1e0dfc 100644 > --- a/net/core/sock.c > +++ b/net/core/sock.c > @@ -2580,7 +2580,7 @@ static void sk_leave_memory_pressure(struct sock *sk) > } > } > > -#define SKB_FRAG_PAGE_ORDER get_order(32768) > +#define SKB_FRAG_PAGE_ORDER get_order(65536) > DEFINE_STATIC_KEY_FALSE(net_high_order_alloc_disable_key); > > /** > > > > >> > >> diff --git a/net/core/sock.c b/net/core/sock.c > >> index a3eea6e0b30a7d43793f567ffa526092c03e3546..6b66b51b61be9f198f6f1c4a3d81b57fa327986a > >> 100644 > >> --- a/net/core/sock.c > >> +++ b/net/core/sock.c > >> @@ -2560,7 +2560,7 @@ static void sk_leave_memory_pressure(struct sock *sk) > >> } > >> } > >> > >> -#define SKB_FRAG_PAGE_ORDER get_order(32768) > >> +#define SKB_FRAG_PAGE_ORDER get_order(65536) > >> DEFINE_STATIC_KEY_FALSE(net_high_order_alloc_disable_key); > >> > >> /** > >> > >> > >> > >>> > > _______________________________________________ > > Linuxarm mailing list -- linuxarm@openeuler.org > > To unsubscribe send an email to linuxarm-leave@openeuler.org > >
On 8/22/21 9:32 PM, Yunsheng Lin wrote: > > I assumed the "either Rx or Tx is cpu bound" meant either Rx or Tx is the > bottleneck? yes. > > It seems iperf3 support the Tx ZC, I retested using the iperf3, Rx settings > is not changed when testing, MTU is 1500: -Z == sendfile API. That works fine to a point and that point is well below 100G. I mean TCP with MSG_ZEROCOPY and SO_ZEROCOPY. > > IOMMU in strict mode: > 1. Tx ZC case: > 22Gbit with Tx being bottleneck(cpu bound) > 2. Tx non-ZC case with pfrag pool enabled: > 40Git with Rx being bottleneck(cpu bound) > 3. Tx non-ZC case with pfrag pool disabled: > 30Git, the bottleneck seems not to be cpu bound, as the Rx and Tx does > not have a single CPU reaching about 100% usage. > >> >> At 1500 MTU lowering CPU usage on the Tx side does not accomplish much >> on throughput since the Rx is 100% cpu. > > As above performance data, enabling ZC does not seems to help when IOMMU > is involved, which has about 30% performance degrade when pfrag pool is > disabled and 50% performance degrade when pfrag pool is enabled. In a past response you should numbers for Tx ZC API with a custom program. That program showed the dramatic reduction in CPU cycles for Tx with the ZC API. > >> >> At 3300 MTU you have ~47% the pps for the same throughput. Lower pps >> reduces Rx processing and lower CPU to process the incoming stream. Then >> using the Tx ZC API you lower the Tx overehad allowing a single stream >> to faster - sending more data which in the end results in much higher >> pps and throughput. At the limit you are CPU bound (both ends in my >> testing as Rx side approaches the max pps, and Tx side as it continually >> tries to send data). >> >> Lowering CPU usage on Tx the side is a win regardless of whether there >> is a big increase on the throughput at 1500 MTU since that configuration >> is an Rx CPU bound problem. Hence, my point that we have a good start >> point for lowering CPU usage on the Tx side; we should improve it rather >> than add per-socket page pools. > > Acctually it is not a per-socket page pools, the page pool is still per > NAPI, this patchset adds multi allocation context to the page pool, so that > the tx can reuse the same page pool with rx, which is quite usefully if the > ARFS is enabled. > >> >> You can stress the Tx side and emphasize its overhead by modifying the >> receiver to drop the data on Rx rather than copy to userspace which is a >> huge bottleneck (e.g., MSG_TRUNC on recv). This allows the single flow > > As the frag page is supported in page pool for Rx, the Rx probably is not > a bottleneck any more, at least not for IOMMU in strict mode. > > It seems iperf3 does not support MSG_TRUNC yet, any testing tool supporting > MSG_TRUNC? Or do I have to hack the kernel or iperf3 tool to do that? https://github.com/dsahern/iperf, mods branch --zc_api is the Tx ZC API; --rx_drop adds MSG_TRUNC to recv. > >> stream to go faster and emphasize Tx bottlenecks as the pps at 3300 >> approaches the top pps at 1500. e.g., doing this with iperf3 shows the >> spinlock overhead with tcp_sendmsg, overhead related to 'select' and >> then gup_pgd_range. > > When IOMMU is in strict mode, the overhead with IOMMU seems to be much > bigger than spinlock(23% to 10%). > > Anyway, I still think ZC mostly benefit to packet which is bigger than a > specific size and IOMMU disabling case. > > >> . >>
On 2021/8/23 23:04, Eric Dumazet wrote: > On Mon, Aug 23, 2021 at 2:25 AM Yunsheng Lin <linyunsheng@huawei.com> wrote: >> >> On 2021/8/18 17:36, Yunsheng Lin wrote: >>> On 2021/8/18 16:57, Eric Dumazet wrote: >>>> On Wed, Aug 18, 2021 at 5:33 AM Yunsheng Lin <linyunsheng@huawei.com> wrote: >>>>> >>>>> This patchset adds the socket to netdev page frag recycling >>>>> support based on the busy polling and page pool infrastructure. >>>> >>>> I really do not see how this can scale to thousands of sockets. >>>> >>>> tcp_mem[] defaults to ~ 9 % of physical memory. >>>> >>>> If you now run tests with thousands of sockets, their skbs will >>>> consume Gigabytes >>>> of memory on typical servers, now backed by order-0 pages (instead of >>>> current order-3 pages) >>>> So IOMMU costs will actually be much bigger. >>> >>> As the page allocator support bulk allocating now, see: >>> https://elixir.bootlin.com/linux/latest/source/net/core/page_pool.c#L252 >>> >>> if the DMA also support batch mapping/unmapping, maybe having a >>> small-sized page pool for thousands of sockets may not be a problem? >>> Christoph Hellwig mentioned the batch DMA operation support in below >>> thread: >>> https://www.spinics.net/lists/netdev/msg666715.html >>> >>> if the batched DMA operation is supported, maybe having the >>> page pool is mainly benefit the case of small number of socket? >>> >>>> >>>> Are we planning to use Gigabyte sized page pools for NIC ? >>>> >>>> Have you tried instead to make TCP frags twice bigger ? >>> >>> Not yet. >>> >>>> This would require less IOMMU mappings. >>>> (Note: This could require some mm help, since PAGE_ALLOC_COSTLY_ORDER >>>> is currently 3, not 4) >>> >>> I am not familiar with mm yet, but I will take a look about that:) >> >> >> It seems PAGE_ALLOC_COSTLY_ORDER is mostly related to pcp page, OOM, memory >> compact and memory isolation, as the test system has a lot of memory installed >> (about 500G, only 3-4G is used), so I used the below patch to test the max >> possible performance improvement when making TCP frags twice bigger, and >> the performance improvement went from about 30Gbit to 32Gbit for one thread >> iperf tcp flow in IOMMU strict mode, > > This is encouraging, and means we can do much better. > > Even with SKB_FRAG_PAGE_ORDER set to 4, typical skbs will need 3 mappings > > 1) One for the headers (in skb->head) > 2) Two page frags, because one TSO packet payload is not a nice power-of-two. > > The first issue can be addressed using a piece of coherent memory (128 > or 256 bytes per entry in TX ring). > Copying the headers can avoid one IOMMU mapping, and improve IOTLB > hits, because all > slots of the TX ring buffer will use one single IOTLB slot. Acctually, the hns3 driver has implemented the bounce buffer for the above case, see: https://elixir.bootlin.com/linux/v5.14-rc7/source/drivers/net/ethernet/hisilicon/hns3/hns3_enet.c#L2042 Enabling the header buffer copying, the performance only improve from about 30Gbit to 32Gbit for one thread iperf tcp flow. So it seems the IOMMU overhead does not only related to how many frag does a skb have, but also related to the length of each frag, as the IOMMU mapping is based on 4K/2M granularity(for arm64), so it may still take a lot of time to write each 4K page entry to the page table when mapping and invalidate each 4K page entry when unmapping. Also, hns3 driver implement the dma_map_sg() to reduce the number of IOMMU mapping/unmapping, the peformance is only about 10%, possibly due to the above reason too, see: https://lkml.org/lkml/2021/6/16/134 > > The second issue can be solved by tweaking a bit > skb_page_frag_refill() to accept an additional parameter > so that the whole skb payload fits in a single order-4 page. I am not sure I understand the above. Are you suggesting passing 'copy' to skb_page_frag_refill(), so that it will allocate a new pages if there is no enough buffer for the caller? > > > and using the pfrag pool, the improvement >> went from about 30Gbit to 40Gbit for the same testing configuation: > > Yes, but you have not provided performance number when 200 (or 1000+) > concurrent flows are running. As the iperf seems to only support 200 concurrent flows(running more threads seems to cause "Connection timed out"), any other performance tool supporting 1000+ concurrent flows? There is 32 cpus on the numa where the nic hw exists, and using taskset to run the iperf in the same numa, as the page pool support page frag for rx now, so the allocating the multi-order pages as skb_page_frag_refill() does won't waste memory any more. The below is the performance data for 200 concurrent iperf tcp flows for one tx queue: throughput node cpu usages pfrag_pool disabled: 31Gbit 5% pfrag_pool-page order 0: 43Gbit 8% pfrag_pool-page order 1: 50Gbit 8% pfrag_pool-page order 2: 70Gbit 10% pfrag_pool-page order 3: 90Gbit 11% The below is the performance data for 200 concurrent iperf tcp flows for 32 tx queues(94.1Gbit is tcp flow line speed for 100Gbit port with mtu 1500): throughput node cpu usages pfrag_pool disabled: 94.1Gbit 23%% pfrag_pool-page order 0: 93.9Gbit 31% pfrag_pool-page order 1: 94.1Gbit 24% pfrag_pool-page order 2: 94.1Gbit 23% pfrag_pool-page order 3: 94.1Gbit 16% So it seems page pool for tx seems promising for large number of sockets too? > > Optimizing singe flow TCP performance while killing performance for > the more common case is not an option. > > >> >> diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h >> index fcb5355..dda20f9 100644 >> --- a/include/linux/mmzone.h >> +++ b/include/linux/mmzone.h >> @@ -37,7 +37,7 @@ >> * coalesce naturally under reasonable reclaim pressure and those which >> * will not. >> */ >> -#define PAGE_ALLOC_COSTLY_ORDER 3 >> +#define PAGE_ALLOC_COSTLY_ORDER 4 >> >> enum migratetype { >> MIGRATE_UNMOVABLE, >> diff --git a/net/core/sock.c b/net/core/sock.c >> index 870a3b7..b1e0dfc 100644 >> --- a/net/core/sock.c >> +++ b/net/core/sock.c >> @@ -2580,7 +2580,7 @@ static void sk_leave_memory_pressure(struct sock *sk) >> } >> } >> >> -#define SKB_FRAG_PAGE_ORDER get_order(32768) >> +#define SKB_FRAG_PAGE_ORDER get_order(65536) >> DEFINE_STATIC_KEY_FALSE(net_high_order_alloc_disable_key); >> >> /** >> >>> >>>> >>>> diff --git a/net/core/sock.c b/net/core/sock.c >>>> index a3eea6e0b30a7d43793f567ffa526092c03e3546..6b66b51b61be9f198f6f1c4a3d81b57fa327986a >>>> 100644 >>>> --- a/net/core/sock.c >>>> +++ b/net/core/sock.c >>>> @@ -2560,7 +2560,7 @@ static void sk_leave_memory_pressure(struct sock *sk) >>>> } >>>> } >>>> >>>> -#define SKB_FRAG_PAGE_ORDER get_order(32768) >>>> +#define SKB_FRAG_PAGE_ORDER get_order(65536) >>>> DEFINE_STATIC_KEY_FALSE(net_high_order_alloc_disable_key); >>>> >>>> /** >>>> >>>> >>>> >>>>> >>> _______________________________________________ >>> Linuxarm mailing list -- linuxarm@openeuler.org >>> To unsubscribe send an email to linuxarm-leave@openeuler.org >>> > . >
On 2021/8/24 11:34, David Ahern wrote: > On 8/22/21 9:32 PM, Yunsheng Lin wrote: >> >> I assumed the "either Rx or Tx is cpu bound" meant either Rx or Tx is the >> bottleneck? > > yes. > >> >> It seems iperf3 support the Tx ZC, I retested using the iperf3, Rx settings >> is not changed when testing, MTU is 1500: > > -Z == sendfile API. That works fine to a point and that point is well > below 100G. > > I mean TCP with MSG_ZEROCOPY and SO_ZEROCOPY. > >> >> IOMMU in strict mode: >> 1. Tx ZC case: >> 22Gbit with Tx being bottleneck(cpu bound) >> 2. Tx non-ZC case with pfrag pool enabled: >> 40Git with Rx being bottleneck(cpu bound) >> 3. Tx non-ZC case with pfrag pool disabled: >> 30Git, the bottleneck seems not to be cpu bound, as the Rx and Tx does >> not have a single CPU reaching about 100% usage. >> >>> >>> At 1500 MTU lowering CPU usage on the Tx side does not accomplish much >>> on throughput since the Rx is 100% cpu. >> >> As above performance data, enabling ZC does not seems to help when IOMMU >> is involved, which has about 30% performance degrade when pfrag pool is >> disabled and 50% performance degrade when pfrag pool is enabled. > > In a past response you should numbers for Tx ZC API with a custom > program. That program showed the dramatic reduction in CPU cycles for Tx > with the ZC API. As I deduced the cpu usage from the cycles in "perf stat -e cycles XX", which does not seem to include the cycles for NAPI polling, which does the tx clean (including dma unmapping) and does not run in the same cpu as msg_zerocopy runs. I retested it using msg_zerocopy: msg_zerocopy cpu usage NAPI polling cpu usage ZC: 23% 70% non-ZC 50% 40% So it seems to match now, sorry for the confusion. > >> >>> >>> At 3300 MTU you have ~47% the pps for the same throughput. Lower pps >>> reduces Rx processing and lower CPU to process the incoming stream. Then >>> using the Tx ZC API you lower the Tx overehad allowing a single stream >>> to faster - sending more data which in the end results in much higher >>> pps and throughput. At the limit you are CPU bound (both ends in my >>> testing as Rx side approaches the max pps, and Tx side as it continually >>> tries to send data). >>> >>> Lowering CPU usage on Tx the side is a win regardless of whether there >>> is a big increase on the throughput at 1500 MTU since that configuration >>> is an Rx CPU bound problem. Hence, my point that we have a good start >>> point for lowering CPU usage on the Tx side; we should improve it rather >>> than add per-socket page pools. >> >> Acctually it is not a per-socket page pools, the page pool is still per >> NAPI, this patchset adds multi allocation context to the page pool, so that >> the tx can reuse the same page pool with rx, which is quite usefully if the >> ARFS is enabled. >> >>> >>> You can stress the Tx side and emphasize its overhead by modifying the >>> receiver to drop the data on Rx rather than copy to userspace which is a >>> huge bottleneck (e.g., MSG_TRUNC on recv). This allows the single flow >> >> As the frag page is supported in page pool for Rx, the Rx probably is not >> a bottleneck any more, at least not for IOMMU in strict mode. >> >> It seems iperf3 does not support MSG_TRUNC yet, any testing tool supporting >> MSG_TRUNC? Or do I have to hack the kernel or iperf3 tool to do that? > > https://github.com/dsahern/iperf, mods branch > > --zc_api is the Tx ZC API; --rx_drop adds MSG_TRUNC to recv. Thanks for sharing the tool. I retested using above iperf, and result is similar to previous result too. > > >> >>> stream to go faster and emphasize Tx bottlenecks as the pps at 3300 >>> approaches the top pps at 1500. e.g., doing this with iperf3 shows the >>> spinlock overhead with tcp_sendmsg, overhead related to 'select' and >>> then gup_pgd_range. >> >> When IOMMU is in strict mode, the overhead with IOMMU seems to be much >> bigger than spinlock(23% to 10%). >> >> Anyway, I still think ZC mostly benefit to packet which is bigger than a >> specific size and IOMMU disabling case. >> >> >>> . >>> > > . >
On 8/23/21 8:04 AM, Eric Dumazet wrote: >> >> >> It seems PAGE_ALLOC_COSTLY_ORDER is mostly related to pcp page, OOM, memory >> compact and memory isolation, as the test system has a lot of memory installed >> (about 500G, only 3-4G is used), so I used the below patch to test the max >> possible performance improvement when making TCP frags twice bigger, and >> the performance improvement went from about 30Gbit to 32Gbit for one thread >> iperf tcp flow in IOMMU strict mode, > > This is encouraging, and means we can do much better. > > Even with SKB_FRAG_PAGE_ORDER set to 4, typical skbs will need 3 mappings > > 1) One for the headers (in skb->head) > 2) Two page frags, because one TSO packet payload is not a nice power-of-two. interesting observation. I have noticed 17 with the ZC API. That might explain the less than expected performance bump with iommu strict mode. > > The first issue can be addressed using a piece of coherent memory (128 > or 256 bytes per entry in TX ring). > Copying the headers can avoid one IOMMU mapping, and improve IOTLB > hits, because all > slots of the TX ring buffer will use one single IOTLB slot. > > The second issue can be solved by tweaking a bit > skb_page_frag_refill() to accept an additional parameter > so that the whole skb payload fits in a single order-4 page. > >
On Wed, Aug 25, 2021 at 9:29 AM David Ahern <dsahern@gmail.com> wrote: > > On 8/23/21 8:04 AM, Eric Dumazet wrote: > >> > >> > >> It seems PAGE_ALLOC_COSTLY_ORDER is mostly related to pcp page, OOM, memory > >> compact and memory isolation, as the test system has a lot of memory installed > >> (about 500G, only 3-4G is used), so I used the below patch to test the max > >> possible performance improvement when making TCP frags twice bigger, and > >> the performance improvement went from about 30Gbit to 32Gbit for one thread > >> iperf tcp flow in IOMMU strict mode, > > > > This is encouraging, and means we can do much better. > > > > Even with SKB_FRAG_PAGE_ORDER set to 4, typical skbs will need 3 mappings > > > > 1) One for the headers (in skb->head) > > 2) Two page frags, because one TSO packet payload is not a nice power-of-two. > > interesting observation. I have noticed 17 with the ZC API. That might > explain the less than expected performance bump with iommu strict mode. Note that if application is using huge pages, things get better after commit 394fcd8a813456b3306c423ec4227ed874dfc08b Author: Eric Dumazet <edumazet@google.com> Date: Thu Aug 20 08:43:59 2020 -0700 net: zerocopy: combine pages in zerocopy_sg_from_iter() Currently, tcp sendmsg(MSG_ZEROCOPY) is building skbs with order-0 fragments. Compared to standard sendmsg(), these skbs usually contain up to 16 fragments on arches with 4KB page sizes, instead of two. This adds considerable costs on various ndo_start_xmit() handlers, especially when IOMMU is in the picture. As high performance applications are often using huge pages, we can try to combine adjacent pages belonging to same compound page. Tested on AMD Rome platform, with IOMMU, nominal single TCP flow speed is roughly doubled (~55Gbit -> ~100Gbit), when user application is using hugepages. For reference, nominal single TCP flow speed on this platform without MSG_ZEROCOPY is ~65Gbit. Signed-off-by: Eric Dumazet <edumazet@google.com> Cc: Willem de Bruijn <willemb@google.com> Signed-off-by: David S. Miller <davem@davemloft.net> Ideally the gup stuff should really directly deal with hugepages, so that we avoid all these crazy refcounting games on the per-huge-page central refcount. > > > > > The first issue can be addressed using a piece of coherent memory (128 > > or 256 bytes per entry in TX ring). > > Copying the headers can avoid one IOMMU mapping, and improve IOTLB > > hits, because all > > slots of the TX ring buffer will use one single IOTLB slot. > > > > The second issue can be solved by tweaking a bit > > skb_page_frag_refill() to accept an additional parameter > > so that the whole skb payload fits in a single order-4 page. > > > >
On 8/25/21 9:32 AM, Eric Dumazet wrote: > On Wed, Aug 25, 2021 at 9:29 AM David Ahern <dsahern@gmail.com> wrote: >> >> On 8/23/21 8:04 AM, Eric Dumazet wrote: >>>> >>>> >>>> It seems PAGE_ALLOC_COSTLY_ORDER is mostly related to pcp page, OOM, memory >>>> compact and memory isolation, as the test system has a lot of memory installed >>>> (about 500G, only 3-4G is used), so I used the below patch to test the max >>>> possible performance improvement when making TCP frags twice bigger, and >>>> the performance improvement went from about 30Gbit to 32Gbit for one thread >>>> iperf tcp flow in IOMMU strict mode, >>> >>> This is encouraging, and means we can do much better. >>> >>> Even with SKB_FRAG_PAGE_ORDER set to 4, typical skbs will need 3 mappings >>> >>> 1) One for the headers (in skb->head) >>> 2) Two page frags, because one TSO packet payload is not a nice power-of-two. >> >> interesting observation. I have noticed 17 with the ZC API. That might >> explain the less than expected performance bump with iommu strict mode. > > Note that if application is using huge pages, things get better after > > commit 394fcd8a813456b3306c423ec4227ed874dfc08b > Author: Eric Dumazet <edumazet@google.com> > Date: Thu Aug 20 08:43:59 2020 -0700 > > net: zerocopy: combine pages in zerocopy_sg_from_iter() > > Currently, tcp sendmsg(MSG_ZEROCOPY) is building skbs with order-0 > fragments. > Compared to standard sendmsg(), these skbs usually contain up to > 16 fragments > on arches with 4KB page sizes, instead of two. > > This adds considerable costs on various ndo_start_xmit() handlers, > especially when IOMMU is in the picture. > > As high performance applications are often using huge pages, > we can try to combine adjacent pages belonging to same > compound page. > > Tested on AMD Rome platform, with IOMMU, nominal single TCP flow speed > is roughly doubled (~55Gbit -> ~100Gbit), when user application > is using hugepages. > > For reference, nominal single TCP flow speed on this platform > without MSG_ZEROCOPY is ~65Gbit. > > Signed-off-by: Eric Dumazet <edumazet@google.com> > Cc: Willem de Bruijn <willemb@google.com> > Signed-off-by: David S. Miller <davem@davemloft.net> > > Ideally the gup stuff should really directly deal with hugepages, so > that we avoid > all these crazy refcounting games on the per-huge-page central refcount. > thanks for the pointer. I need to revisit my past attempt to get iperf3 working with hugepages.
On Wed, Aug 25, 2021 at 9:39 AM David Ahern <dsahern@gmail.com> wrote: > > On 8/25/21 9:32 AM, Eric Dumazet wrote: > > > > thanks for the pointer. I need to revisit my past attempt to get iperf3 > working with hugepages. ANother pointer, just in case this helps. commit 72653ae5303c626ca29fcbcbb8165a894a104adf Author: Eric Dumazet <edumazet@google.com> Date: Thu Aug 20 10:11:17 2020 -0700 selftests: net: tcp_mmap: Use huge pages in send path
On 8/25/21 10:24 AM, Eric Dumazet wrote: > On Wed, Aug 25, 2021 at 9:39 AM David Ahern <dsahern@gmail.com> wrote: >> >> On 8/25/21 9:32 AM, Eric Dumazet wrote: > >>> >> >> thanks for the pointer. I need to revisit my past attempt to get iperf3 >> working with hugepages. > > ANother pointer, just in case this helps. > > commit 72653ae5303c626ca29fcbcbb8165a894a104adf > Author: Eric Dumazet <edumazet@google.com> > Date: Thu Aug 20 10:11:17 2020 -0700 > > selftests: net: tcp_mmap: Use huge pages in send path > very helpful. Thanks. Added support to iperf3, and it does show a big drop in cpu utilization.