| Message ID | 20231107031227.100015-1-xuanzhuo@linux.alibaba.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | virtio-net: support AF_XDP zero copy | expand |
On Tue, 7 Nov 2023 11:12:06 +0800 Xuan Zhuo wrote:
> Please review.
## Form letter - net-next-closed
The merge window for v6.7 has begun and we have already posted our pull
request. Therefore net-next is closed for new drivers, features, code
refactoring and optimizations. We are currently accepting bug fixes only.
Please repost when net-next reopens after Nov 12th.
RFC patches sent for review only are obviously welcome at any time.
See: https://www.kernel.org/doc/html/next/process/maintainer-netdev.html#development-cycle
On Tue, 7 Nov 2023 10:01:48 -0800, Jakub Kicinski <kuba@kernel.org> wrote: > On Tue, 7 Nov 2023 11:12:06 +0800 Xuan Zhuo wrote: > > Please review. > > ## Form letter - net-next-closed > > The merge window for v6.7 has begun and we have already posted our pull > request. Therefore net-next is closed for new drivers, features, code > refactoring and optimizations. We are currently accepting bug fixes only. > > Please repost when net-next reopens after Nov 12th. > > RFC patches sent for review only are obviously welcome at any time. Yes. This is for review. Thanks. > > See: https://www.kernel.org/doc/html/next/process/maintainer-netdev.html#development-cycle > -- > pw-bot: defer
On Tue, Nov 07, 2023 at 11:12:06AM +0800, Xuan Zhuo wrote: > ## AF_XDP > > XDP socket(AF_XDP) is an excellent bypass kernel network framework. The zero > copy feature of xsk (XDP socket) needs to be supported by the driver. The > performance of zero copy is very good. mlx5 and intel ixgbe already support > this feature, This patch set allows virtio-net to support xsk's zerocopy xmit > feature. > > At present, we have completed some preparation: > > 1. vq-reset (virtio spec and kernel code) > 2. virtio-core premapped dma > 3. virtio-net xdp refactor > > So it is time for Virtio-Net to complete the support for the XDP Socket > Zerocopy. > > Virtio-net can not increase the queue num at will, so xsk shares the queue with > kernel. > > On the other hand, Virtio-Net does not support generate interrupt from driver > manually, so when we wakeup tx xmit, we used some tips. If the CPU run by TX > NAPI last time is other CPUs, use IPI to wake up NAPI on the remote CPU. If it > is also the local CPU, then we wake up napi directly. > > This patch set includes some refactor to the virtio-net to let that to support > AF_XDP. > > ## performance > > ENV: Qemu with vhost-user(polling mode). > Host CPU: Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz > > ### virtio PMD in guest with testpmd > > testpmd> show port stats all > > ######################## NIC statistics for port 0 ######################## > RX-packets: 19531092064 RX-missed: 0 RX-bytes: 1093741155584 > RX-errors: 0 > RX-nombuf: 0 > TX-packets: 5959955552 TX-errors: 0 TX-bytes: 371030645664 > > > Throughput (since last show) > Rx-pps: 8861574 Rx-bps: 3969985208 > Tx-pps: 8861493 Tx-bps: 3969962736 > ############################################################################ > > ### AF_XDP PMD in guest with testpmd > > testpmd> show port stats all > > ######################## NIC statistics for port 0 ######################## > RX-packets: 68152727 RX-missed: 0 RX-bytes: 3816552712 > RX-errors: 0 > RX-nombuf: 0 > TX-packets: 68114967 TX-errors: 33216 TX-bytes: 3814438152 > > Throughput (since last show) > Rx-pps: 6333196 Rx-bps: 2837272088 > Tx-pps: 6333227 Tx-bps: 2837285936 > ############################################################################ > > But AF_XDP consumes more CPU for tx and rx napi(100% and 86%). > > ## maintain > > I am currently a reviewer for virtio-net. I commit to maintain AF_XDP support in > virtio-net. > > Please review. > > Thanks. > > v2 > 1. wakeup uses the way of GVE. No send ipi to wakeup napi on remote cpu. > 2. remove rcu. Because we synchronize all operat, so the rcu is not needed. > 3. split the commit "move to virtio_net.h" in last patch set. Just move the > struct/api to header when we use them. > 4. add comments for some code > > v1: > 1. remove two virtio commits. Push this patchset to net-next > 2. squash "virtio_net: virtnet_poll_tx support rescheduled" to xsk: support tx > 3. fix some warnings > > > Xuan Zhuo (21): > virtio_net: rename free_old_xmit_skbs to free_old_xmit > virtio_net: unify the code for recycling the xmit ptr > virtio_net: independent directory > virtio_net: move core structures to virtio_net.h > virtio_net: add prefix virtnet to all struct inside virtio_net.h > virtio_net: separate virtnet_rx_resize() > virtio_net: separate virtnet_tx_resize() > virtio_net: sq support premapped mode > virtio_net: xsk: bind/unbind xsk > virtio_net: xsk: prevent disable tx napi > virtio_net: move some api to header > virtio_net: xsk: tx: support tx > virtio_net: xsk: tx: support wakeup > virtio_net: xsk: tx: virtnet_free_old_xmit() distinguishes xsk buffer > virtio_net: xsk: tx: virtnet_sq_free_unused_buf() check xsk buffer > virtio_net: xsk: rx: introduce add_recvbuf_xsk() > virtio_net: xsk: rx: skip dma unmap when rq is bind with AF_XDP > virtio_net: xsk: rx: introduce receive_xsk() to recv xsk buffer > virtio_net: xsk: rx: virtnet_rq_free_unused_buf() check xsk buffer > virtio_net: update tx timeout record > virtio_net: xdp_features add NETDEV_XDP_ACT_XSK_ZEROCOPY > > MAINTAINERS | 2 +- > drivers/net/Kconfig | 8 +- > drivers/net/Makefile | 2 +- > drivers/net/virtio/Kconfig | 13 + > drivers/net/virtio/Makefile | 8 + > drivers/net/{virtio_net.c => virtio/main.c} | 630 +++++++++----------- > drivers/net/virtio/virtio_net.h | 346 +++++++++++ > drivers/net/virtio/xsk.c | 498 ++++++++++++++++ > drivers/net/virtio/xsk.h | 32 + > 9 files changed, 1174 insertions(+), 365 deletions(-) > create mode 100644 drivers/net/virtio/Kconfig > create mode 100644 drivers/net/virtio/Makefile > rename drivers/net/{virtio_net.c => virtio/main.c} (92%) > create mode 100644 drivers/net/virtio/virtio_net.h > create mode 100644 drivers/net/virtio/xsk.c > create mode 100644 drivers/net/virtio/xsk.h Too much code duplications for my taste. Would there maybe be less if everything was in a single file? > -- > 2.32.0.3.g01195cf9f
On Thu, 9 Nov 2023 03:19:04 -0500, "Michael S. Tsirkin" <mst@redhat.com> wrote: > On Tue, Nov 07, 2023 at 11:12:06AM +0800, Xuan Zhuo wrote: > > ## AF_XDP > > > > XDP socket(AF_XDP) is an excellent bypass kernel network framework. The zero > > copy feature of xsk (XDP socket) needs to be supported by the driver. The > > performance of zero copy is very good. mlx5 and intel ixgbe already support > > this feature, This patch set allows virtio-net to support xsk's zerocopy xmit > > feature. > > > > At present, we have completed some preparation: > > > > 1. vq-reset (virtio spec and kernel code) > > 2. virtio-core premapped dma > > 3. virtio-net xdp refactor > > > > So it is time for Virtio-Net to complete the support for the XDP Socket > > Zerocopy. > > > > Virtio-net can not increase the queue num at will, so xsk shares the queue with > > kernel. > > > > On the other hand, Virtio-Net does not support generate interrupt from driver > > manually, so when we wakeup tx xmit, we used some tips. If the CPU run by TX > > NAPI last time is other CPUs, use IPI to wake up NAPI on the remote CPU. If it > > is also the local CPU, then we wake up napi directly. > > > > This patch set includes some refactor to the virtio-net to let that to support > > AF_XDP. > > > > ## performance > > > > ENV: Qemu with vhost-user(polling mode). > > Host CPU: Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz > > > > ### virtio PMD in guest with testpmd > > > > testpmd> show port stats all > > > > ######################## NIC statistics for port 0 ######################## > > RX-packets: 19531092064 RX-missed: 0 RX-bytes: 1093741155584 > > RX-errors: 0 > > RX-nombuf: 0 > > TX-packets: 5959955552 TX-errors: 0 TX-bytes: 371030645664 > > > > > > Throughput (since last show) > > Rx-pps: 8861574 Rx-bps: 3969985208 > > Tx-pps: 8861493 Tx-bps: 3969962736 > > ############################################################################ > > > > ### AF_XDP PMD in guest with testpmd > > > > testpmd> show port stats all > > > > ######################## NIC statistics for port 0 ######################## > > RX-packets: 68152727 RX-missed: 0 RX-bytes: 3816552712 > > RX-errors: 0 > > RX-nombuf: 0 > > TX-packets: 68114967 TX-errors: 33216 TX-bytes: 3814438152 > > > > Throughput (since last show) > > Rx-pps: 6333196 Rx-bps: 2837272088 > > Tx-pps: 6333227 Tx-bps: 2837285936 > > ############################################################################ > > > > But AF_XDP consumes more CPU for tx and rx napi(100% and 86%). > > > > ## maintain > > > > I am currently a reviewer for virtio-net. I commit to maintain AF_XDP support in > > virtio-net. > > > > Please review. > > > > Thanks. > > > > v2 > > 1. wakeup uses the way of GVE. No send ipi to wakeup napi on remote cpu. > > 2. remove rcu. Because we synchronize all operat, so the rcu is not needed. > > 3. split the commit "move to virtio_net.h" in last patch set. Just move the > > struct/api to header when we use them. > > 4. add comments for some code > > > > v1: > > 1. remove two virtio commits. Push this patchset to net-next > > 2. squash "virtio_net: virtnet_poll_tx support rescheduled" to xsk: support tx > > 3. fix some warnings > > > > > > Xuan Zhuo (21): > > virtio_net: rename free_old_xmit_skbs to free_old_xmit > > virtio_net: unify the code for recycling the xmit ptr > > virtio_net: independent directory > > virtio_net: move core structures to virtio_net.h > > virtio_net: add prefix virtnet to all struct inside virtio_net.h > > virtio_net: separate virtnet_rx_resize() > > virtio_net: separate virtnet_tx_resize() > > virtio_net: sq support premapped mode > > virtio_net: xsk: bind/unbind xsk > > virtio_net: xsk: prevent disable tx napi > > virtio_net: move some api to header > > virtio_net: xsk: tx: support tx > > virtio_net: xsk: tx: support wakeup > > virtio_net: xsk: tx: virtnet_free_old_xmit() distinguishes xsk buffer > > virtio_net: xsk: tx: virtnet_sq_free_unused_buf() check xsk buffer > > virtio_net: xsk: rx: introduce add_recvbuf_xsk() > > virtio_net: xsk: rx: skip dma unmap when rq is bind with AF_XDP > > virtio_net: xsk: rx: introduce receive_xsk() to recv xsk buffer > > virtio_net: xsk: rx: virtnet_rq_free_unused_buf() check xsk buffer > > virtio_net: update tx timeout record > > virtio_net: xdp_features add NETDEV_XDP_ACT_XSK_ZEROCOPY > > > > MAINTAINERS | 2 +- > > drivers/net/Kconfig | 8 +- > > drivers/net/Makefile | 2 +- > > drivers/net/virtio/Kconfig | 13 + > > drivers/net/virtio/Makefile | 8 + > > drivers/net/{virtio_net.c => virtio/main.c} | 630 +++++++++----------- > > drivers/net/virtio/virtio_net.h | 346 +++++++++++ > > drivers/net/virtio/xsk.c | 498 ++++++++++++++++ > > drivers/net/virtio/xsk.h | 32 + > > 9 files changed, 1174 insertions(+), 365 deletions(-) > > create mode 100644 drivers/net/virtio/Kconfig > > create mode 100644 drivers/net/virtio/Makefile > > rename drivers/net/{virtio_net.c => virtio/main.c} (92%) > > create mode 100644 drivers/net/virtio/virtio_net.h > > create mode 100644 drivers/net/virtio/xsk.c > > create mode 100644 drivers/net/virtio/xsk.h > > > Too much code duplications for my taste. Would there maybe be less if > everything was in a single file? What duplication? Yes. If everything is in a single file, something maybe simpler. But I do not like that, that will be more trouble in the future. We want to make the virtnet more like the modern network card, then we will introduce more features to the virtio-net. I think it's time to split the single file to modules. I think letting every module simple is beneficial for the maintaining. About the duplications, I do not understand. No duplication, We just refer the functions from each other. Do you mean the receive_xsk()? Let we resolve this problem if you think it's a duplicate. I think this is the right way. Thanks. > > > > -- > > 2.32.0.3.g01195cf9f > >
## AF_XDP XDP socket(AF_XDP) is an excellent bypass kernel network framework. The zero copy feature of xsk (XDP socket) needs to be supported by the driver. The performance of zero copy is very good. mlx5 and intel ixgbe already support this feature, This patch set allows virtio-net to support xsk's zerocopy xmit feature. At present, we have completed some preparation: 1. vq-reset (virtio spec and kernel code) 2. virtio-core premapped dma 3. virtio-net xdp refactor So it is time for Virtio-Net to complete the support for the XDP Socket Zerocopy. Virtio-net can not increase the queue num at will, so xsk shares the queue with kernel. On the other hand, Virtio-Net does not support generate interrupt from driver manually, so when we wakeup tx xmit, we used some tips. If the CPU run by TX NAPI last time is other CPUs, use IPI to wake up NAPI on the remote CPU. If it is also the local CPU, then we wake up napi directly. This patch set includes some refactor to the virtio-net to let that to support AF_XDP. ## performance ENV: Qemu with vhost-user(polling mode). Host CPU: Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz ### virtio PMD in guest with testpmd testpmd> show port stats all ######################## NIC statistics for port 0 ######################## RX-packets: 19531092064 RX-missed: 0 RX-bytes: 1093741155584 RX-errors: 0 RX-nombuf: 0 TX-packets: 5959955552 TX-errors: 0 TX-bytes: 371030645664 Throughput (since last show) Rx-pps: 8861574 Rx-bps: 3969985208 Tx-pps: 8861493 Tx-bps: 3969962736 ############################################################################ ### AF_XDP PMD in guest with testpmd testpmd> show port stats all ######################## NIC statistics for port 0 ######################## RX-packets: 68152727 RX-missed: 0 RX-bytes: 3816552712 RX-errors: 0 RX-nombuf: 0 TX-packets: 68114967 TX-errors: 33216 TX-bytes: 3814438152 Throughput (since last show) Rx-pps: 6333196 Rx-bps: 2837272088 Tx-pps: 6333227 Tx-bps: 2837285936 ############################################################################ But AF_XDP consumes more CPU for tx and rx napi(100% and 86%). ## maintain I am currently a reviewer for virtio-net. I commit to maintain AF_XDP support in virtio-net. Please review. Thanks. v2 1. wakeup uses the way of GVE. No send ipi to wakeup napi on remote cpu. 2. remove rcu. Because we synchronize all operat, so the rcu is not needed. 3. split the commit "move to virtio_net.h" in last patch set. Just move the struct/api to header when we use them. 4. add comments for some code v1: 1. remove two virtio commits. Push this patchset to net-next 2. squash "virtio_net: virtnet_poll_tx support rescheduled" to xsk: support tx 3. fix some warnings Xuan Zhuo (21): virtio_net: rename free_old_xmit_skbs to free_old_xmit virtio_net: unify the code for recycling the xmit ptr virtio_net: independent directory virtio_net: move core structures to virtio_net.h virtio_net: add prefix virtnet to all struct inside virtio_net.h virtio_net: separate virtnet_rx_resize() virtio_net: separate virtnet_tx_resize() virtio_net: sq support premapped mode virtio_net: xsk: bind/unbind xsk virtio_net: xsk: prevent disable tx napi virtio_net: move some api to header virtio_net: xsk: tx: support tx virtio_net: xsk: tx: support wakeup virtio_net: xsk: tx: virtnet_free_old_xmit() distinguishes xsk buffer virtio_net: xsk: tx: virtnet_sq_free_unused_buf() check xsk buffer virtio_net: xsk: rx: introduce add_recvbuf_xsk() virtio_net: xsk: rx: skip dma unmap when rq is bind with AF_XDP virtio_net: xsk: rx: introduce receive_xsk() to recv xsk buffer virtio_net: xsk: rx: virtnet_rq_free_unused_buf() check xsk buffer virtio_net: update tx timeout record virtio_net: xdp_features add NETDEV_XDP_ACT_XSK_ZEROCOPY MAINTAINERS | 2 +- drivers/net/Kconfig | 8 +- drivers/net/Makefile | 2 +- drivers/net/virtio/Kconfig | 13 + drivers/net/virtio/Makefile | 8 + drivers/net/{virtio_net.c => virtio/main.c} | 630 +++++++++----------- drivers/net/virtio/virtio_net.h | 346 +++++++++++ drivers/net/virtio/xsk.c | 498 ++++++++++++++++ drivers/net/virtio/xsk.h | 32 + 9 files changed, 1174 insertions(+), 365 deletions(-) create mode 100644 drivers/net/virtio/Kconfig create mode 100644 drivers/net/virtio/Makefile rename drivers/net/{virtio_net.c => virtio/main.c} (92%) create mode 100644 drivers/net/virtio/virtio_net.h create mode 100644 drivers/net/virtio/xsk.c create mode 100644 drivers/net/virtio/xsk.h -- 2.32.0.3.g01195cf9f