From patchwork Tue Jan 16 09:42:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13520617 X-Patchwork-Delegate: kuba@kernel.org Received: from out30-112.freemail.mail.aliyun.com (out30-112.freemail.mail.aliyun.com [115.124.30.112]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4B5CA12B8C; Tue, 16 Jan 2024 09:43:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R131e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046060;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=14;SR=0;TI=SMTPD_---0W-lmy0t_1705398194; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W-lmy0t_1705398194) by smtp.aliyun-inc.com; Tue, 16 Jan 2024 17:43:15 +0800 From: Xuan Zhuo To: netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , virtualization@lists.linux.dev, bpf@vger.kernel.org Subject: [PATCH net-next 01/17] virtio_net: separate virtnet_rx_resize() Date: Tue, 16 Jan 2024 17:42:57 +0800 Message-Id: <20240116094313.119939-2-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240116094313.119939-1-xuanzhuo@linux.alibaba.com> References: <20240116094313.119939-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: 1913ebd4ae28 X-Patchwork-Delegate: kuba@kernel.org This patch separates two sub-functions from virtnet_rx_resize(): * virtnet_rx_pause * virtnet_rx_resume Then the subsequent reset rx for xsk can share these two functions. Signed-off-by: Xuan Zhuo Acked-by: Jason Wang --- drivers/net/virtio/main.c | 29 +++++++++++++++++++++-------- drivers/net/virtio/virtio_net.h | 3 +++ 2 files changed, 24 insertions(+), 8 deletions(-) diff --git a/drivers/net/virtio/main.c b/drivers/net/virtio/main.c index 53143f95a3a0..52686beca4ab 100644 --- a/drivers/net/virtio/main.c +++ b/drivers/net/virtio/main.c @@ -2293,28 +2293,41 @@ static netdev_tx_t start_xmit(struct sk_buff *skb, struct net_device *dev) return NETDEV_TX_OK; } -static int virtnet_rx_resize(struct virtnet_info *vi, - struct virtnet_rq *rq, u32 ring_num) +void virtnet_rx_pause(struct virtnet_info *vi, struct virtnet_rq *rq) { bool running = netif_running(vi->dev); - int err, qindex; - - qindex = rq - vi->rq; if (running) { napi_disable(&rq->napi); cancel_work_sync(&rq->dim.work); } +} - err = virtqueue_resize(rq->vq, ring_num, virtnet_rq_free_unused_bufs); - if (err) - netdev_err(vi->dev, "resize rx fail: rx queue index: %d err: %d\n", qindex, err); +void virtnet_rx_resume(struct virtnet_info *vi, struct virtnet_rq *rq) +{ + bool running = netif_running(vi->dev); if (!try_fill_recv(vi, rq, GFP_KERNEL)) schedule_delayed_work(&vi->refill, 0); if (running) virtnet_napi_enable(rq->vq, &rq->napi); +} + +static int virtnet_rx_resize(struct virtnet_info *vi, + struct virtnet_rq *rq, u32 ring_num) +{ + int err, qindex; + + qindex = rq - vi->rq; + + virtnet_rx_pause(vi, rq); + + err = virtqueue_resize(rq->vq, ring_num, virtnet_rq_free_unused_bufs); + if (err) + netdev_err(vi->dev, "resize rx fail: rx queue index: %d err: %d\n", qindex, err); + + virtnet_rx_resume(vi, rq); return err; } diff --git a/drivers/net/virtio/virtio_net.h b/drivers/net/virtio/virtio_net.h index dda144cc91c7..7d06e8fda83b 100644 --- a/drivers/net/virtio/virtio_net.h +++ b/drivers/net/virtio/virtio_net.h @@ -209,4 +209,7 @@ struct virtnet_info { /* failover when STANDBY feature enabled */ struct failover *failover; }; + +void virtnet_rx_pause(struct virtnet_info *vi, struct virtnet_rq *rq); +void virtnet_rx_resume(struct virtnet_info *vi, struct virtnet_rq *rq); #endif From patchwork Tue Jan 16 09:42:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13520618 X-Patchwork-Delegate: kuba@kernel.org Received: from out30-99.freemail.mail.aliyun.com (out30-99.freemail.mail.aliyun.com [115.124.30.99]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 61DE012B9B; Tue, 16 Jan 2024 09:43:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R181e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045176;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=14;SR=0;TI=SMTPD_---0W-lmy1m_1705398195; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W-lmy1m_1705398195) by smtp.aliyun-inc.com; Tue, 16 Jan 2024 17:43:16 +0800 From: Xuan Zhuo To: netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , virtualization@lists.linux.dev, bpf@vger.kernel.org Subject: [PATCH net-next 02/17] virtio_net: separate virtnet_tx_resize() Date: Tue, 16 Jan 2024 17:42:58 +0800 Message-Id: <20240116094313.119939-3-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240116094313.119939-1-xuanzhuo@linux.alibaba.com> References: <20240116094313.119939-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: 1913ebd4ae28 X-Patchwork-Delegate: kuba@kernel.org This patch separates two sub-functions from virtnet_tx_resize(): * virtnet_tx_pause * virtnet_tx_resume Then the subsequent virtnet_tx_reset() can share these two functions. Signed-off-by: Xuan Zhuo Acked-by: Jason Wang --- drivers/net/virtio/main.c | 35 +++++++++++++++++++++++++++------ drivers/net/virtio/virtio_net.h | 2 ++ 2 files changed, 31 insertions(+), 6 deletions(-) diff --git a/drivers/net/virtio/main.c b/drivers/net/virtio/main.c index 52686beca4ab..49c2c1c985f3 100644 --- a/drivers/net/virtio/main.c +++ b/drivers/net/virtio/main.c @@ -2331,12 +2331,11 @@ static int virtnet_rx_resize(struct virtnet_info *vi, return err; } -static int virtnet_tx_resize(struct virtnet_info *vi, - struct virtnet_sq *sq, u32 ring_num) +void virtnet_tx_pause(struct virtnet_info *vi, struct virtnet_sq *sq) { bool running = netif_running(vi->dev); struct netdev_queue *txq; - int err, qindex; + int qindex; qindex = sq - vi->sq; @@ -2357,10 +2356,17 @@ static int virtnet_tx_resize(struct virtnet_info *vi, netif_stop_subqueue(vi->dev, qindex); __netif_tx_unlock_bh(txq); +} - err = virtqueue_resize(sq->vq, ring_num, virtnet_sq_free_unused_bufs); - if (err) - netdev_err(vi->dev, "resize tx fail: tx queue index: %d err: %d\n", qindex, err); +void virtnet_tx_resume(struct virtnet_info *vi, struct virtnet_sq *sq) +{ + bool running = netif_running(vi->dev); + struct netdev_queue *txq; + int qindex; + + qindex = sq - vi->sq; + + txq = netdev_get_tx_queue(vi->dev, qindex); __netif_tx_lock_bh(txq); sq->reset = false; @@ -2369,6 +2375,23 @@ static int virtnet_tx_resize(struct virtnet_info *vi, if (running) virtnet_napi_tx_enable(vi, sq->vq, &sq->napi); +} + +static int virtnet_tx_resize(struct virtnet_info *vi, struct virtnet_sq *sq, + u32 ring_num) +{ + int qindex, err; + + qindex = sq - vi->sq; + + virtnet_tx_pause(vi, sq); + + err = virtqueue_resize(sq->vq, ring_num, virtnet_sq_free_unused_bufs); + if (err) + netdev_err(vi->dev, "resize tx fail: tx queue index: %d err: %d\n", qindex, err); + + virtnet_tx_resume(vi, sq); + return err; } diff --git a/drivers/net/virtio/virtio_net.h b/drivers/net/virtio/virtio_net.h index 7d06e8fda83b..db86338e0dd0 100644 --- a/drivers/net/virtio/virtio_net.h +++ b/drivers/net/virtio/virtio_net.h @@ -212,4 +212,6 @@ struct virtnet_info { void virtnet_rx_pause(struct virtnet_info *vi, struct virtnet_rq *rq); void virtnet_rx_resume(struct virtnet_info *vi, struct virtnet_rq *rq); +void virtnet_tx_pause(struct virtnet_info *vi, struct virtnet_sq *sq); +void virtnet_tx_resume(struct virtnet_info *vi, struct virtnet_sq *sq); #endif From patchwork Tue Jan 16 09:42:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13520619 X-Patchwork-Delegate: kuba@kernel.org Received: from out30-130.freemail.mail.aliyun.com (out30-130.freemail.mail.aliyun.com [115.124.30.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B2C8512E7F; Tue, 16 Jan 2024 09:43:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R171e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045170;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=14;SR=0;TI=SMTPD_---0W-lk6Sm_1705398196; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W-lk6Sm_1705398196) by smtp.aliyun-inc.com; Tue, 16 Jan 2024 17:43:17 +0800 From: Xuan Zhuo To: netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , virtualization@lists.linux.dev, bpf@vger.kernel.org Subject: [PATCH net-next 03/17] virtio_net: xsk: bind/unbind xsk Date: Tue, 16 Jan 2024 17:42:59 +0800 Message-Id: <20240116094313.119939-4-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240116094313.119939-1-xuanzhuo@linux.alibaba.com> References: <20240116094313.119939-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: 1913ebd4ae28 X-Patchwork-Delegate: kuba@kernel.org This patch implement the logic of bind/unbind xsk pool to sq and rq. Signed-off-by: Xuan Zhuo --- drivers/net/virtio/Makefile | 2 +- drivers/net/virtio/main.c | 12 +- drivers/net/virtio/virtio_net.h | 18 +++ drivers/net/virtio/xsk.c | 188 ++++++++++++++++++++++++++++++++ drivers/net/virtio/xsk.h | 7 ++ 5 files changed, 219 insertions(+), 8 deletions(-) create mode 100644 drivers/net/virtio/xsk.c create mode 100644 drivers/net/virtio/xsk.h diff --git a/drivers/net/virtio/Makefile b/drivers/net/virtio/Makefile index 15ed7c97fd4f..8c2a884d2dba 100644 --- a/drivers/net/virtio/Makefile +++ b/drivers/net/virtio/Makefile @@ -5,4 +5,4 @@ obj-$(CONFIG_VIRTIO_NET) += virtio_net.o -virtio_net-y := main.o +virtio_net-y := main.o xsk.o diff --git a/drivers/net/virtio/main.c b/drivers/net/virtio/main.c index 49c2c1c985f3..3856006ac1c7 100644 --- a/drivers/net/virtio/main.c +++ b/drivers/net/virtio/main.c @@ -8,7 +8,6 @@ #include #include #include -#include #include #include #include @@ -17,13 +16,13 @@ #include #include #include -#include #include #include #include #include #include "virtio_net.h" +#include "xsk.h" static int napi_weight = NAPI_POLL_WEIGHT; module_param(napi_weight, int, 0444); @@ -150,9 +149,6 @@ struct virtio_net_common_hdr { }; }; -static void virtnet_rq_free_unused_bufs(struct virtqueue *vq); -static void virtnet_sq_free_unused_bufs(struct virtqueue *vq); - static bool is_xdp_frame(void *ptr) { return (unsigned long)ptr & VIRTIO_XDP_FLAG; @@ -3941,6 +3937,8 @@ static int virtnet_xdp(struct net_device *dev, struct netdev_bpf *xdp) switch (xdp->command) { case XDP_SETUP_PROG: return virtnet_xdp_set(dev, xdp->prog, xdp->extack); + case XDP_SETUP_XSK_POOL: + return virtnet_xsk_pool_setup(dev, xdp); default: return -EINVAL; } @@ -4124,7 +4122,7 @@ static void free_receive_page_frags(struct virtnet_info *vi) } } -static void virtnet_sq_free_unused_bufs(struct virtqueue *vq) +void virtnet_sq_free_unused_bufs(struct virtqueue *vq) { struct virtnet_info *vi = vq->vdev->priv; struct virtio_dma_head *dma; @@ -4152,7 +4150,7 @@ static void virtnet_sq_free_unused_bufs(struct virtqueue *vq) } } -static void virtnet_rq_free_unused_bufs(struct virtqueue *vq) +void virtnet_rq_free_unused_bufs(struct virtqueue *vq) { struct virtnet_info *vi = vq->vdev->priv; struct virtnet_rq *rq; diff --git a/drivers/net/virtio/virtio_net.h b/drivers/net/virtio/virtio_net.h index db86338e0dd0..02dc7650a65c 100644 --- a/drivers/net/virtio/virtio_net.h +++ b/drivers/net/virtio/virtio_net.h @@ -5,6 +5,9 @@ #include #include +#include +#include +#include /* RX packet size EWMA. The average packet size is used to determine the packet * buffer size when refilling RX rings. As the entire RX ring may be refilled @@ -75,6 +78,12 @@ struct virtnet_sq { /* Record whether sq is in reset state. */ bool reset; + + struct { + struct xsk_buff_pool *pool; + + dma_addr_t hdr_dma_address; + } xsk; }; /* Internal representation of a receive virtqueue */ @@ -123,6 +132,13 @@ struct virtnet_rq { /* Record the last dma info to free after new pages is allocated. */ struct virtnet_rq_dma *last_dma; + + struct { + struct xsk_buff_pool *pool; + + /* xdp rxq used by xsk */ + struct xdp_rxq_info xdp_rxq; + } xsk; }; struct virtnet_info { @@ -214,4 +230,6 @@ void virtnet_rx_pause(struct virtnet_info *vi, struct virtnet_rq *rq); void virtnet_rx_resume(struct virtnet_info *vi, struct virtnet_rq *rq); void virtnet_tx_pause(struct virtnet_info *vi, struct virtnet_sq *sq); void virtnet_tx_resume(struct virtnet_info *vi, struct virtnet_sq *sq); +void virtnet_sq_free_unused_bufs(struct virtqueue *vq); +void virtnet_rq_free_unused_bufs(struct virtqueue *vq); #endif diff --git a/drivers/net/virtio/xsk.c b/drivers/net/virtio/xsk.c new file mode 100644 index 000000000000..613a6c63d47c --- /dev/null +++ b/drivers/net/virtio/xsk.c @@ -0,0 +1,188 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* + * virtio-net xsk + */ + +#include "virtio_net.h" +#include "xsk.h" + +static struct virtio_net_hdr_mrg_rxbuf xsk_hdr; + +static int virtnet_rq_bind_xsk_pool(struct virtnet_info *vi, struct virtnet_rq *rq, + struct xsk_buff_pool *pool) +{ + int err, qindex; + + qindex = rq - vi->rq; + + if (pool) { + err = xdp_rxq_info_reg(&rq->xsk.xdp_rxq, vi->dev, qindex, rq->napi.napi_id); + if (err < 0) + return err; + + err = xdp_rxq_info_reg_mem_model(&rq->xsk.xdp_rxq, + MEM_TYPE_XSK_BUFF_POOL, NULL); + if (err < 0) { + xdp_rxq_info_unreg(&rq->xsk.xdp_rxq); + return err; + } + + xsk_pool_set_rxq_info(pool, &rq->xsk.xdp_rxq); + } + + virtnet_rx_pause(vi, rq); + + err = virtqueue_reset(rq->vq, virtnet_rq_free_unused_bufs); + if (err) { + netdev_err(vi->dev, "reset rx fail: rx queue index: %d err: %d\n", qindex, err); + + pool = NULL; + } + + if (!pool) + xdp_rxq_info_unreg(&rq->xsk.xdp_rxq); + + rq->xsk.pool = pool; + + virtnet_rx_resume(vi, rq); + + return err; +} + +static int virtnet_sq_bind_xsk_pool(struct virtnet_info *vi, + struct virtnet_sq *sq, + struct xsk_buff_pool *pool) +{ + int err, qindex; + + qindex = sq - vi->sq; + + virtnet_tx_pause(vi, sq); + + err = virtqueue_reset(sq->vq, virtnet_sq_free_unused_bufs); + if (err) { + pool = NULL; + netdev_err(vi->dev, "reset tx fail: tx queue index: %d err: %d\n", qindex, err); + } + + sq->xsk.pool = pool; + + virtnet_tx_resume(vi, sq); + + return err; +} + +static int virtnet_xsk_pool_enable(struct net_device *dev, + struct xsk_buff_pool *pool, + u16 qid) +{ + struct virtnet_info *vi = netdev_priv(dev); + struct virtnet_rq *rq; + struct virtnet_sq *sq; + struct device *dma_dev; + dma_addr_t hdr_dma; + int err; + + /* In big_packets mode, xdp cannot work, so there is no need to + * initialize xsk of rq. + */ + if (vi->big_packets && !vi->mergeable_rx_bufs) + return -ENOENT; + + if (qid >= vi->curr_queue_pairs) + return -EINVAL; + + sq = &vi->sq[qid]; + rq = &vi->rq[qid]; + + /* xsk tx zerocopy depend on the tx napi. + * + * All xsk packets are actually consumed and sent out from the xsk tx + * queue under the tx napi mechanism. + */ + if (!sq->napi.weight) + return -EPERM; + + if (!virtqueue_get_dma_premapped(rq->vq) || !virtqueue_get_dma_premapped(sq->vq)) + return -EPERM; + + /* For the xsk, the tx and rx should have the same device. But + * vq->dma_dev allows every vq has the respective dma dev. So I check + * the dma dev of vq and sq is the same dev. + */ + if (virtqueue_dma_dev(rq->vq) != virtqueue_dma_dev(sq->vq)) + return -EPERM; + + dma_dev = virtqueue_dma_dev(rq->vq); + if (!dma_dev) + return -EPERM; + + hdr_dma = dma_map_single(dma_dev, &xsk_hdr, vi->hdr_len, DMA_TO_DEVICE); + if (dma_mapping_error(dma_dev, hdr_dma)) + return -ENOMEM; + + err = xsk_pool_dma_map(pool, dma_dev, 0); + if (err) + goto err_xsk_map; + + err = virtnet_rq_bind_xsk_pool(vi, rq, pool); + if (err) + goto err_rq; + + err = virtnet_sq_bind_xsk_pool(vi, sq, pool); + if (err) + goto err_sq; + + /* Now, we do not support tx offset, so all the tx virtnet hdr is zero. + * So all the tx packets can share a single hdr. + */ + sq->xsk.hdr_dma_address = hdr_dma; + + return 0; + +err_sq: + virtnet_rq_bind_xsk_pool(vi, rq, NULL); +err_rq: + xsk_pool_dma_unmap(pool, 0); +err_xsk_map: + dma_unmap_single(dma_dev, hdr_dma, vi->hdr_len, DMA_TO_DEVICE); + return err; +} + +static int virtnet_xsk_pool_disable(struct net_device *dev, u16 qid) +{ + struct virtnet_info *vi = netdev_priv(dev); + struct xsk_buff_pool *pool; + struct device *dma_dev; + struct virtnet_rq *rq; + struct virtnet_sq *sq; + int err1, err2; + + if (qid >= vi->curr_queue_pairs) + return -EINVAL; + + sq = &vi->sq[qid]; + rq = &vi->rq[qid]; + + pool = sq->xsk.pool; + + err1 = virtnet_sq_bind_xsk_pool(vi, sq, NULL); + err2 = virtnet_rq_bind_xsk_pool(vi, rq, NULL); + + xsk_pool_dma_unmap(pool, 0); + + dma_dev = virtqueue_dma_dev(rq->vq); + + dma_unmap_single(dma_dev, sq->xsk.hdr_dma_address, vi->hdr_len, DMA_TO_DEVICE); + + return err1 | err2; +} + +int virtnet_xsk_pool_setup(struct net_device *dev, struct netdev_bpf *xdp) +{ + if (xdp->xsk.pool) + return virtnet_xsk_pool_enable(dev, xdp->xsk.pool, + xdp->xsk.queue_id); + else + return virtnet_xsk_pool_disable(dev, xdp->xsk.queue_id); +} diff --git a/drivers/net/virtio/xsk.h b/drivers/net/virtio/xsk.h new file mode 100644 index 000000000000..1918285c310c --- /dev/null +++ b/drivers/net/virtio/xsk.h @@ -0,0 +1,7 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ + +#ifndef __XSK_H__ +#define __XSK_H__ + +int virtnet_xsk_pool_setup(struct net_device *dev, struct netdev_bpf *xdp); +#endif From patchwork Tue Jan 16 09:43:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13520620 X-Patchwork-Delegate: kuba@kernel.org Received: from out30-100.freemail.mail.aliyun.com (out30-100.freemail.mail.aliyun.com [115.124.30.100]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4B4E8134A6; Tue, 16 Jan 2024 09:43:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R141e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045176;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=14;SR=0;TI=SMTPD_---0W-lmy3B_1705398197; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W-lmy3B_1705398197) by smtp.aliyun-inc.com; Tue, 16 Jan 2024 17:43:18 +0800 From: Xuan Zhuo To: netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , virtualization@lists.linux.dev, bpf@vger.kernel.org Subject: [PATCH net-next 04/17] virtio_net: xsk: prevent disable tx napi Date: Tue, 16 Jan 2024 17:43:00 +0800 Message-Id: <20240116094313.119939-5-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240116094313.119939-1-xuanzhuo@linux.alibaba.com> References: <20240116094313.119939-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: 1913ebd4ae28 X-Patchwork-Delegate: kuba@kernel.org Since xsk's TX queue is consumed by TX NAPI, if sq is bound to xsk, then we must stop tx napi from being disabled. Signed-off-by: Xuan Zhuo Acked-by: Jason Wang --- drivers/net/virtio/main.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/drivers/net/virtio/main.c b/drivers/net/virtio/main.c index 3856006ac1c7..98721c7b3377 100644 --- a/drivers/net/virtio/main.c +++ b/drivers/net/virtio/main.c @@ -3475,7 +3475,7 @@ static int virtnet_set_coalesce(struct net_device *dev, struct netlink_ext_ack *extack) { struct virtnet_info *vi = netdev_priv(dev); - int ret, queue_number, napi_weight; + int ret, queue_number, napi_weight, i; bool update_napi = false; /* Can't change NAPI weight if the link is up */ @@ -3504,6 +3504,14 @@ static int virtnet_set_coalesce(struct net_device *dev, return ret; if (update_napi) { + /* xsk xmit depends on the tx napi. So if xsk is active, + * prevent modifications to tx napi. + */ + for (i = queue_number; i < vi->max_queue_pairs; i++) { + if (vi->sq[i].xsk.pool) + return -EBUSY; + } + for (; queue_number < vi->max_queue_pairs; queue_number++) vi->sq[queue_number].napi.weight = napi_weight; } From patchwork Tue Jan 16 09:43:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13520621 X-Patchwork-Delegate: kuba@kernel.org Received: from out30-97.freemail.mail.aliyun.com (out30-97.freemail.mail.aliyun.com [115.124.30.97]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4E485134DE; Tue, 16 Jan 2024 09:43:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R431e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046059;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=14;SR=0;TI=SMTPD_---0W-lmy3p_1705398199; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W-lmy3p_1705398199) by smtp.aliyun-inc.com; Tue, 16 Jan 2024 17:43:19 +0800 From: Xuan Zhuo To: netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , virtualization@lists.linux.dev, bpf@vger.kernel.org Subject: [PATCH net-next 05/17] virtio_net: move some api to header Date: Tue, 16 Jan 2024 17:43:01 +0800 Message-Id: <20240116094313.119939-6-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240116094313.119939-1-xuanzhuo@linux.alibaba.com> References: <20240116094313.119939-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: 1913ebd4ae28 X-Patchwork-Delegate: kuba@kernel.org __free_old_xmit is_xdp_raw_buffer_queue These two APIs are needed by the xsk part. So this commit move theses to the header. And add prefix "virtnet_". Signed-off-by: Xuan Zhuo --- drivers/net/virtio/main.c | 86 +++------------------------------ drivers/net/virtio/virtio_net.h | 72 +++++++++++++++++++++++++++ 2 files changed, 79 insertions(+), 79 deletions(-) diff --git a/drivers/net/virtio/main.c b/drivers/net/virtio/main.c index 98721c7b3377..89855495e693 100644 --- a/drivers/net/virtio/main.c +++ b/drivers/net/virtio/main.c @@ -45,8 +45,6 @@ module_param(napi_tx, bool, 0644); #define VIRTIO_XDP_TX BIT(0) #define VIRTIO_XDP_REDIR BIT(1) -#define VIRTIO_XDP_FLAG BIT(0) - #define VIRTNET_DRIVER_VERSION "1.0.0" static const unsigned long guest_offloads[] = { @@ -149,71 +147,11 @@ struct virtio_net_common_hdr { }; }; -static bool is_xdp_frame(void *ptr) -{ - return (unsigned long)ptr & VIRTIO_XDP_FLAG; -} - static void *xdp_to_ptr(struct xdp_frame *ptr) { return (void *)((unsigned long)ptr | VIRTIO_XDP_FLAG); } -static struct xdp_frame *ptr_to_xdp(void *ptr) -{ - return (struct xdp_frame *)((unsigned long)ptr & ~VIRTIO_XDP_FLAG); -} - -static void virtnet_sq_unmap_buf(struct virtnet_sq *sq, struct virtio_dma_head *dma) -{ - int i; - - if (!dma) - return; - - for (i = 0; i < dma->next; ++i) - virtqueue_dma_unmap_single_attrs(sq->vq, - dma->items[i].addr, - dma->items[i].length, - DMA_TO_DEVICE, 0); - dma->next = 0; -} - -static void __free_old_xmit(struct virtnet_sq *sq, bool in_napi, - u64 *bytes, u64 *packets) -{ - struct virtio_dma_head *dma; - unsigned int len; - void *ptr; - - if (virtqueue_get_dma_premapped(sq->vq)) { - dma = &sq->dma.head; - dma->num = ARRAY_SIZE(sq->dma.items); - dma->next = 0; - } else { - dma = NULL; - } - - while ((ptr = virtqueue_get_buf_ctx_dma(sq->vq, &len, dma, NULL)) != NULL) { - virtnet_sq_unmap_buf(sq, dma); - - if (!is_xdp_frame(ptr)) { - struct sk_buff *skb = ptr; - - pr_debug("Sent skb %p\n", skb); - - *bytes += skb->len; - napi_consume_skb(skb, in_napi); - } else { - struct xdp_frame *frame = ptr_to_xdp(ptr); - - *bytes += xdp_get_frame_len(frame); - xdp_return_frame(frame); - } - (*packets)++; - } -} - /* Converting between virtqueue no. and kernel tx/rx queue no. * 0:rx0 1:tx0 2:rx1 3:tx1 ... 2N:rxN 2N+1:txN 2N+2:cvq */ @@ -664,7 +602,7 @@ static void free_old_xmit(struct virtnet_sq *sq, bool in_napi) { u64 bytes = 0, packets = 0; - __free_old_xmit(sq, in_napi, &bytes, &packets); + virtnet_free_old_xmit(sq, in_napi, &bytes, &packets); /* Avoid overhead when no packets have been processed * happens when called speculatively from start_xmit. @@ -678,16 +616,6 @@ static void free_old_xmit(struct virtnet_sq *sq, bool in_napi) u64_stats_update_end(&sq->stats.syncp); } -static bool is_xdp_raw_buffer_queue(struct virtnet_info *vi, int q) -{ - if (q < (vi->curr_queue_pairs - vi->xdp_queue_pairs)) - return false; - else if (q < vi->curr_queue_pairs) - return true; - else - return false; -} - static void check_sq_full_and_disable(struct virtnet_info *vi, struct net_device *dev, struct virtnet_sq *sq) @@ -836,7 +764,7 @@ static int virtnet_xdp_xmit(struct net_device *dev, } /* Free up any pending old buffers before queueing new ones. */ - __free_old_xmit(sq, false, &bytes, &packets); + virtnet_free_old_xmit(sq, false, &bytes, &packets); for (i = 0; i < n; i++) { struct xdp_frame *xdpf = frames[i]; @@ -847,7 +775,7 @@ static int virtnet_xdp_xmit(struct net_device *dev, } ret = nxmit; - if (!is_xdp_raw_buffer_queue(vi, sq - vi->sq)) + if (!virtnet_is_xdp_raw_buffer_queue(vi, sq - vi->sq)) check_sq_full_and_disable(vi, dev, sq); if (flags & XDP_XMIT_FLUSH) { @@ -1998,7 +1926,7 @@ static void virtnet_poll_cleantx(struct virtnet_rq *rq) struct virtnet_sq *sq = &vi->sq[index]; struct netdev_queue *txq = netdev_get_tx_queue(vi->dev, index); - if (!sq->napi.weight || is_xdp_raw_buffer_queue(vi, index)) + if (!sq->napi.weight || virtnet_is_xdp_raw_buffer_queue(vi, index)) return; if (__netif_tx_trylock(txq)) { @@ -2148,7 +2076,7 @@ static int virtnet_poll_tx(struct napi_struct *napi, int budget) int opaque; bool done; - if (unlikely(is_xdp_raw_buffer_queue(vi, index))) { + if (unlikely(virtnet_is_xdp_raw_buffer_queue(vi, index))) { /* We don't need to enable cb for XDP */ napi_complete_done(napi, 0); return 0; @@ -4151,10 +4079,10 @@ void virtnet_sq_free_unused_bufs(struct virtqueue *vq) while ((buf = virtqueue_detach_unused_buf_dma(vq, dma)) != NULL) { virtnet_sq_unmap_buf(sq, dma); - if (!is_xdp_frame(buf)) + if (!virtnet_is_xdp_frame(buf)) dev_kfree_skb(buf); else - xdp_return_frame(ptr_to_xdp(buf)); + xdp_return_frame(virtnet_ptr_to_xdp(buf)); } } diff --git a/drivers/net/virtio/virtio_net.h b/drivers/net/virtio/virtio_net.h index 02dc7650a65c..828fef40ba15 100644 --- a/drivers/net/virtio/virtio_net.h +++ b/drivers/net/virtio/virtio_net.h @@ -9,6 +9,8 @@ #include #include +#define VIRTIO_XDP_FLAG BIT(0) + /* RX packet size EWMA. The average packet size is used to determine the packet * buffer size when refilling RX rings. As the entire RX ring may be refilled * at once, the weight is chosen so that the EWMA will be insensitive to short- @@ -226,6 +228,76 @@ struct virtnet_info { struct failover *failover; }; +static inline bool virtnet_is_xdp_frame(void *ptr) +{ + return (unsigned long)ptr & VIRTIO_XDP_FLAG; +} + +static inline struct xdp_frame *virtnet_ptr_to_xdp(void *ptr) +{ + return (struct xdp_frame *)((unsigned long)ptr & ~VIRTIO_XDP_FLAG); +} + +static inline void virtnet_sq_unmap_buf(struct virtnet_sq *sq, struct virtio_dma_head *dma) +{ + int i; + + if (!dma) + return; + + for (i = 0; i < dma->next; ++i) + virtqueue_dma_unmap_single_attrs(sq->vq, + dma->items[i].addr, + dma->items[i].length, + DMA_TO_DEVICE, 0); + dma->next = 0; +} + +static inline void virtnet_free_old_xmit(struct virtnet_sq *sq, bool in_napi, + u64 *bytes, u64 *packets) +{ + struct virtio_dma_head *dma; + unsigned int len; + void *ptr; + + if (virtqueue_get_dma_premapped(sq->vq)) { + dma = &sq->dma.head; + dma->num = ARRAY_SIZE(sq->dma.items); + dma->next = 0; + } else { + dma = NULL; + } + + while ((ptr = virtqueue_get_buf_ctx_dma(sq->vq, &len, dma, NULL)) != NULL) { + virtnet_sq_unmap_buf(sq, dma); + + if (!virtnet_is_xdp_frame(ptr)) { + struct sk_buff *skb = ptr; + + pr_debug("Sent skb %p\n", skb); + + *bytes += skb->len; + napi_consume_skb(skb, in_napi); + } else { + struct xdp_frame *frame = virtnet_ptr_to_xdp(ptr); + + *bytes += xdp_get_frame_len(frame); + xdp_return_frame(frame); + } + (*packets)++; + } +} + +static inline bool virtnet_is_xdp_raw_buffer_queue(struct virtnet_info *vi, int q) +{ + if (q < (vi->curr_queue_pairs - vi->xdp_queue_pairs)) + return false; + else if (q < vi->curr_queue_pairs) + return true; + else + return false; +} + void virtnet_rx_pause(struct virtnet_info *vi, struct virtnet_rq *rq); void virtnet_rx_resume(struct virtnet_info *vi, struct virtnet_rq *rq); void virtnet_tx_pause(struct virtnet_info *vi, struct virtnet_sq *sq); From patchwork Tue Jan 16 09:43:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13520622 X-Patchwork-Delegate: kuba@kernel.org Received: from out30-113.freemail.mail.aliyun.com (out30-113.freemail.mail.aliyun.com [115.124.30.113]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 78B1813ADE; Tue, 16 Jan 2024 09:43:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R351e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045176;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=14;SR=0;TI=SMTPD_---0W-lmy4e_1705398200; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W-lmy4e_1705398200) by smtp.aliyun-inc.com; Tue, 16 Jan 2024 17:43:20 +0800 From: Xuan Zhuo To: netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , virtualization@lists.linux.dev, bpf@vger.kernel.org Subject: [PATCH net-next 06/17] virtio_net: xsk: tx: support xmit xsk buffer Date: Tue, 16 Jan 2024 17:43:02 +0800 Message-Id: <20240116094313.119939-7-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240116094313.119939-1-xuanzhuo@linux.alibaba.com> References: <20240116094313.119939-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: 1913ebd4ae28 X-Patchwork-Delegate: kuba@kernel.org The driver's tx napi is very important for XSK. It is responsible for obtaining data from the XSK queue and sending it out. At the beginning, we need to trigger tx napi. Signed-off-by: Xuan Zhuo --- drivers/net/virtio/main.c | 22 ++++++--- drivers/net/virtio/virtio_net.h | 4 ++ drivers/net/virtio/xsk.c | 87 +++++++++++++++++++++++++++++++++ drivers/net/virtio/xsk.h | 13 +++++ 4 files changed, 120 insertions(+), 6 deletions(-) diff --git a/drivers/net/virtio/main.c b/drivers/net/virtio/main.c index 89855495e693..4575c885acb0 100644 --- a/drivers/net/virtio/main.c +++ b/drivers/net/virtio/main.c @@ -616,9 +616,9 @@ static void free_old_xmit(struct virtnet_sq *sq, bool in_napi) u64_stats_update_end(&sq->stats.syncp); } -static void check_sq_full_and_disable(struct virtnet_info *vi, - struct net_device *dev, - struct virtnet_sq *sq) +void virtnet_check_sq_full_and_disable(struct virtnet_info *vi, + struct net_device *dev, + struct virtnet_sq *sq) { bool use_napi = sq->napi.weight; int qnum; @@ -776,7 +776,7 @@ static int virtnet_xdp_xmit(struct net_device *dev, ret = nxmit; if (!virtnet_is_xdp_raw_buffer_queue(vi, sq - vi->sq)) - check_sq_full_and_disable(vi, dev, sq); + virtnet_check_sq_full_and_disable(vi, dev, sq); if (flags & XDP_XMIT_FLUSH) { if (virtqueue_kick_prepare(sq->vq) && virtqueue_notify(sq->vq)) @@ -2073,6 +2073,7 @@ static int virtnet_poll_tx(struct napi_struct *napi, int budget) struct virtnet_info *vi = sq->vq->vdev->priv; unsigned int index = vq2txq(sq->vq); struct netdev_queue *txq; + bool xsk_busy = false; int opaque; bool done; @@ -2085,11 +2086,20 @@ static int virtnet_poll_tx(struct napi_struct *napi, int budget) txq = netdev_get_tx_queue(vi->dev, index); __netif_tx_lock(txq, raw_smp_processor_id()); virtqueue_disable_cb(sq->vq); - free_old_xmit(sq, true); + + if (sq->xsk.pool) + xsk_busy = virtnet_xsk_xmit(sq, sq->xsk.pool, budget); + else + free_old_xmit(sq, true); if (sq->vq->num_free >= 2 + MAX_SKB_FRAGS) netif_tx_wake_queue(txq); + if (xsk_busy) { + __netif_tx_unlock(txq); + return budget; + } + opaque = virtqueue_enable_cb_prepare(sq->vq); done = napi_complete_done(napi, 0); @@ -2204,7 +2214,7 @@ static netdev_tx_t start_xmit(struct sk_buff *skb, struct net_device *dev) nf_reset_ct(skb); } - check_sq_full_and_disable(vi, dev, sq); + virtnet_check_sq_full_and_disable(vi, dev, sq); if (kick || netif_xmit_stopped(txq)) { if (virtqueue_kick_prepare(sq->vq) && virtqueue_notify(sq->vq)) { diff --git a/drivers/net/virtio/virtio_net.h b/drivers/net/virtio/virtio_net.h index 828fef40ba15..cc735a09a3d9 100644 --- a/drivers/net/virtio/virtio_net.h +++ b/drivers/net/virtio/virtio_net.h @@ -10,6 +10,7 @@ #include #define VIRTIO_XDP_FLAG BIT(0) +#define VIRTIO_XSK_FLAG BIT(1) /* RX packet size EWMA. The average packet size is used to determine the packet * buffer size when refilling RX rings. As the entire RX ring may be refilled @@ -304,4 +305,7 @@ void virtnet_tx_pause(struct virtnet_info *vi, struct virtnet_sq *sq); void virtnet_tx_resume(struct virtnet_info *vi, struct virtnet_sq *sq); void virtnet_sq_free_unused_bufs(struct virtqueue *vq); void virtnet_rq_free_unused_bufs(struct virtqueue *vq); +void virtnet_check_sq_full_and_disable(struct virtnet_info *vi, + struct net_device *dev, + struct virtnet_sq *sq); #endif diff --git a/drivers/net/virtio/xsk.c b/drivers/net/virtio/xsk.c index 613a6c63d47c..d2a96424ade9 100644 --- a/drivers/net/virtio/xsk.c +++ b/drivers/net/virtio/xsk.c @@ -8,6 +8,93 @@ static struct virtio_net_hdr_mrg_rxbuf xsk_hdr; +static void sg_fill_dma(struct scatterlist *sg, dma_addr_t addr, u32 len) +{ + sg->dma_address = addr; + sg->length = len; +} + +static int virtnet_xsk_xmit_one(struct virtnet_sq *sq, + struct xsk_buff_pool *pool, + struct xdp_desc *desc) +{ + struct virtnet_info *vi; + dma_addr_t addr; + + vi = sq->vq->vdev->priv; + + addr = xsk_buff_raw_get_dma(pool, desc->addr); + xsk_buff_raw_dma_sync_for_device(pool, addr, desc->len); + + sg_init_table(sq->sg, 2); + + sg_fill_dma(sq->sg, sq->xsk.hdr_dma_address, vi->hdr_len); + sg_fill_dma(sq->sg + 1, addr, desc->len); + + return virtqueue_add_outbuf(sq->vq, sq->sg, 2, + virtnet_xsk_to_ptr(desc->len), GFP_ATOMIC); +} + +static int virtnet_xsk_xmit_batch(struct virtnet_sq *sq, + struct xsk_buff_pool *pool, + unsigned int budget, + u64 *kicks) +{ + struct xdp_desc *descs = pool->tx_descs; + u32 nb_pkts, max_pkts, i; + bool kick = false; + int err; + + /* Every xsk tx packet needs two desc(virtnet header and packet). So we + * use sq->vq->num_free / 2 as the limitation. + */ + max_pkts = min_t(u32, budget, sq->vq->num_free / 2); + + nb_pkts = xsk_tx_peek_release_desc_batch(pool, max_pkts); + if (!nb_pkts) + return 0; + + for (i = 0; i < nb_pkts; i++) { + err = virtnet_xsk_xmit_one(sq, pool, &descs[i]); + if (unlikely(err)) + break; + + kick = true; + } + + if (kick && virtqueue_kick_prepare(sq->vq) && virtqueue_notify(sq->vq)) + (*kicks)++; + + return i; +} + +bool virtnet_xsk_xmit(struct virtnet_sq *sq, struct xsk_buff_pool *pool, + int budget) +{ + struct virtnet_info *vi = sq->vq->vdev->priv; + u64 bytes = 0, packets = 0, kicks = 0; + int sent; + + virtnet_free_old_xmit(sq, true, &bytes, &packets); + + sent = virtnet_xsk_xmit_batch(sq, pool, budget, &kicks); + + if (!virtnet_is_xdp_raw_buffer_queue(vi, sq - vi->sq)) + virtnet_check_sq_full_and_disable(vi, vi->dev, sq); + + u64_stats_update_begin(&sq->stats.syncp); + u64_stats_add(&sq->stats.packets, packets); + u64_stats_add(&sq->stats.bytes, bytes); + u64_stats_add(&sq->stats.kicks, kicks); + u64_stats_add(&sq->stats.xdp_tx, sent); + u64_stats_update_end(&sq->stats.syncp); + + if (xsk_uses_need_wakeup(pool)) + xsk_set_tx_need_wakeup(pool); + + return sent == budget; +} + static int virtnet_rq_bind_xsk_pool(struct virtnet_info *vi, struct virtnet_rq *rq, struct xsk_buff_pool *pool) { diff --git a/drivers/net/virtio/xsk.h b/drivers/net/virtio/xsk.h index 1918285c310c..73ca8cd5308b 100644 --- a/drivers/net/virtio/xsk.h +++ b/drivers/net/virtio/xsk.h @@ -3,5 +3,18 @@ #ifndef __XSK_H__ #define __XSK_H__ +#define VIRTIO_XSK_FLAG_OFFSET 4 + +static inline void *virtnet_xsk_to_ptr(u32 len) +{ + unsigned long p; + + p = len << VIRTIO_XSK_FLAG_OFFSET; + + return (void *)(p | VIRTIO_XSK_FLAG); +} + int virtnet_xsk_pool_setup(struct net_device *dev, struct netdev_bpf *xdp); +bool virtnet_xsk_xmit(struct virtnet_sq *sq, struct xsk_buff_pool *pool, + int budget); #endif From patchwork Tue Jan 16 09:43:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13520624 X-Patchwork-Delegate: kuba@kernel.org Received: from out30-132.freemail.mail.aliyun.com (out30-132.freemail.mail.aliyun.com [115.124.30.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A18691774D; Tue, 16 Jan 2024 09:43:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R121e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045192;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=14;SR=0;TI=SMTPD_---0W-lmFRG_1705398201; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W-lmFRG_1705398201) by smtp.aliyun-inc.com; Tue, 16 Jan 2024 17:43:22 +0800 From: Xuan Zhuo To: netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , virtualization@lists.linux.dev, bpf@vger.kernel.org Subject: [PATCH net-next 07/17] virtio_net: xsk: tx: support wakeup Date: Tue, 16 Jan 2024 17:43:03 +0800 Message-Id: <20240116094313.119939-8-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240116094313.119939-1-xuanzhuo@linux.alibaba.com> References: <20240116094313.119939-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: 1913ebd4ae28 X-Patchwork-Delegate: kuba@kernel.org xsk wakeup is used to trigger the logic for xsk xmit by xsk framework or user. Virtio-net does not support to actively generate an interruption, so it tries to trigger tx NAPI on the local cpu. Signed-off-by: Xuan Zhuo --- drivers/net/virtio/main.c | 20 ++++++-------------- drivers/net/virtio/virtio_net.h | 9 +++++++++ drivers/net/virtio/xsk.c | 23 +++++++++++++++++++++++ drivers/net/virtio/xsk.h | 1 + 4 files changed, 39 insertions(+), 14 deletions(-) diff --git a/drivers/net/virtio/main.c b/drivers/net/virtio/main.c index 4575c885acb0..cb0a1ea712de 100644 --- a/drivers/net/virtio/main.c +++ b/drivers/net/virtio/main.c @@ -233,15 +233,6 @@ static void disable_delayed_refill(struct virtnet_info *vi) spin_unlock_bh(&vi->refill_lock); } -static void virtqueue_napi_schedule(struct napi_struct *napi, - struct virtqueue *vq) -{ - if (napi_schedule_prep(napi)) { - virtqueue_disable_cb(vq); - __napi_schedule(napi); - } -} - static bool virtqueue_napi_complete(struct napi_struct *napi, struct virtqueue *vq, int processed) { @@ -250,7 +241,7 @@ static bool virtqueue_napi_complete(struct napi_struct *napi, opaque = virtqueue_enable_cb_prepare(vq); if (napi_complete_done(napi, processed)) { if (unlikely(virtqueue_poll(vq, opaque))) - virtqueue_napi_schedule(napi, vq); + virtnet_vq_napi_schedule(napi, vq); else return true; } else { @@ -269,7 +260,7 @@ static void skb_xmit_done(struct virtqueue *vq) virtqueue_disable_cb(vq); if (napi->weight) - virtqueue_napi_schedule(napi, vq); + virtnet_vq_napi_schedule(napi, vq); else /* We were probably waiting for more output buffers. */ netif_wake_subqueue(vi->dev, vq2txq(vq)); @@ -639,7 +630,7 @@ void virtnet_check_sq_full_and_disable(struct virtnet_info *vi, netif_stop_subqueue(dev, qnum); if (use_napi) { if (unlikely(!virtqueue_enable_cb_delayed(sq->vq))) - virtqueue_napi_schedule(&sq->napi, sq->vq); + virtnet_vq_napi_schedule(&sq->napi, sq->vq); } else if (unlikely(!virtqueue_enable_cb_delayed(sq->vq))) { /* More just got used, free them then recheck. */ free_old_xmit(sq, false); @@ -1807,7 +1798,7 @@ static void skb_recv_done(struct virtqueue *rvq) struct virtnet_rq *rq = &vi->rq[vq2rxq(rvq)]; rq->calls++; - virtqueue_napi_schedule(&rq->napi, rvq); + virtnet_vq_napi_schedule(&rq->napi, rvq); } static void virtnet_napi_enable(struct virtqueue *vq, struct napi_struct *napi) @@ -1819,7 +1810,7 @@ static void virtnet_napi_enable(struct virtqueue *vq, struct napi_struct *napi) * Call local_bh_enable after to trigger softIRQ processing. */ local_bh_disable(); - virtqueue_napi_schedule(napi, vq); + virtnet_vq_napi_schedule(napi, vq); local_bh_enable(); } @@ -3969,6 +3960,7 @@ static const struct net_device_ops virtnet_netdev = { .ndo_vlan_rx_kill_vid = virtnet_vlan_rx_kill_vid, .ndo_bpf = virtnet_xdp, .ndo_xdp_xmit = virtnet_xdp_xmit, + .ndo_xsk_wakeup = virtnet_xsk_wakeup, .ndo_features_check = passthru_features_check, .ndo_get_phys_port_name = virtnet_get_phys_port_name, .ndo_set_features = virtnet_set_features, diff --git a/drivers/net/virtio/virtio_net.h b/drivers/net/virtio/virtio_net.h index cc735a09a3d9..c30fb152ecd0 100644 --- a/drivers/net/virtio/virtio_net.h +++ b/drivers/net/virtio/virtio_net.h @@ -299,6 +299,15 @@ static inline bool virtnet_is_xdp_raw_buffer_queue(struct virtnet_info *vi, int return false; } +static inline void virtnet_vq_napi_schedule(struct napi_struct *napi, + struct virtqueue *vq) +{ + if (napi_schedule_prep(napi)) { + virtqueue_disable_cb(vq); + __napi_schedule(napi); + } +} + void virtnet_rx_pause(struct virtnet_info *vi, struct virtnet_rq *rq); void virtnet_rx_resume(struct virtnet_info *vi, struct virtnet_rq *rq); void virtnet_tx_pause(struct virtnet_info *vi, struct virtnet_sq *sq); diff --git a/drivers/net/virtio/xsk.c b/drivers/net/virtio/xsk.c index d2a96424ade9..9e5523ff5707 100644 --- a/drivers/net/virtio/xsk.c +++ b/drivers/net/virtio/xsk.c @@ -95,6 +95,29 @@ bool virtnet_xsk_xmit(struct virtnet_sq *sq, struct xsk_buff_pool *pool, return sent == budget; } +int virtnet_xsk_wakeup(struct net_device *dev, u32 qid, u32 flag) +{ + struct virtnet_info *vi = netdev_priv(dev); + struct virtnet_sq *sq; + + if (!netif_running(dev)) + return -ENETDOWN; + + if (qid >= vi->curr_queue_pairs) + return -EINVAL; + + sq = &vi->sq[qid]; + + if (napi_if_scheduled_mark_missed(&sq->napi)) + return 0; + + local_bh_disable(); + virtnet_vq_napi_schedule(&sq->napi, sq->vq); + local_bh_enable(); + + return 0; +} + static int virtnet_rq_bind_xsk_pool(struct virtnet_info *vi, struct virtnet_rq *rq, struct xsk_buff_pool *pool) { diff --git a/drivers/net/virtio/xsk.h b/drivers/net/virtio/xsk.h index 73ca8cd5308b..1bd19dcda649 100644 --- a/drivers/net/virtio/xsk.h +++ b/drivers/net/virtio/xsk.h @@ -17,4 +17,5 @@ static inline void *virtnet_xsk_to_ptr(u32 len) int virtnet_xsk_pool_setup(struct net_device *dev, struct netdev_bpf *xdp); bool virtnet_xsk_xmit(struct virtnet_sq *sq, struct xsk_buff_pool *pool, int budget); +int virtnet_xsk_wakeup(struct net_device *dev, u32 qid, u32 flag); #endif From patchwork Tue Jan 16 09:43:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13520623 X-Patchwork-Delegate: kuba@kernel.org Received: from out30-110.freemail.mail.aliyun.com (out30-110.freemail.mail.aliyun.com [115.124.30.110]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 87CCB175B3; Tue, 16 Jan 2024 09:43:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R301e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046059;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=14;SR=0;TI=SMTPD_---0W-lk6W-_1705398202; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W-lk6W-_1705398202) by smtp.aliyun-inc.com; Tue, 16 Jan 2024 17:43:23 +0800 From: Xuan Zhuo To: netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , virtualization@lists.linux.dev, bpf@vger.kernel.org Subject: [PATCH net-next 08/17] virtio_net: xsk: tx: handle the transmitted xsk buffer Date: Tue, 16 Jan 2024 17:43:04 +0800 Message-Id: <20240116094313.119939-9-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240116094313.119939-1-xuanzhuo@linux.alibaba.com> References: <20240116094313.119939-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: 1913ebd4ae28 X-Patchwork-Delegate: kuba@kernel.org virtnet_free_old_xmit distinguishes three type ptr(skb, xdp frame, xsk buffer) by the last bits of the pointer. Signed-off-by: Xuan Zhuo --- drivers/net/virtio/virtio_net.h | 30 ++++++++++++++++++++++++++---- drivers/net/virtio/xsk.c | 33 ++++++++++++++++++++++++++------- drivers/net/virtio/xsk.h | 5 +++++ 3 files changed, 57 insertions(+), 11 deletions(-) diff --git a/drivers/net/virtio/virtio_net.h b/drivers/net/virtio/virtio_net.h index c30fb152ecd0..480b533bd61e 100644 --- a/drivers/net/virtio/virtio_net.h +++ b/drivers/net/virtio/virtio_net.h @@ -229,6 +229,11 @@ struct virtnet_info { struct failover *failover; }; +static inline bool virtnet_is_skb_ptr(void *ptr) +{ + return !((unsigned long)ptr & (VIRTIO_XDP_FLAG | VIRTIO_XSK_FLAG)); +} + static inline bool virtnet_is_xdp_frame(void *ptr) { return (unsigned long)ptr & VIRTIO_XDP_FLAG; @@ -239,6 +244,9 @@ static inline struct xdp_frame *virtnet_ptr_to_xdp(void *ptr) return (struct xdp_frame *)((unsigned long)ptr & ~VIRTIO_XDP_FLAG); } +static inline u32 virtnet_ptr_to_xsk(void *ptr); +void virtnet_xsk_completed(struct virtnet_sq *sq, int num); + static inline void virtnet_sq_unmap_buf(struct virtnet_sq *sq, struct virtio_dma_head *dma) { int i; @@ -254,8 +262,8 @@ static inline void virtnet_sq_unmap_buf(struct virtnet_sq *sq, struct virtio_dma dma->next = 0; } -static inline void virtnet_free_old_xmit(struct virtnet_sq *sq, bool in_napi, - u64 *bytes, u64 *packets) +static inline void __virtnet_free_old_xmit(struct virtnet_sq *sq, bool in_napi, + u64 *bytes, u64 *packets, u64 *xsk) { struct virtio_dma_head *dma; unsigned int len; @@ -272,23 +280,37 @@ static inline void virtnet_free_old_xmit(struct virtnet_sq *sq, bool in_napi, while ((ptr = virtqueue_get_buf_ctx_dma(sq->vq, &len, dma, NULL)) != NULL) { virtnet_sq_unmap_buf(sq, dma); - if (!virtnet_is_xdp_frame(ptr)) { + if (virtnet_is_skb_ptr(ptr)) { struct sk_buff *skb = ptr; pr_debug("Sent skb %p\n", skb); *bytes += skb->len; napi_consume_skb(skb, in_napi); - } else { + } else if (virtnet_is_xdp_frame(ptr)) { struct xdp_frame *frame = virtnet_ptr_to_xdp(ptr); *bytes += xdp_get_frame_len(frame); xdp_return_frame(frame); + } else { + *bytes += virtnet_ptr_to_xsk(ptr); + (*xsk)++; } (*packets)++; } } +static inline void virtnet_free_old_xmit(struct virtnet_sq *sq, bool in_napi, + u64 *bytes, u64 *packets) +{ + u64 xsknum = 0; + + __virtnet_free_old_xmit(sq, in_napi, bytes, packets, &xsknum); + + if (xsknum) + virtnet_xsk_completed(sq, xsknum); +} + static inline bool virtnet_is_xdp_raw_buffer_queue(struct virtnet_info *vi, int q) { if (q < (vi->curr_queue_pairs - vi->xdp_queue_pairs)) diff --git a/drivers/net/virtio/xsk.c b/drivers/net/virtio/xsk.c index 9e5523ff5707..0c6a8f92ae38 100644 --- a/drivers/net/virtio/xsk.c +++ b/drivers/net/virtio/xsk.c @@ -73,9 +73,13 @@ bool virtnet_xsk_xmit(struct virtnet_sq *sq, struct xsk_buff_pool *pool, { struct virtnet_info *vi = sq->vq->vdev->priv; u64 bytes = 0, packets = 0, kicks = 0; + u64 xsknum = 0; int sent; - virtnet_free_old_xmit(sq, true, &bytes, &packets); + /* Avoid to wakeup napi meanless, so call __virtnet_free_old_xmit. */ + __virtnet_free_old_xmit(sq, true, &bytes, &packets, &xsknum); + if (xsknum) + xsk_tx_completed(sq->xsk.pool, xsknum); sent = virtnet_xsk_xmit_batch(sq, pool, budget, &kicks); @@ -95,6 +99,16 @@ bool virtnet_xsk_xmit(struct virtnet_sq *sq, struct xsk_buff_pool *pool, return sent == budget; } +static void xsk_wakeup(struct virtnet_sq *sq) +{ + if (napi_if_scheduled_mark_missed(&sq->napi)) + return; + + local_bh_disable(); + virtnet_vq_napi_schedule(&sq->napi, sq->vq); + local_bh_enable(); +} + int virtnet_xsk_wakeup(struct net_device *dev, u32 qid, u32 flag) { struct virtnet_info *vi = netdev_priv(dev); @@ -108,14 +122,19 @@ int virtnet_xsk_wakeup(struct net_device *dev, u32 qid, u32 flag) sq = &vi->sq[qid]; - if (napi_if_scheduled_mark_missed(&sq->napi)) - return 0; + xsk_wakeup(sq); + return 0; +} - local_bh_disable(); - virtnet_vq_napi_schedule(&sq->napi, sq->vq); - local_bh_enable(); +void virtnet_xsk_completed(struct virtnet_sq *sq, int num) +{ + xsk_tx_completed(sq->xsk.pool, num); - return 0; + /* If this is called by rx poll, start_xmit and xdp xmit we should + * wakeup the tx napi to consume the xsk tx queue, because the tx + * interrupt may not be triggered. + */ + xsk_wakeup(sq); } static int virtnet_rq_bind_xsk_pool(struct virtnet_info *vi, struct virtnet_rq *rq, diff --git a/drivers/net/virtio/xsk.h b/drivers/net/virtio/xsk.h index 1bd19dcda649..7ebc9bda7aee 100644 --- a/drivers/net/virtio/xsk.h +++ b/drivers/net/virtio/xsk.h @@ -14,6 +14,11 @@ static inline void *virtnet_xsk_to_ptr(u32 len) return (void *)(p | VIRTIO_XSK_FLAG); } +static inline u32 virtnet_ptr_to_xsk(void *ptr) +{ + return ((unsigned long)ptr) >> VIRTIO_XSK_FLAG_OFFSET; +} + int virtnet_xsk_pool_setup(struct net_device *dev, struct netdev_bpf *xdp); bool virtnet_xsk_xmit(struct virtnet_sq *sq, struct xsk_buff_pool *pool, int budget); From patchwork Tue Jan 16 09:43:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13520625 X-Patchwork-Delegate: kuba@kernel.org Received: from out30-97.freemail.mail.aliyun.com (out30-97.freemail.mail.aliyun.com [115.124.30.97]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E9E6018B04; Tue, 16 Jan 2024 09:43:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R691e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045192;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=14;SR=0;TI=SMTPD_---0W-lmy6k_1705398203; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W-lmy6k_1705398203) by smtp.aliyun-inc.com; Tue, 16 Jan 2024 17:43:24 +0800 From: Xuan Zhuo To: netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , virtualization@lists.linux.dev, bpf@vger.kernel.org Subject: [PATCH net-next 09/17] virtio_net: xsk: tx: free the unused xsk buffer Date: Tue, 16 Jan 2024 17:43:05 +0800 Message-Id: <20240116094313.119939-10-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240116094313.119939-1-xuanzhuo@linux.alibaba.com> References: <20240116094313.119939-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: 1913ebd4ae28 X-Patchwork-Delegate: kuba@kernel.org virtnet_sq_free_unused_buf() check xsk buffer. Signed-off-by: Xuan Zhuo Acked-by: Jason Wang --- drivers/net/virtio/main.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/drivers/net/virtio/main.c b/drivers/net/virtio/main.c index cb0a1ea712de..e64d52e7d95b 100644 --- a/drivers/net/virtio/main.c +++ b/drivers/net/virtio/main.c @@ -4081,10 +4081,12 @@ void virtnet_sq_free_unused_bufs(struct virtqueue *vq) while ((buf = virtqueue_detach_unused_buf_dma(vq, dma)) != NULL) { virtnet_sq_unmap_buf(sq, dma); - if (!virtnet_is_xdp_frame(buf)) + if (virtnet_is_skb_ptr(buf)) dev_kfree_skb(buf); - else + else if (virtnet_is_xdp_frame(buf)) xdp_return_frame(virtnet_ptr_to_xdp(buf)); + else + xsk_tx_completed(sq->xsk.pool, 1); } } From patchwork Tue Jan 16 09:43:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13520626 X-Patchwork-Delegate: kuba@kernel.org Received: from out30-111.freemail.mail.aliyun.com (out30-111.freemail.mail.aliyun.com [115.124.30.111]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 591DC19471; Tue, 16 Jan 2024 09:43:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R141e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046060;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=14;SR=0;TI=SMTPD_---0W-lk6XF_1705398204; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W-lk6XF_1705398204) by smtp.aliyun-inc.com; Tue, 16 Jan 2024 17:43:25 +0800 From: Xuan Zhuo To: netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , virtualization@lists.linux.dev, bpf@vger.kernel.org Subject: [PATCH net-next 10/17] virtio_net: separate receive_mergeable Date: Tue, 16 Jan 2024 17:43:06 +0800 Message-Id: <20240116094313.119939-11-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240116094313.119939-1-xuanzhuo@linux.alibaba.com> References: <20240116094313.119939-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: 1913ebd4ae28 X-Patchwork-Delegate: kuba@kernel.org This commit separates the function receive_mergeable(), put the logic of appending frag to the skb as an independent function. The subsequent commit will reuse it. Signed-off-by: Xuan Zhuo --- drivers/net/virtio/main.c | 77 ++++++++++++++++++++------------- drivers/net/virtio/virtio_net.h | 4 ++ 2 files changed, 51 insertions(+), 30 deletions(-) diff --git a/drivers/net/virtio/main.c b/drivers/net/virtio/main.c index e64d52e7d95b..3ea75c62b384 100644 --- a/drivers/net/virtio/main.c +++ b/drivers/net/virtio/main.c @@ -1405,6 +1405,49 @@ static struct sk_buff *receive_mergeable_xdp(struct net_device *dev, return NULL; } +struct sk_buff *virtnet_skb_append_frag(struct sk_buff *head_skb, + struct sk_buff *curr_skb, + struct page *page, void *buf, + int len, int truesize) +{ + int num_skb_frags; + int offset; + + num_skb_frags = skb_shinfo(curr_skb)->nr_frags; + if (unlikely(num_skb_frags == MAX_SKB_FRAGS)) { + struct sk_buff *nskb = alloc_skb(0, GFP_ATOMIC); + + if (unlikely(!nskb)) + return NULL; + + if (curr_skb == head_skb) + skb_shinfo(curr_skb)->frag_list = nskb; + else + curr_skb->next = nskb; + curr_skb = nskb; + head_skb->truesize += nskb->truesize; + num_skb_frags = 0; + } + + if (curr_skb != head_skb) { + head_skb->data_len += len; + head_skb->len += len; + head_skb->truesize += truesize; + } + + offset = buf - page_address(page); + if (skb_can_coalesce(curr_skb, num_skb_frags, page, offset)) { + put_page(page); + skb_coalesce_rx_frag(curr_skb, num_skb_frags - 1, + len, truesize); + } else { + skb_add_rx_frag(curr_skb, num_skb_frags, page, + offset, len, truesize); + } + + return curr_skb; +} + static struct sk_buff *receive_mergeable(struct net_device *dev, struct virtnet_info *vi, struct virtnet_rq *rq, @@ -1454,8 +1497,6 @@ static struct sk_buff *receive_mergeable(struct net_device *dev, if (unlikely(!curr_skb)) goto err_skb; while (--num_buf) { - int num_skb_frags; - buf = virtnet_rq_get_buf(rq, &len, &ctx); if (unlikely(!buf)) { pr_debug("%s: rx error: %d buffers out of %d missing\n", @@ -1480,34 +1521,10 @@ static struct sk_buff *receive_mergeable(struct net_device *dev, goto err_skb; } - num_skb_frags = skb_shinfo(curr_skb)->nr_frags; - if (unlikely(num_skb_frags == MAX_SKB_FRAGS)) { - struct sk_buff *nskb = alloc_skb(0, GFP_ATOMIC); - - if (unlikely(!nskb)) - goto err_skb; - if (curr_skb == head_skb) - skb_shinfo(curr_skb)->frag_list = nskb; - else - curr_skb->next = nskb; - curr_skb = nskb; - head_skb->truesize += nskb->truesize; - num_skb_frags = 0; - } - if (curr_skb != head_skb) { - head_skb->data_len += len; - head_skb->len += len; - head_skb->truesize += truesize; - } - offset = buf - page_address(page); - if (skb_can_coalesce(curr_skb, num_skb_frags, page, offset)) { - put_page(page); - skb_coalesce_rx_frag(curr_skb, num_skb_frags - 1, - len, truesize); - } else { - skb_add_rx_frag(curr_skb, num_skb_frags, page, - offset, len, truesize); - } + curr_skb = virtnet_skb_append_frag(head_skb, curr_skb, page, + buf, len, truesize); + if (!curr_skb) + goto err_skb; } ewma_pkt_len_add(&rq->mrg_avg_pkt_len, head_skb->len); diff --git a/drivers/net/virtio/virtio_net.h b/drivers/net/virtio/virtio_net.h index 480b533bd61e..1ca92eea8203 100644 --- a/drivers/net/virtio/virtio_net.h +++ b/drivers/net/virtio/virtio_net.h @@ -339,4 +339,8 @@ void virtnet_rq_free_unused_bufs(struct virtqueue *vq); void virtnet_check_sq_full_and_disable(struct virtnet_info *vi, struct net_device *dev, struct virtnet_sq *sq); +struct sk_buff *virtnet_skb_append_frag(struct sk_buff *head_skb, + struct sk_buff *curr_skb, + struct page *page, void *buf, + int len, int truesize); #endif From patchwork Tue Jan 16 09:43:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13520627 X-Patchwork-Delegate: kuba@kernel.org Received: from out30-113.freemail.mail.aliyun.com (out30-113.freemail.mail.aliyun.com [115.124.30.113]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3BF6B1B5BF; Tue, 16 Jan 2024 09:43:28 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R381e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046060;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=14;SR=0;TI=SMTPD_---0W-lmy7q_1705398205; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W-lmy7q_1705398205) by smtp.aliyun-inc.com; Tue, 16 Jan 2024 17:43:26 +0800 From: Xuan Zhuo To: netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , virtualization@lists.linux.dev, bpf@vger.kernel.org Subject: [PATCH net-next 11/17] virtio_net: separate receive_buf Date: Tue, 16 Jan 2024 17:43:07 +0800 Message-Id: <20240116094313.119939-12-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240116094313.119939-1-xuanzhuo@linux.alibaba.com> References: <20240116094313.119939-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: 1913ebd4ae28 X-Patchwork-Delegate: kuba@kernel.org This commit separates the function receive_buf(), then we wrap the logic of handling the skb to an independent function virtnet_receive_done(). The subsequent commit will reuse it. Signed-off-by: Xuan Zhuo --- drivers/net/virtio/main.c | 56 ++++++++++++++++++++++----------------- 1 file changed, 32 insertions(+), 24 deletions(-) diff --git a/drivers/net/virtio/main.c b/drivers/net/virtio/main.c index 3ea75c62b384..2f2141c837e6 100644 --- a/drivers/net/virtio/main.c +++ b/drivers/net/virtio/main.c @@ -1569,32 +1569,11 @@ static void virtio_skb_set_hash(const struct virtio_net_hdr_v1_hash *hdr_hash, skb_set_hash(skb, __le32_to_cpu(hdr_hash->hash_value), rss_hash_type); } -static void receive_buf(struct virtnet_info *vi, struct virtnet_rq *rq, - void *buf, unsigned int len, void **ctx, - unsigned int *xdp_xmit, - struct virtnet_rq_stats *stats) +static void virtnet_receive_done(struct virtnet_info *vi, struct virtnet_rq *rq, + struct sk_buff *skb) { - struct net_device *dev = vi->dev; - struct sk_buff *skb; struct virtio_net_common_hdr *hdr; - - if (unlikely(len < vi->hdr_len + ETH_HLEN)) { - pr_debug("%s: short packet %i\n", dev->name, len); - DEV_STATS_INC(dev, rx_length_errors); - virtnet_rq_free_buf(vi, rq, buf); - return; - } - - if (vi->mergeable_rx_bufs) - skb = receive_mergeable(dev, vi, rq, buf, ctx, len, xdp_xmit, - stats); - else if (vi->big_packets) - skb = receive_big(dev, vi, rq, buf, len, stats); - else - skb = receive_small(dev, vi, rq, buf, ctx, len, xdp_xmit, stats); - - if (unlikely(!skb)) - return; + struct net_device *dev = vi->dev; hdr = skb_vnet_common_hdr(skb); if (dev->features & NETIF_F_RXHASH && vi->has_rss_hash_report) @@ -1624,6 +1603,35 @@ static void receive_buf(struct virtnet_info *vi, struct virtnet_rq *rq, dev_kfree_skb(skb); } +static void receive_buf(struct virtnet_info *vi, struct virtnet_rq *rq, + void *buf, unsigned int len, void **ctx, + unsigned int *xdp_xmit, + struct virtnet_rq_stats *stats) +{ + struct net_device *dev = vi->dev; + struct sk_buff *skb; + + if (unlikely(len < vi->hdr_len + ETH_HLEN)) { + pr_debug("%s: short packet %i\n", dev->name, len); + DEV_STATS_INC(dev, rx_length_errors); + virtnet_rq_free_buf(vi, rq, buf); + return; + } + + if (vi->mergeable_rx_bufs) + skb = receive_mergeable(dev, vi, rq, buf, ctx, len, xdp_xmit, + stats); + else if (vi->big_packets) + skb = receive_big(dev, vi, rq, buf, len, stats); + else + skb = receive_small(dev, vi, rq, buf, ctx, len, xdp_xmit, stats); + + if (unlikely(!skb)) + return; + + virtnet_receive_done(vi, rq, skb); +} + /* Unlike mergeable buffers, all buffers are allocated to the * same size, except for the headroom. For this reason we do * not need to use mergeable_len_to_ctx here - it is enough From patchwork Tue Jan 16 09:43:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13520628 X-Patchwork-Delegate: kuba@kernel.org Received: from out30-100.freemail.mail.aliyun.com (out30-100.freemail.mail.aliyun.com [115.124.30.100]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DFFF01B949; Tue, 16 Jan 2024 09:43:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R131e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046049;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=14;SR=0;TI=SMTPD_---0W-lj1YV_1705398206; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W-lj1YV_1705398206) by smtp.aliyun-inc.com; Tue, 16 Jan 2024 17:43:27 +0800 From: Xuan Zhuo To: netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , virtualization@lists.linux.dev, bpf@vger.kernel.org Subject: [PATCH net-next 12/17] virtio_net: xsk: rx: support fill with xsk buffer Date: Tue, 16 Jan 2024 17:43:08 +0800 Message-Id: <20240116094313.119939-13-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240116094313.119939-1-xuanzhuo@linux.alibaba.com> References: <20240116094313.119939-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: 1913ebd4ae28 X-Patchwork-Delegate: kuba@kernel.org Implement the logic of filling rq with XSK buffers. Signed-off-by: Xuan Zhuo --- drivers/net/virtio/main.c | 9 +++++- drivers/net/virtio/virtio_net.h | 2 ++ drivers/net/virtio/xsk.c | 51 ++++++++++++++++++++++++++++++++- drivers/net/virtio/xsk.h | 2 ++ 4 files changed, 62 insertions(+), 2 deletions(-) diff --git a/drivers/net/virtio/main.c b/drivers/net/virtio/main.c index 2f2141c837e6..fb8dbce11e53 100644 --- a/drivers/net/virtio/main.c +++ b/drivers/net/virtio/main.c @@ -1794,6 +1794,11 @@ static bool try_fill_recv(struct virtnet_info *vi, struct virtnet_rq *rq, int err; bool oom; + if (rq->xsk.pool) { + err = virtnet_add_recvbuf_xsk(vi, rq, rq->xsk.pool, gfp); + goto kick; + } + do { if (vi->mergeable_rx_bufs) err = add_recvbuf_mergeable(vi, rq, gfp); @@ -1802,10 +1807,11 @@ static bool try_fill_recv(struct virtnet_info *vi, struct virtnet_rq *rq, else err = add_recvbuf_small(vi, rq, gfp); - oom = err == -ENOMEM; if (err) break; } while (rq->vq->num_free); + +kick: if (virtqueue_kick_prepare(rq->vq) && virtqueue_notify(rq->vq)) { unsigned long flags; @@ -1814,6 +1820,7 @@ static bool try_fill_recv(struct virtnet_info *vi, struct virtnet_rq *rq, u64_stats_update_end_irqrestore(&rq->stats.syncp, flags); } + oom = err == -ENOMEM; return !oom; } diff --git a/drivers/net/virtio/virtio_net.h b/drivers/net/virtio/virtio_net.h index 1ca92eea8203..c07a1e58bf0f 100644 --- a/drivers/net/virtio/virtio_net.h +++ b/drivers/net/virtio/virtio_net.h @@ -141,6 +141,8 @@ struct virtnet_rq { /* xdp rxq used by xsk */ struct xdp_rxq_info xdp_rxq; + + struct xdp_buff **xsk_buffs; } xsk; }; diff --git a/drivers/net/virtio/xsk.c b/drivers/net/virtio/xsk.c index 0c6a8f92ae38..c54cd08e9c77 100644 --- a/drivers/net/virtio/xsk.c +++ b/drivers/net/virtio/xsk.c @@ -14,6 +14,47 @@ static void sg_fill_dma(struct scatterlist *sg, dma_addr_t addr, u32 len) sg->length = len; } +int virtnet_add_recvbuf_xsk(struct virtnet_info *vi, struct virtnet_rq *rq, + struct xsk_buff_pool *pool, gfp_t gfp) +{ + struct xdp_buff **xsk_buffs; + dma_addr_t addr; + u32 len, i; + int err = 0; + int num; + + xsk_buffs = rq->xsk.xsk_buffs; + + num = xsk_buff_alloc_batch(pool, xsk_buffs, rq->vq->num_free); + if (!num) + return -ENOMEM; + + len = xsk_pool_get_rx_frame_size(pool) + vi->hdr_len; + + for (i = 0; i < num; ++i) { + /* use the part of XDP_PACKET_HEADROOM as the virtnet hdr space */ + addr = xsk_buff_xdp_get_dma(xsk_buffs[i]) - vi->hdr_len; + + sg_init_table(rq->sg, 1); + sg_fill_dma(rq->sg, addr, len); + + err = virtqueue_add_inbuf(rq->vq, rq->sg, 1, xsk_buffs[i], gfp); + if (err) + goto err; + } + + return num; + +err: + if (i) + err = i; + + for (; i < num; ++i) + xsk_buff_free(xsk_buffs[i]); + + return err; +} + static int virtnet_xsk_xmit_one(struct virtnet_sq *sq, struct xsk_buff_pool *pool, struct xdp_desc *desc) @@ -210,7 +251,7 @@ static int virtnet_xsk_pool_enable(struct net_device *dev, struct virtnet_sq *sq; struct device *dma_dev; dma_addr_t hdr_dma; - int err; + int err, size; /* In big_packets mode, xdp cannot work, so there is no need to * initialize xsk of rq. @@ -246,6 +287,12 @@ static int virtnet_xsk_pool_enable(struct net_device *dev, if (!dma_dev) return -EPERM; + size = virtqueue_get_vring_size(rq->vq); + + rq->xsk.xsk_buffs = kcalloc(size, sizeof(*rq->xsk.xsk_buffs), GFP_KERNEL); + if (!rq->xsk.xsk_buffs) + return -ENOMEM; + hdr_dma = dma_map_single(dma_dev, &xsk_hdr, vi->hdr_len, DMA_TO_DEVICE); if (dma_mapping_error(dma_dev, hdr_dma)) return -ENOMEM; @@ -304,6 +351,8 @@ static int virtnet_xsk_pool_disable(struct net_device *dev, u16 qid) dma_unmap_single(dma_dev, sq->xsk.hdr_dma_address, vi->hdr_len, DMA_TO_DEVICE); + kfree(rq->xsk.xsk_buffs); + return err1 | err2; } diff --git a/drivers/net/virtio/xsk.h b/drivers/net/virtio/xsk.h index 7ebc9bda7aee..bef41a3f954e 100644 --- a/drivers/net/virtio/xsk.h +++ b/drivers/net/virtio/xsk.h @@ -23,4 +23,6 @@ int virtnet_xsk_pool_setup(struct net_device *dev, struct netdev_bpf *xdp); bool virtnet_xsk_xmit(struct virtnet_sq *sq, struct xsk_buff_pool *pool, int budget); int virtnet_xsk_wakeup(struct net_device *dev, u32 qid, u32 flag); +int virtnet_add_recvbuf_xsk(struct virtnet_info *vi, struct virtnet_rq *rq, + struct xsk_buff_pool *pool, gfp_t gfp); #endif From patchwork Tue Jan 16 09:43:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13520629 X-Patchwork-Delegate: kuba@kernel.org Received: from out30-111.freemail.mail.aliyun.com (out30-111.freemail.mail.aliyun.com [115.124.30.111]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 129671B95B; Tue, 16 Jan 2024 09:43:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R141e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046050;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=14;SR=0;TI=SMTPD_---0W-lmFUh_1705398207; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W-lmFUh_1705398207) by smtp.aliyun-inc.com; Tue, 16 Jan 2024 17:43:28 +0800 From: Xuan Zhuo To: netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , virtualization@lists.linux.dev, bpf@vger.kernel.org Subject: [PATCH net-next 13/17] virtio_net: xsk: rx: support recv merge mode Date: Tue, 16 Jan 2024 17:43:09 +0800 Message-Id: <20240116094313.119939-14-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240116094313.119939-1-xuanzhuo@linux.alibaba.com> References: <20240116094313.119939-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: 1913ebd4ae28 X-Patchwork-Delegate: kuba@kernel.org The virtnet_xdp_handler() is re-used. But 1. We need to copy data to create skb for XDP_PASS. 2. We need to call xsk_buff_free() to release the buffer. 3. The handle for xdp_buff is difference. If we pushed this logic into existing receive handle(merge and small), we would have to maintain code scattered inside merge and small (and big). So I think it is a good choice for us to put the xsk code into an independent function. Signed-off-by: Xuan Zhuo --- drivers/net/virtio/main.c | 40 ++++-- drivers/net/virtio/virtio_net.h | 4 + drivers/net/virtio/xsk.c | 216 ++++++++++++++++++++++++++++++++ drivers/net/virtio/xsk.h | 4 + 4 files changed, 251 insertions(+), 13 deletions(-) diff --git a/drivers/net/virtio/main.c b/drivers/net/virtio/main.c index fb8dbce11e53..35f0daaea7e5 100644 --- a/drivers/net/virtio/main.c +++ b/drivers/net/virtio/main.c @@ -801,10 +801,10 @@ static void put_xdp_frags(struct xdp_buff *xdp) } } -static int virtnet_xdp_handler(struct bpf_prog *xdp_prog, struct xdp_buff *xdp, - struct net_device *dev, - unsigned int *xdp_xmit, - struct virtnet_rq_stats *stats) +int virtnet_xdp_handler(struct bpf_prog *xdp_prog, struct xdp_buff *xdp, + struct net_device *dev, + unsigned int *xdp_xmit, + struct virtnet_rq_stats *stats) { struct xdp_frame *xdpf; int err; @@ -1897,23 +1897,37 @@ static int virtnet_receive(struct virtnet_rq *rq, int budget, { struct virtnet_info *vi = rq->vq->vdev->priv; struct virtnet_rq_stats stats = {}; + void *buf, *ctx, *pctx = NULL; unsigned int len; int packets = 0; - void *buf; int i; - if (!vi->big_packets || vi->mergeable_rx_bufs) { - void *ctx; + if (rq->xsk.pool) { + struct sk_buff *skb; + + while (packets < budget) { + buf = virtqueue_get_buf(rq->vq, &len); + if (!buf) + break; + + skb = virtnet_receive_xsk_buf(vi, rq, buf, len, xdp_xmit, &stats); + if (skb) + virtnet_receive_done(vi, rq, skb); - while (packets < budget && - (buf = virtnet_rq_get_buf(rq, &len, &ctx))) { - receive_buf(vi, rq, buf, len, ctx, xdp_xmit, &stats); packets++; } } else { - while (packets < budget && - (buf = virtnet_rq_get_buf(rq, &len, NULL)) != NULL) { - receive_buf(vi, rq, buf, len, NULL, xdp_xmit, &stats); + if (!vi->big_packets || vi->mergeable_rx_bufs) + pctx = &ctx; + else + ctx = NULL; + + while (packets < budget) { + buf = virtnet_rq_get_buf(rq, &len, pctx); + if (!buf) + break; + + receive_buf(vi, rq, buf, len, ctx, xdp_xmit, &stats); packets++; } } diff --git a/drivers/net/virtio/virtio_net.h b/drivers/net/virtio/virtio_net.h index c07a1e58bf0f..98dcf1825448 100644 --- a/drivers/net/virtio/virtio_net.h +++ b/drivers/net/virtio/virtio_net.h @@ -345,4 +345,8 @@ struct sk_buff *virtnet_skb_append_frag(struct sk_buff *head_skb, struct sk_buff *curr_skb, struct page *page, void *buf, int len, int truesize); +int virtnet_xdp_handler(struct bpf_prog *xdp_prog, struct xdp_buff *xdp, + struct net_device *dev, + unsigned int *xdp_xmit, + struct virtnet_rq_stats *stats); #endif diff --git a/drivers/net/virtio/xsk.c b/drivers/net/virtio/xsk.c index c54cd08e9c77..6f4fa32b1184 100644 --- a/drivers/net/virtio/xsk.c +++ b/drivers/net/virtio/xsk.c @@ -14,6 +14,222 @@ static void sg_fill_dma(struct scatterlist *sg, dma_addr_t addr, u32 len) sg->length = len; } +static void xsk_drop_follow_bufs(struct net_device *dev, + struct virtnet_rq *rq, + u32 num_buf, + struct virtnet_rq_stats *stats) +{ + struct xdp_buff *xdp; + u32 len; + + while (num_buf-- > 1) { + xdp = virtqueue_get_buf(rq->vq, &len); + if (unlikely(!xdp)) { + pr_debug("%s: rx error: %d buffers missing\n", + dev->name, num_buf); + DEV_STATS_INC(dev, rx_length_errors); + break; + } + u64_stats_add(&stats->bytes, len); + xsk_buff_free(xdp); + } +} + +static struct xdp_buff *buf_to_xdp(struct virtnet_info *vi, + struct virtnet_rq *rq, void *buf, u32 len) +{ + struct xdp_buff *xdp; + u32 bufsize; + + xdp = (struct xdp_buff *)buf; + + bufsize = xsk_pool_get_rx_frame_size(rq->xsk.pool) + vi->hdr_len; + + if (unlikely(len > bufsize)) { + pr_debug("%s: rx error: len %u exceeds truesize %u\n", + vi->dev->name, len, bufsize); + DEV_STATS_INC(vi->dev, rx_length_errors); + xsk_buff_free(xdp); + return NULL; + } + + xsk_buff_set_size(xdp, len); + xsk_buff_dma_sync_for_cpu(xdp, rq->xsk.pool); + + return xdp; +} + +static int xsk_append_merge_buffer(struct virtnet_info *vi, + struct virtnet_rq *rq, + struct sk_buff *head_skb, + u32 num_buf, + struct virtio_net_hdr_mrg_rxbuf *hdr, + struct virtnet_rq_stats *stats) +{ + struct sk_buff *curr_skb; + struct xdp_buff *xdp; + u32 len, truesize; + struct page *page; + void *buf; + + curr_skb = head_skb; + + while (--num_buf) { + buf = virtqueue_get_buf(rq->vq, &len); + if (unlikely(!buf)) { + pr_debug("%s: rx error: %d buffers out of %d missing\n", + vi->dev->name, num_buf, + virtio16_to_cpu(vi->vdev, + hdr->num_buffers)); + DEV_STATS_INC(vi->dev, rx_length_errors); + return -EINVAL; + } + + u64_stats_add(&stats->bytes, len); + + xdp = buf_to_xdp(vi, rq, buf, len); + if (!xdp) + goto err; + + buf = napi_alloc_frag(len); + if (!buf) { + xsk_buff_free(xdp); + goto err; + } + + memcpy(buf, xdp->data - vi->hdr_len, len); + + xsk_buff_free(xdp); + + page = virt_to_page(buf); + + truesize = len; + + curr_skb = virtnet_skb_append_frag(head_skb, curr_skb, page, + buf, len, truesize); + if (!curr_skb) { + put_page(page); + goto err; + } + } + + return 0; + +err: + xsk_drop_follow_bufs(vi->dev, rq, num_buf, stats); + return -EINVAL; +} + +static struct sk_buff *xdp_construct_skb(struct virtnet_rq *rq, + struct xdp_buff *xdp) +{ + unsigned int metasize = xdp->data - xdp->data_meta; + struct sk_buff *skb; + unsigned int size; + + size = xdp->data_end - xdp->data_hard_start; + skb = napi_alloc_skb(&rq->napi, size); + if (unlikely(!skb)) { + xsk_buff_free(xdp); + return NULL; + } + + skb_reserve(skb, xdp->data_meta - xdp->data_hard_start); + + size = xdp->data_end - xdp->data_meta; + memcpy(__skb_put(skb, size), xdp->data_meta, size); + + if (metasize) { + __skb_pull(skb, metasize); + skb_metadata_set(skb, metasize); + } + + xsk_buff_free(xdp); + + return skb; +} + +static struct sk_buff *virtnet_receive_xsk_merge(struct net_device *dev, struct virtnet_info *vi, + struct virtnet_rq *rq, struct xdp_buff *xdp, + unsigned int *xdp_xmit, + struct virtnet_rq_stats *stats) +{ + struct virtio_net_hdr_mrg_rxbuf *hdr; + struct bpf_prog *prog; + struct sk_buff *skb; + u32 ret, num_buf; + + hdr = xdp->data - vi->hdr_len; + num_buf = virtio16_to_cpu(vi->vdev, hdr->num_buffers); + + ret = XDP_PASS; + rcu_read_lock(); + prog = rcu_dereference(rq->xdp_prog); + /* TODO: support multi buffer. */ + if (prog && num_buf == 1) + ret = virtnet_xdp_handler(prog, xdp, dev, xdp_xmit, stats); + rcu_read_unlock(); + + switch (ret) { + case XDP_PASS: + skb = xdp_construct_skb(rq, xdp); + if (!skb) + goto drop_bufs; + + if (xsk_append_merge_buffer(vi, rq, skb, num_buf, hdr, stats)) { + dev_kfree_skb(skb); + goto drop; + } + + return skb; + + case XDP_TX: + case XDP_REDIRECT: + return NULL; + + default: + /* drop packet */ + xsk_buff_free(xdp); + } + +drop_bufs: + xsk_drop_follow_bufs(dev, rq, num_buf, stats); + +drop: + u64_stats_inc(&stats->drops); + return NULL; +} + +struct sk_buff *virtnet_receive_xsk_buf(struct virtnet_info *vi, struct virtnet_rq *rq, + void *buf, u32 len, + unsigned int *xdp_xmit, + struct virtnet_rq_stats *stats) +{ + struct net_device *dev = vi->dev; + struct sk_buff *skb = NULL; + struct xdp_buff *xdp; + + len -= vi->hdr_len; + + u64_stats_add(&stats->bytes, len); + + xdp = buf_to_xdp(vi, rq, buf, len); + if (!xdp) + return NULL; + + if (unlikely(len < ETH_HLEN)) { + pr_debug("%s: short packet %i\n", dev->name, len); + DEV_STATS_INC(dev, rx_length_errors); + xsk_buff_free(xdp); + return NULL; + } + + if (vi->mergeable_rx_bufs) + skb = virtnet_receive_xsk_merge(dev, vi, rq, xdp, xdp_xmit, stats); + + return skb; +} + int virtnet_add_recvbuf_xsk(struct virtnet_info *vi, struct virtnet_rq *rq, struct xsk_buff_pool *pool, gfp_t gfp) { diff --git a/drivers/net/virtio/xsk.h b/drivers/net/virtio/xsk.h index bef41a3f954e..e78fcd0a4946 100644 --- a/drivers/net/virtio/xsk.h +++ b/drivers/net/virtio/xsk.h @@ -25,4 +25,8 @@ bool virtnet_xsk_xmit(struct virtnet_sq *sq, struct xsk_buff_pool *pool, int virtnet_xsk_wakeup(struct net_device *dev, u32 qid, u32 flag); int virtnet_add_recvbuf_xsk(struct virtnet_info *vi, struct virtnet_rq *rq, struct xsk_buff_pool *pool, gfp_t gfp); +struct sk_buff *virtnet_receive_xsk_buf(struct virtnet_info *vi, struct virtnet_rq *rq, + void *buf, u32 len, + unsigned int *xdp_xmit, + struct virtnet_rq_stats *stats); #endif From patchwork Tue Jan 16 09:43:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13520630 X-Patchwork-Delegate: kuba@kernel.org Received: from out30-111.freemail.mail.aliyun.com (out30-111.freemail.mail.aliyun.com [115.124.30.111]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D4DC11B966; Tue, 16 Jan 2024 09:43:31 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R841e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045176;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=14;SR=0;TI=SMTPD_---0W-lmFVF_1705398208; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W-lmFVF_1705398208) by smtp.aliyun-inc.com; Tue, 16 Jan 2024 17:43:29 +0800 From: Xuan Zhuo To: netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , virtualization@lists.linux.dev, bpf@vger.kernel.org Subject: [PATCH net-next 14/17] virtio_net: xsk: rx: support recv small mode Date: Tue, 16 Jan 2024 17:43:10 +0800 Message-Id: <20240116094313.119939-15-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240116094313.119939-1-xuanzhuo@linux.alibaba.com> References: <20240116094313.119939-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: 1913ebd4ae28 X-Patchwork-Delegate: kuba@kernel.org receive the xsk buffer for small mode. Signed-off-by: Xuan Zhuo --- drivers/net/virtio/xsk.c | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/drivers/net/virtio/xsk.c b/drivers/net/virtio/xsk.c index 6f4fa32b1184..a73559faadf6 100644 --- a/drivers/net/virtio/xsk.c +++ b/drivers/net/virtio/xsk.c @@ -200,6 +200,37 @@ static struct sk_buff *virtnet_receive_xsk_merge(struct net_device *dev, struct return NULL; } +static struct sk_buff *virtnet_receive_xsk_small(struct net_device *dev, struct virtnet_info *vi, + struct virtnet_rq *rq, struct xdp_buff *xdp, + unsigned int *xdp_xmit, + struct virtnet_rq_stats *stats) +{ + struct bpf_prog *prog; + u32 ret; + + ret = XDP_PASS; + rcu_read_lock(); + prog = rcu_dereference(rq->xdp_prog); + if (prog) + ret = virtnet_xdp_handler(prog, xdp, dev, xdp_xmit, stats); + rcu_read_unlock(); + + switch (ret) { + case XDP_PASS: + return xdp_construct_skb(rq, xdp); + + case XDP_TX: + case XDP_REDIRECT: + return NULL; + + default: + /* drop packet */ + xsk_buff_free(xdp); + u64_stats_inc(&stats->drops); + return NULL; + } +} + struct sk_buff *virtnet_receive_xsk_buf(struct virtnet_info *vi, struct virtnet_rq *rq, void *buf, u32 len, unsigned int *xdp_xmit, @@ -226,6 +257,8 @@ struct sk_buff *virtnet_receive_xsk_buf(struct virtnet_info *vi, struct virtnet_ if (vi->mergeable_rx_bufs) skb = virtnet_receive_xsk_merge(dev, vi, rq, xdp, xdp_xmit, stats); + else + skb = virtnet_receive_xsk_small(dev, vi, rq, xdp, xdp_xmit, stats); return skb; } From patchwork Tue Jan 16 09:43:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13520631 X-Patchwork-Delegate: kuba@kernel.org Received: from out30-101.freemail.mail.aliyun.com (out30-101.freemail.mail.aliyun.com [115.124.30.101]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4E9781BC25; Tue, 16 Jan 2024 09:43:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R201e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045192;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=14;SR=0;TI=SMTPD_---0W-lmyA0_1705398209; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W-lmyA0_1705398209) by smtp.aliyun-inc.com; Tue, 16 Jan 2024 17:43:30 +0800 From: Xuan Zhuo To: netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , virtualization@lists.linux.dev, bpf@vger.kernel.org Subject: [PATCH net-next 15/17] virtio_net: xsk: rx: free the unused xsk buffer Date: Tue, 16 Jan 2024 17:43:11 +0800 Message-Id: <20240116094313.119939-16-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240116094313.119939-1-xuanzhuo@linux.alibaba.com> References: <20240116094313.119939-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: 1913ebd4ae28 X-Patchwork-Delegate: kuba@kernel.org Since this will be called in other circumstances(freeze), we must check whether it is xsk's buffer in this function. It cannot be judged outside this function. Signed-off-by: Xuan Zhuo --- drivers/net/virtio/main.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/drivers/net/virtio/main.c b/drivers/net/virtio/main.c index 35f0daaea7e5..81d390781a4f 100644 --- a/drivers/net/virtio/main.c +++ b/drivers/net/virtio/main.c @@ -4146,6 +4146,14 @@ void virtnet_rq_free_unused_bufs(struct virtqueue *vq) rq = &vi->rq[i]; while ((buf = virtqueue_detach_unused_buf(vq)) != NULL) { + if (rq->xsk.pool) { + struct xdp_buff *xdp; + + xdp = (struct xdp_buff *)buf; + xsk_buff_free(xdp); + continue; + } + if (virtqueue_get_dma_premapped(rq->vq)) virtnet_rq_unmap(rq, buf, 0); From patchwork Tue Jan 16 09:43:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13520632 X-Patchwork-Delegate: kuba@kernel.org Received: from out30-133.freemail.mail.aliyun.com (out30-133.freemail.mail.aliyun.com [115.124.30.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EBCFB1BC39; Tue, 16 Jan 2024 09:43:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R511e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045168;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=14;SR=0;TI=SMTPD_---0W-lnfDx_1705398210; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W-lnfDx_1705398210) by smtp.aliyun-inc.com; Tue, 16 Jan 2024 17:43:31 +0800 From: Xuan Zhuo To: netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , virtualization@lists.linux.dev, bpf@vger.kernel.org Subject: [PATCH net-next 16/17] virtio_net: update tx timeout record Date: Tue, 16 Jan 2024 17:43:12 +0800 Message-Id: <20240116094313.119939-17-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240116094313.119939-1-xuanzhuo@linux.alibaba.com> References: <20240116094313.119939-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: 1913ebd4ae28 X-Patchwork-Delegate: kuba@kernel.org If send queue sent some packets, we update the tx timeout record to prevent the tx timeout. Signed-off-by: Xuan Zhuo Acked-by: Jason Wang --- drivers/net/virtio/xsk.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/drivers/net/virtio/xsk.c b/drivers/net/virtio/xsk.c index a73559faadf6..1ecb7bf626e5 100644 --- a/drivers/net/virtio/xsk.c +++ b/drivers/net/virtio/xsk.c @@ -376,6 +376,13 @@ bool virtnet_xsk_xmit(struct virtnet_sq *sq, struct xsk_buff_pool *pool, if (!virtnet_is_xdp_raw_buffer_queue(vi, sq - vi->sq)) virtnet_check_sq_full_and_disable(vi, vi->dev, sq); + if (packets) { + struct netdev_queue *txq; + + txq = netdev_get_tx_queue(vi->dev, sq - vi->sq); + txq_trans_cond_update(txq); + } + u64_stats_update_begin(&sq->stats.syncp); u64_stats_add(&sq->stats.packets, packets); u64_stats_add(&sq->stats.bytes, bytes); From patchwork Tue Jan 16 09:43:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13520633 X-Patchwork-Delegate: kuba@kernel.org Received: from out30-110.freemail.mail.aliyun.com (out30-110.freemail.mail.aliyun.com [115.124.30.110]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1140C1BC49; Tue, 16 Jan 2024 09:43:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R111e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045170;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=14;SR=0;TI=SMTPD_---0W-lj1bN_1705398211; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W-lj1bN_1705398211) by smtp.aliyun-inc.com; Tue, 16 Jan 2024 17:43:32 +0800 From: Xuan Zhuo To: netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , virtualization@lists.linux.dev, bpf@vger.kernel.org Subject: [PATCH net-next 17/17] virtio_net: xdp_features add NETDEV_XDP_ACT_XSK_ZEROCOPY Date: Tue, 16 Jan 2024 17:43:13 +0800 Message-Id: <20240116094313.119939-18-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240116094313.119939-1-xuanzhuo@linux.alibaba.com> References: <20240116094313.119939-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: 1913ebd4ae28 X-Patchwork-Delegate: kuba@kernel.org Now, we supported AF_XDP(xsk). Add NETDEV_XDP_ACT_XSK_ZEROCOPY to xdp_features. Signed-off-by: Xuan Zhuo --- drivers/net/virtio/main.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/drivers/net/virtio/main.c b/drivers/net/virtio/main.c index 81d390781a4f..176d549b0c80 100644 --- a/drivers/net/virtio/main.c +++ b/drivers/net/virtio/main.c @@ -4563,7 +4563,8 @@ static int virtnet_probe(struct virtio_device *vdev) dev->hw_features |= NETIF_F_GRO_HW; dev->vlan_features = dev->features; - dev->xdp_features = NETDEV_XDP_ACT_BASIC | NETDEV_XDP_ACT_REDIRECT; + dev->xdp_features = NETDEV_XDP_ACT_BASIC | NETDEV_XDP_ACT_REDIRECT | + NETDEV_XDP_ACT_XSK_ZEROCOPY; /* MTU range: 68 - 65535 */ dev->min_mtu = MIN_MTU;