From patchwork Wed May 8 02:23:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13658025 Received: from out30-112.freemail.mail.aliyun.com (out30-112.freemail.mail.aliyun.com [115.124.30.112]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0A34FAD51; Wed, 8 May 2024 02:23:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.112 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715135021; cv=none; b=n/guA52rxDnpHZeRnErHr9axv1cgqJdUCEDmni+chp+R9F3G6IEpQ7w6H5YhXxTgo2Ia06ThnlOiyMZbtUROlBEg7uwwKRrJBtncsJ1uuvox5EPxQSPzkkw3WwHyGTY7w4+065n50xXeNQa1vNqlU4QUowRnPWUyFmsOILnWjGU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715135021; c=relaxed/simple; bh=uSsC4xDTN3LDP3RFkCWZ2rhdoU0+wzZ9rlLw7WMlxgk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=dfJ6+ASUY2lfCdRJWoH9yXhETERNLUH77Y4Y1Hyhc7XxVo7hjhF+qgbb0GHiTyxCqQ18M/TL7ycwUcOwhUcxYr4KT/pXX//3dfTnHZmFtpM+mbkYI7BdN+tRffD3M8xMOZrOf8JAcO7CDe7D6XaiGVHfTCQx+77MnILMm6NQN3c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=Wj6WR/7A; arc=none smtp.client-ip=115.124.30.112 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="Wj6WR/7A" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1715135015; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=F4tNh2S1osiccFF9Xd2gla4ltUsL6ZG1ZCfJ9TJshpU=; b=Wj6WR/7ANbOavJxL2zDfBO/IhRrjRjraTT600Z/PGvhoyt+h5yCJ80zqkzaVtUgSjXdyIyzdkYJiM2DSB6ij55LaRXatnJBTdDmq2HRn8O/f1kRqoC2nwN/E8YIuYHJmx0O+f88z9tv927e6L3pTHldrey1i4w1PsKmgX37Q0ko= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R101e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033037067112;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=14;SR=0;TI=SMTPD_---0W61qWpS_1715135013; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W61qWpS_1715135013) by smtp.aliyun-inc.com; Wed, 08 May 2024 10:23:34 +0800 From: Xuan Zhuo To: virtualization@lists.linux.dev Cc: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , netdev@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH vhost 1/5] virtio_ring: introduce vring_need_unmap_buffer Date: Wed, 8 May 2024 10:23:27 +0800 Message-Id: <20240508022331.63751-2-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240508022331.63751-1-xuanzhuo@linux.alibaba.com> References: <20240508022331.63751-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: a0fef46c457b To make the code readable, introduce vring_need_unmap_buffer() to replace do_unmap. use_dma_api premapped -> vring_need_unmap_buffer() 1. false false false 2. true false true 3. true true false Signed-off-by: Xuan Zhuo Acked-by: Jason Wang --- drivers/virtio/virtio_ring.c | 27 ++++++++++++--------------- 1 file changed, 12 insertions(+), 15 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index a939104d551f..d7059bacb593 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -175,11 +175,6 @@ struct vring_virtqueue { /* Do DMA mapping by driver */ bool premapped; - /* Do unmap or not for desc. Just when premapped is False and - * use_dma_api is true, this is true. - */ - bool do_unmap; - /* Head of free buffer list. */ unsigned int free_head; /* Number we've added since last sync. */ @@ -295,6 +290,11 @@ static bool vring_use_dma_api(const struct virtio_device *vdev) return false; } +static bool vring_need_unmap_buffer(const struct vring_virtqueue *vring) +{ + return vring->use_dma_api && !vring->premapped; +} + size_t virtio_max_dma_size(const struct virtio_device *vdev) { size_t max_segment_size = SIZE_MAX; @@ -443,7 +443,7 @@ static void vring_unmap_one_split_indirect(const struct vring_virtqueue *vq, { u16 flags; - if (!vq->do_unmap) + if (!vring_need_unmap_buffer(vq)) return; flags = virtio16_to_cpu(vq->vq.vdev, desc->flags); @@ -473,7 +473,7 @@ static unsigned int vring_unmap_one_split(const struct vring_virtqueue *vq, (flags & VRING_DESC_F_WRITE) ? DMA_FROM_DEVICE : DMA_TO_DEVICE); } else { - if (!vq->do_unmap) + if (!vring_need_unmap_buffer(vq)) goto out; dma_unmap_page(vring_dma_dev(vq), @@ -641,7 +641,7 @@ static inline int virtqueue_add_split(struct virtqueue *_vq, } /* Last one doesn't continue. */ desc[prev].flags &= cpu_to_virtio16(_vq->vdev, ~VRING_DESC_F_NEXT); - if (!indirect && vq->do_unmap) + if (!indirect && vring_need_unmap_buffer(vq)) vq->split.desc_extra[prev & (vq->split.vring.num - 1)].flags &= ~VRING_DESC_F_NEXT; @@ -800,7 +800,7 @@ static void detach_buf_split(struct vring_virtqueue *vq, unsigned int head, VRING_DESC_F_INDIRECT)); BUG_ON(len == 0 || len % sizeof(struct vring_desc)); - if (vq->do_unmap) { + if (vring_need_unmap_buffer(vq)) { for (j = 0; j < len / sizeof(struct vring_desc); j++) vring_unmap_one_split_indirect(vq, &indir_desc[j]); } @@ -1230,7 +1230,7 @@ static void vring_unmap_extra_packed(const struct vring_virtqueue *vq, (flags & VRING_DESC_F_WRITE) ? DMA_FROM_DEVICE : DMA_TO_DEVICE); } else { - if (!vq->do_unmap) + if (!vring_need_unmap_buffer(vq)) return; dma_unmap_page(vring_dma_dev(vq), @@ -1245,7 +1245,7 @@ static void vring_unmap_desc_packed(const struct vring_virtqueue *vq, { u16 flags; - if (!vq->do_unmap) + if (!vring_need_unmap_buffer(vq)) return; flags = le16_to_cpu(desc->flags); @@ -1626,7 +1626,7 @@ static void detach_buf_packed(struct vring_virtqueue *vq, if (!desc) return; - if (vq->do_unmap) { + if (vring_need_unmap_buffer(vq)) { len = vq->packed.desc_extra[id].len; for (i = 0; i < len / sizeof(struct vring_packed_desc); i++) @@ -2080,7 +2080,6 @@ static struct virtqueue *vring_create_virtqueue_packed(struct virtio_device *vde vq->dma_dev = dma_dev; vq->use_dma_api = vring_use_dma_api(vdev); vq->premapped = false; - vq->do_unmap = vq->use_dma_api; vq->indirect = virtio_has_feature(vdev, VIRTIO_RING_F_INDIRECT_DESC) && !cfg_vq_get(cfg, vq, ctx); @@ -2621,7 +2620,6 @@ static struct virtqueue *__vring_new_virtqueue(struct virtio_device *vdev, vq->dma_dev = tp_cfg->dma_dev; vq->use_dma_api = vring_use_dma_api(vdev); vq->premapped = false; - vq->do_unmap = vq->use_dma_api; vq->indirect = virtio_has_feature(vdev, VIRTIO_RING_F_INDIRECT_DESC) && !cfg_vq_get(cfg, vq, ctx); @@ -2747,7 +2745,6 @@ int virtqueue_set_dma_premapped(struct virtqueue *_vq) } vq->premapped = true; - vq->do_unmap = false; END_USE(vq); From patchwork Wed May 8 02:23:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13658026 Received: from out30-97.freemail.mail.aliyun.com (out30-97.freemail.mail.aliyun.com [115.124.30.97]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 73BD9BA2E; Wed, 8 May 2024 02:23:39 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.97 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715135021; cv=none; b=lQg6ozmzta9SvJzemLyt9JNYWyyML8UXpwvf3Wqk6KnPXE0Qbe8VI9iywwYyKDSKZr86JJfmyJS6m8Ok7ppROmlJNMtvuj+kS/x6UyiTLkhaI3vMpbxQ3v4VCQ5dAXyEnZ9VuCD+xYXe+NajfSs+Jz6ScRwTbNhWvYSldwoCtN4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715135021; c=relaxed/simple; bh=6QiS0/5CmZ0pf2KdZ/qXZV/mvc3EUYF6n+eahZDi2Hk=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=CPgRT7V3HvqZOb6WwumKUySfaFCAO37nBswl+jmu+RgyhbDA40BEGGCUf3E3RW/XssdUF4shXx1kYWzvbBhLBzKDAdUad5FpDfPqXkaGVXHU4hgWQM1n1oWLyA1CLZCCFKXZviwva/XI1zTUgSPtyDp8PEkhNkZE7UR3DkNfoAo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=uMgItCbJ; arc=none smtp.client-ip=115.124.30.97 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="uMgItCbJ" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1715135016; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=nOMJaXwRPFppSnIk7SgaJEXuyHcqypZO3nEf/f3kmLk=; b=uMgItCbJ8l6UArY6xz6DQYWVEQsfvjWdh54xCpLPIZY35cnYZGmxHS+PoMuamuMbIo6k0FW6+6AxCgT5HIKX85CP+Rj04U/1bb6En5x5BNaMiMTcqoEg/UUexdwcE30eFruoj2fTBoXyTZiDIjHe8M3iFD2v5aax28P/ISKKFMo= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R951e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033022160150;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=14;SR=0;TI=SMTPD_---0W61ibMG_1715135014; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W61ibMG_1715135014) by smtp.aliyun-inc.com; Wed, 08 May 2024 10:23:36 +0800 From: Xuan Zhuo To: virtualization@lists.linux.dev Cc: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , netdev@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH vhost 2/5] virtio_ring: introduce dma map api for page Date: Wed, 8 May 2024 10:23:28 +0800 Message-Id: <20240508022331.63751-3-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240508022331.63751-1-xuanzhuo@linux.alibaba.com> References: <20240508022331.63751-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: a0fef46c457b The virtio-net sq will use these APIs to map the scatterlist. For scatterlist, the page dma APIs are more appropriate. dma_addr_t virtqueue_dma_map_page_attrs(struct virtqueue *_vq, struct page *page, size_t offset, size_t size, enum dma_data_direction dir, unsigned long attrs); void virtqueue_dma_unmap_page_attrs(struct virtqueue *_vq, dma_addr_t addr, size_t size, enum dma_data_direction dir, unsigned long attrs); Signed-off-by: Xuan Zhuo --- drivers/virtio/virtio_ring.c | 52 ++++++++++++++++++++++++++++++++++++ include/linux/virtio.h | 7 +++++ 2 files changed, 59 insertions(+) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index d7059bacb593..653c7ea24fb7 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -3092,6 +3092,58 @@ void virtqueue_dma_unmap_single_attrs(struct virtqueue *_vq, dma_addr_t addr, } EXPORT_SYMBOL_GPL(virtqueue_dma_unmap_single_attrs); +/** + * virtqueue_dma_map_page_attrs - map DMA for _vq + * @_vq: the struct virtqueue we're talking about. + * @page: the page to do dma + * @offset: the offset inside the page + * @size: the size of the page to do dma + * @dir: DMA direction + * @attrs: DMA Attrs + * + * The caller calls this to do dma mapping in advance. The DMA address can be + * passed to this _vq when it is in pre-mapped mode. + * + * return DMA address. Caller should check that by virtqueue_dma_mapping_error(). + */ +dma_addr_t virtqueue_dma_map_page_attrs(struct virtqueue *_vq, struct page *page, + size_t offset, size_t size, + enum dma_data_direction dir, + unsigned long attrs) +{ + struct vring_virtqueue *vq = to_vvq(_vq); + + if (!vq->use_dma_api) + return page_to_phys(page) + offset; + + return dma_map_page_attrs(vring_dma_dev(vq), page, offset, size, dir, attrs); +} +EXPORT_SYMBOL_GPL(virtqueue_dma_map_page_attrs); + +/** + * virtqueue_dma_unmap_page_attrs - unmap DMA for _vq + * @_vq: the struct virtqueue we're talking about. + * @addr: the dma address to unmap + * @size: the size of the buffer + * @dir: DMA direction + * @attrs: DMA Attrs + * + * Unmap the address that is mapped by the virtqueue_dma_map_* APIs. + * + */ +void virtqueue_dma_unmap_page_attrs(struct virtqueue *_vq, dma_addr_t addr, + size_t size, enum dma_data_direction dir, + unsigned long attrs) +{ + struct vring_virtqueue *vq = to_vvq(_vq); + + if (!vq->use_dma_api) + return; + + dma_unmap_page_attrs(vring_dma_dev(vq), addr, size, dir, attrs); +} +EXPORT_SYMBOL_GPL(virtqueue_dma_unmap_page_attrs); + /** * virtqueue_dma_mapping_error - check dma address * @_vq: the struct virtqueue we're talking about. diff --git a/include/linux/virtio.h b/include/linux/virtio.h index b0201747a263..d92aa7be2c44 100644 --- a/include/linux/virtio.h +++ b/include/linux/virtio.h @@ -225,6 +225,13 @@ dma_addr_t virtqueue_dma_map_single_attrs(struct virtqueue *_vq, void *ptr, size void virtqueue_dma_unmap_single_attrs(struct virtqueue *_vq, dma_addr_t addr, size_t size, enum dma_data_direction dir, unsigned long attrs); +dma_addr_t virtqueue_dma_map_page_attrs(struct virtqueue *_vq, struct page *page, + size_t offset, size_t size, + enum dma_data_direction dir, + unsigned long attrs); +void virtqueue_dma_unmap_page_attrs(struct virtqueue *_vq, dma_addr_t addr, + size_t size, enum dma_data_direction dir, + unsigned long attrs); int virtqueue_dma_mapping_error(struct virtqueue *_vq, dma_addr_t addr); bool virtqueue_dma_need_sync(struct virtqueue *_vq, dma_addr_t addr); From patchwork Wed May 8 02:23:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13658029 Received: from out30-124.freemail.mail.aliyun.com (out30-124.freemail.mail.aliyun.com [115.124.30.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CD6B21A2C25; Wed, 8 May 2024 02:23:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715135028; cv=none; b=gvMsGpz6c8TSMR003aBXG5Zk6d8pvGcTHGYztnF7vHUZOT2eue+zJDVF6IFODwuU9YjJjUKalJg/AOAkXUQNMusnueDzk5ySj8nhGfoH0AoEYrn9SzazxjmKGguvmYGj7ObloYtinF2OJpn6FG55t95SvZ1KZsDmR69slCyHiPk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715135028; c=relaxed/simple; bh=D0LZB1jqUO4M2Lu/Kb0xMb37tvHIxAOAL5WwMRt0I2I=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=MBdt088zmjHnwkLB+XxNlD0B5wbc6cLAEBQTF9cbm+ElWWGzIkgruFwSX/+NOKkgI2OIGJlPWL3kitxbLr1aStSuvn0rq+hHzlfsNKJC+TVYP1YDKbjW1qHq+VmsHmJ/53xR1D2GRDYIc3182Kc8EwAJu/wXoSTaJhbI6/2ZDI4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=EHGcPwfa; arc=none smtp.client-ip=115.124.30.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="EHGcPwfa" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1715135018; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=SguPATYh4XBUzhYH0ZZ/UftrHp+cShC1ybwgVBb53fA=; b=EHGcPwfabjBuCdSS18AOZjjp3EkmFKd0RMFrpuToizVH/hPs6t9U+Q5ub54xUFujCHEpNRHLDU2TlEs1AKflfighzowspjcjQcYqLWWCFv1AN/mHxTj+hHpyYeg03zjQb7PcDefPap6f0EAqxyErvxdyj1smq4m9066VUtOOJ7s= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R571e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033037067111;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=14;SR=0;TI=SMTPD_---0W61t.UO_1715135016; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W61t.UO_1715135016) by smtp.aliyun-inc.com; Wed, 08 May 2024 10:23:37 +0800 From: Xuan Zhuo To: virtualization@lists.linux.dev Cc: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , netdev@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH vhost 3/5] virtio_ring: introduce virtqueue_dma_map_sg_attrs Date: Wed, 8 May 2024 10:23:29 +0800 Message-Id: <20240508022331.63751-4-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240508022331.63751-1-xuanzhuo@linux.alibaba.com> References: <20240508022331.63751-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: a0fef46c457b Introduce a helper to do dma map for scatterlist. That can be used by other drivers. Signed-off-by: Xuan Zhuo --- drivers/virtio/virtio_ring.c | 32 ++++++++++++++++++++++++++++++++ include/linux/virtio.h | 3 +++ 2 files changed, 35 insertions(+) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 653c7ea24fb7..dab7f7fa8ec1 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -3162,6 +3162,38 @@ int virtqueue_dma_mapping_error(struct virtqueue *_vq, dma_addr_t addr) } EXPORT_SYMBOL_GPL(virtqueue_dma_mapping_error); +/** + * virtqueue_dma_map_sg_attrs - map scatterlist addr DMA for _vq + * @_vq: the struct virtqueue we're talking about. + * @sg: the scatterlist to do dma + * @dir: DMA direction + * @attrs: DMA Attrs + * + * The caller calls this to do dma mapping in advance. The sg can be + * passed to this _vq when it is in pre-mapped mode. + * + * Returns zero or a negative error. + * 0: success + * -ENOMEM: dma map error + */ +int virtqueue_dma_map_sg_attrs(struct virtqueue *_vq, struct scatterlist *sg, + enum dma_data_direction dir, unsigned long attrs) +{ + dma_addr_t addr; + int err; + + addr = virtqueue_dma_map_page_attrs(_vq, sg_page(sg), sg->offset, + sg->length, dir, attrs); + err = virtqueue_dma_mapping_error(_vq, addr); + if (err) + return err; + + sg->dma_address = addr; + + return 0; +} +EXPORT_SYMBOL_GPL(virtqueue_dma_map_sg_attrs); + /** * virtqueue_dma_need_sync - check a dma address needs sync * @_vq: the struct virtqueue we're talking about. diff --git a/include/linux/virtio.h b/include/linux/virtio.h index d92aa7be2c44..18694d201698 100644 --- a/include/linux/virtio.h +++ b/include/linux/virtio.h @@ -234,6 +234,9 @@ void virtqueue_dma_unmap_page_attrs(struct virtqueue *_vq, dma_addr_t addr, unsigned long attrs); int virtqueue_dma_mapping_error(struct virtqueue *_vq, dma_addr_t addr); +int virtqueue_dma_map_sg_attrs(struct virtqueue *_vq, struct scatterlist *sg, + enum dma_data_direction dir, unsigned long attrs); + bool virtqueue_dma_need_sync(struct virtqueue *_vq, dma_addr_t addr); void virtqueue_dma_sync_single_range_for_cpu(struct virtqueue *_vq, dma_addr_t addr, unsigned long offset, size_t size, From patchwork Wed May 8 02:23:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13658027 X-Patchwork-Delegate: kuba@kernel.org Received: from out30-97.freemail.mail.aliyun.com (out30-97.freemail.mail.aliyun.com [115.124.30.97]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4D301DF42; Wed, 8 May 2024 02:23:41 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.97 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715135024; cv=none; b=fgfIuG+1hKx2MzVS3bvGOVf3kkVsCGcFfyvQlDxjrXjcKOxKOQccR42myM4Her8xDNJUyw373qgmAuAWyFAlDvyr6f3NSM1SsGsXvZdKk9RR3xXevwF7PGni1wa6VHrGzGBZTolYjf4kKP8u8VPtuDyZnScY/G9V89NIFuvF2I8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715135024; c=relaxed/simple; bh=Ua6MmFtyWLBM+RkYjU3PVACyYS7R+vmMM2aRQ9xBNqA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=CgkTEZ2xSrW764BnxoRbwB1I78dyAjXFP8CRHIo4DPmGbQKI+ts0aoestMfstfEBJZcW2UilTt3tn1V9z5oMXglXfIfjnQgDK1JNi4zwL/5STKbhUAq+3iSHFC9IjDWJNZE7YyPhdbcLc1CtSArgKEDmc/9oXlEKtIsHUulWV0M= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=EcXQfE3e; arc=none smtp.client-ip=115.124.30.97 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="EcXQfE3e" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1715135020; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=EO7wn5Fu8gX5iQFZQOKzHSIik4RiTxHVbWyieF/rF/o=; b=EcXQfE3eG8+Rk7Gc7zRgma1Fhdlr0gfE2bAXQyQV0AHPwDDUYYjruReM3xBYM3qISm+pV4G1J0s4EmPyi71izplG+hP9I9oRh1CqZgI/l/h5eDl4Gd8DZuemXUo+Qo0E/1awxFc4JEmziKBSSux0HM0UnbZwwRiUBNa4DMA9m0A= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R171e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033045075189;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=14;SR=0;TI=SMTPD_---0W61rwQZ_1715135018; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W61rwQZ_1715135018) by smtp.aliyun-inc.com; Wed, 08 May 2024 10:23:39 +0800 From: Xuan Zhuo To: virtualization@lists.linux.dev Cc: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , netdev@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH vhost 4/5] virtio_ring: virtqueue_set_dma_premapped() support to disable Date: Wed, 8 May 2024 10:23:30 +0800 Message-Id: <20240508022331.63751-5-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240508022331.63751-1-xuanzhuo@linux.alibaba.com> References: <20240508022331.63751-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: a0fef46c457b X-Patchwork-Delegate: kuba@kernel.org virtio-net sq will only enable premapped mode when the sq is bound to the af-xdp. So we need the helper (virtqueue_set_dma_premapped) to enable the premapped mode when af-xdp binds to the sq. And to disable the premapped mode when af-xdp unbinds to the sq. Signed-off-by: Xuan Zhuo --- drivers/net/virtio_net.c | 2 +- drivers/virtio/virtio_ring.c | 7 ++++--- include/linux/virtio.h | 2 +- 3 files changed, 6 insertions(+), 5 deletions(-) diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index 3e8694837a29..a28a84101d5b 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -795,7 +795,7 @@ static void virtnet_rq_set_premapped(struct virtnet_info *vi) for (i = 0; i < vi->max_queue_pairs; i++) /* error never happen */ - BUG_ON(virtqueue_set_dma_premapped(vi->rq[i].vq)); + BUG_ON(virtqueue_set_dma_premapped(vi->rq[i].vq, true)); } static void virtnet_rq_unmap_free_buf(struct virtqueue *vq, void *buf) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index dab7f7fa8ec1..78145ad43370 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -2712,8 +2712,9 @@ EXPORT_SYMBOL_GPL(virtqueue_resize); /** * virtqueue_set_dma_premapped - set the vring premapped mode * @_vq: the struct virtqueue we're talking about. + * @premapped: bool enable/disable the premapped mode * - * Enable the premapped mode of the vq. + * Enable/disable the premapped mode of the vq. * * The vring in premapped mode does not do dma internally, so the driver must * do dma mapping in advance. The driver must pass the dma_address through @@ -2730,7 +2731,7 @@ EXPORT_SYMBOL_GPL(virtqueue_resize); * 0: success. * -EINVAL: the vq is in use. */ -int virtqueue_set_dma_premapped(struct virtqueue *_vq) +int virtqueue_set_dma_premapped(struct virtqueue *_vq, bool premapped) { struct vring_virtqueue *vq = to_vvq(_vq); u32 num; @@ -2744,7 +2745,7 @@ int virtqueue_set_dma_premapped(struct virtqueue *_vq) return -EINVAL; } - vq->premapped = true; + vq->premapped = premapped; END_USE(vq); diff --git a/include/linux/virtio.h b/include/linux/virtio.h index 18694d201698..78687d9a8986 100644 --- a/include/linux/virtio.h +++ b/include/linux/virtio.h @@ -81,7 +81,7 @@ bool virtqueue_enable_cb(struct virtqueue *vq); unsigned virtqueue_enable_cb_prepare(struct virtqueue *vq); -int virtqueue_set_dma_premapped(struct virtqueue *_vq); +int virtqueue_set_dma_premapped(struct virtqueue *_vq, bool premapped); bool virtqueue_poll(struct virtqueue *vq, unsigned); From patchwork Wed May 8 02:23:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13658028 X-Patchwork-Delegate: kuba@kernel.org Received: from out30-119.freemail.mail.aliyun.com (out30-119.freemail.mail.aliyun.com [115.124.30.119]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BFD19FC1C; Wed, 8 May 2024 02:23:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.119 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715135026; cv=none; b=jnOvZP098Xd/TYWn55IGlNbQ+zxz0/W7HekXN40cQRE/QsnNpUz7+o0vfEJYqNvtBmKUgx4xH/h+FWlA7Hp43MiLAXuo549dCTww8+7cQ8PNvdp8TcB+gOrJ6VxIHdOSk2dtG5kU0JsbL5sxiC1J8iIfF0UUTXN9buOgOtdIDjQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715135026; c=relaxed/simple; bh=UF+TAPOrGWJnhbGhBoW8JR4hZpOu6/pKeUA7xX+lqGM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=O55Z50e4ehocu+giE6eZ/RpegfhKeG8wTdVkZx0dt8fUnWb/DFnou3BONbTgkC+HloqDSS1l3a3/1UOOzVGhm9J7I/K1hEOzC4goWG7DkkM7sr/z/QoMt2M0qT2UaqSeoULmj8+S5zCQorZKhRAY2QCqzcQ8v5ZucEkogtTVOe8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=a3WTRRMN; arc=none smtp.client-ip=115.124.30.119 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="a3WTRRMN" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1715135021; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=+16xHV+E+hZsSbFAcLStp0O+Mu92bYJGwyc/2BUXSR0=; b=a3WTRRMNhbbTp6p+hwq397f4zXDWivn/I4qKA4Lh/5o79zKg8PCl2mKAdqEYv7DR2bPPNNRfroWmBGq2OlZ3xkkA99Ax2eCJkGskVAVXgubN/Y9tugCwPFHvtFhpmlVnP8P1YAhxjMPQsXXF9pHLCTuHJPN55MMMAik1HM0jn1w= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R141e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033037067110;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=14;SR=0;TI=SMTPD_---0W61t.VX_1715135019; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W61t.VX_1715135019) by smtp.aliyun-inc.com; Wed, 08 May 2024 10:23:40 +0800 From: Xuan Zhuo To: virtualization@lists.linux.dev Cc: "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , netdev@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH vhost 5/5] virtio_net: sq support premapped mode Date: Wed, 8 May 2024 10:23:31 +0800 Message-Id: <20240508022331.63751-6-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240508022331.63751-1-xuanzhuo@linux.alibaba.com> References: <20240508022331.63751-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: a0fef46c457b X-Patchwork-Delegate: kuba@kernel.org If the xsk is enabling, the xsk tx will share the send queue. But the xsk requires that the send queue use the premapped mode. So the send queue must support premapped mode when it is bound to af-xdp. * virtnet_sq_set_premapped(sq, true) is used to enable premapped mode. In this mode, the driver will record the dma info when skb or xdp frame is sent. Currently, the SQ premapped mode is operational only with af-xdp. In this mode, af-xdp, the kernel stack, and xdp tx/redirect will share the same SQ. Af-xdp independently manages its DMA. The kernel stack and xdp tx/redirect utilize this DMA metadata to manage the DMA info. If the indirect descriptor feature be supported, the volume of DMA details we need to maintain becomes quite substantial. Here, we have a cap on the amount of DMA info we manage. If the kernel stack and xdp tx/redirect attempt to use more descriptors, virtnet_add_outbuf() will return an -ENOMEM error. But the af-xdp can work continually. * virtnet_sq_set_premapped(sq, false) is used to disable premapped mode. Signed-off-by: Xuan Zhuo --- drivers/net/virtio_net.c | 210 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 207 insertions(+), 3 deletions(-) diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index a28a84101d5b..7274daed4993 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -24,6 +24,7 @@ #include #include #include +#include static int napi_weight = NAPI_POLL_WEIGHT; module_param(napi_weight, int, 0444); @@ -47,6 +48,7 @@ module_param(napi_tx, bool, 0644); #define VIRTIO_XDP_REDIR BIT(1) #define VIRTIO_XDP_FLAG BIT(0) +#define VIRTIO_DMA_FLAG BIT(1) /* RX packet size EWMA. The average packet size is used to determine the packet * buffer size when refilling RX rings. As the entire RX ring may be refilled @@ -146,6 +148,25 @@ struct virtnet_rq_dma { u16 need_sync; }; +struct virtnet_sq_dma { + union { + struct virtnet_sq_dma *next; + void *data; + }; + dma_addr_t addr; + u32 len; + u8 num; +}; + +struct virtnet_sq_dma_info { + /* record for kfree */ + void *p; + + u32 free_num; + + struct virtnet_sq_dma *free; +}; + /* Internal representation of a send virtqueue */ struct send_queue { /* Virtqueue associated with this send _queue */ @@ -165,6 +186,11 @@ struct send_queue { /* Record whether sq is in reset state. */ bool reset; + + /* SQ is premapped mode or not. */ + bool premapped; + + struct virtnet_sq_dma_info dmainfo; }; /* Internal representation of a receive virtqueue */ @@ -374,6 +400,173 @@ static struct xdp_frame *ptr_to_xdp(void *ptr) return (struct xdp_frame *)((unsigned long)ptr & ~VIRTIO_XDP_FLAG); } +static void virtnet_sq_unmap(struct send_queue *sq, void **data) +{ + struct virtnet_sq_dma *head, *tail, *p; + unsigned long ptr; + int i; + + ptr = (unsigned long)*data; + + if (!(ptr & VIRTIO_DMA_FLAG)) + return; + + head = (void *)(ptr & ~VIRTIO_DMA_FLAG); + + p = head; + + for (i = 0; i < head->num; ++i) { + virtqueue_dma_unmap_page_attrs(sq->vq, p->addr, p->len, + DMA_TO_DEVICE, 0); + tail = p; + p = p->next; + } + + *data = tail->data; + + tail->next = sq->dmainfo.free; + sq->dmainfo.free = head; + sq->dmainfo.free_num += head->num; +} + +static void *virtnet_dma_chain_update(struct send_queue *sq, + struct virtnet_sq_dma *head, + struct virtnet_sq_dma *tail, + u8 num, void *data) +{ + sq->dmainfo.free = tail->next; + sq->dmainfo.free_num -= num; + head->num = num; + + tail->data = data; + + return (void *)((unsigned long)head | VIRTIO_DMA_FLAG); +} + +static struct virtnet_sq_dma *virtnet_sq_map_sg(struct send_queue *sq, int num, void *data) +{ + struct virtnet_sq_dma *head, *tail, *p; + struct scatterlist *sg; + int i; + + if (num > sq->dmainfo.free_num) + return NULL; + + head = sq->dmainfo.free; + p = head; + + tail = NULL; + + for (i = 0; i < num; ++i) { + if (virtqueue_dma_map_sg_attrs(sq->vq, &sq->sg[i], DMA_TO_DEVICE, 0)) + goto err; + + tail = p; + tail->addr = sg->dma_address; + tail->len = sg->length; + + p = p->next; + } + + return virtnet_dma_chain_update(sq, head, tail, num, data); + +err: + if (tail) { + data = virtnet_dma_chain_update(sq, head, tail, i, data); + virtnet_sq_unmap(sq, &data); + } + + return NULL; +} + +static int virtnet_add_outbuf(struct send_queue *sq, u8 num, void *data) +{ + int ret; + + if (sq->premapped) { + data = virtnet_sq_map_sg(sq, num, data); + if (!data) + return -ENOMEM; + } + + ret = virtqueue_add_outbuf(sq->vq, sq->sg, num, data, GFP_ATOMIC); + if (ret && sq->premapped) + virtnet_sq_unmap(sq, &data); + + return ret; +} + +static int virtnet_sq_alloc_dma_meta(struct send_queue *sq) +{ + struct virtnet_sq_dma *d; + int num, i; + + num = virtqueue_get_vring_size(sq->vq); + + /* Currently, the SQ premapped mode is operational only with af-xdp. In + * this mode, af-xdp, the kernel stack, and xdp tx/redirect will share + * the same SQ. Af-xdp independently manages its DMA. The kernel stack + * and xdp tx/redirect utilize this DMA metadata to manage the DMA info. + * + * If the indirect descriptor feature be supported, the volume of DMA + * details we need to maintain becomes quite substantial. Here, we have + * a cap on the amount of DMA info we manage, effectively limiting it to + * twice the size of the ring buffer. + * + * If the kernel stack and xdp tx/redirect attempt to use more + * descriptors than allowed by this double ring buffer size, + * virtnet_add_outbuf() will return an -ENOMEM error. But the af-xdp can + * work continually. + */ + if (virtio_has_feature(sq->vq->vdev, VIRTIO_RING_F_INDIRECT_DESC)) + num = num * 2; + + sq->dmainfo.free = kcalloc(num, sizeof(*sq->dmainfo.free), GFP_KERNEL); + if (!sq->dmainfo.free) + return -ENOMEM; + + sq->dmainfo.p = sq->dmainfo.free; + sq->dmainfo.free_num = num; + + for (i = 0; i < num; ++i) { + d = &sq->dmainfo.free[i]; + d->next = d + 1; + } + + d->next = NULL; + + return 0; +} + +static void virtnet_sq_free_dma_meta(struct send_queue *sq) +{ + kfree(sq->dmainfo.p); + + sq->dmainfo.p = NULL; + sq->dmainfo.free = NULL; + sq->dmainfo.free_num = 0; +} + +int virtnet_sq_set_premapped(struct send_queue *sq, bool premapped); + +/* This function must be called immediately after creating the vq, or after vq + * reset, and before adding any buffers to it. + */ +int virtnet_sq_set_premapped(struct send_queue *sq, bool premapped) +{ + if (premapped) { + if (virtnet_sq_alloc_dma_meta(sq)) + return -ENOMEM; + } else { + virtnet_sq_free_dma_meta(sq); + } + + BUG_ON(virtqueue_set_dma_premapped(sq->vq, premapped)); + + sq->premapped = premapped; + return 0; +} + static void __free_old_xmit(struct send_queue *sq, bool in_napi, struct virtnet_sq_free_stats *stats) { @@ -383,6 +576,8 @@ static void __free_old_xmit(struct send_queue *sq, bool in_napi, while ((ptr = virtqueue_get_buf(sq->vq, &len)) != NULL) { ++stats->packets; + virtnet_sq_unmap(sq, &ptr); + if (!is_xdp_frame(ptr)) { struct sk_buff *skb = ptr; @@ -915,8 +1110,7 @@ static int __virtnet_xdp_xmit_one(struct virtnet_info *vi, skb_frag_size(frag), skb_frag_off(frag)); } - err = virtqueue_add_outbuf(sq->vq, sq->sg, nr_frags + 1, - xdp_to_ptr(xdpf), GFP_ATOMIC); + err = virtnet_add_outbuf(sq, nr_frags + 1, xdp_to_ptr(xdpf)); if (unlikely(err)) return -ENOSPC; /* Caller handle free/refcnt */ @@ -2380,7 +2574,7 @@ static int xmit_skb(struct send_queue *sq, struct sk_buff *skb) return num_sg; num_sg++; } - return virtqueue_add_outbuf(sq->vq, sq->sg, num_sg, skb, GFP_ATOMIC); + return virtnet_add_outbuf(sq, num_sg, skb); } static netdev_tx_t start_xmit(struct sk_buff *skb, struct net_device *dev) @@ -4215,6 +4409,8 @@ static void virtnet_free_queues(struct virtnet_info *vi) for (i = 0; i < vi->max_queue_pairs; i++) { __netif_napi_del(&vi->rq[i].napi); __netif_napi_del(&vi->sq[i].napi); + + virtnet_sq_free_dma_meta(&vi->sq[i]); } /* We called __netif_napi_del(), @@ -4263,6 +4459,14 @@ static void free_receive_page_frags(struct virtnet_info *vi) static void virtnet_sq_free_unused_buf(struct virtqueue *vq, void *buf) { + struct virtnet_info *vi = vq->vdev->priv; + struct send_queue *sq; + int i = vq2rxq(vq); + + sq = &vi->sq[i]; + + virtnet_sq_unmap(sq, &buf); + if (!is_xdp_frame(buf)) dev_kfree_skb(buf); else