From patchwork Thu Feb 24 11:03:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 12758354 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DEC3BC4167B for ; Thu, 24 Feb 2022 11:04:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233723AbiBXLEl (ORCPT ); Thu, 24 Feb 2022 06:04:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33720 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233702AbiBXLEh (ORCPT ); Thu, 24 Feb 2022 06:04:37 -0500 Received: from out30-43.freemail.mail.aliyun.com (out30-43.freemail.mail.aliyun.com [115.124.30.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 149EE28F969; Thu, 24 Feb 2022 03:04:06 -0800 (PST) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R621e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04423;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=11;SR=0;TI=SMTPD_---0V5Nr1qt_1645700643; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0V5Nr1qt_1645700643) by smtp.aliyun-inc.com(127.0.0.1); Thu, 24 Feb 2022 19:04:04 +0800 From: Xuan Zhuo To: virtualization@lists.linux-foundation.org, netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , "David S. Miller" , Jakub Kicinski , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , bpf@vger.kernel.org Subject: [PATCH v2 1/9] virtio_ring: rename vring_unmap_state_packed() to vring_unmap_extra_packed() Date: Thu, 24 Feb 2022 19:03:54 +0800 Message-Id: <20220224110402.108161-2-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20220224110402.108161-1-xuanzhuo@linux.alibaba.com> References: <20220224110402.108161-1-xuanzhuo@linux.alibaba.com> MIME-Version: 1.0 X-Git-Hash: d02e086a8668 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org The actual parameter handled by vring_unmap_state_packed() is that vring_desc_extra, so this function should use "extra" instead of "state". Signed-off-by: Xuan Zhuo --- drivers/virtio/virtio_ring.c | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 962f1477b1fa..7cf3ae057833 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -984,24 +984,24 @@ static struct virtqueue *vring_create_virtqueue_split( * Packed ring specific functions - *_packed(). */ -static void vring_unmap_state_packed(const struct vring_virtqueue *vq, - struct vring_desc_extra *state) +static void vring_unmap_extra_packed(const struct vring_virtqueue *vq, + struct vring_desc_extra *extra) { u16 flags; if (!vq->use_dma_api) return; - flags = state->flags; + flags = extra->flags; if (flags & VRING_DESC_F_INDIRECT) { dma_unmap_single(vring_dma_dev(vq), - state->addr, state->len, + extra->addr, extra->len, (flags & VRING_DESC_F_WRITE) ? DMA_FROM_DEVICE : DMA_TO_DEVICE); } else { dma_unmap_page(vring_dma_dev(vq), - state->addr, state->len, + extra->addr, extra->len, (flags & VRING_DESC_F_WRITE) ? DMA_FROM_DEVICE : DMA_TO_DEVICE); } @@ -1303,8 +1303,7 @@ static inline int virtqueue_add_packed(struct virtqueue *_vq, for (n = 0; n < total_sg; n++) { if (i == err_idx) break; - vring_unmap_state_packed(vq, - &vq->packed.desc_extra[curr]); + vring_unmap_extra_packed(vq, &vq->packed.desc_extra[curr]); curr = vq->packed.desc_extra[curr].next; i++; if (i >= vq->packed.vring.num) @@ -1383,8 +1382,8 @@ static void detach_buf_packed(struct vring_virtqueue *vq, if (unlikely(vq->use_dma_api)) { curr = id; for (i = 0; i < state->num; i++) { - vring_unmap_state_packed(vq, - &vq->packed.desc_extra[curr]); + vring_unmap_extra_packed(vq, + &vq->packed.desc_extra[curr]); curr = vq->packed.desc_extra[curr].next; } } From patchwork Thu Feb 24 11:03:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 12758355 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3E17C43217 for ; Thu, 24 Feb 2022 11:04:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233724AbiBXLEm (ORCPT ); Thu, 24 Feb 2022 06:04:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33732 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233706AbiBXLEi (ORCPT ); Thu, 24 Feb 2022 06:04:38 -0500 Received: from out30-133.freemail.mail.aliyun.com (out30-133.freemail.mail.aliyun.com [115.124.30.133]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EC83D28F977; Thu, 24 Feb 2022 03:04:08 -0800 (PST) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R711e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04395;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=11;SR=0;TI=SMTPD_---0V5O7ats_1645700644; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0V5O7ats_1645700644) by smtp.aliyun-inc.com(127.0.0.1); Thu, 24 Feb 2022 19:04:05 +0800 From: Xuan Zhuo To: virtualization@lists.linux-foundation.org, netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , "David S. Miller" , Jakub Kicinski , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , bpf@vger.kernel.org Subject: [PATCH v2 2/9] virtio_ring: remove flags check for unmap split indirect desc Date: Thu, 24 Feb 2022 19:03:55 +0800 Message-Id: <20220224110402.108161-3-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20220224110402.108161-1-xuanzhuo@linux.alibaba.com> References: <20220224110402.108161-1-xuanzhuo@linux.alibaba.com> MIME-Version: 1.0 X-Git-Hash: d02e086a8668 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org When calling vring_unmap_one_split_indirect(), it will not encounter the situation that the flags contains VRING_DESC_F_INDIRECT. So remove this logic. Signed-off-by: Xuan Zhuo --- drivers/virtio/virtio_ring.c | 18 +++++------------- 1 file changed, 5 insertions(+), 13 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 7cf3ae057833..fadd0a7503e9 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -379,19 +379,11 @@ static void vring_unmap_one_split_indirect(const struct vring_virtqueue *vq, flags = virtio16_to_cpu(vq->vq.vdev, desc->flags); - if (flags & VRING_DESC_F_INDIRECT) { - dma_unmap_single(vring_dma_dev(vq), - virtio64_to_cpu(vq->vq.vdev, desc->addr), - virtio32_to_cpu(vq->vq.vdev, desc->len), - (flags & VRING_DESC_F_WRITE) ? - DMA_FROM_DEVICE : DMA_TO_DEVICE); - } else { - dma_unmap_page(vring_dma_dev(vq), - virtio64_to_cpu(vq->vq.vdev, desc->addr), - virtio32_to_cpu(vq->vq.vdev, desc->len), - (flags & VRING_DESC_F_WRITE) ? - DMA_FROM_DEVICE : DMA_TO_DEVICE); - } + dma_unmap_page(vring_dma_dev(vq), + virtio64_to_cpu(vq->vq.vdev, desc->addr), + virtio32_to_cpu(vq->vq.vdev, desc->len), + (flags & VRING_DESC_F_WRITE) ? + DMA_FROM_DEVICE : DMA_TO_DEVICE); } static unsigned int vring_unmap_one_split(const struct vring_virtqueue *vq, From patchwork Thu Feb 24 11:03:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 12758357 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 281BEC4332F for ; Thu, 24 Feb 2022 11:04:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233755AbiBXLEs (ORCPT ); Thu, 24 Feb 2022 06:04:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33756 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233713AbiBXLEl (ORCPT ); Thu, 24 Feb 2022 06:04:41 -0500 Received: from out199-18.us.a.mail.aliyun.com (out199-18.us.a.mail.aliyun.com [47.90.199.18]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 38AD528F96B; Thu, 24 Feb 2022 03:04:10 -0800 (PST) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R191e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04395;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=11;SR=0;TI=SMTPD_---0V5Nr1rK_1645700645; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0V5Nr1rK_1645700645) by smtp.aliyun-inc.com(127.0.0.1); Thu, 24 Feb 2022 19:04:06 +0800 From: Xuan Zhuo To: virtualization@lists.linux-foundation.org, netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , "David S. Miller" , Jakub Kicinski , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , bpf@vger.kernel.org Subject: [PATCH v2 3/9] virtio_ring: remove flags check for unmap packed indirect desc Date: Thu, 24 Feb 2022 19:03:56 +0800 Message-Id: <20220224110402.108161-4-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20220224110402.108161-1-xuanzhuo@linux.alibaba.com> References: <20220224110402.108161-1-xuanzhuo@linux.alibaba.com> MIME-Version: 1.0 X-Git-Hash: d02e086a8668 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org When calling vring_unmap_desc_packed(), it will not encounter the situation that the flags contains VRING_DESC_F_INDIRECT. So remove this logic. Signed-off-by: Xuan Zhuo --- drivers/virtio/virtio_ring.c | 18 +++++------------- 1 file changed, 5 insertions(+), 13 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index fadd0a7503e9..cfb028ca238e 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -1009,19 +1009,11 @@ static void vring_unmap_desc_packed(const struct vring_virtqueue *vq, flags = le16_to_cpu(desc->flags); - if (flags & VRING_DESC_F_INDIRECT) { - dma_unmap_single(vring_dma_dev(vq), - le64_to_cpu(desc->addr), - le32_to_cpu(desc->len), - (flags & VRING_DESC_F_WRITE) ? - DMA_FROM_DEVICE : DMA_TO_DEVICE); - } else { - dma_unmap_page(vring_dma_dev(vq), - le64_to_cpu(desc->addr), - le32_to_cpu(desc->len), - (flags & VRING_DESC_F_WRITE) ? - DMA_FROM_DEVICE : DMA_TO_DEVICE); - } + dma_unmap_page(vring_dma_dev(vq), + le64_to_cpu(desc->addr), + le32_to_cpu(desc->len), + (flags & VRING_DESC_F_WRITE) ? + DMA_FROM_DEVICE : DMA_TO_DEVICE); } static struct vring_packed_desc *alloc_indirect_packed(unsigned int total_sg, From patchwork Thu Feb 24 11:03:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 12758356 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51511C433F5 for ; Thu, 24 Feb 2022 11:04:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233756AbiBXLEo (ORCPT ); Thu, 24 Feb 2022 06:04:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33746 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233711AbiBXLEk (ORCPT ); Thu, 24 Feb 2022 06:04:40 -0500 Received: from out30-43.freemail.mail.aliyun.com (out30-43.freemail.mail.aliyun.com [115.124.30.43]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C06F028F969; Thu, 24 Feb 2022 03:04:10 -0800 (PST) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R101e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04395;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=11;SR=0;TI=SMTPD_---0V5O5g83_1645700647; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0V5O5g83_1645700647) by smtp.aliyun-inc.com(127.0.0.1); Thu, 24 Feb 2022 19:04:07 +0800 From: Xuan Zhuo To: virtualization@lists.linux-foundation.org, netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , "David S. Miller" , Jakub Kicinski , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , bpf@vger.kernel.org Subject: [PATCH v2 4/9] virtio_ring: virtqueue_add() support premapped Date: Thu, 24 Feb 2022 19:03:57 +0800 Message-Id: <20220224110402.108161-5-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20220224110402.108161-1-xuanzhuo@linux.alibaba.com> References: <20220224110402.108161-1-xuanzhuo@linux.alibaba.com> MIME-Version: 1.0 X-Git-Hash: d02e086a8668 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org virtuque_add() adds parameter premapped. Signed-off-by: Xuan Zhuo --- drivers/virtio/virtio_ring.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index cfb028ca238e..2f9572c3628c 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -1780,7 +1780,8 @@ static inline int virtqueue_add(struct virtqueue *_vq, unsigned int in_sgs, void *data, void *ctx, - gfp_t gfp) + gfp_t gfp, + bool premapped) { struct vring_virtqueue *vq = to_vvq(_vq); @@ -1821,7 +1822,7 @@ int virtqueue_add_sgs(struct virtqueue *_vq, total_sg++; } return virtqueue_add(_vq, sgs, total_sg, out_sgs, in_sgs, - data, NULL, gfp); + data, NULL, gfp, false); } EXPORT_SYMBOL_GPL(virtqueue_add_sgs); @@ -1843,7 +1844,7 @@ int virtqueue_add_outbuf(struct virtqueue *vq, void *data, gfp_t gfp) { - return virtqueue_add(vq, &sg, num, 1, 0, data, NULL, gfp); + return virtqueue_add(vq, &sg, num, 1, 0, data, NULL, gfp, false); } EXPORT_SYMBOL_GPL(virtqueue_add_outbuf); @@ -1865,7 +1866,7 @@ int virtqueue_add_inbuf(struct virtqueue *vq, void *data, gfp_t gfp) { - return virtqueue_add(vq, &sg, num, 0, 1, data, NULL, gfp); + return virtqueue_add(vq, &sg, num, 0, 1, data, NULL, gfp, false); } EXPORT_SYMBOL_GPL(virtqueue_add_inbuf); @@ -1889,7 +1890,7 @@ int virtqueue_add_inbuf_ctx(struct virtqueue *vq, void *ctx, gfp_t gfp) { - return virtqueue_add(vq, &sg, num, 0, 1, data, ctx, gfp); + return virtqueue_add(vq, &sg, num, 0, 1, data, ctx, gfp, false); } EXPORT_SYMBOL_GPL(virtqueue_add_inbuf_ctx); From patchwork Thu Feb 24 11:03:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 12758358 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 595C2C43217 for ; Thu, 24 Feb 2022 11:04:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233765AbiBXLEu (ORCPT ); Thu, 24 Feb 2022 06:04:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33758 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233720AbiBXLEl (ORCPT ); Thu, 24 Feb 2022 06:04:41 -0500 Received: from out30-45.freemail.mail.aliyun.com (out30-45.freemail.mail.aliyun.com [115.124.30.45]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9D45328F970; Thu, 24 Feb 2022 03:04:11 -0800 (PST) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R111e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04395;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=11;SR=0;TI=SMTPD_---0V5Nr1rv_1645700648; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0V5Nr1rv_1645700648) by smtp.aliyun-inc.com(127.0.0.1); Thu, 24 Feb 2022 19:04:08 +0800 From: Xuan Zhuo To: virtualization@lists.linux-foundation.org, netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , "David S. Miller" , Jakub Kicinski , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , bpf@vger.kernel.org Subject: [PATCH v2 5/9] virtio_ring: split: virtqueue_add_split() support premapped Date: Thu, 24 Feb 2022 19:03:58 +0800 Message-Id: <20220224110402.108161-6-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20220224110402.108161-1-xuanzhuo@linux.alibaba.com> References: <20220224110402.108161-1-xuanzhuo@linux.alibaba.com> MIME-Version: 1.0 X-Git-Hash: d02e086a8668 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org virtqueue_add_split() only supports virtual addresses, dma is completed in virtqueue_add_split(). In some scenarios (such as the AF_XDP scenario), the memory is allocated and DMA is completed in advance, so it is necessary for us to support passing the DMA address to virtqueue_add_split(). And record this premapped information in desc_extra[head]->premapped, which can be skipped when executing dma unmap. Signed-off-by: Xuan Zhuo --- drivers/virtio/virtio_ring.c | 58 +++++++++++++++++++++++++++--------- 1 file changed, 44 insertions(+), 14 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 2f9572c3628c..35cb804dcf2d 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -83,6 +83,7 @@ struct vring_desc_extra { u32 len; /* Descriptor length. */ u16 flags; /* Descriptor flags. */ u16 next; /* The next desc state in a list. */ + bool premapped; }; struct vring_virtqueue { @@ -387,7 +388,7 @@ static void vring_unmap_one_split_indirect(const struct vring_virtqueue *vq, } static unsigned int vring_unmap_one_split(const struct vring_virtqueue *vq, - unsigned int i) + unsigned int i, bool premapped) { struct vring_desc_extra *extra = vq->split.desc_extra; u16 flags; @@ -404,6 +405,9 @@ static unsigned int vring_unmap_one_split(const struct vring_virtqueue *vq, (flags & VRING_DESC_F_WRITE) ? DMA_FROM_DEVICE : DMA_TO_DEVICE); } else { + if (premapped) + goto out; + dma_unmap_page(vring_dma_dev(vq), extra[i].addr, extra[i].len, @@ -474,7 +478,8 @@ static inline int virtqueue_add_split(struct virtqueue *_vq, unsigned int in_sgs, void *data, void *ctx, - gfp_t gfp) + gfp_t gfp, + bool premapped) { struct vring_virtqueue *vq = to_vvq(_vq); struct scatterlist *sg; @@ -535,9 +540,16 @@ static inline int virtqueue_add_split(struct virtqueue *_vq, for (n = 0; n < out_sgs; n++) { for (sg = sgs[n]; sg; sg = sg_next(sg)) { - dma_addr_t addr = vring_map_one_sg(vq, sg, DMA_TO_DEVICE); - if (vring_mapping_error(vq, addr)) - goto unmap_release; + dma_addr_t addr; + + if (premapped) { + addr = sg_dma_address(sg); + + } else { + addr = vring_map_one_sg(vq, sg, DMA_TO_DEVICE); + if (vring_mapping_error(vq, addr)) + goto unmap_release; + } prev = i; /* Note that we trust indirect descriptor @@ -550,9 +562,16 @@ static inline int virtqueue_add_split(struct virtqueue *_vq, } for (; n < (out_sgs + in_sgs); n++) { for (sg = sgs[n]; sg; sg = sg_next(sg)) { - dma_addr_t addr = vring_map_one_sg(vq, sg, DMA_FROM_DEVICE); - if (vring_mapping_error(vq, addr)) - goto unmap_release; + dma_addr_t addr; + + if (premapped) { + addr = sg_dma_address(sg); + + } else { + addr = vring_map_one_sg(vq, sg, DMA_FROM_DEVICE); + if (vring_mapping_error(vq, addr)) + goto unmap_release; + } prev = i; /* Note that we trust indirect descriptor @@ -602,6 +621,8 @@ static inline int virtqueue_add_split(struct virtqueue *_vq, else vq->split.desc_state[head].indir_desc = ctx; + vq->split.desc_extra[head].premapped = premapped; + /* Put entry in available array (but don't update avail->idx until they * do sync). */ avail = vq->split.avail_idx_shadow & (vq->split.vring.num - 1); @@ -626,6 +647,9 @@ static inline int virtqueue_add_split(struct virtqueue *_vq, return 0; unmap_release: + if (premapped) + goto unmap_free; + err_idx = i; if (indirect) @@ -640,9 +664,10 @@ static inline int virtqueue_add_split(struct virtqueue *_vq, vring_unmap_one_split_indirect(vq, &desc[i]); i = virtio16_to_cpu(_vq->vdev, desc[i].next); } else - i = vring_unmap_one_split(vq, i); + i = vring_unmap_one_split(vq, i, false); } +unmap_free: if (indirect) kfree(desc); @@ -686,20 +711,23 @@ static void detach_buf_split(struct vring_virtqueue *vq, unsigned int head, { unsigned int i, j; __virtio16 nextflag = cpu_to_virtio16(vq->vq.vdev, VRING_DESC_F_NEXT); + bool premapped; /* Clear data ptr. */ vq->split.desc_state[head].data = NULL; + premapped = vq->split.desc_extra[head].premapped; + /* Put back on free list: unmap first-level descriptors and find end */ i = head; while (vq->split.vring.desc[i].flags & nextflag) { - vring_unmap_one_split(vq, i); + vring_unmap_one_split(vq, i, premapped); i = vq->split.desc_extra[i].next; vq->vq.num_free++; } - vring_unmap_one_split(vq, i); + vring_unmap_one_split(vq, i, premapped); vq->split.desc_extra[i].next = vq->free_head; vq->free_head = head; @@ -721,8 +749,10 @@ static void detach_buf_split(struct vring_virtqueue *vq, unsigned int head, VRING_DESC_F_INDIRECT)); BUG_ON(len == 0 || len % sizeof(struct vring_desc)); - for (j = 0; j < len / sizeof(struct vring_desc); j++) - vring_unmap_one_split_indirect(vq, &indir_desc[j]); + if (!premapped) { + for (j = 0; j < len / sizeof(struct vring_desc); j++) + vring_unmap_one_split_indirect(vq, &indir_desc[j]); + } kfree(indir_desc); vq->split.desc_state[head].indir_desc = NULL; @@ -1788,7 +1818,7 @@ static inline int virtqueue_add(struct virtqueue *_vq, return vq->packed_ring ? virtqueue_add_packed(_vq, sgs, total_sg, out_sgs, in_sgs, data, ctx, gfp) : virtqueue_add_split(_vq, sgs, total_sg, - out_sgs, in_sgs, data, ctx, gfp); + out_sgs, in_sgs, data, ctx, gfp, premapped); } /** From patchwork Thu Feb 24 11:03:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 12758360 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C61A9C4332F for ; Thu, 24 Feb 2022 11:04:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233769AbiBXLEw (ORCPT ); Thu, 24 Feb 2022 06:04:52 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33790 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233745AbiBXLEn (ORCPT ); Thu, 24 Feb 2022 06:04:43 -0500 Received: from out30-45.freemail.mail.aliyun.com (out30-45.freemail.mail.aliyun.com [115.124.30.45]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A905E28F969; Thu, 24 Feb 2022 03:04:12 -0800 (PST) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R971e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04423;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=11;SR=0;TI=SMTPD_---0V5O7aur_1645700649; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0V5O7aur_1645700649) by smtp.aliyun-inc.com(127.0.0.1); Thu, 24 Feb 2022 19:04:09 +0800 From: Xuan Zhuo To: virtualization@lists.linux-foundation.org, netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , "David S. Miller" , Jakub Kicinski , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , bpf@vger.kernel.org Subject: [PATCH v2 6/9] virtio_ring: packed: virtqueue_add_packed() support premapped Date: Thu, 24 Feb 2022 19:03:59 +0800 Message-Id: <20220224110402.108161-7-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20220224110402.108161-1-xuanzhuo@linux.alibaba.com> References: <20220224110402.108161-1-xuanzhuo@linux.alibaba.com> MIME-Version: 1.0 X-Git-Hash: d02e086a8668 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org virtqueue_add_packed() only supports virtual addresses, dma is completed in virtqueue_add_packed(). In some scenarios (such as the AF_XDP scenario), the memory is allocated and DMA is completed in advance, so it is necessary for us to support passing the DMA address to virtqueue_add_packed(). Record this premapped information in desc_extra[id]->premapped, which can be skipped when executing dma unmap. Signed-off-by: Xuan Zhuo --- drivers/virtio/virtio_ring.c | 76 +++++++++++++++++++++++++----------- 1 file changed, 54 insertions(+), 22 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 35cb804dcf2d..c8d63cae8b33 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -1007,7 +1007,8 @@ static struct virtqueue *vring_create_virtqueue_split( */ static void vring_unmap_extra_packed(const struct vring_virtqueue *vq, - struct vring_desc_extra *extra) + struct vring_desc_extra *extra, + bool premapped) { u16 flags; @@ -1022,6 +1023,9 @@ static void vring_unmap_extra_packed(const struct vring_virtqueue *vq, (flags & VRING_DESC_F_WRITE) ? DMA_FROM_DEVICE : DMA_TO_DEVICE); } else { + if (premapped) + return; + dma_unmap_page(vring_dma_dev(vq), extra->addr, extra->len, (flags & VRING_DESC_F_WRITE) ? @@ -1069,7 +1073,8 @@ static int virtqueue_add_indirect_packed(struct vring_virtqueue *vq, unsigned int out_sgs, unsigned int in_sgs, void *data, - gfp_t gfp) + gfp_t gfp, + bool premapped) { struct vring_packed_desc *desc; struct scatterlist *sg; @@ -1095,10 +1100,15 @@ static int virtqueue_add_indirect_packed(struct vring_virtqueue *vq, for (n = 0; n < out_sgs + in_sgs; n++) { for (sg = sgs[n]; sg; sg = sg_next(sg)) { - addr = vring_map_one_sg(vq, sg, n < out_sgs ? - DMA_TO_DEVICE : DMA_FROM_DEVICE); - if (vring_mapping_error(vq, addr)) - goto unmap_release; + if (premapped) { + addr = sg_dma_address(sg); + + } else { + addr = vring_map_one_sg(vq, sg, n < out_sgs ? + DMA_TO_DEVICE : DMA_FROM_DEVICE); + if (vring_mapping_error(vq, addr)) + goto unmap_release; + } desc[i].flags = cpu_to_le16(n < out_sgs ? 0 : VRING_DESC_F_WRITE); @@ -1128,6 +1138,8 @@ static int virtqueue_add_indirect_packed(struct vring_virtqueue *vq, vq->packed.avail_used_flags; } + vq->packed.desc_extra[id].premapped = premapped; + /* * A driver MUST NOT make the first descriptor in the list * available before all subsequent descriptors comprising @@ -1166,10 +1178,11 @@ static int virtqueue_add_indirect_packed(struct vring_virtqueue *vq, return 0; unmap_release: - err_idx = i; - - for (i = 0; i < err_idx; i++) - vring_unmap_desc_packed(vq, &desc[i]); + if (!premapped) { + err_idx = i; + for (i = 0; i < err_idx; i++) + vring_unmap_desc_packed(vq, &desc[i]); + } kfree(desc); @@ -1184,7 +1197,8 @@ static inline int virtqueue_add_packed(struct virtqueue *_vq, unsigned int in_sgs, void *data, void *ctx, - gfp_t gfp) + gfp_t gfp, + bool premapped) { struct vring_virtqueue *vq = to_vvq(_vq); struct vring_packed_desc *desc; @@ -1210,7 +1224,7 @@ static inline int virtqueue_add_packed(struct virtqueue *_vq, if (virtqueue_use_indirect(_vq, total_sg)) { err = virtqueue_add_indirect_packed(vq, sgs, total_sg, out_sgs, - in_sgs, data, gfp); + in_sgs, data, gfp, premapped); if (err != -ENOMEM) { END_USE(vq); return err; @@ -1242,10 +1256,17 @@ static inline int virtqueue_add_packed(struct virtqueue *_vq, c = 0; for (n = 0; n < out_sgs + in_sgs; n++) { for (sg = sgs[n]; sg; sg = sg_next(sg)) { - dma_addr_t addr = vring_map_one_sg(vq, sg, n < out_sgs ? - DMA_TO_DEVICE : DMA_FROM_DEVICE); - if (vring_mapping_error(vq, addr)) - goto unmap_release; + dma_addr_t addr; + + if (premapped) { + addr = sg_dma_address(sg); + + } else { + addr = vring_map_one_sg(vq, sg, n < out_sgs ? + DMA_TO_DEVICE : DMA_FROM_DEVICE); + if (vring_mapping_error(vq, addr)) + goto unmap_release; + } flags = cpu_to_le16(vq->packed.avail_used_flags | (++c == total_sg ? 0 : VRING_DESC_F_NEXT) | @@ -1265,6 +1286,7 @@ static inline int virtqueue_add_packed(struct virtqueue *_vq, vq->packed.desc_extra[curr].flags = le16_to_cpu(flags); } + prev = curr; curr = vq->packed.desc_extra[curr].next; @@ -1293,6 +1315,8 @@ static inline int virtqueue_add_packed(struct virtqueue *_vq, vq->packed.desc_state[id].indir_desc = ctx; vq->packed.desc_state[id].last = prev; + vq->packed.desc_extra[id].premapped = premapped; + /* * A driver MUST NOT make the first descriptor in the list * available before all subsequent descriptors comprising @@ -1308,22 +1332,26 @@ static inline int virtqueue_add_packed(struct virtqueue *_vq, return 0; unmap_release: + vq->packed.avail_used_flags = avail_used_flags; + + if (premapped) + goto unmap_free; + err_idx = i; i = head; curr = vq->free_head; - vq->packed.avail_used_flags = avail_used_flags; - for (n = 0; n < total_sg; n++) { if (i == err_idx) break; - vring_unmap_extra_packed(vq, &vq->packed.desc_extra[curr]); + vring_unmap_extra_packed(vq, &vq->packed.desc_extra[curr], false); curr = vq->packed.desc_extra[curr].next; i++; if (i >= vq->packed.vring.num) i = 0; } +unmap_free: END_USE(vq); return -EIO; } @@ -1383,9 +1411,12 @@ static void detach_buf_packed(struct vring_virtqueue *vq, struct vring_desc_state_packed *state = NULL; struct vring_packed_desc *desc; unsigned int i, curr; + bool premapped; state = &vq->packed.desc_state[id]; + premapped = vq->packed.desc_extra[state->last].premapped; + /* Clear data ptr. */ state->data = NULL; @@ -1397,7 +1428,8 @@ static void detach_buf_packed(struct vring_virtqueue *vq, curr = id; for (i = 0; i < state->num; i++) { vring_unmap_extra_packed(vq, - &vq->packed.desc_extra[curr]); + &vq->packed.desc_extra[curr], + premapped); curr = vq->packed.desc_extra[curr].next; } } @@ -1410,7 +1442,7 @@ static void detach_buf_packed(struct vring_virtqueue *vq, if (!desc) return; - if (vq->use_dma_api) { + if (vq->use_dma_api && !premapped) { len = vq->packed.desc_extra[id].len; for (i = 0; i < len / sizeof(struct vring_packed_desc); i++) @@ -1816,7 +1848,7 @@ static inline int virtqueue_add(struct virtqueue *_vq, struct vring_virtqueue *vq = to_vvq(_vq); return vq->packed_ring ? virtqueue_add_packed(_vq, sgs, total_sg, - out_sgs, in_sgs, data, ctx, gfp) : + out_sgs, in_sgs, data, ctx, gfp, premapped) : virtqueue_add_split(_vq, sgs, total_sg, out_sgs, in_sgs, data, ctx, gfp, premapped); } From patchwork Thu Feb 24 11:04:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 12758359 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22617C433F5 for ; Thu, 24 Feb 2022 11:04:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233770AbiBXLEx (ORCPT ); Thu, 24 Feb 2022 06:04:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33792 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233754AbiBXLEn (ORCPT ); Thu, 24 Feb 2022 06:04:43 -0500 Received: from out30-44.freemail.mail.aliyun.com (out30-44.freemail.mail.aliyun.com [115.124.30.44]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D534428F972; Thu, 24 Feb 2022 03:04:13 -0800 (PST) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R601e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04423;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=11;SR=0;TI=SMTPD_---0V5O.G4g_1645700650; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0V5O.G4g_1645700650) by smtp.aliyun-inc.com(127.0.0.1); Thu, 24 Feb 2022 19:04:11 +0800 From: Xuan Zhuo To: virtualization@lists.linux-foundation.org, netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , "David S. Miller" , Jakub Kicinski , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , bpf@vger.kernel.org Subject: [PATCH v2 7/9] virtio_ring: add api virtio_dma_map() for advance dma Date: Thu, 24 Feb 2022 19:04:00 +0800 Message-Id: <20220224110402.108161-8-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20220224110402.108161-1-xuanzhuo@linux.alibaba.com> References: <20220224110402.108161-1-xuanzhuo@linux.alibaba.com> MIME-Version: 1.0 X-Git-Hash: d02e086a8668 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Added virtio_dma_map() to map DMA addresses for virtual memory in advance. The purpose of adding this function is to check vring_use_dma_api() for virtio dma operation and get vdev->dev.parent as the parameter of dma_map_page(). Added virtio_dma_unmap() for unmap DMA address. Signed-off-by: Xuan Zhuo --- drivers/virtio/virtio_ring.c | 63 ++++++++++++++++++++++++++++++++++++ include/linux/virtio.h | 7 ++++ 2 files changed, 70 insertions(+) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index c8d63cae8b33..6a1e0ef5498a 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -2508,4 +2508,67 @@ const struct vring *virtqueue_get_vring(struct virtqueue *vq) } EXPORT_SYMBOL_GPL(virtqueue_get_vring); +/** + * virtio_dma_map - get the DMA addr of the memory for virtio device + * @dev: virtio device + * @addr: the addr to DMA + * @length: memory length + * @dir: DMA direction + * + * Returns the DMA addr. + */ +dma_addr_t virtio_dma_map(struct device *dev, void *addr, unsigned int length, + enum dma_data_direction dir) +{ + struct virtio_device *vdev = dev_to_virtio(dev); + struct page *page; + size_t offset; + + page = virt_to_page(addr); + offset = offset_in_page(addr); + + if (!vring_use_dma_api(vdev)) + return page_to_phys(page) + offset; + + return dma_map_page(vdev->dev.parent, page, offset, length, dir); +} +EXPORT_SYMBOL_GPL(virtio_dma_map); + +/** + * virtio_dma_mapping_error - check dma address + * @dev: virtio device + * @addr: DMA address + * + * Returns 0 means dma valid. Other means invalid dma address. + */ +int virtio_dma_mapping_error(struct device *dev, dma_addr_t addr) +{ + struct virtio_device *vdev = dev_to_virtio(dev); + + if (!vring_use_dma_api(vdev)) + return 0; + + return dma_mapping_error(vdev->dev.parent, addr); +} +EXPORT_SYMBOL_GPL(virtio_dma_mapping_error); + +/** + * virtio_dma_unmap - unmap DMA addr + * @dev: virtio device + * @dma: DMA address + * @length: memory length + * @dir: DMA direction + */ +void virtio_dma_unmap(struct device *dev, dma_addr_t dma, unsigned int length, + enum dma_data_direction dir) +{ + struct virtio_device *vdev = dev_to_virtio(dev); + + if (!vring_use_dma_api(vdev)) + return; + + dma_unmap_page(vdev->dev.parent, dma, length, dir); +} +EXPORT_SYMBOL_GPL(virtio_dma_unmap); + MODULE_LICENSE("GPL"); diff --git a/include/linux/virtio.h b/include/linux/virtio.h index 72292a62cd90..e86ea3b1a124 100644 --- a/include/linux/virtio.h +++ b/include/linux/virtio.h @@ -9,6 +9,7 @@ #include #include #include +#include /** * virtqueue - a queue to register buffers for sending or receiving. @@ -196,4 +197,10 @@ void unregister_virtio_driver(struct virtio_driver *drv); #define module_virtio_driver(__virtio_driver) \ module_driver(__virtio_driver, register_virtio_driver, \ unregister_virtio_driver) + +dma_addr_t virtio_dma_map(struct device *dev, void *addr, unsigned int length, + enum dma_data_direction dir); +int virtio_dma_mapping_error(struct device *dev, dma_addr_t addr); +void virtio_dma_unmap(struct device *dev, dma_addr_t dma, unsigned int length, + enum dma_data_direction dir); #endif /* _LINUX_VIRTIO_H */ From patchwork Thu Feb 24 11:04:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 12758361 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3777EC43217 for ; Thu, 24 Feb 2022 11:04:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233771AbiBXLEx (ORCPT ); Thu, 24 Feb 2022 06:04:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33810 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232420AbiBXLEo (ORCPT ); Thu, 24 Feb 2022 06:04:44 -0500 Received: from out30-44.freemail.mail.aliyun.com (out30-44.freemail.mail.aliyun.com [115.124.30.44]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9D30728F97A; Thu, 24 Feb 2022 03:04:14 -0800 (PST) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R511e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04400;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=11;SR=0;TI=SMTPD_---0V5OACdw_1645700651; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0V5OACdw_1645700651) by smtp.aliyun-inc.com(127.0.0.1); Thu, 24 Feb 2022 19:04:12 +0800 From: Xuan Zhuo To: virtualization@lists.linux-foundation.org, netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , "David S. Miller" , Jakub Kicinski , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , bpf@vger.kernel.org Subject: [PATCH v2 8/9] virtio_ring: introduce virtqueue_add_outbuf_premapped() Date: Thu, 24 Feb 2022 19:04:01 +0800 Message-Id: <20220224110402.108161-9-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20220224110402.108161-1-xuanzhuo@linux.alibaba.com> References: <20220224110402.108161-1-xuanzhuo@linux.alibaba.com> MIME-Version: 1.0 X-Git-Hash: d02e086a8668 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Introduce virtqueue_add_outbuf_premapped() to submit premapped sgs. Signed-off-by: Xuan Zhuo --- drivers/virtio/virtio_ring.c | 25 +++++++++++++++++++++++++ include/linux/virtio.h | 5 +++++ 2 files changed, 30 insertions(+) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 6a1e0ef5498a..a76615aca2fd 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -1910,6 +1910,31 @@ int virtqueue_add_outbuf(struct virtqueue *vq, } EXPORT_SYMBOL_GPL(virtqueue_add_outbuf); +/** + * virtqueue_add_outbuf_premapped - expose output buffers to other end + * @vq: the struct virtqueue we're talking about. + * @sg: scatterlist (must be well-formed and terminated!) + * @num: the number of entries in @sg readable by other side + * @data: the token identifying the buffer. + * @gfp: how to do memory allocations (if necessary). + * + * Caller must ensure we don't call this with other virtqueue operations + * at the same time (except where noted). + * + * It is required that all addrs have completed DMA operations. And use + * sg->dma_address, sg->length to pass addr and length. + * + * Returns zero or a negative error (ie. ENOSPC, ENOMEM, EIO). + */ +int virtqueue_add_outbuf_premapped(struct virtqueue *vq, + struct scatterlist *sg, unsigned int num, + void *data, + gfp_t gfp) +{ + return virtqueue_add(vq, &sg, num, 1, 0, data, NULL, gfp, true); +} +EXPORT_SYMBOL_GPL(virtqueue_add_outbuf_premapped); + /** * virtqueue_add_inbuf - expose input buffers to other end * @vq: the struct virtqueue we're talking about. diff --git a/include/linux/virtio.h b/include/linux/virtio.h index e86ea3b1a124..dbe5a5f8a1ff 100644 --- a/include/linux/virtio.h +++ b/include/linux/virtio.h @@ -40,6 +40,11 @@ int virtqueue_add_outbuf(struct virtqueue *vq, void *data, gfp_t gfp); +int virtqueue_add_outbuf_premapped(struct virtqueue *vq, + struct scatterlist *sg, unsigned int num, + void *data, + gfp_t gfp); + int virtqueue_add_inbuf(struct virtqueue *vq, struct scatterlist sg[], unsigned int num, void *data, From patchwork Thu Feb 24 11:04:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 12758362 X-Patchwork-Delegate: kuba@kernel.org Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68C95C433EF for ; Thu, 24 Feb 2022 11:04:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233772AbiBXLEy (ORCPT ); Thu, 24 Feb 2022 06:04:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33828 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233758AbiBXLEq (ORCPT ); Thu, 24 Feb 2022 06:04:46 -0500 Received: from out30-131.freemail.mail.aliyun.com (out30-131.freemail.mail.aliyun.com [115.124.30.131]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0138928F97D; Thu, 24 Feb 2022 03:04:15 -0800 (PST) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R161e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04357;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=11;SR=0;TI=SMTPD_---0V5Nr1si_1645700652; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0V5Nr1si_1645700652) by smtp.aliyun-inc.com(127.0.0.1); Thu, 24 Feb 2022 19:04:13 +0800 From: Xuan Zhuo To: virtualization@lists.linux-foundation.org, netdev@vger.kernel.org Cc: "Michael S. Tsirkin" , Jason Wang , "David S. Miller" , Jakub Kicinski , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , bpf@vger.kernel.org Subject: [PATCH v2 9/9] virtio_net: xdp xmit use virtio dma api Date: Thu, 24 Feb 2022 19:04:02 +0800 Message-Id: <20220224110402.108161-10-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20220224110402.108161-1-xuanzhuo@linux.alibaba.com> References: <20220224110402.108161-1-xuanzhuo@linux.alibaba.com> MIME-Version: 1.0 X-Git-Hash: d02e086a8668 Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org X-Patchwork-Delegate: kuba@kernel.org XDP xmit uses virtio dma api for DMA operations. No longer let virtio core manage DMA address. To record the DMA address, allocate a space in the xdp_frame headroom to store the DMA address. Introduce virtnet_return_xdp_frame() to release the xdp frame and complete the dma unmap operation. Signed-off-by: Xuan Zhuo --- drivers/net/virtio_net.c | 42 +++++++++++++++++++++++++++++++++------- 1 file changed, 35 insertions(+), 7 deletions(-) diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index a801ea40908f..0efbf7992a95 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -321,6 +321,20 @@ static struct page *get_a_page(struct receive_queue *rq, gfp_t gfp_mask) return p; } +static void virtnet_return_xdp_frame(struct send_queue *sq, + struct xdp_frame *frame) +{ + struct virtnet_info *vi = sq->vq->vdev->priv; + dma_addr_t *p_addr, addr; + + p_addr = frame->data - sizeof(*p_addr); + addr = *p_addr; + + virtio_dma_unmap(&vi->vdev->dev, addr, frame->len, DMA_TO_DEVICE); + + xdp_return_frame(frame); +} + static void virtqueue_napi_schedule(struct napi_struct *napi, struct virtqueue *vq) { @@ -504,9 +518,11 @@ static int __virtnet_xdp_xmit_one(struct virtnet_info *vi, struct xdp_frame *xdpf) { struct virtio_net_hdr_mrg_rxbuf *hdr; + struct device *dev = &vi->vdev->dev; + dma_addr_t addr, *p_addr; int err; - if (unlikely(xdpf->headroom < vi->hdr_len)) + if (unlikely(xdpf->headroom < vi->hdr_len + sizeof(addr))) return -EOVERFLOW; /* Make room for virtqueue hdr (also change xdpf->headroom?) */ @@ -516,10 +532,21 @@ static int __virtnet_xdp_xmit_one(struct virtnet_info *vi, memset(hdr, 0, vi->hdr_len); xdpf->len += vi->hdr_len; - sg_init_one(sq->sg, xdpf->data, xdpf->len); + p_addr = xdpf->data - sizeof(addr); + + addr = virtio_dma_map(dev, xdpf->data, xdpf->len, DMA_TO_DEVICE); + + if (virtio_dma_mapping_error(dev, addr)) + return -ENOMEM; + + *p_addr = addr; + + sg_init_table(sq->sg, 1); + sq->sg->dma_address = addr; + sq->sg->length = xdpf->len; - err = virtqueue_add_outbuf(sq->vq, sq->sg, 1, xdp_to_ptr(xdpf), - GFP_ATOMIC); + err = virtqueue_add_outbuf_premapped(sq->vq, sq->sg, 1, + xdp_to_ptr(xdpf), GFP_ATOMIC); if (unlikely(err)) return -ENOSPC; /* Caller handle free/refcnt */ @@ -600,7 +627,7 @@ static int virtnet_xdp_xmit(struct net_device *dev, struct xdp_frame *frame = ptr_to_xdp(ptr); bytes += frame->len; - xdp_return_frame(frame); + virtnet_return_xdp_frame(sq, frame); } else { struct sk_buff *skb = ptr; @@ -1486,7 +1513,7 @@ static void free_old_xmit_skbs(struct send_queue *sq, bool in_napi) struct xdp_frame *frame = ptr_to_xdp(ptr); bytes += frame->len; - xdp_return_frame(frame); + virtnet_return_xdp_frame(sq, frame); } packets++; } @@ -2815,7 +2842,8 @@ static void free_unused_bufs(struct virtnet_info *vi) if (!is_xdp_frame(buf)) dev_kfree_skb(buf); else - xdp_return_frame(ptr_to_xdp(buf)); + virtnet_return_xdp_frame(vi->sq + i, + ptr_to_xdp(buf)); } }