From patchwork Tue Jan 30 03:05:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13536795 Received: from out30-112.freemail.mail.aliyun.com (out30-112.freemail.mail.aliyun.com [115.124.30.112]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C26C03716D; Tue, 30 Jan 2024 03:06:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.112 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706583980; cv=none; b=UCeQXcZ7qr8R22gQ/13HT3t+VlA0/TWEKNuGtXq0S4z8v8/wBWNhVhAxzs8prz9VWAUqrrice0OjMMHdjwyqw3PJzUIvaVRHzM+w05X/ThDqi6y52QNDCySImCSySLEy40hSmfy9Z2rSKOHiw5V9n1dnR02LwzixZOQLrS+50Bg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706583980; c=relaxed/simple; bh=w7D3FAxacz1cRePcp5nH/LN7vWiwihTU58oyIbIbYOM=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=qaOlRSwhs7C5baZSJ4LAvKa1kS7HNGXwesqAixlRz2+2pxTuuT60fpJCFPwxL9xZxjZZ3uUmnKOlns6CN9OEGkzlQmHqEHP98Fg7yLd4AAtJp7jRlLa2gioPLCuDCfzQFThJJK11jnNg9QFFGDGYlJwHYXBbCA8hoVPtO/XwlVg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=X458/85w; arc=none smtp.client-ip=115.124.30.112 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="X458/85w" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1706583969; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=HjElYrqlM2BjB7Xez3T79umxevfUAwxmJHZN/Z/xJIo=; b=X458/85wciOgK9Q19zfrfqlP7ZH61ROte7J1XebwTOOVSBkFx1zBbfnqwyLSpE606S3I3yueS5EMqZhZjJjOXbkIQcxbrhwPYiP9Q1KmOMQDZPY0AvTjb80CluVCXZnQoCPrBSlSx8LMPR0OZ7M+B8JlDLb6j7IoihYL62LX91Q= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R201e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046056;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=37;SR=0;TI=SMTPD_---0W.eaYrD_1706583966; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W.eaYrD_1706583966) by smtp.aliyun-inc.com; Tue, 30 Jan 2024 11:06:07 +0800 From: Xuan Zhuo To: virtualization@lists.linux.dev Cc: Richard Weinberger , Anton Ivanov , Johannes Berg , "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Hans de Goede , =?utf-8?q?Ilpo_J=C3=A4rvinen?= , Vadim Pasternak , Bjorn Andersson , Mathieu Poirier , Cornelia Huck , Halil Pasic , Eric Farman , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , Benjamin Berg , Yang Li , linux-um@lists.infradead.org, netdev@vger.kernel.org, platform-driver-x86@vger.kernel.org, linux-remoteproc@vger.kernel.org, linux-s390@vger.kernel.org, kvm@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH 01/14] virtio_ring: introduce vring_need_unmap_buffer Date: Tue, 30 Jan 2024 11:05:51 +0800 Message-Id: <20240130030604.108463-2-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240130030604.108463-1-xuanzhuo@linux.alibaba.com> References: <20240130030604.108463-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: linux-remoteproc@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: ce068f9b825d To make the code readable, introduce vring_need_unmap_buffer() to replace do_unmap. use_dma_api premapped -> vring_need_unmap_buffer() 1. false false false 2. true false true 3. true true false Signed-off-by: Xuan Zhuo --- drivers/virtio/virtio_ring.c | 33 +++++++++++++++------------------ 1 file changed, 15 insertions(+), 18 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 49299b1f9ec7..4677831e6c26 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -175,11 +175,6 @@ struct vring_virtqueue { /* Do DMA mapping by driver */ bool premapped; - /* Do unmap or not for desc. Just when premapped is False and - * use_dma_api is true, this is true. - */ - bool do_unmap; - /* Head of free buffer list. */ unsigned int free_head; /* Number we've added since last sync. */ @@ -297,6 +292,11 @@ static bool vring_use_dma_api(const struct virtio_device *vdev) return false; } +static bool vring_need_unmap_buffer(const struct vring_virtqueue *vring) +{ + return vring->use_dma_api && !vring->premapped; +} + size_t virtio_max_dma_size(const struct virtio_device *vdev) { size_t max_segment_size = SIZE_MAX; @@ -445,7 +445,7 @@ static void vring_unmap_one_split_indirect(const struct vring_virtqueue *vq, { u16 flags; - if (!vq->do_unmap) + if (!vring_need_unmap_buffer(vq)) return; flags = virtio16_to_cpu(vq->vq.vdev, desc->flags); @@ -475,7 +475,7 @@ static unsigned int vring_unmap_one_split(const struct vring_virtqueue *vq, (flags & VRING_DESC_F_WRITE) ? DMA_FROM_DEVICE : DMA_TO_DEVICE); } else { - if (!vq->do_unmap) + if (!vring_need_unmap_buffer(vq)) goto out; dma_unmap_page(vring_dma_dev(vq), @@ -643,7 +643,7 @@ static inline int virtqueue_add_split(struct virtqueue *_vq, } /* Last one doesn't continue. */ desc[prev].flags &= cpu_to_virtio16(_vq->vdev, ~VRING_DESC_F_NEXT); - if (!indirect && vq->do_unmap) + if (!indirect && vring_need_unmap_buffer(vq)) vq->split.desc_extra[prev & (vq->split.vring.num - 1)].flags &= ~VRING_DESC_F_NEXT; @@ -802,7 +802,7 @@ static void detach_buf_split(struct vring_virtqueue *vq, unsigned int head, VRING_DESC_F_INDIRECT)); BUG_ON(len == 0 || len % sizeof(struct vring_desc)); - if (vq->do_unmap) { + if (vring_need_unmap_buffer(vq)) { for (j = 0; j < len / sizeof(struct vring_desc); j++) vring_unmap_one_split_indirect(vq, &indir_desc[j]); } @@ -1236,7 +1236,7 @@ static void vring_unmap_extra_packed(const struct vring_virtqueue *vq, (flags & VRING_DESC_F_WRITE) ? DMA_FROM_DEVICE : DMA_TO_DEVICE); } else { - if (!vq->do_unmap) + if (!vring_need_unmap_buffer(vq)) return; dma_unmap_page(vring_dma_dev(vq), @@ -1251,7 +1251,7 @@ static void vring_unmap_desc_packed(const struct vring_virtqueue *vq, { u16 flags; - if (!vq->do_unmap) + if (!vring_need_unmap_buffer(vq)) return; flags = le16_to_cpu(desc->flags); @@ -1340,7 +1340,7 @@ static int virtqueue_add_indirect_packed(struct vring_virtqueue *vq, sizeof(struct vring_packed_desc)); vq->packed.vring.desc[head].id = cpu_to_le16(id); - if (vq->do_unmap) { + if (vring_need_unmap_buffer(vq)) { vq->packed.desc_extra[id].addr = addr; vq->packed.desc_extra[id].len = total_sg * sizeof(struct vring_packed_desc); @@ -1481,7 +1481,7 @@ static inline int virtqueue_add_packed(struct virtqueue *_vq, desc[i].len = cpu_to_le32(sg->length); desc[i].id = cpu_to_le16(id); - if (unlikely(vq->do_unmap)) { + if (unlikely(vring_need_unmap_buffer(vq))) { vq->packed.desc_extra[curr].addr = addr; vq->packed.desc_extra[curr].len = sg->length; vq->packed.desc_extra[curr].flags = @@ -1615,7 +1615,7 @@ static void detach_buf_packed(struct vring_virtqueue *vq, vq->free_head = id; vq->vq.num_free += state->num; - if (unlikely(vq->do_unmap)) { + if (unlikely(vring_need_unmap_buffer(vq))) { curr = id; for (i = 0; i < state->num; i++) { vring_unmap_extra_packed(vq, @@ -1632,7 +1632,7 @@ static void detach_buf_packed(struct vring_virtqueue *vq, if (!desc) return; - if (vq->do_unmap) { + if (vring_need_unmap_buffer(vq)) { len = vq->packed.desc_extra[id].len; for (i = 0; i < len / sizeof(struct vring_packed_desc); i++) @@ -2091,7 +2091,6 @@ static struct virtqueue *vring_create_virtqueue_packed( vq->dma_dev = dma_dev; vq->use_dma_api = vring_use_dma_api(vdev); vq->premapped = false; - vq->do_unmap = vq->use_dma_api; vq->indirect = virtio_has_feature(vdev, VIRTIO_RING_F_INDIRECT_DESC) && !context; @@ -2636,7 +2635,6 @@ static struct virtqueue *__vring_new_virtqueue(unsigned int index, vq->dma_dev = dma_dev; vq->use_dma_api = vring_use_dma_api(vdev); vq->premapped = false; - vq->do_unmap = vq->use_dma_api; vq->indirect = virtio_has_feature(vdev, VIRTIO_RING_F_INDIRECT_DESC) && !context; @@ -2804,7 +2802,6 @@ int virtqueue_set_dma_premapped(struct virtqueue *_vq) } vq->premapped = true; - vq->do_unmap = false; END_USE(vq); From patchwork Tue Jan 30 03:05:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13536796 Received: from out30-118.freemail.mail.aliyun.com (out30-118.freemail.mail.aliyun.com [115.124.30.118]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1470437700; Tue, 30 Jan 2024 03:06:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.118 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706583981; cv=none; b=B35RkyDCnt71Wv1c33HHTXzk0YW87FMNc1t2jg2iESf0AoLACtIhs8yKLhubWurC0tEWokzF9shmbvPsrDLG/qflNiNahaKZxWRWxT5Zv2sle/jQfES4VWL1BMdzJYwlH94vmy5A1N8oLI47BrAEoMJpOULpKHQpNTGAwEsftPw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706583981; c=relaxed/simple; bh=OLUJgu2GtDakzRNx8JjX/f1thHwVn9dH+TV16TIJ45Y=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=D6SwQm7LD1OJidzkbuXTTs43v5ILAKb2DcalWOJrG1JhTVgKxtpgZSFy0xfV/1kuJjuUSVdbM9aQgCJ+KqS883qTl5CfLXSOFNSFYblcAUnJFqveBYwQ/MLyzN9flJBSbu27Up1HH38vog/xXA9YNNG5tVHt9X99KTx821rO+zc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=yCq54cMp; arc=none smtp.client-ip=115.124.30.118 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="yCq54cMp" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1706583971; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=M/gA0U5hbdGfXbE+ate8yVaxvtF+5l5LRTg2RSnUFsU=; b=yCq54cMpsn4Wlm9+7BvkrOgHL3xqgtJ5ij+TnWcyv3pqHKkdNrl14VlDPdiiA6XWqYuYYk2is9O+HgOAtkXI4XURhXQ2E3fKcCMB1ZM0YM52P0pBcq51Sejs63soT82a4JKIVjIsA33YEsjRgmAY+j/TcOLXrJVL3n317rCELZU= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R181e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046050;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=37;SR=0;TI=SMTPD_---0W.eeC8U_1706583968; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W.eeC8U_1706583968) by smtp.aliyun-inc.com; Tue, 30 Jan 2024 11:06:09 +0800 From: Xuan Zhuo To: virtualization@lists.linux.dev Cc: Richard Weinberger , Anton Ivanov , Johannes Berg , "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Hans de Goede , =?utf-8?q?Ilpo_J=C3=A4rvinen?= , Vadim Pasternak , Bjorn Andersson , Mathieu Poirier , Cornelia Huck , Halil Pasic , Eric Farman , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , Benjamin Berg , Yang Li , linux-um@lists.infradead.org, netdev@vger.kernel.org, platform-driver-x86@vger.kernel.org, linux-remoteproc@vger.kernel.org, linux-s390@vger.kernel.org, kvm@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH 02/14] virtio_ring: packed: remove double check of the unmap ops Date: Tue, 30 Jan 2024 11:05:52 +0800 Message-Id: <20240130030604.108463-3-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240130030604.108463-1-xuanzhuo@linux.alibaba.com> References: <20240130030604.108463-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: linux-remoteproc@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: ce068f9b825d In the functions vring_unmap_extra_packed and vring_unmap_desc_packed, multiple checks are made whether unmap is performed and whether it is INDIRECT. These two functions are usually called in a loop, and we should put the check outside the loop. And we unmap the descs with VRING_DESC_F_INDIRECT on the same path with other descs, that make the thing more complex. If we distinguish the descs with VRING_DESC_F_INDIRECT before unmap, thing will be clearer. 1. only one desc of the desc table is used, we do not need the loop 2. the called unmap api is difference from the other desc 3. the vq->premapped is not needed to check 4. the vq->indirect is not needed to check 5. the state->indir_desc must not be null Signed-off-by: Xuan Zhuo --- drivers/virtio/virtio_ring.c | 76 ++++++++++++++++++------------------ 1 file changed, 39 insertions(+), 37 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 4677831e6c26..7280a1706cca 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -1220,6 +1220,7 @@ static u16 packed_last_used(u16 last_used_idx) return last_used_idx & ~(-(1 << VRING_PACKED_EVENT_F_WRAP_CTR)); } +/* caller must check vring_need_unmap_buffer() */ static void vring_unmap_extra_packed(const struct vring_virtqueue *vq, const struct vring_desc_extra *extra) { @@ -1227,33 +1228,18 @@ static void vring_unmap_extra_packed(const struct vring_virtqueue *vq, flags = extra->flags; - if (flags & VRING_DESC_F_INDIRECT) { - if (!vq->use_dma_api) - return; - - dma_unmap_single(vring_dma_dev(vq), - extra->addr, extra->len, - (flags & VRING_DESC_F_WRITE) ? - DMA_FROM_DEVICE : DMA_TO_DEVICE); - } else { - if (!vring_need_unmap_buffer(vq)) - return; - - dma_unmap_page(vring_dma_dev(vq), - extra->addr, extra->len, - (flags & VRING_DESC_F_WRITE) ? - DMA_FROM_DEVICE : DMA_TO_DEVICE); - } + dma_unmap_page(vring_dma_dev(vq), + extra->addr, extra->len, + (flags & VRING_DESC_F_WRITE) ? + DMA_FROM_DEVICE : DMA_TO_DEVICE); } +/* caller must check vring_need_unmap_buffer() */ static void vring_unmap_desc_packed(const struct vring_virtqueue *vq, const struct vring_packed_desc *desc) { u16 flags; - if (!vring_need_unmap_buffer(vq)) - return; - flags = le16_to_cpu(desc->flags); dma_unmap_page(vring_dma_dev(vq), @@ -1329,7 +1315,7 @@ static int virtqueue_add_indirect_packed(struct vring_virtqueue *vq, total_sg * sizeof(struct vring_packed_desc), DMA_TO_DEVICE); if (vring_mapping_error(vq, addr)) { - if (vq->premapped) + if (!vring_need_unmap_buffer(vq)) goto free_desc; goto unmap_release; @@ -1344,10 +1330,11 @@ static int virtqueue_add_indirect_packed(struct vring_virtqueue *vq, vq->packed.desc_extra[id].addr = addr; vq->packed.desc_extra[id].len = total_sg * sizeof(struct vring_packed_desc); - vq->packed.desc_extra[id].flags = VRING_DESC_F_INDIRECT | - vq->packed.avail_used_flags; } + vq->packed.desc_extra[id].flags = VRING_DESC_F_INDIRECT | + vq->packed.avail_used_flags; + /* * A driver MUST NOT make the first descriptor in the list * available before all subsequent descriptors comprising @@ -1388,6 +1375,8 @@ static int virtqueue_add_indirect_packed(struct vring_virtqueue *vq, unmap_release: err_idx = i; + WARN_ON(!vring_need_unmap_buffer(vq)); + for (i = 0; i < err_idx; i++) vring_unmap_desc_packed(vq, &desc[i]); @@ -1484,9 +1473,10 @@ static inline int virtqueue_add_packed(struct virtqueue *_vq, if (unlikely(vring_need_unmap_buffer(vq))) { vq->packed.desc_extra[curr].addr = addr; vq->packed.desc_extra[curr].len = sg->length; - vq->packed.desc_extra[curr].flags = - le16_to_cpu(flags); } + + vq->packed.desc_extra[curr].flags = le16_to_cpu(flags); + prev = curr; curr = vq->packed.desc_extra[curr].next; @@ -1536,6 +1526,8 @@ static inline int virtqueue_add_packed(struct virtqueue *_vq, vq->packed.avail_used_flags = avail_used_flags; + WARN_ON(!vring_need_unmap_buffer(vq)); + for (n = 0; n < total_sg; n++) { if (i == err_idx) break; @@ -1605,7 +1597,9 @@ static void detach_buf_packed(struct vring_virtqueue *vq, struct vring_desc_state_packed *state = NULL; struct vring_packed_desc *desc; unsigned int i, curr; + u16 flags; + flags = vq->packed.desc_extra[id].flags; state = &vq->packed.desc_state[id]; /* Clear data ptr. */ @@ -1615,22 +1609,32 @@ static void detach_buf_packed(struct vring_virtqueue *vq, vq->free_head = id; vq->vq.num_free += state->num; - if (unlikely(vring_need_unmap_buffer(vq))) { - curr = id; - for (i = 0; i < state->num; i++) { - vring_unmap_extra_packed(vq, - &vq->packed.desc_extra[curr]); - curr = vq->packed.desc_extra[curr].next; + if (!(flags & VRING_DESC_F_INDIRECT)) { + if (vring_need_unmap_buffer(vq)) { + curr = id; + for (i = 0; i < state->num; i++) { + vring_unmap_extra_packed(vq, + &vq->packed.desc_extra[curr]); + curr = vq->packed.desc_extra[curr].next; + } } - } - if (vq->indirect) { + if (ctx) + *ctx = state->indir_desc; + } else { + const struct vring_desc_extra *extra; u32 len; + if (vq->use_dma_api) { + extra = &vq->packed.desc_extra[id]; + dma_unmap_single(vring_dma_dev(vq), + extra->addr, extra->len, + (flags & VRING_DESC_F_WRITE) ? + DMA_FROM_DEVICE : DMA_TO_DEVICE); + } + /* Free the indirect table, if any, now that it's unmapped. */ desc = state->indir_desc; - if (!desc) - return; if (vring_need_unmap_buffer(vq)) { len = vq->packed.desc_extra[id].len; @@ -1640,8 +1644,6 @@ static void detach_buf_packed(struct vring_virtqueue *vq, } kfree(desc); state->indir_desc = NULL; - } else if (ctx) { - *ctx = state->indir_desc; } } From patchwork Tue Jan 30 03:05:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13536792 Received: from out30-132.freemail.mail.aliyun.com (out30-132.freemail.mail.aliyun.com [115.124.30.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 201A437162; Tue, 30 Jan 2024 03:06:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.132 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706583978; cv=none; b=DK7gg1KR7CWQpgPenUZOW5d+TZhmJMSOoE1Fwko0asdzDzz9HQapWuImgA7O8dNUv2jbQeQZ/H3WoI/bIaRIVBpHjP2T7BCp7YiOrG6QOIwP+rd0ksjctuCxBY/upETDooDqSte2OKo/zYXb90Lwh/sEBJjTsZzg9eSwSTtqBtk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706583978; c=relaxed/simple; bh=wtKZ3JnkbyRol+1QUCI3g+aTLGcAE86kgna4ZI/a4ik=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=pguTZvfnQ3rYu2aRpY+gqr00+i2Qi++xTDDyOtoTkxyZQ2bvzk3txiIUFMUxvPTm5lafZak1tWk//UqGGWRV2d6um18ZH2kRpql274iVUBJImAB7jutJratAK2mPYutgV+3ctIBM/kM8wgEXaPZtVB6+yNE7dEDy0r40dcDBwf8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=AWD2I1sC; arc=none smtp.client-ip=115.124.30.132 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="AWD2I1sC" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1706583973; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=PHE2stYM55HtJOlk3lTfdUsq+qQbKVYlz5rHB+a8ViQ=; b=AWD2I1sC09CoJTw9zRRWnPuHg0jkguqT9jVH931twGafk3YTHd+Rd8u1ZhMX84t6SU5PWVLvihmIwdaSMfg3XIKgryq7+fWBKkLVPLPazZEZ2zaXKVOQO8AfLQ985V9Lnm9gmlV+M1uxw2q1aiJq/vbmieDvU3logMcbOlXbFac= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R171e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045192;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=37;SR=0;TI=SMTPD_---0W.efIQZ_1706583970; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W.efIQZ_1706583970) by smtp.aliyun-inc.com; Tue, 30 Jan 2024 11:06:11 +0800 From: Xuan Zhuo To: virtualization@lists.linux.dev Cc: Richard Weinberger , Anton Ivanov , Johannes Berg , "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Hans de Goede , =?utf-8?q?Ilpo_J=C3=A4rvinen?= , Vadim Pasternak , Bjorn Andersson , Mathieu Poirier , Cornelia Huck , Halil Pasic , Eric Farman , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , Benjamin Berg , Yang Li , linux-um@lists.infradead.org, netdev@vger.kernel.org, platform-driver-x86@vger.kernel.org, linux-remoteproc@vger.kernel.org, linux-s390@vger.kernel.org, kvm@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH 03/14] virtio_ring: packed: structure the indirect desc table Date: Tue, 30 Jan 2024 11:05:53 +0800 Message-Id: <20240130030604.108463-4-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240130030604.108463-1-xuanzhuo@linux.alibaba.com> References: <20240130030604.108463-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: linux-remoteproc@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: ce068f9b825d This commit structure the indirect desc table. Then we can get the desc num directly when doing unmap. And save the dma info to the struct, then the indirect will not use the dma fields of the desc_extra. The subsequent commits will make the dma fields are optional. But for the indirect case, we must record the dma info. Signed-off-by: Xuan Zhuo --- drivers/virtio/virtio_ring.c | 63 ++++++++++++++++++++---------------- 1 file changed, 35 insertions(+), 28 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 7280a1706cca..dd03bc5a81fe 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -72,9 +72,16 @@ struct vring_desc_state_split { struct vring_desc *indir_desc; /* Indirect descriptor, if any. */ }; +struct vring_packed_desc_indir { + dma_addr_t addr; /* Descriptor Array DMA addr. */ + u32 len; /* Descriptor Array length. */ + u32 num; + struct vring_packed_desc desc[]; +}; + struct vring_desc_state_packed { void *data; /* Data for callback. */ - struct vring_packed_desc *indir_desc; /* Indirect descriptor, if any. */ + struct vring_packed_desc_indir *indir_desc; /* Indirect descriptor, if any. */ u16 num; /* Descriptor list length. */ u16 last; /* The last desc state in a list. */ }; @@ -1249,10 +1256,13 @@ static void vring_unmap_desc_packed(const struct vring_virtqueue *vq, DMA_FROM_DEVICE : DMA_TO_DEVICE); } -static struct vring_packed_desc *alloc_indirect_packed(unsigned int total_sg, +static struct vring_packed_desc_indir *alloc_indirect_packed(unsigned int total_sg, gfp_t gfp) { - struct vring_packed_desc *desc; + struct vring_packed_desc_indir *in_desc; + u32 size; + + size = struct_size(in_desc, desc, total_sg); /* * We require lowmem mappings for the descriptors because @@ -1261,9 +1271,10 @@ static struct vring_packed_desc *alloc_indirect_packed(unsigned int total_sg, */ gfp &= ~__GFP_HIGHMEM; - desc = kmalloc_array(total_sg, sizeof(struct vring_packed_desc), gfp); - return desc; + in_desc = kmalloc(size, gfp); + + return in_desc; } static int virtqueue_add_indirect_packed(struct vring_virtqueue *vq, @@ -1274,6 +1285,7 @@ static int virtqueue_add_indirect_packed(struct vring_virtqueue *vq, void *data, gfp_t gfp) { + struct vring_packed_desc_indir *in_desc; struct vring_packed_desc *desc; struct scatterlist *sg; unsigned int i, n, err_idx; @@ -1281,10 +1293,12 @@ static int virtqueue_add_indirect_packed(struct vring_virtqueue *vq, dma_addr_t addr; head = vq->packed.next_avail_idx; - desc = alloc_indirect_packed(total_sg, gfp); - if (!desc) + in_desc = alloc_indirect_packed(total_sg, gfp); + if (!in_desc) return -ENOMEM; + desc = in_desc->desc; + if (unlikely(vq->vq.num_free < 1)) { pr_debug("Can't add buf len 1 - avail = 0\n"); kfree(desc); @@ -1321,17 +1335,15 @@ static int virtqueue_add_indirect_packed(struct vring_virtqueue *vq, goto unmap_release; } + in_desc->num = i; + in_desc->addr = addr; + in_desc->len = total_sg * sizeof(struct vring_packed_desc); + vq->packed.vring.desc[head].addr = cpu_to_le64(addr); vq->packed.vring.desc[head].len = cpu_to_le32(total_sg * sizeof(struct vring_packed_desc)); vq->packed.vring.desc[head].id = cpu_to_le16(id); - if (vring_need_unmap_buffer(vq)) { - vq->packed.desc_extra[id].addr = addr; - vq->packed.desc_extra[id].len = total_sg * - sizeof(struct vring_packed_desc); - } - vq->packed.desc_extra[id].flags = VRING_DESC_F_INDIRECT | vq->packed.avail_used_flags; @@ -1362,7 +1374,7 @@ static int virtqueue_add_indirect_packed(struct vring_virtqueue *vq, /* Store token and indirect buffer state. */ vq->packed.desc_state[id].num = 1; vq->packed.desc_state[id].data = data; - vq->packed.desc_state[id].indir_desc = desc; + vq->packed.desc_state[id].indir_desc = in_desc; vq->packed.desc_state[id].last = id; vq->num_added += 1; @@ -1381,7 +1393,7 @@ static int virtqueue_add_indirect_packed(struct vring_virtqueue *vq, vring_unmap_desc_packed(vq, &desc[i]); free_desc: - kfree(desc); + kfree(in_desc); END_USE(vq); return -ENOMEM; @@ -1595,7 +1607,6 @@ static void detach_buf_packed(struct vring_virtqueue *vq, unsigned int id, void **ctx) { struct vring_desc_state_packed *state = NULL; - struct vring_packed_desc *desc; unsigned int i, curr; u16 flags; @@ -1621,28 +1632,24 @@ static void detach_buf_packed(struct vring_virtqueue *vq, if (ctx) *ctx = state->indir_desc; + } else { - const struct vring_desc_extra *extra; - u32 len; + struct vring_packed_desc_indir *in_desc; + + in_desc = state->indir_desc; if (vq->use_dma_api) { - extra = &vq->packed.desc_extra[id]; dma_unmap_single(vring_dma_dev(vq), - extra->addr, extra->len, + in_desc->addr, in_desc->len, (flags & VRING_DESC_F_WRITE) ? DMA_FROM_DEVICE : DMA_TO_DEVICE); } - /* Free the indirect table, if any, now that it's unmapped. */ - desc = state->indir_desc; - if (vring_need_unmap_buffer(vq)) { - len = vq->packed.desc_extra[id].len; - for (i = 0; i < len / sizeof(struct vring_packed_desc); - i++) - vring_unmap_desc_packed(vq, &desc[i]); + for (i = 0; i < in_desc->num; i++) + vring_unmap_desc_packed(vq, &in_desc->desc[i]); } - kfree(desc); + kfree(in_desc); state->indir_desc = NULL; } } From patchwork Tue Jan 30 03:05:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13536794 Received: from out30-113.freemail.mail.aliyun.com (out30-113.freemail.mail.aliyun.com [115.124.30.113]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0D30137169; Tue, 30 Jan 2024 03:06:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.113 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706583979; cv=none; b=Mb3IeFytdAaMaZtpWmpdm1gQHRlvVaXx5aHifVLNIQGfI6/1uOSTIk2esv/IsWALikJIJm/c20vqvpCrzCXYYhI+X0s8+xYQ5+4CLKkh5foQXp/IAkMUvXK1YVfk2R4fAslxG9wVltsf8A4Fam9pxEndO+RFU5qUH32YFQ/Or9Q= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706583979; c=relaxed/simple; bh=csVqBjWHLd0wx8pt5oq5iqtKVwsT+VuwzwY4U+gqvHU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Ip05okV+75oOb8Nfl34DeBBOTJtlFPGMfC4cyArUHMfGZvPH8kZelDmWf6box12cboOFe/uAU+MVy+nCQW0oZhaKWA/uHMZX63e05WH+IfQ2HMG737f7EkDBaplQ7IF56hrYPTA6Vo6dBKXOevzubot/ryCcfsAOdbTbqVnWjzY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=Rg0+Na16; arc=none smtp.client-ip=115.124.30.113 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="Rg0+Na16" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1706583974; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=HrZuvU9yrR6T2DolteXZPfHuA2cDqF8W4GegYsiFG0w=; b=Rg0+Na16isyLrkJKyV5IXWoWZ9qXy8theO2Eb/e39H+BEkhUgdvcs5BXjxbdhm7vTcvAsaJbYWp+8l2o6eqWEVHX30Lv3L89euGbKRuEBTmbkPXmrXVYQqM6nUNWOyJp7gz/TKiJvPiIkJ8JiF4GR9ZareyoRtiLRJsBEdtTF0w= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R661e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045170;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=37;SR=0;TI=SMTPD_---0W.eeC9v_1706583971; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W.eeC9v_1706583971) by smtp.aliyun-inc.com; Tue, 30 Jan 2024 11:06:12 +0800 From: Xuan Zhuo To: virtualization@lists.linux.dev Cc: Richard Weinberger , Anton Ivanov , Johannes Berg , "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Hans de Goede , =?utf-8?q?Ilpo_J=C3=A4rvinen?= , Vadim Pasternak , Bjorn Andersson , Mathieu Poirier , Cornelia Huck , Halil Pasic , Eric Farman , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , Benjamin Berg , Yang Li , linux-um@lists.infradead.org, netdev@vger.kernel.org, platform-driver-x86@vger.kernel.org, linux-remoteproc@vger.kernel.org, linux-s390@vger.kernel.org, kvm@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH 04/14] virtio_ring: split: remove double check of the unmap ops Date: Tue, 30 Jan 2024 11:05:54 +0800 Message-Id: <20240130030604.108463-5-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240130030604.108463-1-xuanzhuo@linux.alibaba.com> References: <20240130030604.108463-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: linux-remoteproc@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: ce068f9b825d In the functions vring_unmap_one_split and vring_unmap_one_split_indirect, multiple checks are made whether unmap is performed and whether it is INDIRECT. These two functions are usually called in a loop, and we should put the check outside the loop. And we unmap the descs with VRING_DESC_F_INDIRECT on the same path with other descs, that make the thing more complex. If we distinguish the descs with VRING_DESC_F_INDIRECT before unmap, thing will be clearer. 1. only one desc of the desc table is used, we do not need the loop 2. the called unmap api is difference from the other desc 3. the vq->premapped is not needed to check 4. the vq->indirect is not needed to check 5. the state->indir_desc must not be null Signed-off-by: Xuan Zhuo --- drivers/virtio/virtio_ring.c | 80 ++++++++++++++++++------------------ 1 file changed, 39 insertions(+), 41 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index dd03bc5a81fe..2b41fdbce975 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -452,9 +452,6 @@ static void vring_unmap_one_split_indirect(const struct vring_virtqueue *vq, { u16 flags; - if (!vring_need_unmap_buffer(vq)) - return; - flags = virtio16_to_cpu(vq->vq.vdev, desc->flags); dma_unmap_page(vring_dma_dev(vq), @@ -472,27 +469,12 @@ static unsigned int vring_unmap_one_split(const struct vring_virtqueue *vq, flags = extra[i].flags; - if (flags & VRING_DESC_F_INDIRECT) { - if (!vq->use_dma_api) - goto out; - - dma_unmap_single(vring_dma_dev(vq), - extra[i].addr, - extra[i].len, - (flags & VRING_DESC_F_WRITE) ? - DMA_FROM_DEVICE : DMA_TO_DEVICE); - } else { - if (!vring_need_unmap_buffer(vq)) - goto out; - - dma_unmap_page(vring_dma_dev(vq), - extra[i].addr, - extra[i].len, - (flags & VRING_DESC_F_WRITE) ? - DMA_FROM_DEVICE : DMA_TO_DEVICE); - } + dma_unmap_page(vring_dma_dev(vq), + extra[i].addr, + extra[i].len, + (flags & VRING_DESC_F_WRITE) ? + DMA_FROM_DEVICE : DMA_TO_DEVICE); -out: return extra[i].next; } @@ -660,7 +642,7 @@ static inline int virtqueue_add_split(struct virtqueue *_vq, vq, desc, total_sg * sizeof(struct vring_desc), DMA_TO_DEVICE); if (vring_mapping_error(vq, addr)) { - if (vq->premapped) + if (!vring_need_unmap_buffer(vq)) goto free_indirect; goto unmap_release; @@ -713,6 +695,9 @@ static inline int virtqueue_add_split(struct virtqueue *_vq, return 0; unmap_release: + + WARN_ON(!vring_need_unmap_buffer(vq)); + err_idx = i; if (indirect) @@ -774,34 +759,42 @@ static void detach_buf_split(struct vring_virtqueue *vq, unsigned int head, { unsigned int i, j; __virtio16 nextflag = cpu_to_virtio16(vq->vq.vdev, VRING_DESC_F_NEXT); + u16 flags; /* Clear data ptr. */ vq->split.desc_state[head].data = NULL; + flags = vq->split.desc_extra[head].flags; /* Put back on free list: unmap first-level descriptors and find end */ i = head; - while (vq->split.vring.desc[i].flags & nextflag) { - vring_unmap_one_split(vq, i); - i = vq->split.desc_extra[i].next; - vq->vq.num_free++; - } - - vring_unmap_one_split(vq, i); - vq->split.desc_extra[i].next = vq->free_head; - vq->free_head = head; + if (!(flags & VRING_DESC_F_INDIRECT)) { + while (vq->split.vring.desc[i].flags & nextflag) { + if (vring_need_unmap_buffer(vq)) + vring_unmap_one_split(vq, i); + i = vq->split.desc_extra[i].next; + vq->vq.num_free++; + } - /* Plus final descriptor */ - vq->vq.num_free++; + if (vring_need_unmap_buffer(vq)) + vring_unmap_one_split(vq, i); - if (vq->indirect) { + if (ctx) + *ctx = vq->split.desc_state[head].indir_desc; + } else { struct vring_desc *indir_desc = vq->split.desc_state[head].indir_desc; u32 len; - /* Free the indirect table, if any, now that it's unmapped. */ - if (!indir_desc) - return; + if (vq->use_dma_api) { + struct vring_desc_extra *extra = vq->split.desc_extra; + + dma_unmap_single(vring_dma_dev(vq), + extra[i].addr, + extra[i].len, + (flags & VRING_DESC_F_WRITE) ? + DMA_FROM_DEVICE : DMA_TO_DEVICE); + } len = vq->split.desc_extra[head].len; @@ -816,9 +809,14 @@ static void detach_buf_split(struct vring_virtqueue *vq, unsigned int head, kfree(indir_desc); vq->split.desc_state[head].indir_desc = NULL; - } else if (ctx) { - *ctx = vq->split.desc_state[head].indir_desc; + } + + vq->split.desc_extra[i].next = vq->free_head; + vq->free_head = head; + + /* Plus final descriptor */ + vq->vq.num_free++; } static bool more_used_split(const struct vring_virtqueue *vq) From patchwork Tue Jan 30 03:05:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13536799 Received: from out30-119.freemail.mail.aliyun.com (out30-119.freemail.mail.aliyun.com [115.124.30.119]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5EBAB433CD; Tue, 30 Jan 2024 03:06:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.119 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706583986; cv=none; b=AuHjMmzMVn4IJpXtPH03VtgueAp2SbhVzD6r0Xe/eFyBQbt/1winGvh/z8cZ3Mx8NPN6lzcD83CpNEyQRkVIiaFHz/9BTpXRWUVFksN/WnacuEhU/NH6wbi7Nm0FqDICbaYGSrxCHpL1GeSpyqYuaKZdFaFNeHdYSVqyFuY6yAE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706583986; c=relaxed/simple; bh=6G571sJ4EdFAIcFL28AlPWeFbDahxvOWW2JSK/6QgjY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=T8dvanm5keMBV7+E0/TQggsub142PZSREvWC0DrXNtfJs/8GIv0rw52WUXG7+6PVkseubqpjv/Lx/JkPTQuIbHWSMxZlx4jZ66zJdAclA4sZcCNjF2ZJJ4eYkUv3vA5Qkqb7mxxepndbIKbI4P0Hz7/b8poDmsgCDHLymsovmtk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=Hj3koiwO; arc=none smtp.client-ip=115.124.30.119 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="Hj3koiwO" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1706583976; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=z3UTRsbRsFcB2sqFN3XBDD7wrtKUw8K4tiUA1HwO3kw=; b=Hj3koiwO8iHFD0m5xzlwE8GYQFDHQ7bEldOE+q+irZoGNWNogbFo2TuLzeUjnw3RP4s/DfpX7pTWViwkGGjj1IFAqty2RB1GQ3CsMxnb4+Tsles431kGW2FRau1fysa7GilH6fLkWLpeYIRj3lAUg/kMgrU5jziQQybIwPpalRU= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R101e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046050;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=37;SR=0;TI=SMTPD_---0W.eeCAv_1706583973; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W.eeCAv_1706583973) by smtp.aliyun-inc.com; Tue, 30 Jan 2024 11:06:14 +0800 From: Xuan Zhuo To: virtualization@lists.linux.dev Cc: Richard Weinberger , Anton Ivanov , Johannes Berg , "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Hans de Goede , =?utf-8?q?Ilpo_J=C3=A4rvinen?= , Vadim Pasternak , Bjorn Andersson , Mathieu Poirier , Cornelia Huck , Halil Pasic , Eric Farman , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , Benjamin Berg , Yang Li , linux-um@lists.infradead.org, netdev@vger.kernel.org, platform-driver-x86@vger.kernel.org, linux-remoteproc@vger.kernel.org, linux-s390@vger.kernel.org, kvm@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH 05/14] virtio_ring: split: structure the indirect desc table Date: Tue, 30 Jan 2024 11:05:55 +0800 Message-Id: <20240130030604.108463-6-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240130030604.108463-1-xuanzhuo@linux.alibaba.com> References: <20240130030604.108463-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: linux-remoteproc@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: ce068f9b825d This commit structure the indirect desc table. Then we can get the desc num directly when doing unmap. And save the dma info to the struct, then the indirect will not use the dma fields of the desc_extra. The subsequent commits will make the dma fields are optional. But for the indirect case, we must record the dma info. Signed-off-by: Xuan Zhuo --- drivers/virtio/virtio_ring.c | 86 ++++++++++++++++++++++-------------- 1 file changed, 52 insertions(+), 34 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 2b41fdbce975..831667a57429 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -67,9 +67,16 @@ #define LAST_ADD_TIME_INVALID(vq) #endif +struct vring_split_desc_indir { + dma_addr_t addr; /* Descriptor Array DMA addr. */ + u32 len; /* Descriptor Array length. */ + u32 num; + struct vring_desc desc[]; +}; + struct vring_desc_state_split { void *data; /* Data for callback. */ - struct vring_desc *indir_desc; /* Indirect descriptor, if any. */ + struct vring_split_desc_indir *indir_desc; /* Indirect descriptor, if any. */ }; struct vring_packed_desc_indir { @@ -478,12 +485,16 @@ static unsigned int vring_unmap_one_split(const struct vring_virtqueue *vq, return extra[i].next; } -static struct vring_desc *alloc_indirect_split(struct virtqueue *_vq, - unsigned int total_sg, - gfp_t gfp) +static struct vring_split_desc_indir *alloc_indirect_split(struct virtqueue *_vq, + unsigned int total_sg, + gfp_t gfp) { + struct vring_split_desc_indir *in_desc; struct vring_desc *desc; unsigned int i; + u32 size; + + size = struct_size(in_desc, desc, total_sg); /* * We require lowmem mappings for the descriptors because @@ -492,13 +503,16 @@ static struct vring_desc *alloc_indirect_split(struct virtqueue *_vq, */ gfp &= ~__GFP_HIGHMEM; - desc = kmalloc_array(total_sg, sizeof(struct vring_desc), gfp); - if (!desc) + in_desc = kmalloc(size, gfp); + if (!in_desc) return NULL; + desc = in_desc->desc; + for (i = 0; i < total_sg; i++) desc[i].next = cpu_to_virtio16(_vq->vdev, i + 1); - return desc; + + return in_desc; } static inline unsigned int virtqueue_add_desc_split(struct virtqueue *vq, @@ -540,6 +554,7 @@ static inline int virtqueue_add_split(struct virtqueue *_vq, gfp_t gfp) { struct vring_virtqueue *vq = to_vvq(_vq); + struct vring_split_desc_indir *in_desc; struct scatterlist *sg; struct vring_desc *desc; unsigned int i, n, avail, descs_used, prev, err_idx; @@ -562,9 +577,13 @@ static inline int virtqueue_add_split(struct virtqueue *_vq, head = vq->free_head; - if (virtqueue_use_indirect(vq, total_sg)) - desc = alloc_indirect_split(_vq, total_sg, gfp); - else { + if (virtqueue_use_indirect(vq, total_sg)) { + in_desc = alloc_indirect_split(_vq, total_sg, gfp); + if (!in_desc) + desc = NULL; + else + desc = in_desc->desc; + } else { desc = NULL; WARN_ON_ONCE(total_sg > vq->split.vring.num && !vq->indirect); } @@ -637,10 +656,10 @@ static inline int virtqueue_add_split(struct virtqueue *_vq, ~VRING_DESC_F_NEXT; if (indirect) { + u32 size = total_sg * sizeof(struct vring_desc); + /* Now that the indirect table is filled in, map it. */ - dma_addr_t addr = vring_map_single( - vq, desc, total_sg * sizeof(struct vring_desc), - DMA_TO_DEVICE); + dma_addr_t addr = vring_map_single(vq, desc, size, DMA_TO_DEVICE); if (vring_mapping_error(vq, addr)) { if (!vring_need_unmap_buffer(vq)) goto free_indirect; @@ -648,11 +667,17 @@ static inline int virtqueue_add_split(struct virtqueue *_vq, goto unmap_release; } - virtqueue_add_desc_split(_vq, vq->split.vring.desc, - head, addr, - total_sg * sizeof(struct vring_desc), - VRING_DESC_F_INDIRECT, - false); + desc = &vq->split.vring.desc[head]; + + desc->flags = cpu_to_virtio16(_vq->vdev, VRING_DESC_F_INDIRECT); + desc->addr = cpu_to_virtio64(_vq->vdev, addr); + desc->len = cpu_to_virtio32(_vq->vdev, size); + + vq->split.desc_extra[head].flags = VRING_DESC_F_INDIRECT; + + in_desc->addr = addr; + in_desc->len = size; + in_desc->num = total_sg; } /* We're using some buffers from the free list. */ @@ -667,7 +692,7 @@ static inline int virtqueue_add_split(struct virtqueue *_vq, /* Store token and indirect buffer state. */ vq->split.desc_state[head].data = data; if (indirect) - vq->split.desc_state[head].indir_desc = desc; + vq->split.desc_state[head].indir_desc = in_desc; else vq->split.desc_state[head].indir_desc = ctx; @@ -717,7 +742,7 @@ static inline int virtqueue_add_split(struct virtqueue *_vq, free_indirect: if (indirect) - kfree(desc); + kfree(in_desc); END_USE(vq); return -ENOMEM; @@ -782,34 +807,27 @@ static void detach_buf_split(struct vring_virtqueue *vq, unsigned int head, if (ctx) *ctx = vq->split.desc_state[head].indir_desc; } else { - struct vring_desc *indir_desc = - vq->split.desc_state[head].indir_desc; - u32 len; + struct vring_split_desc_indir *in_desc; - if (vq->use_dma_api) { - struct vring_desc_extra *extra = vq->split.desc_extra; + in_desc = vq->split.desc_state[head].indir_desc; + if (vq->use_dma_api) { dma_unmap_single(vring_dma_dev(vq), - extra[i].addr, - extra[i].len, + in_desc->addr, in_desc->len, (flags & VRING_DESC_F_WRITE) ? DMA_FROM_DEVICE : DMA_TO_DEVICE); } - len = vq->split.desc_extra[head].len; - BUG_ON(!(vq->split.desc_extra[head].flags & VRING_DESC_F_INDIRECT)); - BUG_ON(len == 0 || len % sizeof(struct vring_desc)); if (vring_need_unmap_buffer(vq)) { - for (j = 0; j < len / sizeof(struct vring_desc); j++) - vring_unmap_one_split_indirect(vq, &indir_desc[j]); + for (j = 0; j < in_desc->num; j++) + vring_unmap_one_split_indirect(vq, &in_desc->desc[j]); } - kfree(indir_desc); + kfree(in_desc); vq->split.desc_state[head].indir_desc = NULL; - } vq->split.desc_extra[i].next = vq->free_head; From patchwork Tue Jan 30 03:05:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13536797 Received: from out30-119.freemail.mail.aliyun.com (out30-119.freemail.mail.aliyun.com [115.124.30.119]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D3F02376F3; Tue, 30 Jan 2024 03:06:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.119 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706583981; cv=none; b=i7EaHe96CcblgyCSo9zxNQMLtT8m88hKC05gu10sbQX/qoFKFCEtNnvmqE6+lq5yrnqKQqa6mF+Ql4FnVXthPVtWSDd4WkG4a/jHGd9DErCZWMVsOZYKm/asj6BfeLWYW+NG58Nd5LLY0HdaB70XCOnrQ1l0QaP2TJEiC/O5DVo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706583981; c=relaxed/simple; bh=sioBitvVODKWFEi3It7y9hTCuYhRhlwoPb8vZ+NKaXo=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=WH4PaLJ8aM0R5WC3fJ+pdKn+BRphOcK8v8heBctYoKY0q50qMvqs029uMWUxtXE9McGjYfOpzCEcEuyy5GXYLc5bJR1lrwS38PZC+4Iu/rozwv8AmSlsR++tjTMeWCxPOIOdP2WUD4YBZKBq+aCnGO9ByCNPVyPJJx6Jg5EVd6o= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=nvViitah; arc=none smtp.client-ip=115.124.30.119 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="nvViitah" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1706583977; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=aJW45Loje2yp/4dQUt3QyiyDsaI3FGnc4OA+IXxlF9E=; b=nvViitahQB6xEW9ju5Mdk/hTwNdSncmPesEPhwPpTa0hy9MMiJ5RKqgaGziZDZ0Z0rq83xiCva8nAFZUSotU29eBux1bUsmWaN0tLpkH3DYitb3w66lR9Uo2MkB6FYUzUGfRN65lRcJDNao6XKllBA7bg4zeAYXRG8zyNL5FS6k= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R721e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046056;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=37;SR=0;TI=SMTPD_---0W.eaYvT_1706583975; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W.eaYvT_1706583975) by smtp.aliyun-inc.com; Tue, 30 Jan 2024 11:06:16 +0800 From: Xuan Zhuo To: virtualization@lists.linux.dev Cc: Richard Weinberger , Anton Ivanov , Johannes Berg , "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Hans de Goede , =?utf-8?q?Ilpo_J=C3=A4rvinen?= , Vadim Pasternak , Bjorn Andersson , Mathieu Poirier , Cornelia Huck , Halil Pasic , Eric Farman , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , Benjamin Berg , Yang Li , linux-um@lists.infradead.org, netdev@vger.kernel.org, platform-driver-x86@vger.kernel.org, linux-remoteproc@vger.kernel.org, linux-s390@vger.kernel.org, kvm@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH 06/14] virtio_ring: no store dma info when unmap is not needed Date: Tue, 30 Jan 2024 11:05:56 +0800 Message-Id: <20240130030604.108463-7-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240130030604.108463-1-xuanzhuo@linux.alibaba.com> References: <20240130030604.108463-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: linux-remoteproc@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: ce068f9b825d As discussed: http://lore.kernel.org/all/CACGkMEug-=C+VQhkMYSgUKMC==04m7-uem_yC21bgGkKZh845w@mail.gmail.com When the vq is premapped mode, the driver manages the dma info is a good way. So this commit make the virtio core not to store the dma info and release the memory which is used to store the dma info. If the use_dma_api is false, the memory is also not allocated. Signed-off-by: Xuan Zhuo --- drivers/virtio/virtio_ring.c | 89 ++++++++++++++++++++++++++++-------- 1 file changed, 70 insertions(+), 19 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 831667a57429..5bea25167259 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -94,12 +94,15 @@ struct vring_desc_state_packed { }; struct vring_desc_extra { - dma_addr_t addr; /* Descriptor DMA addr. */ - u32 len; /* Descriptor length. */ u16 flags; /* Descriptor flags. */ u16 next; /* The next desc state in a list. */ }; +struct vring_desc_dma { + dma_addr_t addr; /* Descriptor DMA addr. */ + u32 len; /* Descriptor length. */ +}; + struct vring_virtqueue_split { /* Actual memory layout for this queue. */ struct vring vring; @@ -116,6 +119,7 @@ struct vring_virtqueue_split { /* Per-descriptor state. */ struct vring_desc_state_split *desc_state; struct vring_desc_extra *desc_extra; + struct vring_desc_dma *desc_dma; /* DMA address and size information */ dma_addr_t queue_dma_addr; @@ -156,6 +160,7 @@ struct vring_virtqueue_packed { /* Per-descriptor state. */ struct vring_desc_state_packed *desc_state; struct vring_desc_extra *desc_extra; + struct vring_desc_dma *desc_dma; /* DMA address and size information */ dma_addr_t ring_dma_addr; @@ -472,13 +477,14 @@ static unsigned int vring_unmap_one_split(const struct vring_virtqueue *vq, unsigned int i) { struct vring_desc_extra *extra = vq->split.desc_extra; + struct vring_desc_dma *dma = vq->split.desc_dma; u16 flags; flags = extra[i].flags; dma_unmap_page(vring_dma_dev(vq), - extra[i].addr, - extra[i].len, + dma[i].addr, + dma[i].len, (flags & VRING_DESC_F_WRITE) ? DMA_FROM_DEVICE : DMA_TO_DEVICE); @@ -535,8 +541,11 @@ static inline unsigned int virtqueue_add_desc_split(struct virtqueue *vq, next = extra[i].next; desc[i].next = cpu_to_virtio16(vq->vdev, next); - extra[i].addr = addr; - extra[i].len = len; + if (vring->split.desc_dma) { + vring->split.desc_dma[i].addr = addr; + vring->split.desc_dma[i].len = len; + } + extra[i].flags = flags; } else next = virtio16_to_cpu(vq->vdev, desc[i].next); @@ -1072,16 +1081,26 @@ static void virtqueue_vring_attach_split(struct vring_virtqueue *vq, vq->free_head = 0; } -static int vring_alloc_state_extra_split(struct vring_virtqueue_split *vring_split) +static int vring_alloc_state_extra_split(struct vring_virtqueue_split *vring_split, + bool need_unmap) { struct vring_desc_state_split *state; struct vring_desc_extra *extra; + struct vring_desc_dma *dma; u32 num = vring_split->vring.num; state = kmalloc_array(num, sizeof(struct vring_desc_state_split), GFP_KERNEL); if (!state) goto err_state; + if (need_unmap) { + dma = kmalloc_array(num, sizeof(struct vring_desc_dma), GFP_KERNEL); + if (!dma) + goto err_dma; + } else { + dma = NULL; + } + extra = vring_alloc_desc_extra(num); if (!extra) goto err_extra; @@ -1090,9 +1109,12 @@ static int vring_alloc_state_extra_split(struct vring_virtqueue_split *vring_spl vring_split->desc_state = state; vring_split->desc_extra = extra; + vring_split->desc_dma = dma; return 0; err_extra: + kfree(dma); +err_dma: kfree(state); err_state: return -ENOMEM; @@ -1108,6 +1130,7 @@ static void vring_free_split(struct vring_virtqueue_split *vring_split, kfree(vring_split->desc_state); kfree(vring_split->desc_extra); + kfree(vring_split->desc_dma); } static int vring_alloc_queue_split(struct vring_virtqueue_split *vring_split, @@ -1209,7 +1232,8 @@ static int virtqueue_resize_split(struct virtqueue *_vq, u32 num) if (err) goto err; - err = vring_alloc_state_extra_split(&vring_split); + err = vring_alloc_state_extra_split(&vring_split, + vring_need_unmap_buffer(vq)); if (err) goto err_state_extra; @@ -1245,14 +1269,16 @@ static u16 packed_last_used(u16 last_used_idx) /* caller must check vring_need_unmap_buffer() */ static void vring_unmap_extra_packed(const struct vring_virtqueue *vq, - const struct vring_desc_extra *extra) + unsigned int i) { + const struct vring_desc_extra *extra = &vq->packed.desc_extra[i]; + const struct vring_desc_dma *dma = &vq->packed.desc_dma[i]; u16 flags; flags = extra->flags; dma_unmap_page(vring_dma_dev(vq), - extra->addr, extra->len, + dma->addr, dma->len, (flags & VRING_DESC_F_WRITE) ? DMA_FROM_DEVICE : DMA_TO_DEVICE); } @@ -1499,8 +1525,8 @@ static inline int virtqueue_add_packed(struct virtqueue *_vq, desc[i].id = cpu_to_le16(id); if (unlikely(vring_need_unmap_buffer(vq))) { - vq->packed.desc_extra[curr].addr = addr; - vq->packed.desc_extra[curr].len = sg->length; + vq->packed.desc_dma[curr].addr = addr; + vq->packed.desc_dma[curr].len = sg->length; } vq->packed.desc_extra[curr].flags = le16_to_cpu(flags); @@ -1559,7 +1585,7 @@ static inline int virtqueue_add_packed(struct virtqueue *_vq, for (n = 0; n < total_sg; n++) { if (i == err_idx) break; - vring_unmap_extra_packed(vq, &vq->packed.desc_extra[curr]); + vring_unmap_extra_packed(vq, curr); curr = vq->packed.desc_extra[curr].next; i++; if (i >= vq->packed.vring.num) @@ -1640,8 +1666,7 @@ static void detach_buf_packed(struct vring_virtqueue *vq, if (vring_need_unmap_buffer(vq)) { curr = id; for (i = 0; i < state->num; i++) { - vring_unmap_extra_packed(vq, - &vq->packed.desc_extra[curr]); + vring_unmap_extra_packed(vq, curr); curr = vq->packed.desc_extra[curr].next; } } @@ -1955,6 +1980,7 @@ static void vring_free_packed(struct vring_virtqueue_packed *vring_packed, kfree(vring_packed->desc_state); kfree(vring_packed->desc_extra); + kfree(vring_packed->desc_dma); } static int vring_alloc_queue_packed(struct vring_virtqueue_packed *vring_packed, @@ -2011,10 +2037,12 @@ static int vring_alloc_queue_packed(struct vring_virtqueue_packed *vring_packed, return -ENOMEM; } -static int vring_alloc_state_extra_packed(struct vring_virtqueue_packed *vring_packed) +static int vring_alloc_state_extra_packed(struct vring_virtqueue_packed *vring_packed, + bool need_unmap) { struct vring_desc_state_packed *state; struct vring_desc_extra *extra; + struct vring_desc_dma *dma; u32 num = vring_packed->vring.num; state = kmalloc_array(num, sizeof(struct vring_desc_state_packed), GFP_KERNEL); @@ -2023,6 +2051,14 @@ static int vring_alloc_state_extra_packed(struct vring_virtqueue_packed *vring_p memset(state, 0, num * sizeof(struct vring_desc_state_packed)); + if (need_unmap) { + dma = kmalloc_array(num, sizeof(struct vring_desc_dma), GFP_KERNEL); + if (!dma) + goto err_desc_dma; + } else { + dma = NULL; + } + extra = vring_alloc_desc_extra(num); if (!extra) goto err_desc_extra; @@ -2033,6 +2069,8 @@ static int vring_alloc_state_extra_packed(struct vring_virtqueue_packed *vring_p return 0; err_desc_extra: + kfree(dma); +err_desc_dma: kfree(state); err_desc_state: return -ENOMEM; @@ -2124,7 +2162,8 @@ static struct virtqueue *vring_create_virtqueue_packed( if (virtio_has_feature(vdev, VIRTIO_F_ORDER_PLATFORM)) vq->weak_barriers = false; - err = vring_alloc_state_extra_packed(&vring_packed); + err = vring_alloc_state_extra_packed(&vring_packed, + vring_need_unmap_buffer(vq)); if (err) goto err_state_extra; @@ -2156,7 +2195,8 @@ static int virtqueue_resize_packed(struct virtqueue *_vq, u32 num) if (vring_alloc_queue_packed(&vring_packed, vdev, num, vring_dma_dev(vq))) goto err_ring; - err = vring_alloc_state_extra_packed(&vring_packed); + err = vring_alloc_state_extra_packed(&vring_packed, + vring_need_unmap_buffer(vq)); if (err) goto err_state_extra; @@ -2668,7 +2708,8 @@ static struct virtqueue *__vring_new_virtqueue(unsigned int index, if (virtio_has_feature(vdev, VIRTIO_F_ORDER_PLATFORM)) vq->weak_barriers = false; - err = vring_alloc_state_extra_split(vring_split); + err = vring_alloc_state_extra_split(vring_split, + vring_need_unmap_buffer(vq)); if (err) { kfree(vq); return NULL; @@ -2828,6 +2869,14 @@ int virtqueue_set_dma_premapped(struct virtqueue *_vq) vq->premapped = true; + if (vq->packed_ring) { + kfree(vq->packed.desc_dma); + vq->packed.desc_dma = NULL; + } else { + kfree(vq->split.desc_dma); + vq->split.desc_dma = NULL; + } + END_USE(vq); return 0; @@ -2917,6 +2966,7 @@ static void vring_free(struct virtqueue *_vq) kfree(vq->packed.desc_state); kfree(vq->packed.desc_extra); + kfree(vq->packed.desc_dma); } else { vring_free_queue(vq->vq.vdev, vq->split.queue_size_in_bytes, @@ -2928,6 +2978,7 @@ static void vring_free(struct virtqueue *_vq) if (!vq->packed_ring) { kfree(vq->split.desc_state); kfree(vq->split.desc_extra); + kfree(vq->split.desc_dma); } } From patchwork Tue Jan 30 03:05:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13536798 Received: from out30-124.freemail.mail.aliyun.com (out30-124.freemail.mail.aliyun.com [115.124.30.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1AE773839B; Tue, 30 Jan 2024 03:06:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706583984; cv=none; b=mgeXD1/IdStzwWnAJCLd/WH/KUasBoy5/ci4zX5mzK/eLKHLcGkDZ/ljvIX2dtkVzgY02vkEIPBTCOgRT+unmXzcey75i5OYU2Eq/Qbqf4MobUgwI+eHTN1viPkzyluRR0XNqKWziM0t+eJjG50ijLKztfDaB613cThOiG1FB2c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706583984; c=relaxed/simple; bh=bd5Dk80ZYYRCN8tMXCNkKTwe9A1yIuLwZTinBtI7sws=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=XHbBajr9mL0cR+CAtkF+fB5lvsyvVITEBVk+UZFKzV5D2RAEmhdSzxRrRrQEzT5BvyjYd8A5eZffwKWozVRB8rkZotlVmPNiGXh+yShNbjBDTyOsBuIcn45CihuF0ZD00ulXRC1ErOruv5R2EcbTaTUgyEwS4Y/rgfRj/enUzHQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=BMCGvDKM; arc=none smtp.client-ip=115.124.30.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="BMCGvDKM" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1706583979; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=fz+AMEHj0im9h4q3+MyxuwaBPX6/NgaxFXXW55WfyB8=; b=BMCGvDKMejPc05SD23bOaJA1zP4/BGdvcRag0KZ0QjcKU1Wkkj7mSeEHSQjcjBdI9Oo3J5PrhRpQFCFVjAFwPhfhh06HqTXSKksGqCUhMomHVaPhWF3tKy8QS73Aa0yshohYriRUML1nzBZoqkmZ473lyziwzr1wZlSCxmJIdis= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R151e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046059;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=37;SR=0;TI=SMTPD_---0W.eaYw6_1706583976; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W.eaYw6_1706583976) by smtp.aliyun-inc.com; Tue, 30 Jan 2024 11:06:17 +0800 From: Xuan Zhuo To: virtualization@lists.linux.dev Cc: Richard Weinberger , Anton Ivanov , Johannes Berg , "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Hans de Goede , =?utf-8?q?Ilpo_J=C3=A4rvinen?= , Vadim Pasternak , Bjorn Andersson , Mathieu Poirier , Cornelia Huck , Halil Pasic , Eric Farman , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , Benjamin Berg , Yang Li , linux-um@lists.infradead.org, netdev@vger.kernel.org, platform-driver-x86@vger.kernel.org, linux-remoteproc@vger.kernel.org, linux-s390@vger.kernel.org, kvm@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH 07/14] virtio_ring: introduce dma map api for page Date: Tue, 30 Jan 2024 11:05:57 +0800 Message-Id: <20240130030604.108463-8-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240130030604.108463-1-xuanzhuo@linux.alibaba.com> References: <20240130030604.108463-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: linux-remoteproc@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: ce068f9b825d The virtio-net sq will use these APIs to map the scatterlist. For scatterlist, the page dma APIs are more appropriate. dma_addr_t virtqueue_dma_map_page_attrs(struct virtqueue *_vq, struct page *page, size_t offset, size_t size, enum dma_data_direction dir, unsigned long attrs); void virtqueue_dma_unmap_page_attrs(struct virtqueue *_vq, dma_addr_t addr, size_t size, enum dma_data_direction dir, unsigned long attrs); Signed-off-by: Xuan Zhuo --- drivers/virtio/virtio_ring.c | 52 ++++++++++++++++++++++++++++++++++++ include/linux/virtio.h | 7 +++++ 2 files changed, 59 insertions(+) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 5bea25167259..05aa5ae4f932 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -3230,6 +3230,58 @@ void virtqueue_dma_unmap_single_attrs(struct virtqueue *_vq, dma_addr_t addr, } EXPORT_SYMBOL_GPL(virtqueue_dma_unmap_single_attrs); +/** + * virtqueue_dma_map_page_attrs - map DMA for _vq + * @_vq: the struct virtqueue we're talking about. + * @page: the page to do dma + * @offset: the offset inside the page + * @size: the size of the page to do dma + * @dir: DMA direction + * @attrs: DMA Attrs + * + * The caller calls this to do dma mapping in advance. The DMA address can be + * passed to this _vq when it is in pre-mapped mode. + * + * return DMA address. Caller should check that by virtqueue_dma_mapping_error(). + */ +dma_addr_t virtqueue_dma_map_page_attrs(struct virtqueue *_vq, struct page *page, + size_t offset, size_t size, + enum dma_data_direction dir, + unsigned long attrs) +{ + struct vring_virtqueue *vq = to_vvq(_vq); + + if (!vq->use_dma_api) + return page_to_phys(page) + offset; + + return dma_map_page_attrs(vring_dma_dev(vq), page, offset, size, dir, attrs); +} +EXPORT_SYMBOL_GPL(virtqueue_dma_map_page_attrs); + +/** + * virtqueue_dma_unmap_page_attrs - unmap DMA for _vq + * @_vq: the struct virtqueue we're talking about. + * @addr: the dma address to unmap + * @size: the size of the buffer + * @dir: DMA direction + * @attrs: DMA Attrs + * + * Unmap the address that is mapped by the virtqueue_dma_map_* APIs. + * + */ +void virtqueue_dma_unmap_page_attrs(struct virtqueue *_vq, dma_addr_t addr, + size_t size, enum dma_data_direction dir, + unsigned long attrs) +{ + struct vring_virtqueue *vq = to_vvq(_vq); + + if (!vq->use_dma_api) + return; + + dma_unmap_page_attrs(vring_dma_dev(vq), addr, size, dir, attrs); +} +EXPORT_SYMBOL_GPL(virtqueue_dma_unmap_page_attrs); + /** * virtqueue_dma_mapping_error - check dma address * @_vq: the struct virtqueue we're talking about. diff --git a/include/linux/virtio.h b/include/linux/virtio.h index 4cc614a38376..856b9f16e47b 100644 --- a/include/linux/virtio.h +++ b/include/linux/virtio.h @@ -217,6 +217,13 @@ dma_addr_t virtqueue_dma_map_single_attrs(struct virtqueue *_vq, void *ptr, size void virtqueue_dma_unmap_single_attrs(struct virtqueue *_vq, dma_addr_t addr, size_t size, enum dma_data_direction dir, unsigned long attrs); +dma_addr_t virtqueue_dma_map_page_attrs(struct virtqueue *_vq, struct page *page, + size_t offset, size_t size, + enum dma_data_direction dir, + unsigned long attrs); +void virtqueue_dma_unmap_page_attrs(struct virtqueue *_vq, dma_addr_t addr, + size_t size, enum dma_data_direction dir, + unsigned long attrs); int virtqueue_dma_mapping_error(struct virtqueue *_vq, dma_addr_t addr); bool virtqueue_dma_need_sync(struct virtqueue *_vq, dma_addr_t addr); From patchwork Tue Jan 30 03:05:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13536800 Received: from out30-99.freemail.mail.aliyun.com (out30-99.freemail.mail.aliyun.com [115.124.30.99]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0FAE33F9E3; Tue, 30 Jan 2024 03:06:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.99 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706583987; cv=none; b=G1nyKVEhMTnmgg1Smx0BZCPinU4auJrpeiKqmyB+wm7y9LkGPGQ3wt411VbN51Bnv8vQfJ2lBT/AzWqsBn7FkdtoITZy+U8xZvSmKFI8PPRXT75K0IvxeEHggOpPCqCxDWOlX1KVm8E9330QBqnA9eOB+KV7WFyH2wfi2Vpfw+M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706583987; c=relaxed/simple; bh=dXd2NP/xdM6sjyjiS36mOci044QBHI4Od1zyfs3Vm+w=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=O+NTdEqAD4YqhAeguGxKmbjs95k1PGUWkPq7HEnhAGzVYq0k1mvNSc3ajAFFLfWeFrftznVzrRFI/xvdMO4B5jjLaAd4+sCV0x6rj48o3sKMQpaYOVF8rhtAgZ5ie7FYFOfNtmvLuv23o4O6tzgvGsuLEz/M8so9TNYSyG6Imsk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=t6R9NffG; arc=none smtp.client-ip=115.124.30.99 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="t6R9NffG" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1706583981; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=6VIRaVPIabk7E1XlRduLJQgjTOkxYWFJWN/eQZeIAbw=; b=t6R9NffG5vLcorb9LMTZMX5NU4HTetAvCPRc3PVMrFcmPLOeEwvZECGfwtvRoDXOFcqm2gWJmA6sacu2aIBBq27rdlotMRUIYP9fAdJ2kGt8Zr2FiCzyKvOi6xqxphdKmR0dQjovKEnQMuliaQRR/C5SAj/Y5D17gV9dWAsFpEY= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R121e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045192;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=37;SR=0;TI=SMTPD_---0W.eeCCy_1706583978; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W.eeCCy_1706583978) by smtp.aliyun-inc.com; Tue, 30 Jan 2024 11:06:19 +0800 From: Xuan Zhuo To: virtualization@lists.linux.dev Cc: Richard Weinberger , Anton Ivanov , Johannes Berg , "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Hans de Goede , =?utf-8?q?Ilpo_J=C3=A4rvinen?= , Vadim Pasternak , Bjorn Andersson , Mathieu Poirier , Cornelia Huck , Halil Pasic , Eric Farman , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , Benjamin Berg , Yang Li , linux-um@lists.infradead.org, netdev@vger.kernel.org, platform-driver-x86@vger.kernel.org, linux-remoteproc@vger.kernel.org, linux-s390@vger.kernel.org, kvm@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH 08/14] virtio: find_vqs introduce premapped parameter Date: Tue, 30 Jan 2024 11:05:58 +0800 Message-Id: <20240130030604.108463-9-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240130030604.108463-1-xuanzhuo@linux.alibaba.com> References: <20240130030604.108463-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: linux-remoteproc@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: ce068f9b825d If the premapped mode is enabled, the dma array(struct vring_desc_dma) of virtio core will not be allocated. That is judged when find_vqs() is called. To avoid allocating dma array in find_vqs() and releasing it immediately by virtqueue_set_dma_premapped(). This patch introduces a new parameter to find_vqs(). Then we can judge should we allocate the dma array(struct vring_desc_dma) or not inside find_vqs(). The driver must check the premapped mode of every vq after find_vqs(). Signed-off-by: Xuan Zhuo --- arch/um/drivers/virtio_uml.c | 5 ++-- drivers/platform/mellanox/mlxbf-tmfifo.c | 3 ++- drivers/remoteproc/remoteproc_virtio.c | 9 ++++--- drivers/s390/virtio/virtio_ccw.c | 8 ++++-- drivers/virtio/virtio_mmio.c | 8 +++--- drivers/virtio/virtio_pci_common.c | 15 ++++++++---- drivers/virtio/virtio_pci_common.h | 2 ++ drivers/virtio/virtio_pci_legacy.c | 3 ++- drivers/virtio/virtio_pci_modern.c | 6 +++-- drivers/virtio/virtio_ring.c | 31 ++++++++++++++++++------ drivers/virtio/virtio_vdpa.c | 6 +++-- include/linux/virtio_config.h | 11 ++++++--- include/linux/virtio_ring.h | 3 +++ tools/virtio/linux/virtio.h | 1 + tools/virtio/virtio_test.c | 2 +- tools/virtio/vringh_test.c | 10 ++++---- 16 files changed, 85 insertions(+), 38 deletions(-) diff --git a/arch/um/drivers/virtio_uml.c b/arch/um/drivers/virtio_uml.c index 8adca2000e51..171214c06411 100644 --- a/arch/um/drivers/virtio_uml.c +++ b/arch/um/drivers/virtio_uml.c @@ -1016,7 +1016,7 @@ static struct virtqueue *vu_setup_vq(struct virtio_device *vdev, static int vu_find_vqs(struct virtio_device *vdev, unsigned nvqs, struct virtqueue *vqs[], vq_callback_t *callbacks[], const char * const names[], const bool *ctx, - struct irq_affinity *desc) + const bool *premapped, struct irq_affinity *desc) { struct virtio_uml_device *vu_dev = to_virtio_uml_device(vdev); int i, queue_idx = 0, rc; @@ -1037,7 +1037,8 @@ static int vu_find_vqs(struct virtio_device *vdev, unsigned nvqs, } vqs[i] = vu_setup_vq(vdev, queue_idx++, callbacks[i], names[i], - ctx ? ctx[i] : false); + ctx ? ctx[i] : false, + premapped ? premapped[i] : false); if (IS_ERR(vqs[i])) { rc = PTR_ERR(vqs[i]); goto error_setup; diff --git a/drivers/platform/mellanox/mlxbf-tmfifo.c b/drivers/platform/mellanox/mlxbf-tmfifo.c index 5c683b4eaf10..a7b477c80c16 100644 --- a/drivers/platform/mellanox/mlxbf-tmfifo.c +++ b/drivers/platform/mellanox/mlxbf-tmfifo.c @@ -994,6 +994,7 @@ static int mlxbf_tmfifo_virtio_find_vqs(struct virtio_device *vdev, vq_callback_t *callbacks[], const char * const names[], const bool *ctx, + const bool *premapped, struct irq_affinity *desc) { struct mlxbf_tmfifo_vdev *tm_vdev = mlxbf_vdev_to_tmfifo(vdev); @@ -1015,7 +1016,7 @@ static int mlxbf_tmfifo_virtio_find_vqs(struct virtio_device *vdev, size = vring_size(vring->num, vring->align); memset(vring->va, 0, size); vq = vring_new_virtqueue(i, vring->num, vring->align, vdev, - false, false, vring->va, + false, false, premapped, vring->va, mlxbf_tmfifo_virtio_notify, callbacks[i], names[i]); if (!vq) { diff --git a/drivers/remoteproc/remoteproc_virtio.c b/drivers/remoteproc/remoteproc_virtio.c index 83d76915a6ad..774f1325524b 100644 --- a/drivers/remoteproc/remoteproc_virtio.c +++ b/drivers/remoteproc/remoteproc_virtio.c @@ -103,7 +103,7 @@ EXPORT_SYMBOL(rproc_vq_interrupt); static struct virtqueue *rp_find_vq(struct virtio_device *vdev, unsigned int id, void (*callback)(struct virtqueue *vq), - const char *name, bool ctx) + const char *name, bool ctx, bool premapped) { struct rproc_vdev *rvdev = vdev_to_rvdev(vdev); struct rproc *rproc = vdev_to_rproc(vdev); @@ -144,7 +144,8 @@ static struct virtqueue *rp_find_vq(struct virtio_device *vdev, * the 'weak' smp barriers, since we're talking with a real device. */ vq = vring_new_virtqueue(id, num, rvring->align, vdev, false, ctx, - addr, rproc_virtio_notify, callback, name); + premapped, addr, rproc_virtio_notify, callback, + name); if (!vq) { dev_err(dev, "vring_new_virtqueue %s failed\n", name); rproc_free_vring(rvring); @@ -185,6 +186,7 @@ static int rproc_virtio_find_vqs(struct virtio_device *vdev, unsigned int nvqs, vq_callback_t *callbacks[], const char * const names[], const bool * ctx, + const bool * premapped, struct irq_affinity *desc) { int i, ret, queue_idx = 0; @@ -196,7 +198,8 @@ static int rproc_virtio_find_vqs(struct virtio_device *vdev, unsigned int nvqs, } vqs[i] = rp_find_vq(vdev, queue_idx++, callbacks[i], names[i], - ctx ? ctx[i] : false); + ctx ? ctx[i] : false, + premapped ? premapped[i] : false); if (IS_ERR(vqs[i])) { ret = PTR_ERR(vqs[i]); goto error; diff --git a/drivers/s390/virtio/virtio_ccw.c b/drivers/s390/virtio/virtio_ccw.c index ac67576301bf..b27132994dc9 100644 --- a/drivers/s390/virtio/virtio_ccw.c +++ b/drivers/s390/virtio/virtio_ccw.c @@ -501,6 +501,7 @@ static void virtio_ccw_del_vqs(struct virtio_device *vdev) static struct virtqueue *virtio_ccw_setup_vq(struct virtio_device *vdev, int i, vq_callback_t *callback, const char *name, bool ctx, + bool premapped, struct ccw1 *ccw) { struct virtio_ccw_device *vcdev = to_vc_device(vdev); @@ -539,7 +540,7 @@ static struct virtqueue *virtio_ccw_setup_vq(struct virtio_device *vdev, may_reduce = vcdev->revision > 0; vq = vring_create_virtqueue(i, info->num, KVM_VIRTIO_CCW_RING_ALIGN, vdev, true, may_reduce, ctx, - notify, callback, name); + premapped, notify, callback, name); if (!vq) { /* For now, we fail if we can't get the requested size. */ @@ -655,6 +656,7 @@ static int virtio_ccw_find_vqs(struct virtio_device *vdev, unsigned nvqs, vq_callback_t *callbacks[], const char * const names[], const bool *ctx, + const bool *premapped, struct irq_affinity *desc) { struct virtio_ccw_device *vcdev = to_vc_device(vdev); @@ -673,7 +675,9 @@ static int virtio_ccw_find_vqs(struct virtio_device *vdev, unsigned nvqs, } vqs[i] = virtio_ccw_setup_vq(vdev, queue_idx++, callbacks[i], - names[i], ctx ? ctx[i] : false, + names[i], + ctx ? ctx[i] : false, + premapped ? premapped[i] : false, ccw); if (IS_ERR(vqs[i])) { ret = PTR_ERR(vqs[i]); diff --git a/drivers/virtio/virtio_mmio.c b/drivers/virtio/virtio_mmio.c index 59892a31cf76..1df38cceafd3 100644 --- a/drivers/virtio/virtio_mmio.c +++ b/drivers/virtio/virtio_mmio.c @@ -371,7 +371,7 @@ static void vm_synchronize_cbs(struct virtio_device *vdev) static struct virtqueue *vm_setup_vq(struct virtio_device *vdev, unsigned int index, void (*callback)(struct virtqueue *vq), - const char *name, bool ctx) + const char *name, bool ctx, bool premapped) { struct virtio_mmio_device *vm_dev = to_virtio_mmio_device(vdev); bool (*notify)(struct virtqueue *vq); @@ -414,7 +414,7 @@ static struct virtqueue *vm_setup_vq(struct virtio_device *vdev, unsigned int in /* Create the vring */ vq = vring_create_virtqueue(index, num, VIRTIO_MMIO_VRING_ALIGN, vdev, - true, true, ctx, notify, callback, name); + true, true, ctx, premapped, notify, callback, name); if (!vq) { err = -ENOMEM; goto error_new_virtqueue; @@ -492,6 +492,7 @@ static int vm_find_vqs(struct virtio_device *vdev, unsigned int nvqs, vq_callback_t *callbacks[], const char * const names[], const bool *ctx, + const bool *premapped, struct irq_affinity *desc) { struct virtio_mmio_device *vm_dev = to_virtio_mmio_device(vdev); @@ -516,7 +517,8 @@ static int vm_find_vqs(struct virtio_device *vdev, unsigned int nvqs, } vqs[i] = vm_setup_vq(vdev, queue_idx++, callbacks[i], names[i], - ctx ? ctx[i] : false); + ctx ? ctx[i] : false, + premapped ? premapped[i] : false); if (IS_ERR(vqs[i])) { vm_del_vqs(vdev); return PTR_ERR(vqs[i]); diff --git a/drivers/virtio/virtio_pci_common.c b/drivers/virtio/virtio_pci_common.c index 1d21d1a1b3f5..9787b880e658 100644 --- a/drivers/virtio/virtio_pci_common.c +++ b/drivers/virtio/virtio_pci_common.c @@ -175,6 +175,7 @@ static struct virtqueue *vp_setup_vq(struct virtio_device *vdev, unsigned int in void (*callback)(struct virtqueue *vq), const char *name, bool ctx, + bool premapped, u16 msix_vec) { struct virtio_pci_device *vp_dev = to_vp_device(vdev); @@ -187,7 +188,7 @@ static struct virtqueue *vp_setup_vq(struct virtio_device *vdev, unsigned int in return ERR_PTR(-ENOMEM); vq = vp_dev->setup_vq(vp_dev, info, index, callback, name, ctx, - msix_vec); + premapped, msix_vec); if (IS_ERR(vq)) goto out_info; @@ -285,6 +286,7 @@ static int vp_find_vqs_msix(struct virtio_device *vdev, unsigned int nvqs, struct virtqueue *vqs[], vq_callback_t *callbacks[], const char * const names[], bool per_vq_vectors, const bool *ctx, + const bool *premapped, struct irq_affinity *desc) { struct virtio_pci_device *vp_dev = to_vp_device(vdev); @@ -327,6 +329,7 @@ static int vp_find_vqs_msix(struct virtio_device *vdev, unsigned int nvqs, msix_vec = VP_MSIX_VQ_VECTOR; vqs[i] = vp_setup_vq(vdev, queue_idx++, callbacks[i], names[i], ctx ? ctx[i] : false, + premapped ? premapped[i] : false, msix_vec); if (IS_ERR(vqs[i])) { err = PTR_ERR(vqs[i]); @@ -357,7 +360,7 @@ static int vp_find_vqs_msix(struct virtio_device *vdev, unsigned int nvqs, static int vp_find_vqs_intx(struct virtio_device *vdev, unsigned int nvqs, struct virtqueue *vqs[], vq_callback_t *callbacks[], - const char * const names[], const bool *ctx) + const char * const names[], const bool *ctx, const bool *premapped) { struct virtio_pci_device *vp_dev = to_vp_device(vdev); int i, err, queue_idx = 0; @@ -380,6 +383,7 @@ static int vp_find_vqs_intx(struct virtio_device *vdev, unsigned int nvqs, } vqs[i] = vp_setup_vq(vdev, queue_idx++, callbacks[i], names[i], ctx ? ctx[i] : false, + premapped ? premapped[i] : false, VIRTIO_MSI_NO_VECTOR); if (IS_ERR(vqs[i])) { err = PTR_ERR(vqs[i]); @@ -397,23 +401,24 @@ static int vp_find_vqs_intx(struct virtio_device *vdev, unsigned int nvqs, int vp_find_vqs(struct virtio_device *vdev, unsigned int nvqs, struct virtqueue *vqs[], vq_callback_t *callbacks[], const char * const names[], const bool *ctx, + const bool *premapped, struct irq_affinity *desc) { int err; /* Try MSI-X with one vector per queue. */ - err = vp_find_vqs_msix(vdev, nvqs, vqs, callbacks, names, true, ctx, desc); + err = vp_find_vqs_msix(vdev, nvqs, vqs, callbacks, names, true, ctx, premapped, desc); if (!err) return 0; /* Fallback: MSI-X with one vector for config, one shared for queues. */ - err = vp_find_vqs_msix(vdev, nvqs, vqs, callbacks, names, false, ctx, desc); + err = vp_find_vqs_msix(vdev, nvqs, vqs, callbacks, names, false, ctx, premapped, desc); if (!err) return 0; /* Is there an interrupt? If not give up. */ if (!(to_vp_device(vdev)->pci_dev->irq)) return err; /* Finally fall back to regular interrupts. */ - return vp_find_vqs_intx(vdev, nvqs, vqs, callbacks, names, ctx); + return vp_find_vqs_intx(vdev, nvqs, vqs, callbacks, names, ctx, premapped); } const char *vp_bus_name(struct virtio_device *vdev) diff --git a/drivers/virtio/virtio_pci_common.h b/drivers/virtio/virtio_pci_common.h index 4b773bd7c58c..3270bfcaa2ce 100644 --- a/drivers/virtio/virtio_pci_common.h +++ b/drivers/virtio/virtio_pci_common.h @@ -82,6 +82,7 @@ struct virtio_pci_device { void (*callback)(struct virtqueue *vq), const char *name, bool ctx, + bool premapped, u16 msix_vec); void (*del_vq)(struct virtio_pci_vq_info *info); @@ -112,6 +113,7 @@ void vp_del_vqs(struct virtio_device *vdev); int vp_find_vqs(struct virtio_device *vdev, unsigned int nvqs, struct virtqueue *vqs[], vq_callback_t *callbacks[], const char * const names[], const bool *ctx, + const bool *premapped, struct irq_affinity *desc); const char *vp_bus_name(struct virtio_device *vdev); diff --git a/drivers/virtio/virtio_pci_legacy.c b/drivers/virtio/virtio_pci_legacy.c index d9cbb02b35a1..3582fc1f61e2 100644 --- a/drivers/virtio/virtio_pci_legacy.c +++ b/drivers/virtio/virtio_pci_legacy.c @@ -113,6 +113,7 @@ static struct virtqueue *setup_vq(struct virtio_pci_device *vp_dev, void (*callback)(struct virtqueue *vq), const char *name, bool ctx, + bool premapped, u16 msix_vec) { struct virtqueue *vq; @@ -130,7 +131,7 @@ static struct virtqueue *setup_vq(struct virtio_pci_device *vp_dev, /* create the vring */ vq = vring_create_virtqueue(index, num, VIRTIO_PCI_VRING_ALIGN, &vp_dev->vdev, - true, false, ctx, + true, false, ctx, premapped, vp_notify, callback, name); if (!vq) return ERR_PTR(-ENOMEM); diff --git a/drivers/virtio/virtio_pci_modern.c b/drivers/virtio/virtio_pci_modern.c index ee6a386d250b..f410ba3d531f 100644 --- a/drivers/virtio/virtio_pci_modern.c +++ b/drivers/virtio/virtio_pci_modern.c @@ -339,6 +339,7 @@ static struct virtqueue *setup_vq(struct virtio_pci_device *vp_dev, void (*callback)(struct virtqueue *vq), const char *name, bool ctx, + bool premapped, u16 msix_vec) { @@ -366,7 +367,7 @@ static struct virtqueue *setup_vq(struct virtio_pci_device *vp_dev, /* create the vring */ vq = vring_create_virtqueue(index, num, SMP_CACHE_BYTES, &vp_dev->vdev, - true, true, ctx, + true, true, ctx, premapped, notify, callback, name); if (!vq) return ERR_PTR(-ENOMEM); @@ -394,11 +395,12 @@ static int vp_modern_find_vqs(struct virtio_device *vdev, unsigned int nvqs, struct virtqueue *vqs[], vq_callback_t *callbacks[], const char * const names[], const bool *ctx, + const bool *premapped, struct irq_affinity *desc) { struct virtio_pci_device *vp_dev = to_vp_device(vdev); struct virtqueue *vq; - int rc = vp_find_vqs(vdev, nvqs, vqs, callbacks, names, ctx, desc); + int rc = vp_find_vqs(vdev, nvqs, vqs, callbacks, names, ctx, premapped, desc); if (rc) return rc; diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 05aa5ae4f932..11a131c22f7e 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -242,6 +242,7 @@ static struct virtqueue *__vring_new_virtqueue(unsigned int index, struct virtio_device *vdev, bool weak_barriers, bool context, + bool premapped, bool (*notify)(struct virtqueue *), void (*callback)(struct virtqueue *), const char *name, @@ -1192,6 +1193,7 @@ static struct virtqueue *vring_create_virtqueue_split( bool weak_barriers, bool may_reduce_num, bool context, + bool premapped, bool (*notify)(struct virtqueue *), void (*callback)(struct virtqueue *), const char *name, @@ -1207,7 +1209,7 @@ static struct virtqueue *vring_create_virtqueue_split( return NULL; vq = __vring_new_virtqueue(index, &vring_split, vdev, weak_barriers, - context, notify, callback, name, dma_dev); + context, premapped, notify, callback, name, dma_dev); if (!vq) { vring_free_split(&vring_split, vdev, dma_dev); return NULL; @@ -2121,6 +2123,7 @@ static struct virtqueue *vring_create_virtqueue_packed( bool weak_barriers, bool may_reduce_num, bool context, + bool premapped, bool (*notify)(struct virtqueue *), void (*callback)(struct virtqueue *), const char *name, @@ -2153,7 +2156,11 @@ static struct virtqueue *vring_create_virtqueue_packed( vq->packed_ring = true; vq->dma_dev = dma_dev; vq->use_dma_api = vring_use_dma_api(vdev); - vq->premapped = false; + + if (premapped && vq->use_dma_api) + vq->premapped = true; + else + vq->premapped = false; vq->indirect = virtio_has_feature(vdev, VIRTIO_RING_F_INDIRECT_DESC) && !context; @@ -2668,6 +2675,7 @@ static struct virtqueue *__vring_new_virtqueue(unsigned int index, struct virtio_device *vdev, bool weak_barriers, bool context, + bool premapped, bool (*notify)(struct virtqueue *), void (*callback)(struct virtqueue *), const char *name, @@ -2699,7 +2707,11 @@ static struct virtqueue *__vring_new_virtqueue(unsigned int index, #endif vq->dma_dev = dma_dev; vq->use_dma_api = vring_use_dma_api(vdev); - vq->premapped = false; + + if (premapped && vq->use_dma_api) + vq->premapped = true; + else + vq->premapped = false; vq->indirect = virtio_has_feature(vdev, VIRTIO_RING_F_INDIRECT_DESC) && !context; @@ -2734,6 +2746,7 @@ struct virtqueue *vring_create_virtqueue( bool weak_barriers, bool may_reduce_num, bool context, + bool premapped, bool (*notify)(struct virtqueue *), void (*callback)(struct virtqueue *), const char *name) @@ -2742,11 +2755,11 @@ struct virtqueue *vring_create_virtqueue( if (virtio_has_feature(vdev, VIRTIO_F_RING_PACKED)) return vring_create_virtqueue_packed(index, num, vring_align, vdev, weak_barriers, may_reduce_num, - context, notify, callback, name, vdev->dev.parent); + context, premapped, notify, callback, name, vdev->dev.parent); return vring_create_virtqueue_split(index, num, vring_align, vdev, weak_barriers, may_reduce_num, - context, notify, callback, name, vdev->dev.parent); + context, premapped, notify, callback, name, vdev->dev.parent); } EXPORT_SYMBOL_GPL(vring_create_virtqueue); @@ -2758,6 +2771,7 @@ struct virtqueue *vring_create_virtqueue_dma( bool weak_barriers, bool may_reduce_num, bool context, + bool premapped, bool (*notify)(struct virtqueue *), void (*callback)(struct virtqueue *), const char *name, @@ -2767,11 +2781,11 @@ struct virtqueue *vring_create_virtqueue_dma( if (virtio_has_feature(vdev, VIRTIO_F_RING_PACKED)) return vring_create_virtqueue_packed(index, num, vring_align, vdev, weak_barriers, may_reduce_num, - context, notify, callback, name, dma_dev); + context, premapped, notify, callback, name, dma_dev); return vring_create_virtqueue_split(index, num, vring_align, vdev, weak_barriers, may_reduce_num, - context, notify, callback, name, dma_dev); + context, premapped, notify, callback, name, dma_dev); } EXPORT_SYMBOL_GPL(vring_create_virtqueue_dma); @@ -2923,6 +2937,7 @@ struct virtqueue *vring_new_virtqueue(unsigned int index, struct virtio_device *vdev, bool weak_barriers, bool context, + bool premapped, void *pages, bool (*notify)(struct virtqueue *vq), void (*callback)(struct virtqueue *vq), @@ -2935,7 +2950,7 @@ struct virtqueue *vring_new_virtqueue(unsigned int index, vring_init(&vring_split.vring, num, pages, vring_align); return __vring_new_virtqueue(index, &vring_split, vdev, weak_barriers, - context, notify, callback, name, + context, premapped, notify, callback, name, vdev->dev.parent); } EXPORT_SYMBOL_GPL(vring_new_virtqueue); diff --git a/drivers/virtio/virtio_vdpa.c b/drivers/virtio/virtio_vdpa.c index 8d63e5923d24..87296dc5c8c4 100644 --- a/drivers/virtio/virtio_vdpa.c +++ b/drivers/virtio/virtio_vdpa.c @@ -358,6 +358,7 @@ static int virtio_vdpa_find_vqs(struct virtio_device *vdev, unsigned int nvqs, vq_callback_t *callbacks[], const char * const names[], const bool *ctx, + const bool *premapped, struct irq_affinity *desc) { struct virtio_vdpa_device *vd_dev = to_virtio_vdpa_device(vdev); @@ -382,8 +383,9 @@ static int virtio_vdpa_find_vqs(struct virtio_device *vdev, unsigned int nvqs, } vqs[i] = virtio_vdpa_setup_vq(vdev, queue_idx++, - callbacks[i], names[i], ctx ? - ctx[i] : false); + callbacks[i], names[i], + ctx ? ctx[i] : false, + premapped ? premapped[i] : false); if (IS_ERR(vqs[i])) { err = PTR_ERR(vqs[i]); goto err_setup_vq; diff --git a/include/linux/virtio_config.h b/include/linux/virtio_config.h index 2b3438de2c4d..45b5a980561e 100644 --- a/include/linux/virtio_config.h +++ b/include/linux/virtio_config.h @@ -57,6 +57,9 @@ typedef void vq_callback_t(struct virtqueue *); * include a NULL entry for vqs that do not need a callback * names: array of virtqueue names (mainly for debugging) * include a NULL entry for vqs unused by driver + * premapped: array of virtqueue premapped mode + * virtio core will try to set the vq to premapped mode + * The driver must check the premapped mode after find_vqs(). * Returns 0 on success or error status * @del_vqs: free virtqueues found by find_vqs(). * @synchronize_cbs: synchronize with the virtqueue callbacks (optional) @@ -106,6 +109,7 @@ struct virtio_config_ops { int (*find_vqs)(struct virtio_device *, unsigned nvqs, struct virtqueue *vqs[], vq_callback_t *callbacks[], const char * const names[], const bool *ctx, + const bool *premapped, struct irq_affinity *desc); void (*del_vqs)(struct virtio_device *); void (*synchronize_cbs)(struct virtio_device *); @@ -214,7 +218,7 @@ struct virtqueue *virtio_find_single_vq(struct virtio_device *vdev, const char *names[] = { n }; struct virtqueue *vq; int err = vdev->config->find_vqs(vdev, 1, &vq, callbacks, names, NULL, - NULL); + NULL, NULL); if (err < 0) return ERR_PTR(err); return vq; @@ -226,7 +230,8 @@ int virtio_find_vqs(struct virtio_device *vdev, unsigned nvqs, const char * const names[], struct irq_affinity *desc) { - return vdev->config->find_vqs(vdev, nvqs, vqs, callbacks, names, NULL, desc); + return vdev->config->find_vqs(vdev, nvqs, vqs, callbacks, names, NULL, + NULL, desc); } static inline @@ -236,7 +241,7 @@ int virtio_find_vqs_ctx(struct virtio_device *vdev, unsigned nvqs, struct irq_affinity *desc) { return vdev->config->find_vqs(vdev, nvqs, vqs, callbacks, names, ctx, - desc); + NULL, desc); } /** diff --git a/include/linux/virtio_ring.h b/include/linux/virtio_ring.h index 9b33df741b63..91a6608d7001 100644 --- a/include/linux/virtio_ring.h +++ b/include/linux/virtio_ring.h @@ -73,6 +73,7 @@ struct virtqueue *vring_create_virtqueue(unsigned int index, bool weak_barriers, bool may_reduce_num, bool ctx, + bool premapped, bool (*notify)(struct virtqueue *vq), void (*callback)(struct virtqueue *vq), const char *name); @@ -88,6 +89,7 @@ struct virtqueue *vring_create_virtqueue_dma(unsigned int index, bool weak_barriers, bool may_reduce_num, bool ctx, + bool premapped, bool (*notify)(struct virtqueue *vq), void (*callback)(struct virtqueue *vq), const char *name, @@ -103,6 +105,7 @@ struct virtqueue *vring_new_virtqueue(unsigned int index, struct virtio_device *vdev, bool weak_barriers, bool ctx, + bool premapped, void *pages, bool (*notify)(struct virtqueue *vq), void (*callback)(struct virtqueue *vq), diff --git a/tools/virtio/linux/virtio.h b/tools/virtio/linux/virtio.h index 5d3440f474dd..dbbb426ed44d 100644 --- a/tools/virtio/linux/virtio.h +++ b/tools/virtio/linux/virtio.h @@ -63,6 +63,7 @@ struct virtqueue *vring_new_virtqueue(unsigned int index, struct virtio_device *vdev, bool weak_barriers, bool ctx, + bool premapped, void *pages, bool (*notify)(struct virtqueue *vq), void (*callback)(struct virtqueue *vq), diff --git a/tools/virtio/virtio_test.c b/tools/virtio/virtio_test.c index 028f54e6854a..f7bbf5bf580a 100644 --- a/tools/virtio/virtio_test.c +++ b/tools/virtio/virtio_test.c @@ -102,7 +102,7 @@ static void vq_reset(struct vq_info *info, int num, struct virtio_device *vdev) memset(info->ring, 0, vring_size(num, 4096)); vring_init(&info->vring, num, info->ring, 4096); - info->vq = vring_new_virtqueue(info->idx, num, 4096, vdev, true, false, + info->vq = vring_new_virtqueue(info->idx, num, 4096, vdev, true, false, false, info->ring, vq_notify, vq_callback, "test"); assert(info->vq); info->vq->priv = info; diff --git a/tools/virtio/vringh_test.c b/tools/virtio/vringh_test.c index 98ff808d6f0c..84f50bd115e0 100644 --- a/tools/virtio/vringh_test.c +++ b/tools/virtio/vringh_test.c @@ -317,7 +317,7 @@ static int parallel_test(u64 features, err(1, "Could not set affinity to cpu %u", first_cpu); vq = vring_new_virtqueue(0, RINGSIZE, ALIGN, &gvdev.vdev, true, - false, guest_map, + false, false, guest_map, fast_vringh ? no_notify_host : parallel_notify_host, never_callback_guest, "guest vq"); @@ -391,7 +391,7 @@ static int parallel_test(u64 features, /* Swallow all notifies at once. */ if (read(to_guest[0], buf, sizeof(buf)) < 1) break; - + receives++; virtqueue_disable_cb(vq); continue; @@ -424,7 +424,7 @@ static int parallel_test(u64 features, continue; if (read(to_guest[0], buf, sizeof(buf)) < 1) break; - + receives++; virtqueue_disable_cb(vq); } @@ -485,7 +485,7 @@ int main(int argc, char *argv[]) memset(__user_addr_min, 0, vring_size(RINGSIZE, ALIGN)); /* Set up guest side. */ - vq = vring_new_virtqueue(0, RINGSIZE, ALIGN, &vdev, true, false, + vq = vring_new_virtqueue(0, RINGSIZE, ALIGN, &vdev, true, false, false, __user_addr_min, never_notify_host, never_callback_guest, "guest vq"); @@ -669,7 +669,7 @@ int main(int argc, char *argv[]) /* Force creation of direct, which we modify. */ __virtio_clear_bit(&vdev, VIRTIO_RING_F_INDIRECT_DESC); vq = vring_new_virtqueue(0, RINGSIZE, ALIGN, &vdev, true, - false, __user_addr_min, + false, false, __user_addr_min, never_notify_host, never_callback_guest, "guest vq"); From patchwork Tue Jan 30 03:05:59 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13536803 Received: from out30-100.freemail.mail.aliyun.com (out30-100.freemail.mail.aliyun.com [115.124.30.100]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8A5EB55E68; Tue, 30 Jan 2024 03:06:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.100 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706583993; cv=none; b=XShbmnq7zgUnxRsE26Ns7kTbxhYWhjGybnWFIwAnjcK2XjtAFi5KWvw6lZAxI4i7fjXA4qVCZXaLzrkGZHovw18CLYjkV9jW1klaGSh/cHvxml7jYn5POKTB5JhEdwM/pW6g0TYlSlTHmsVI60G50Xz2BO8U2dVYUaqGx7DSbW4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706583993; c=relaxed/simple; bh=7d8yZ0XltJ9CN6IXq8sv7pKckuDmlQaD3bgvZ2VD1lQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=J4Ot3TISCb4tK8JAWvKni0UDzvHHzfUjJ4RN0rc7/NrznREYwWSLisl2CI36YJjqATNl8rjegPJNK2YwAE/rNghQekaEseK56Jm1fa6yvAWEXHbZC/WpDb4JAupi0EvCpAtgiufwHBc1lWELUeZmVLUUFIriJzKgcAU8MERxreU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=EpBPppam; arc=none smtp.client-ip=115.124.30.100 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="EpBPppam" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1706583983; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=qt3C0IgNOS3wKLdgDjV5YUtjOHXeCb5dvcmcCFV4wb8=; b=EpBPppamsHt3nyv6oQa8GVkXZzHtJvZrZeyWBxLKpx4yXlUX0vQ/AlE8sOGxkZRfoDNNwn4P5MFDLcEacKVgqeOs8oPcX4sAYxd3FhYxzb7QqIK3E8OjhfF6LlN37YDrNTXZFp6Y6zk+IcHgOkZ6NKvpSuQ/BbBX/qHeVFKSeO0= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R181e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045170;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=37;SR=0;TI=SMTPD_---0W.eeCDd_1706583979; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W.eeCDd_1706583979) by smtp.aliyun-inc.com; Tue, 30 Jan 2024 11:06:20 +0800 From: Xuan Zhuo To: virtualization@lists.linux.dev Cc: Richard Weinberger , Anton Ivanov , Johannes Berg , "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Hans de Goede , =?utf-8?q?Ilpo_J=C3=A4rvinen?= , Vadim Pasternak , Bjorn Andersson , Mathieu Poirier , Cornelia Huck , Halil Pasic , Eric Farman , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , Benjamin Berg , Yang Li , linux-um@lists.infradead.org, netdev@vger.kernel.org, platform-driver-x86@vger.kernel.org, linux-remoteproc@vger.kernel.org, linux-s390@vger.kernel.org, kvm@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH 09/14] virtio_ring: export premapped to driver by struct virtqueue Date: Tue, 30 Jan 2024 11:05:59 +0800 Message-Id: <20240130030604.108463-10-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240130030604.108463-1-xuanzhuo@linux.alibaba.com> References: <20240130030604.108463-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: linux-remoteproc@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: ce068f9b825d Export the premapped to drivers, then drivers can check the vq premapped mode after the find_vqs(). Because the find_vqs() just try to enable the vq premapped mode, the driver must check that after find_vqs(). Signed-off-by: Xuan Zhuo --- drivers/virtio/virtio_ring.c | 17 +++++++---------- include/linux/virtio.h | 1 + 2 files changed, 8 insertions(+), 10 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index 11a131c22f7e..c0d8522332b2 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -191,9 +191,6 @@ struct vring_virtqueue { /* Host publishes avail event idx */ bool event; - /* Do DMA mapping by driver */ - bool premapped; - /* Head of free buffer list. */ unsigned int free_head; /* Number we've added since last sync. */ @@ -314,7 +311,7 @@ static bool vring_use_dma_api(const struct virtio_device *vdev) static bool vring_need_unmap_buffer(const struct vring_virtqueue *vring) { - return vring->use_dma_api && !vring->premapped; + return vring->use_dma_api && !vring->vq.premapped; } size_t virtio_max_dma_size(const struct virtio_device *vdev) @@ -386,7 +383,7 @@ static struct device *vring_dma_dev(const struct vring_virtqueue *vq) static int vring_map_one_sg(const struct vring_virtqueue *vq, struct scatterlist *sg, enum dma_data_direction direction, dma_addr_t *addr) { - if (vq->premapped) { + if (vq->vq.premapped) { *addr = sg_dma_address(sg); return 0; } @@ -2158,9 +2155,9 @@ static struct virtqueue *vring_create_virtqueue_packed( vq->use_dma_api = vring_use_dma_api(vdev); if (premapped && vq->use_dma_api) - vq->premapped = true; + vq->vq.premapped = true; else - vq->premapped = false; + vq->vq.premapped = false; vq->indirect = virtio_has_feature(vdev, VIRTIO_RING_F_INDIRECT_DESC) && !context; @@ -2709,9 +2706,9 @@ static struct virtqueue *__vring_new_virtqueue(unsigned int index, vq->use_dma_api = vring_use_dma_api(vdev); if (premapped && vq->use_dma_api) - vq->premapped = true; + vq->vq.premapped = true; else - vq->premapped = false; + vq->vq.premapped = false; vq->indirect = virtio_has_feature(vdev, VIRTIO_RING_F_INDIRECT_DESC) && !context; @@ -2881,7 +2878,7 @@ int virtqueue_set_dma_premapped(struct virtqueue *_vq) return -EINVAL; } - vq->premapped = true; + vq->vq.premapped = true; if (vq->packed_ring) { kfree(vq->packed.desc_dma); diff --git a/include/linux/virtio.h b/include/linux/virtio.h index 856b9f16e47b..2e279469277c 100644 --- a/include/linux/virtio.h +++ b/include/linux/virtio.h @@ -36,6 +36,7 @@ struct virtqueue { unsigned int num_free; unsigned int num_max; bool reset; + bool premapped; void *priv; }; From patchwork Tue Jan 30 03:06:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13536801 Received: from out30-132.freemail.mail.aliyun.com (out30-132.freemail.mail.aliyun.com [115.124.30.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1B62D53E0A; Tue, 30 Jan 2024 03:06:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.132 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706583990; cv=none; b=bho1bBRYGzQ3Tsr4qaDbzf2sqMTrWyb+ERZj2Wo5k70neoXe/plLVpnoaj6Sc/5wCDCd6gLGA7dIRlkCpOoa7l3+U50bslDPCNp1znXiF6CdRe+IexY68VTtyLYPnWwe8E4ny017mV/5cvIkO0lhdHcrUaTv+dusJjcin9GX0Kk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706583990; c=relaxed/simple; bh=tmhjzYNCYCrDHond5B6w87b0fu++aBkzE0b6FJjDiXU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=qqVjdl395s50RYXR90dyYGWc9zWYzLXXuhJxpKx///xRh1yIFoPk6ppKFvFgWXWcN/h2STPzHV4KVjo1ggxOVm2KoLl5+V1X2DRYgup1Vf7icpBE8DfJjCm4HQt8is/yvArJi4TUhX1TV0cKVGpXz82G3b/WE3LhAt9l9aKfDKQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=Ywvb1IEe; arc=none smtp.client-ip=115.124.30.132 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="Ywvb1IEe" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1706583985; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=NS9ceng5ziq2Yn2+9KrlE+PH2QFHreCvTPLpx0wusIg=; b=Ywvb1IEeUVL4yr5O0hJN7sb5UHSdq/MiFNL1QHJ4RLquvlb1XTTsnRMFOQ+ypuruNxCw7TzS6G7FWEwyWRiyhqiZNwOOSQHc7rSANCFsY+E6ID1i9MiWobGA6Uu64NzAYyEvG9OcRGrITF1BFZTNfizElou3JEYQri9FIW/gn/Q= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R171e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046059;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=37;SR=0;TI=SMTPD_---0W.eaYxi_1706583981; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W.eaYxi_1706583981) by smtp.aliyun-inc.com; Tue, 30 Jan 2024 11:06:22 +0800 From: Xuan Zhuo To: virtualization@lists.linux.dev Cc: Richard Weinberger , Anton Ivanov , Johannes Berg , "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Hans de Goede , =?utf-8?q?Ilpo_J=C3=A4rvinen?= , Vadim Pasternak , Bjorn Andersson , Mathieu Poirier , Cornelia Huck , Halil Pasic , Eric Farman , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , Benjamin Berg , Yang Li , linux-um@lists.infradead.org, netdev@vger.kernel.org, platform-driver-x86@vger.kernel.org, linux-remoteproc@vger.kernel.org, linux-s390@vger.kernel.org, kvm@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH 10/14] virtio_net: set premapped mode by find_vqs() Date: Tue, 30 Jan 2024 11:06:00 +0800 Message-Id: <20240130030604.108463-11-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240130030604.108463-1-xuanzhuo@linux.alibaba.com> References: <20240130030604.108463-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: linux-remoteproc@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: ce068f9b825d Now, the virtio core can set the premapped mode by find_vqs(). If the premapped can be enabled, the dma array will not be allocated. So virtio-net use the api of find_vqs to enable the premapped. Judge the premapped mode by the vq->premapped instead of saving local variable. Signed-off-by: Xuan Zhuo --- drivers/net/virtio_net.c | 50 ++++++++++++++--------------------- include/linux/virtio_config.h | 10 +++---- 2 files changed, 25 insertions(+), 35 deletions(-) diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index 51b1868d2f22..660459b4c3e3 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -196,9 +196,6 @@ struct receive_queue { /* Record the last dma info to free after new pages is allocated. */ struct virtnet_rq_dma *last_dma; - - /* Do dma by self */ - bool do_dma; }; /* This structure can contain rss message with maximum settings for indirection table and keysize @@ -638,7 +635,7 @@ static void *virtnet_rq_get_buf(struct receive_queue *rq, u32 *len, void **ctx) void *buf; buf = virtqueue_get_buf_ctx(rq->vq, len, ctx); - if (buf && rq->do_dma) + if (buf && rq->vq->premapped) virtnet_rq_unmap(rq, buf, *len); return buf; @@ -651,7 +648,7 @@ static void virtnet_rq_init_one_sg(struct receive_queue *rq, void *buf, u32 len) u32 offset; void *head; - if (!rq->do_dma) { + if (!rq->vq->premapped) { sg_init_one(rq->sg, buf, len); return; } @@ -681,7 +678,7 @@ static void *virtnet_rq_alloc(struct receive_queue *rq, u32 size, gfp_t gfp) head = page_address(alloc_frag->page); - if (rq->do_dma) { + if (rq->vq->premapped) { dma = head; /* new pages */ @@ -727,22 +724,6 @@ static void *virtnet_rq_alloc(struct receive_queue *rq, u32 size, gfp_t gfp) return buf; } -static void virtnet_rq_set_premapped(struct virtnet_info *vi) -{ - int i; - - /* disable for big mode */ - if (!vi->mergeable_rx_bufs && vi->big_packets) - return; - - for (i = 0; i < vi->max_queue_pairs; i++) { - if (virtqueue_set_dma_premapped(vi->rq[i].vq)) - continue; - - vi->rq[i].do_dma = true; - } -} - static void virtnet_rq_unmap_free_buf(struct virtqueue *vq, void *buf) { struct virtnet_info *vi = vq->vdev->priv; @@ -751,7 +732,7 @@ static void virtnet_rq_unmap_free_buf(struct virtqueue *vq, void *buf) rq = &vi->rq[i]; - if (rq->do_dma) + if (rq->vq->premapped) virtnet_rq_unmap(rq, buf, 0); virtnet_rq_free_buf(vi, rq, buf); @@ -1846,7 +1827,7 @@ static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq, err = virtqueue_add_inbuf_ctx(rq->vq, rq->sg, 1, buf, ctx, gfp); if (err < 0) { - if (rq->do_dma) + if (rq->vq->premapped) virtnet_rq_unmap(rq, buf, 0); put_page(virt_to_head_page(buf)); } @@ -1961,7 +1942,7 @@ static int add_recvbuf_mergeable(struct virtnet_info *vi, ctx = mergeable_len_to_ctx(len + room, headroom); err = virtqueue_add_inbuf_ctx(rq->vq, rq->sg, 1, buf, ctx, gfp); if (err < 0) { - if (rq->do_dma) + if (rq->vq->premapped) virtnet_rq_unmap(rq, buf, 0); put_page(virt_to_head_page(buf)); } @@ -4030,7 +4011,7 @@ static void free_receive_page_frags(struct virtnet_info *vi) int i; for (i = 0; i < vi->max_queue_pairs; i++) if (vi->rq[i].alloc_frag.page) { - if (vi->rq[i].do_dma && vi->rq[i].last_dma) + if (vi->rq[i].vq->premapped && vi->rq[i].last_dma) virtnet_rq_unmap(&vi->rq[i], vi->rq[i].last_dma, 0); put_page(vi->rq[i].alloc_frag.page); } @@ -4099,6 +4080,7 @@ static int virtnet_find_vqs(struct virtnet_info *vi) int ret = -ENOMEM; int i, total_vqs; const char **names; + bool *premapped; bool *ctx; /* We expect 1 RX virtqueue followed by 1 TX virtqueue, followed by @@ -4122,8 +4104,13 @@ static int virtnet_find_vqs(struct virtnet_info *vi) ctx = kcalloc(total_vqs, sizeof(*ctx), GFP_KERNEL); if (!ctx) goto err_ctx; + + premapped = kcalloc(total_vqs, sizeof(*ctx), GFP_KERNEL); + if (!ctx) + goto err_premapped; } else { ctx = NULL; + premapped = NULL; } /* Parameters for control virtqueue, if any */ @@ -4142,10 +4129,13 @@ static int virtnet_find_vqs(struct virtnet_info *vi) names[txq2vq(i)] = vi->sq[i].name; if (ctx) ctx[rxq2vq(i)] = true; + + if (premapped) + premapped[rxq2vq(i)] = true; } - ret = virtio_find_vqs_ctx(vi->vdev, total_vqs, vqs, callbacks, - names, ctx, NULL); + ret = virtio_find_vqs_ctx_premapped(vi->vdev, total_vqs, vqs, callbacks, + names, ctx, premapped, NULL); if (ret) goto err_find; @@ -4165,6 +4155,8 @@ static int virtnet_find_vqs(struct virtnet_info *vi) err_find: + kfree(premapped); +err_premapped: kfree(ctx); err_ctx: kfree(names); @@ -4234,8 +4226,6 @@ static int init_vqs(struct virtnet_info *vi) if (ret) goto err_free; - virtnet_rq_set_premapped(vi); - cpus_read_lock(); virtnet_set_affinity(vi); cpus_read_unlock(); diff --git a/include/linux/virtio_config.h b/include/linux/virtio_config.h index 45b5a980561e..908e68a48c99 100644 --- a/include/linux/virtio_config.h +++ b/include/linux/virtio_config.h @@ -235,13 +235,13 @@ int virtio_find_vqs(struct virtio_device *vdev, unsigned nvqs, } static inline -int virtio_find_vqs_ctx(struct virtio_device *vdev, unsigned nvqs, - struct virtqueue *vqs[], vq_callback_t *callbacks[], - const char * const names[], const bool *ctx, - struct irq_affinity *desc) +int virtio_find_vqs_ctx_premapped(struct virtio_device *vdev, unsigned int nvqs, + struct virtqueue *vqs[], vq_callback_t *callbacks[], + const char * const names[], const bool *ctx, + const bool *premapped, struct irq_affinity *desc) { return vdev->config->find_vqs(vdev, nvqs, vqs, callbacks, names, ctx, - NULL, desc); + premapped, desc); } /** From patchwork Tue Jan 30 03:06:01 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13536805 Received: from out30-110.freemail.mail.aliyun.com (out30-110.freemail.mail.aliyun.com [115.124.30.110]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8FAB236B11; Tue, 30 Jan 2024 03:06:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.110 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706583997; cv=none; b=OdhADW65NoTCEjfo08EXD0me0W4mU8mRvtrXsLtlF6RuwLkSGtzBR+NBvoSf93IeZ/9YRr0Td31uXS29sHZFnr7yNctiSwU2o5ypEgujwNb9lXvfdPPuO/bR1eP+YE6oBSQMfDmvzvEyYT/bYSkSvAWK0Pr0si47QJSBDOYxz50= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706583997; c=relaxed/simple; bh=AlSdtc7beZToOsVWR2g1ctsI2OR5sZGiz84rHZSuX+c=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=JOKdb8nFT1Oqtsyit7iwcaWSOTR8/7TrOhV+KR08yE1yTi7s2p/e55G3aMQ6bOeDQVCnExHuaMDA5Q02+TxOW1WNwDbm7aG0v89I63KeioYdt8ZSAX6kw+/B5tPxf2+nSORtVTQEnPxI9u0XFtIqPxBNuf9XyjUpDQf9rWfaYTM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=NQf9FRHE; arc=none smtp.client-ip=115.124.30.110 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="NQf9FRHE" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1706583987; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=kSHh4YpKjoP6sPNLBvdLdn6B7QAgtZdbQF3H99FXUtU=; b=NQf9FRHE2ZL4CsHQtE2znGDoo4yUNqtycyU/Hrspxu08zPVYFO3Swz9wQsIXFkhuX63TXj9v7J7qoxX1nyGiuKr78S8n/5fo7az9o5wKxJDVL3LRr9yNozV1j9pLuGaq0bNe6/hDCFF/Jpr9wTKbX+hFuGdY1ho9mezZpY7f6s0= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R831e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046056;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=37;SR=0;TI=SMTPD_---0W.eaYyC_1706583982; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W.eaYyC_1706583982) by smtp.aliyun-inc.com; Tue, 30 Jan 2024 11:06:23 +0800 From: Xuan Zhuo To: virtualization@lists.linux.dev Cc: Richard Weinberger , Anton Ivanov , Johannes Berg , "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Hans de Goede , =?utf-8?q?Ilpo_J=C3=A4rvinen?= , Vadim Pasternak , Bjorn Andersson , Mathieu Poirier , Cornelia Huck , Halil Pasic , Eric Farman , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , Benjamin Berg , Yang Li , linux-um@lists.infradead.org, netdev@vger.kernel.org, platform-driver-x86@vger.kernel.org, linux-remoteproc@vger.kernel.org, linux-s390@vger.kernel.org, kvm@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH 11/14] virtio_ring: remove api of setting vq premapped Date: Tue, 30 Jan 2024 11:06:01 +0800 Message-Id: <20240130030604.108463-12-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240130030604.108463-1-xuanzhuo@linux.alibaba.com> References: <20240130030604.108463-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: linux-remoteproc@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: ce068f9b825d The virtio-net sets the premapped mode by find_vqs(). So the below API is not used by anyone, we remove it. int virtqueue_set_dma_premapped(struct virtqueue *_vq); Signed-off-by: Xuan Zhuo --- drivers/virtio/virtio_ring.c | 56 ------------------------------------ include/linux/virtio.h | 2 -- 2 files changed, 58 deletions(-) diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index c0d8522332b2..25ea35499af9 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -2838,62 +2838,6 @@ int virtqueue_resize(struct virtqueue *_vq, u32 num, } EXPORT_SYMBOL_GPL(virtqueue_resize); -/** - * virtqueue_set_dma_premapped - set the vring premapped mode - * @_vq: the struct virtqueue we're talking about. - * - * Enable the premapped mode of the vq. - * - * The vring in premapped mode does not do dma internally, so the driver must - * do dma mapping in advance. The driver must pass the dma_address through - * dma_address of scatterlist. When the driver got a used buffer from - * the vring, it has to unmap the dma address. - * - * This function must be called immediately after creating the vq, or after vq - * reset, and before adding any buffers to it. - * - * Caller must ensure we don't call this with other virtqueue operations - * at the same time (except where noted). - * - * Returns zero or a negative error. - * 0: success. - * -EINVAL: vring does not use the dma api, so we can not enable premapped mode. - */ -int virtqueue_set_dma_premapped(struct virtqueue *_vq) -{ - struct vring_virtqueue *vq = to_vvq(_vq); - u32 num; - - START_USE(vq); - - num = vq->packed_ring ? vq->packed.vring.num : vq->split.vring.num; - - if (num != vq->vq.num_free) { - END_USE(vq); - return -EINVAL; - } - - if (!vq->use_dma_api) { - END_USE(vq); - return -EINVAL; - } - - vq->vq.premapped = true; - - if (vq->packed_ring) { - kfree(vq->packed.desc_dma); - vq->packed.desc_dma = NULL; - } else { - kfree(vq->split.desc_dma); - vq->split.desc_dma = NULL; - } - - END_USE(vq); - - return 0; -} -EXPORT_SYMBOL_GPL(virtqueue_set_dma_premapped); - /** * virtqueue_reset - detach and recycle all unused buffers * @_vq: the struct virtqueue we're talking about. diff --git a/include/linux/virtio.h b/include/linux/virtio.h index 2e279469277c..8a8d282982ad 100644 --- a/include/linux/virtio.h +++ b/include/linux/virtio.h @@ -82,8 +82,6 @@ bool virtqueue_enable_cb(struct virtqueue *vq); unsigned virtqueue_enable_cb_prepare(struct virtqueue *vq); -int virtqueue_set_dma_premapped(struct virtqueue *_vq); - bool virtqueue_poll(struct virtqueue *vq, unsigned); bool virtqueue_enable_cb_delayed(struct virtqueue *vq); From patchwork Tue Jan 30 03:06:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13536806 Received: from out30-110.freemail.mail.aliyun.com (out30-110.freemail.mail.aliyun.com [115.124.30.110]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7CD775821B; Tue, 30 Jan 2024 03:06:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.110 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706583998; cv=none; b=sXodO724HPehPfD+8Dqe8ZfnDqFA0hQmgn59+Ib/m2B7K487O/Pki65F8+MA8wKKK1F26smwGDFH+cde7WAV1K3OVvdc7ighBtE+FERqBUd02T5ltOtI+LJARJwPvYy1bVBaMwUcrENn3vs5KCeIuCGIxfS/H5FZl+NqZIDK1S4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706583998; c=relaxed/simple; bh=WqRon/t36axLWnC2m31FaAfXbOKepA9MEJ60/dbxVng=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=Ouj7Dv83nt8BsfFin0qHb0q3W+joS3QR6dFFb+8BcsvCNjIHvldhakjXUWAFN0BA8eeZXINF2aLDUKp6cHlBBcZpXhgUhiYf4/qqjFfbgtCVaG/572XLBW1qGEGGCY2XwKPQl2UfuJ1V5moRJia/W46g4b78B2k+QAnccFPZVKA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=FIM9MhTL; arc=none smtp.client-ip=115.124.30.110 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="FIM9MhTL" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1706583987; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=zoyfNHAJCmyTBAcX034M6esKyxN0JDrAC5N4EOL9MWs=; b=FIM9MhTLCX0ci5s9qL7Dga/6zkISe+Ipf1Yc6tAs/WQ4jzoap457SJ29JNl9dRF0K3pPB2VnxcgeI3qxwXFw8X87LS00t/mUbCGFKZLNrNsosfmdg8i0sppb1MBRdkwv2x37nn7n13niekBBJfxoZV8fM+2ViJauHVWv8pidCqM= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R161e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046059;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=37;SR=0;TI=SMTPD_---0W.efIX0_1706583984; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W.efIX0_1706583984) by smtp.aliyun-inc.com; Tue, 30 Jan 2024 11:06:25 +0800 From: Xuan Zhuo To: virtualization@lists.linux.dev Cc: Richard Weinberger , Anton Ivanov , Johannes Berg , "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Hans de Goede , =?utf-8?q?Ilpo_J=C3=A4rvinen?= , Vadim Pasternak , Bjorn Andersson , Mathieu Poirier , Cornelia Huck , Halil Pasic , Eric Farman , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , Benjamin Berg , Yang Li , linux-um@lists.infradead.org, netdev@vger.kernel.org, platform-driver-x86@vger.kernel.org, linux-remoteproc@vger.kernel.org, linux-s390@vger.kernel.org, kvm@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH 12/14] virtio_net: unify the code for recycling the xmit ptr Date: Tue, 30 Jan 2024 11:06:02 +0800 Message-Id: <20240130030604.108463-13-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240130030604.108463-1-xuanzhuo@linux.alibaba.com> References: <20240130030604.108463-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: linux-remoteproc@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: ce068f9b825d There are two completely similar and independent implementations. This is inconvenient for the subsequent addition of new types. So extract a function from this piece of code and call this function uniformly to recover old xmit ptr. Signed-off-by: Xuan Zhuo Acked-by: Jason Wang --- drivers/net/virtio_net.c | 66 +++++++++++++++++----------------------- 1 file changed, 28 insertions(+), 38 deletions(-) diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index 660459b4c3e3..b56f8617700f 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -348,6 +348,30 @@ static struct xdp_frame *ptr_to_xdp(void *ptr) return (struct xdp_frame *)((unsigned long)ptr & ~VIRTIO_XDP_FLAG); } +static void __free_old_xmit(struct send_queue *sq, bool in_napi, + u64 *bytes, u64 *packets) +{ + unsigned int len; + void *ptr; + + while ((ptr = virtqueue_get_buf(sq->vq, &len)) != NULL) { + if (!is_xdp_frame(ptr)) { + struct sk_buff *skb = ptr; + + pr_debug("Sent skb %p\n", skb); + + *bytes += skb->len; + napi_consume_skb(skb, in_napi); + } else { + struct xdp_frame *frame = ptr_to_xdp(ptr); + + *bytes += xdp_get_frame_len(frame); + xdp_return_frame(frame); + } + (*packets)++; + } +} + /* Converting between virtqueue no. and kernel tx/rx queue no. * 0:rx0 1:tx0 2:rx1 3:tx1 ... 2N:rxN 2N+1:txN 2N+2:cvq */ @@ -740,27 +764,9 @@ static void virtnet_rq_unmap_free_buf(struct virtqueue *vq, void *buf) static void free_old_xmit_skbs(struct send_queue *sq, bool in_napi) { - unsigned int len; - unsigned int packets = 0; - unsigned int bytes = 0; - void *ptr; - - while ((ptr = virtqueue_get_buf(sq->vq, &len)) != NULL) { - if (likely(!is_xdp_frame(ptr))) { - struct sk_buff *skb = ptr; - - pr_debug("Sent skb %p\n", skb); + u64 bytes = 0, packets = 0; - bytes += skb->len; - napi_consume_skb(skb, in_napi); - } else { - struct xdp_frame *frame = ptr_to_xdp(ptr); - - bytes += xdp_get_frame_len(frame); - xdp_return_frame(frame); - } - packets++; - } + __free_old_xmit(sq, in_napi, &bytes, &packets); /* Avoid overhead when no packets have been processed * happens when called speculatively from start_xmit. @@ -910,14 +916,11 @@ static int virtnet_xdp_xmit(struct net_device *dev, { struct virtnet_info *vi = netdev_priv(dev); struct receive_queue *rq = vi->rq; + u64 bytes = 0, packets = 0; struct bpf_prog *xdp_prog; struct send_queue *sq; - unsigned int len; - int packets = 0; - int bytes = 0; int nxmit = 0; int kicks = 0; - void *ptr; int ret; int i; @@ -936,20 +939,7 @@ static int virtnet_xdp_xmit(struct net_device *dev, } /* Free up any pending old buffers before queueing new ones. */ - while ((ptr = virtqueue_get_buf(sq->vq, &len)) != NULL) { - if (likely(is_xdp_frame(ptr))) { - struct xdp_frame *frame = ptr_to_xdp(ptr); - - bytes += xdp_get_frame_len(frame); - xdp_return_frame(frame); - } else { - struct sk_buff *skb = ptr; - - bytes += skb->len; - napi_consume_skb(skb, false); - } - packets++; - } + __free_old_xmit(sq, false, &bytes, &packets); for (i = 0; i < n; i++) { struct xdp_frame *xdpf = frames[i]; From patchwork Tue Jan 30 03:06:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13536802 Received: from out30-98.freemail.mail.aliyun.com (out30-98.freemail.mail.aliyun.com [115.124.30.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2B844374DB; Tue, 30 Jan 2024 03:06:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.98 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706583993; cv=none; b=DrYSCvUzdJWYfkQ38V/sUzjjBgHrtdbst5noH3vpN1irDQoPaIMjigXzU/PsgCyojGxLEBT15U/Dmsonr9WoVfECATVdx6NVtzP69zx0JaHkPJBlREFAr4ZCtaME0U9VRPfTwQiHsackq/ZvxVzoUfKKXAvdwUsXPjocPBaGNKE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706583993; c=relaxed/simple; bh=WrNUrVkKsRBGMAcCLsS54ceMN+IjqBq6LF2Ggr3xROw=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=MdDyA0o9lX2PkgIDpKqh5+S7eUsVECbtvZuISY8P97gTQ48yzgdT/Asj4WKvItLy2ruY5ztZJeaber7nTrXJ7m++fKQYMN6EyPY3DhQqMrGHBQ/munoIOjBTzZwEGOWUdx/CZfaoRHUOYUTQs5qvYFWGyigSOFP4Hjvlz1ZtfJg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=UOlWV3MD; arc=none smtp.client-ip=115.124.30.98 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="UOlWV3MD" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1706583988; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=In+uheSTb8yqlBUteGC87q6ZU8VjjfZZMdZdMrF6NfA=; b=UOlWV3MDco3yheBv1fDfb8PJjpkS/s4sPIm9NoB8txCJG7mGdbhiukP05hTrhiymJXkf5AMxq1D2IVlKey3s4EM/P5z5jrOfOpcHT4Cc+iEGyGMX9e4pU4uWDiHRJngkGNkAfGvPIRTtoOF6ssVg2EVyE18oXwWL6HPjRirDVoY= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R291e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045176;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=37;SR=0;TI=SMTPD_---0W.eeCGK_1706583985; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W.eeCGK_1706583985) by smtp.aliyun-inc.com; Tue, 30 Jan 2024 11:06:26 +0800 From: Xuan Zhuo To: virtualization@lists.linux.dev Cc: Richard Weinberger , Anton Ivanov , Johannes Berg , "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Hans de Goede , =?utf-8?q?Ilpo_J=C3=A4rvinen?= , Vadim Pasternak , Bjorn Andersson , Mathieu Poirier , Cornelia Huck , Halil Pasic , Eric Farman , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , Benjamin Berg , Yang Li , linux-um@lists.infradead.org, netdev@vger.kernel.org, platform-driver-x86@vger.kernel.org, linux-remoteproc@vger.kernel.org, linux-s390@vger.kernel.org, kvm@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH 13/14] virtio_net: rename free_old_xmit_skbs to free_old_xmit Date: Tue, 30 Jan 2024 11:06:03 +0800 Message-Id: <20240130030604.108463-14-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240130030604.108463-1-xuanzhuo@linux.alibaba.com> References: <20240130030604.108463-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: linux-remoteproc@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: ce068f9b825d Since free_old_xmit_skbs not only deals with skb, but also xdp frame and subsequent added xsk, so change the name of this function to free_old_xmit. Signed-off-by: Xuan Zhuo Acked-by: Jason Wang --- drivers/net/virtio_net.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index b56f8617700f..1118aa0ebc53 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -762,7 +762,7 @@ static void virtnet_rq_unmap_free_buf(struct virtqueue *vq, void *buf) virtnet_rq_free_buf(vi, rq, buf); } -static void free_old_xmit_skbs(struct send_queue *sq, bool in_napi) +static void free_old_xmit(struct send_queue *sq, bool in_napi) { u64 bytes = 0, packets = 0; @@ -816,7 +816,7 @@ static void check_sq_full_and_disable(struct virtnet_info *vi, virtqueue_napi_schedule(&sq->napi, sq->vq); } else if (unlikely(!virtqueue_enable_cb_delayed(sq->vq))) { /* More just got used, free them then recheck. */ - free_old_xmit_skbs(sq, false); + free_old_xmit(sq, false); if (sq->vq->num_free >= 2+MAX_SKB_FRAGS) { netif_start_subqueue(dev, qnum); virtqueue_disable_cb(sq->vq); @@ -2111,7 +2111,7 @@ static void virtnet_poll_cleantx(struct receive_queue *rq) do { virtqueue_disable_cb(sq->vq); - free_old_xmit_skbs(sq, true); + free_old_xmit(sq, true); } while (unlikely(!virtqueue_enable_cb_delayed(sq->vq))); if (sq->vq->num_free >= 2 + MAX_SKB_FRAGS) @@ -2233,7 +2233,7 @@ static int virtnet_poll_tx(struct napi_struct *napi, int budget) txq = netdev_get_tx_queue(vi->dev, index); __netif_tx_lock(txq, raw_smp_processor_id()); virtqueue_disable_cb(sq->vq); - free_old_xmit_skbs(sq, true); + free_old_xmit(sq, true); if (sq->vq->num_free >= 2 + MAX_SKB_FRAGS) netif_tx_wake_queue(txq); @@ -2323,7 +2323,7 @@ static netdev_tx_t start_xmit(struct sk_buff *skb, struct net_device *dev) if (use_napi) virtqueue_disable_cb(sq->vq); - free_old_xmit_skbs(sq, false); + free_old_xmit(sq, false); } while (use_napi && kick && unlikely(!virtqueue_enable_cb_delayed(sq->vq))); From patchwork Tue Jan 30 03:06:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xuan Zhuo X-Patchwork-Id: 13536804 Received: from out30-124.freemail.mail.aliyun.com (out30-124.freemail.mail.aliyun.com [115.124.30.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 46A2A5813B; Tue, 30 Jan 2024 03:06:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=115.124.30.124 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706583997; cv=none; b=lkGpEhnLZDuao6WChQR056/F8i6aqywEIfdGlHgs2DwAzes3pnDB+lIcac10bl+26T8he33K3uSfCZt1DDKghI3giK2cfol5Rd9znxL+hMgFhoDU+xwKCebyLjkW5QJIi+dQoJJyQ3vodUUYQf/t/6i0N5ldKwdj/VlIflDSXlU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1706583997; c=relaxed/simple; bh=CAlEWt4vumuPcH4ua+wCQwPEZkkLaIhlPIZH53Cr+P8=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=eSe2iX8KAmMFMt59HJuuPTRYGhTcackEIOVgAiKPr0xd3dxz5MmSGD20oYZHzJkJ50b0T1teSNyh6OGMv51RE/in9ESSSKmm8FT6gObPJ3R9y1ViglEWjAht8bybTtSagieykuAvRVqbPz+XRvmEDj3KUGc3Oi6S7IhfBp8pMIg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com; spf=pass smtp.mailfrom=linux.alibaba.com; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b=esMBzXVK; arc=none smtp.client-ip=115.124.30.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.alibaba.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.alibaba.com header.i=@linux.alibaba.com header.b="esMBzXVK" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1706583992; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=k3jrHVzeGbkw+MbEmyF2IW48EPUzLK1XdzB5/asDKLs=; b=esMBzXVKr8SB+eH4Vrq0+w9ocluwyOH21HI/uzdjRSrz4EXzY/Z5BPccs9Kl27QeapbXTj+xcFFv6GT4yxdLbmnPaeqLar5eoED2Cu9DiPmyZ6+I5MrmV4AEHNseBepI7y0RhSfsl9oY2wf0BD0PZSjuFW5Pn2cltnthTAzv7KY= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R591e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045192;MF=xuanzhuo@linux.alibaba.com;NM=1;PH=DS;RN=37;SR=0;TI=SMTPD_---0W.efIYH_1706583987; Received: from localhost(mailfrom:xuanzhuo@linux.alibaba.com fp:SMTPD_---0W.efIYH_1706583987) by smtp.aliyun-inc.com; Tue, 30 Jan 2024 11:06:28 +0800 From: Xuan Zhuo To: virtualization@lists.linux.dev Cc: Richard Weinberger , Anton Ivanov , Johannes Berg , "Michael S. Tsirkin" , Jason Wang , Xuan Zhuo , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Hans de Goede , =?utf-8?q?Ilpo_J=C3=A4rvinen?= , Vadim Pasternak , Bjorn Andersson , Mathieu Poirier , Cornelia Huck , Halil Pasic , Eric Farman , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , Christian Borntraeger , Sven Schnelle , Alexei Starovoitov , Daniel Borkmann , Jesper Dangaard Brouer , John Fastabend , Benjamin Berg , Yang Li , linux-um@lists.infradead.org, netdev@vger.kernel.org, platform-driver-x86@vger.kernel.org, linux-remoteproc@vger.kernel.org, linux-s390@vger.kernel.org, kvm@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH 14/14] virtio_net: sq support premapped mode Date: Tue, 30 Jan 2024 11:06:04 +0800 Message-Id: <20240130030604.108463-15-xuanzhuo@linux.alibaba.com> X-Mailer: git-send-email 2.32.0.3.g01195cf9f In-Reply-To: <20240130030604.108463-1-xuanzhuo@linux.alibaba.com> References: <20240130030604.108463-1-xuanzhuo@linux.alibaba.com> Precedence: bulk X-Mailing-List: linux-remoteproc@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Git-Hash: ce068f9b825d If the xsk is enabling, the xsk tx will share the send queue. But the xsk requires that the send queue use the premapped mode. So the send queue must support premapped mode. Signed-off-by: Xuan Zhuo --- drivers/net/virtio_net.c | 167 ++++++++++++++++++++++++++++++++++++++- 1 file changed, 163 insertions(+), 4 deletions(-) diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index 1118aa0ebc53..e007759c60ba 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -46,6 +46,7 @@ module_param(napi_tx, bool, 0644); #define VIRTIO_XDP_REDIR BIT(1) #define VIRTIO_XDP_FLAG BIT(0) +#define VIRTIO_DMA_FLAG BIT(1) /* RX packet size EWMA. The average packet size is used to determine the packet * buffer size when refilling RX rings. As the entire RX ring may be refilled @@ -140,6 +141,21 @@ struct virtnet_rq_dma { u16 need_sync; }; +struct virtnet_sq_dma { + union { + struct virtnet_sq_dma *next; + void *data; + }; + dma_addr_t addr; + u32 len; + bool is_tail; +}; + +struct virtnet_sq_dma_head { + struct virtnet_sq_dma *free; + struct virtnet_sq_dma *head; +}; + /* Internal representation of a send virtqueue */ struct send_queue { /* Virtqueue associated with this send _queue */ @@ -159,6 +175,8 @@ struct send_queue { /* Record whether sq is in reset state. */ bool reset; + + struct virtnet_sq_dma_head dmainfo; }; /* Internal representation of a receive virtqueue */ @@ -348,6 +366,131 @@ static struct xdp_frame *ptr_to_xdp(void *ptr) return (struct xdp_frame *)((unsigned long)ptr & ~VIRTIO_XDP_FLAG); } +static inline void *virtnet_sq_unmap(struct send_queue *sq, void *data) +{ + struct virtnet_sq_dma *head, *tail; + + if (!((unsigned long)data & VIRTIO_DMA_FLAG)) + return data; + + head = (void *)((unsigned long)data & ~VIRTIO_DMA_FLAG); + + tail = head; + + while (true) { + virtqueue_dma_unmap_page_attrs(sq->vq, tail->addr, tail->len, + DMA_TO_DEVICE, 0); + + if (tail->is_tail) + break; + + tail = tail->next; + } + + data = tail->data; + tail->is_tail = false; + + tail->next = sq->dmainfo.free; + sq->dmainfo.free = head; + + return data; +} + +static void *virtnet_sq_dma_splice(struct send_queue *sq, + struct virtnet_sq_dma *head, + struct virtnet_sq_dma *tail, + void *data) +{ + sq->dmainfo.free = tail->next; + + tail->is_tail = true; + tail->data = data; + + head = (void *)((unsigned long)head | VIRTIO_DMA_FLAG); + + return head; +} + +static struct virtnet_sq_dma *virtnet_sq_map_sg(struct send_queue *sq, int nents, void *data) +{ + struct virtnet_sq_dma *head, *tail, *p; + struct scatterlist *sg; + dma_addr_t addr; + int i; + + head = sq->dmainfo.free; + p = head; + + tail = NULL; + + for_each_sg(sq->sg, sg, nents, i) { + addr = virtqueue_dma_map_page_attrs(sq->vq, sg_page(sg), + sg->offset, sg->length, + DMA_TO_DEVICE, 0); + if (virtqueue_dma_mapping_error(sq->vq, addr)) + goto err; + + sg->dma_address = addr; + + tail = p; + tail->addr = sg->dma_address; + tail->len = sg->length; + + p = p->next; + } + + return virtnet_sq_dma_splice(sq, head, tail, data); + +err: + if (tail) + virtnet_sq_unmap(sq, virtnet_sq_dma_splice(sq, head, tail, data)); + + return NULL; +} + +static int virtnet_add_outbuf(struct send_queue *sq, u32 num, void *data) +{ + int ret; + + if (sq->vq->premapped) { + data = virtnet_sq_map_sg(sq, num, data); + if (!data) + return -ENOMEM; + } + + ret = virtqueue_add_outbuf(sq->vq, sq->sg, num, data, GFP_ATOMIC); + if (ret && sq->vq->premapped) + virtnet_sq_unmap(sq, data); + + return ret; +} + +static int virtnet_sq_init_dma_mate(struct send_queue *sq) +{ + struct virtnet_sq_dma *d; + int size, i; + + size = virtqueue_get_vring_size(sq->vq); + + size += MAX_SKB_FRAGS + 2; + + sq->dmainfo.head = kcalloc(size, sizeof(*sq->dmainfo.head), GFP_KERNEL); + if (!sq->dmainfo.head) + return -ENOMEM; + + sq->dmainfo.free = sq->dmainfo.head; + + for (i = 0; i < size; ++i) { + d = &sq->dmainfo.head[i]; + d->is_tail = false; + d->next = d + 1; + } + + d->next = NULL; + + return 0; +} + static void __free_old_xmit(struct send_queue *sq, bool in_napi, u64 *bytes, u64 *packets) { @@ -355,6 +498,8 @@ static void __free_old_xmit(struct send_queue *sq, bool in_napi, void *ptr; while ((ptr = virtqueue_get_buf(sq->vq, &len)) != NULL) { + ptr = virtnet_sq_unmap(sq, ptr); + if (!is_xdp_frame(ptr)) { struct sk_buff *skb = ptr; @@ -865,8 +1010,7 @@ static int __virtnet_xdp_xmit_one(struct virtnet_info *vi, skb_frag_size(frag), skb_frag_off(frag)); } - err = virtqueue_add_outbuf(sq->vq, sq->sg, nr_frags + 1, - xdp_to_ptr(xdpf), GFP_ATOMIC); + err = virtnet_add_outbuf(sq, nr_frags + 1, xdp_to_ptr(xdpf)); if (unlikely(err)) return -ENOSPC; /* Caller handle free/refcnt */ @@ -2305,7 +2449,7 @@ static int xmit_skb(struct send_queue *sq, struct sk_buff *skb) return num_sg; num_sg++; } - return virtqueue_add_outbuf(sq->vq, sq->sg, num_sg, skb, GFP_ATOMIC); + return virtnet_add_outbuf(sq, num_sg, skb); } static netdev_tx_t start_xmit(struct sk_buff *skb, struct net_device *dev) @@ -3961,6 +4105,8 @@ static void virtnet_free_queues(struct virtnet_info *vi) for (i = 0; i < vi->max_queue_pairs; i++) { __netif_napi_del(&vi->rq[i].napi); __netif_napi_del(&vi->sq[i].napi); + + kfree(vi->sq[i].dmainfo.head); } /* We called __netif_napi_del(), @@ -4009,6 +4155,14 @@ static void free_receive_page_frags(struct virtnet_info *vi) static void virtnet_sq_free_unused_buf(struct virtqueue *vq, void *buf) { + struct virtnet_info *vi = vq->vdev->priv; + struct send_queue *sq; + int i = vq2rxq(vq); + + sq = &vi->sq[i]; + + buf = virtnet_sq_unmap(sq, buf); + if (!is_xdp_frame(buf)) dev_kfree_skb(buf); else @@ -4120,8 +4274,10 @@ static int virtnet_find_vqs(struct virtnet_info *vi) if (ctx) ctx[rxq2vq(i)] = true; - if (premapped) + if (premapped) { premapped[rxq2vq(i)] = true; + premapped[txq2vq(i)] = true; + } } ret = virtio_find_vqs_ctx_premapped(vi->vdev, total_vqs, vqs, callbacks, @@ -4139,6 +4295,9 @@ static int virtnet_find_vqs(struct virtnet_info *vi) vi->rq[i].vq = vqs[rxq2vq(i)]; vi->rq[i].min_buf_len = mergeable_min_buf_len(vi, vi->rq[i].vq); vi->sq[i].vq = vqs[txq2vq(i)]; + + if (vi->sq[i].vq->premapped) + virtnet_sq_init_dma_mate(&vi->sq[i]); } /* run here: ret == 0. */