| Message ID | 20240229072044.77388-1-xuanzhuo@linux.alibaba.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | virtio: drivers maintain dma info for premapped vq | expand |
On Thu, Feb 29, 2024 at 03:20:25PM +0800, Xuan Zhuo wrote: > As discussed: > http://lore.kernel.org/all/CACGkMEvq0No8QGC46U4mGsMtuD44fD_cfLcPaVmJ3rHYqRZxYg@mail.gmail.com > > If the virtio is premapped mode, the driver should manage the dma info by self. > So the virtio core should not store the dma info. > So we can release the memory used to store the dma info. > > But if the desc_extra has not dma info, we face a new question, > it is hard to get the dma info of the desc with indirect flag. > For split mode, that is easy from desc, but for the packed mode, > it is hard to get the dma info from the desc. And for hardening > the dma unmap is saft, we should store the dma info of indirect > descs. > > So I introduce the "structure the indirect desc table" to > allocate space to store dma info with the desc table. > > On the other side, we mix the descs with indirect flag > with other descs together to share the unmap api. That > is complex. I found if we we distinguish the descs with > VRING_DESC_F_INDIRECT before unmap, thing will be clearer. > > Because of the dma array is allocated in the find_vqs(), > so I introduce a new parameter to find_vqs(). > > Note: > this is on the top of > [PATCH vhost v1] virtio: packed: fix unmap leak for indirect desc table > http://lore.kernel.org/all/20240223071833.26095-1-xuanzhuo@linux.alibaba.com > > Please review. > > Thanks > > v3: > 1. fix the conflict with the vp_modern_create_avq(). Okay but are you going to address huge memory waste all this is causing for - people who never do zero copy - systems where dma unmap is a nop ? You should address all comments when you post a new version, not just what was expedient, or alternatively tag patch as RFC and explain in commit log that you plan to do it later. > v2: > 1. change the dma item of virtio-net, every item have MAX_SKB_FRAGS + 2 > addr + len pairs. > 2. introduce virtnet_sq_free_stats for __free_old_xmit > > v1: > 1. rename transport_vq_config to vq_transport_config > 2. virtio-net set dma meta number to (ring-size + 1)(MAX_SKB_FRGAS +2) > 3. introduce virtqueue_dma_map_sg_attrs > 4. separate vring_create_virtqueue to an independent commit > > > > Xuan Zhuo (19): > virtio_ring: introduce vring_need_unmap_buffer > virtio_ring: packed: remove double check of the unmap ops > virtio_ring: packed: structure the indirect desc table > virtio_ring: split: remove double check of the unmap ops > virtio_ring: split: structure the indirect desc table > virtio_ring: no store dma info when unmap is not needed > virtio: find_vqs: pass struct instead of multi parameters > virtio: vring_create_virtqueue: pass struct instead of multi > parameters > virtio: vring_new_virtqueue(): pass struct instead of multi parameters > virtio_ring: simplify the parameters of the funcs related to > vring_create/new_virtqueue() > virtio: find_vqs: add new parameter premapped > virtio_ring: export premapped to driver by struct virtqueue > virtio_net: set premapped mode by find_vqs() > virtio_ring: remove api of setting vq premapped > virtio_ring: introduce dma map api for page > virtio_ring: introduce virtqueue_dma_map_sg_attrs > virtio_net: unify the code for recycling the xmit ptr > virtio_net: rename free_old_xmit_skbs to free_old_xmit > virtio_net: sq support premapped mode > > arch/um/drivers/virtio_uml.c | 31 +- > drivers/net/virtio_net.c | 283 ++++++--- > drivers/platform/mellanox/mlxbf-tmfifo.c | 24 +- > drivers/remoteproc/remoteproc_virtio.c | 31 +- > drivers/s390/virtio/virtio_ccw.c | 33 +- > drivers/virtio/virtio_mmio.c | 30 +- > drivers/virtio/virtio_pci_common.c | 59 +- > drivers/virtio/virtio_pci_common.h | 9 +- > drivers/virtio/virtio_pci_legacy.c | 16 +- > drivers/virtio/virtio_pci_modern.c | 38 +- > drivers/virtio/virtio_ring.c | 698 ++++++++++++----------- > drivers/virtio/virtio_vdpa.c | 45 +- > include/linux/virtio.h | 13 +- > include/linux/virtio_config.h | 48 +- > include/linux/virtio_ring.h | 82 +-- > tools/virtio/virtio_test.c | 4 +- > tools/virtio/vringh_test.c | 28 +- > 17 files changed, 847 insertions(+), 625 deletions(-) > > -- > 2.32.0.3.g01195cf9f
On Thu, 29 Feb 2024 03:21:14 -0500, "Michael S. Tsirkin" <mst@redhat.com> wrote: > On Thu, Feb 29, 2024 at 03:20:25PM +0800, Xuan Zhuo wrote: > > As discussed: > > http://lore.kernel.org/all/CACGkMEvq0No8QGC46U4mGsMtuD44fD_cfLcPaVmJ3rHYqRZxYg@mail.gmail.com > > > > If the virtio is premapped mode, the driver should manage the dma info by self. > > So the virtio core should not store the dma info. > > So we can release the memory used to store the dma info. > > > > But if the desc_extra has not dma info, we face a new question, > > it is hard to get the dma info of the desc with indirect flag. > > For split mode, that is easy from desc, but for the packed mode, > > it is hard to get the dma info from the desc. And for hardening > > the dma unmap is saft, we should store the dma info of indirect > > descs. > > > > So I introduce the "structure the indirect desc table" to > > allocate space to store dma info with the desc table. > > > > On the other side, we mix the descs with indirect flag > > with other descs together to share the unmap api. That > > is complex. I found if we we distinguish the descs with > > VRING_DESC_F_INDIRECT before unmap, thing will be clearer. > > > > Because of the dma array is allocated in the find_vqs(), > > so I introduce a new parameter to find_vqs(). > > > > Note: > > this is on the top of > > [PATCH vhost v1] virtio: packed: fix unmap leak for indirect desc table > > http://lore.kernel.org/all/20240223071833.26095-1-xuanzhuo@linux.alibaba.com > > > > Please review. > > > > Thanks > > > > v3: > > 1. fix the conflict with the vp_modern_create_avq(). > > Okay but are you going to address huge memory waste all this is causing for > - people who never do zero copy > - systems where dma unmap is a nop > > ? > > You should address all comments when you post a new version, not just > what was expedient, or alternatively tag patch as RFC and explain > in commit log that you plan to do it later. Do you miss this one? http://lore.kernel.org/all/1708997579.5613105-1-xuanzhuo@linux.alibaba.com I asked you. But I didnot recv your answer. Thanks. > > > v2: > > 1. change the dma item of virtio-net, every item have MAX_SKB_FRAGS + 2 > > addr + len pairs. > > 2. introduce virtnet_sq_free_stats for __free_old_xmit > > > > v1: > > 1. rename transport_vq_config to vq_transport_config > > 2. virtio-net set dma meta number to (ring-size + 1)(MAX_SKB_FRGAS +2) > > 3. introduce virtqueue_dma_map_sg_attrs > > 4. separate vring_create_virtqueue to an independent commit > > > > > > > > Xuan Zhuo (19): > > virtio_ring: introduce vring_need_unmap_buffer > > virtio_ring: packed: remove double check of the unmap ops > > virtio_ring: packed: structure the indirect desc table > > virtio_ring: split: remove double check of the unmap ops > > virtio_ring: split: structure the indirect desc table > > virtio_ring: no store dma info when unmap is not needed > > virtio: find_vqs: pass struct instead of multi parameters > > virtio: vring_create_virtqueue: pass struct instead of multi > > parameters > > virtio: vring_new_virtqueue(): pass struct instead of multi parameters > > virtio_ring: simplify the parameters of the funcs related to > > vring_create/new_virtqueue() > > virtio: find_vqs: add new parameter premapped > > virtio_ring: export premapped to driver by struct virtqueue > > virtio_net: set premapped mode by find_vqs() > > virtio_ring: remove api of setting vq premapped > > virtio_ring: introduce dma map api for page > > virtio_ring: introduce virtqueue_dma_map_sg_attrs > > virtio_net: unify the code for recycling the xmit ptr > > virtio_net: rename free_old_xmit_skbs to free_old_xmit > > virtio_net: sq support premapped mode > > > > arch/um/drivers/virtio_uml.c | 31 +- > > drivers/net/virtio_net.c | 283 ++++++--- > > drivers/platform/mellanox/mlxbf-tmfifo.c | 24 +- > > drivers/remoteproc/remoteproc_virtio.c | 31 +- > > drivers/s390/virtio/virtio_ccw.c | 33 +- > > drivers/virtio/virtio_mmio.c | 30 +- > > drivers/virtio/virtio_pci_common.c | 59 +- > > drivers/virtio/virtio_pci_common.h | 9 +- > > drivers/virtio/virtio_pci_legacy.c | 16 +- > > drivers/virtio/virtio_pci_modern.c | 38 +- > > drivers/virtio/virtio_ring.c | 698 ++++++++++++----------- > > drivers/virtio/virtio_vdpa.c | 45 +- > > include/linux/virtio.h | 13 +- > > include/linux/virtio_config.h | 48 +- > > include/linux/virtio_ring.h | 82 +-- > > tools/virtio/virtio_test.c | 4 +- > > tools/virtio/vringh_test.c | 28 +- > > 17 files changed, 847 insertions(+), 625 deletions(-) > > > > -- > > 2.32.0.3.g01195cf9f >
On Thu, Feb 29, 2024 at 05:02:37PM +0800, Xuan Zhuo wrote: > On Thu, 29 Feb 2024 03:21:14 -0500, "Michael S. Tsirkin" <mst@redhat.com> wrote: > > On Thu, Feb 29, 2024 at 03:20:25PM +0800, Xuan Zhuo wrote: > > > As discussed: > > > http://lore.kernel.org/all/CACGkMEvq0No8QGC46U4mGsMtuD44fD_cfLcPaVmJ3rHYqRZxYg@mail.gmail.com > > > > > > If the virtio is premapped mode, the driver should manage the dma info by self. > > > So the virtio core should not store the dma info. > > > So we can release the memory used to store the dma info. > > > > > > But if the desc_extra has not dma info, we face a new question, > > > it is hard to get the dma info of the desc with indirect flag. > > > For split mode, that is easy from desc, but for the packed mode, > > > it is hard to get the dma info from the desc. And for hardening > > > the dma unmap is saft, we should store the dma info of indirect > > > descs. > > > > > > So I introduce the "structure the indirect desc table" to > > > allocate space to store dma info with the desc table. > > > > > > On the other side, we mix the descs with indirect flag > > > with other descs together to share the unmap api. That > > > is complex. I found if we we distinguish the descs with > > > VRING_DESC_F_INDIRECT before unmap, thing will be clearer. > > > > > > Because of the dma array is allocated in the find_vqs(), > > > so I introduce a new parameter to find_vqs(). > > > > > > Note: > > > this is on the top of > > > [PATCH vhost v1] virtio: packed: fix unmap leak for indirect desc table > > > http://lore.kernel.org/all/20240223071833.26095-1-xuanzhuo@linux.alibaba.com > > > > > > Please review. > > > > > > Thanks > > > > > > v3: > > > 1. fix the conflict with the vp_modern_create_avq(). > > > > Okay but are you going to address huge memory waste all this is causing for > > - people who never do zero copy > > - systems where dma unmap is a nop > > > > ? > > > > You should address all comments when you post a new version, not just > > what was expedient, or alternatively tag patch as RFC and explain > > in commit log that you plan to do it later. > > > Do you miss this one? > http://lore.kernel.org/all/1708997579.5613105-1-xuanzhuo@linux.alibaba.com I did. The answer is that no, you don't get to regress memory usage for lots of people then fix it up. So the patchset is big, I guess it will take a couple of cycles to merge gradually. > I asked you. But I didnot recv your answer. > > Thanks. > > > > > > > v2: > > > 1. change the dma item of virtio-net, every item have MAX_SKB_FRAGS + 2 > > > addr + len pairs. > > > 2. introduce virtnet_sq_free_stats for __free_old_xmit > > > > > > v1: > > > 1. rename transport_vq_config to vq_transport_config > > > 2. virtio-net set dma meta number to (ring-size + 1)(MAX_SKB_FRGAS +2) > > > 3. introduce virtqueue_dma_map_sg_attrs > > > 4. separate vring_create_virtqueue to an independent commit > > > > > > > > > > > > Xuan Zhuo (19): > > > virtio_ring: introduce vring_need_unmap_buffer > > > virtio_ring: packed: remove double check of the unmap ops > > > virtio_ring: packed: structure the indirect desc table > > > virtio_ring: split: remove double check of the unmap ops > > > virtio_ring: split: structure the indirect desc table > > > virtio_ring: no store dma info when unmap is not needed > > > virtio: find_vqs: pass struct instead of multi parameters > > > virtio: vring_create_virtqueue: pass struct instead of multi > > > parameters > > > virtio: vring_new_virtqueue(): pass struct instead of multi parameters > > > virtio_ring: simplify the parameters of the funcs related to > > > vring_create/new_virtqueue() > > > virtio: find_vqs: add new parameter premapped > > > virtio_ring: export premapped to driver by struct virtqueue > > > virtio_net: set premapped mode by find_vqs() > > > virtio_ring: remove api of setting vq premapped > > > virtio_ring: introduce dma map api for page > > > virtio_ring: introduce virtqueue_dma_map_sg_attrs > > > virtio_net: unify the code for recycling the xmit ptr > > > virtio_net: rename free_old_xmit_skbs to free_old_xmit > > > virtio_net: sq support premapped mode > > > > > > arch/um/drivers/virtio_uml.c | 31 +- > > > drivers/net/virtio_net.c | 283 ++++++--- > > > drivers/platform/mellanox/mlxbf-tmfifo.c | 24 +- > > > drivers/remoteproc/remoteproc_virtio.c | 31 +- > > > drivers/s390/virtio/virtio_ccw.c | 33 +- > > > drivers/virtio/virtio_mmio.c | 30 +- > > > drivers/virtio/virtio_pci_common.c | 59 +- > > > drivers/virtio/virtio_pci_common.h | 9 +- > > > drivers/virtio/virtio_pci_legacy.c | 16 +- > > > drivers/virtio/virtio_pci_modern.c | 38 +- > > > drivers/virtio/virtio_ring.c | 698 ++++++++++++----------- > > > drivers/virtio/virtio_vdpa.c | 45 +- > > > include/linux/virtio.h | 13 +- > > > include/linux/virtio_config.h | 48 +- > > > include/linux/virtio_ring.h | 82 +-- > > > tools/virtio/virtio_test.c | 4 +- > > > tools/virtio/vringh_test.c | 28 +- > > > 17 files changed, 847 insertions(+), 625 deletions(-) > > > > > > -- > > > 2.32.0.3.g01195cf9f > >
On Thu, 29 Feb 2024 04:34:20 -0500, "Michael S. Tsirkin" <mst@redhat.com> wrote: > On Thu, Feb 29, 2024 at 05:02:37PM +0800, Xuan Zhuo wrote: > > On Thu, 29 Feb 2024 03:21:14 -0500, "Michael S. Tsirkin" <mst@redhat.com> wrote: > > > On Thu, Feb 29, 2024 at 03:20:25PM +0800, Xuan Zhuo wrote: > > > > As discussed: > > > > http://lore.kernel.org/all/CACGkMEvq0No8QGC46U4mGsMtuD44fD_cfLcPaVmJ3rHYqRZxYg@mail.gmail.com > > > > > > > > If the virtio is premapped mode, the driver should manage the dma info by self. > > > > So the virtio core should not store the dma info. > > > > So we can release the memory used to store the dma info. > > > > > > > > But if the desc_extra has not dma info, we face a new question, > > > > it is hard to get the dma info of the desc with indirect flag. > > > > For split mode, that is easy from desc, but for the packed mode, > > > > it is hard to get the dma info from the desc. And for hardening > > > > the dma unmap is saft, we should store the dma info of indirect > > > > descs. > > > > > > > > So I introduce the "structure the indirect desc table" to > > > > allocate space to store dma info with the desc table. > > > > > > > > On the other side, we mix the descs with indirect flag > > > > with other descs together to share the unmap api. That > > > > is complex. I found if we we distinguish the descs with > > > > VRING_DESC_F_INDIRECT before unmap, thing will be clearer. > > > > > > > > Because of the dma array is allocated in the find_vqs(), > > > > so I introduce a new parameter to find_vqs(). > > > > > > > > Note: > > > > this is on the top of > > > > [PATCH vhost v1] virtio: packed: fix unmap leak for indirect desc table > > > > http://lore.kernel.org/all/20240223071833.26095-1-xuanzhuo@linux.alibaba.com > > > > > > > > Please review. > > > > > > > > Thanks > > > > > > > > v3: > > > > 1. fix the conflict with the vp_modern_create_avq(). > > > > > > Okay but are you going to address huge memory waste all this is causing for > > > - people who never do zero copy > > > - systems where dma unmap is a nop > > > > > > ? > > > > > > You should address all comments when you post a new version, not just > > > what was expedient, or alternatively tag patch as RFC and explain > > > in commit log that you plan to do it later. > > > > > > Do you miss this one? > > http://lore.kernel.org/all/1708997579.5613105-1-xuanzhuo@linux.alibaba.com > > > I did. The answer is that no, you don't get to regress memory usage > for lots of people then fix it up. > So the patchset is big, I guess it will take a couple of cycles to > merge gradually. I see. But I hope we can review the other part of this patch set. @Jason @Michael I don't want us to give up on this plan for some other reason after I've done so much work. Thanks. > > > I asked you. But I didnot recv your answer. > > > > Thanks. > > > > > > > > > > > v2: > > > > 1. change the dma item of virtio-net, every item have MAX_SKB_FRAGS + 2 > > > > addr + len pairs. > > > > 2. introduce virtnet_sq_free_stats for __free_old_xmit > > > > > > > > v1: > > > > 1. rename transport_vq_config to vq_transport_config > > > > 2. virtio-net set dma meta number to (ring-size + 1)(MAX_SKB_FRGAS +2) > > > > 3. introduce virtqueue_dma_map_sg_attrs > > > > 4. separate vring_create_virtqueue to an independent commit > > > > > > > > > > > > > > > > Xuan Zhuo (19): > > > > virtio_ring: introduce vring_need_unmap_buffer > > > > virtio_ring: packed: remove double check of the unmap ops > > > > virtio_ring: packed: structure the indirect desc table > > > > virtio_ring: split: remove double check of the unmap ops > > > > virtio_ring: split: structure the indirect desc table > > > > virtio_ring: no store dma info when unmap is not needed > > > > virtio: find_vqs: pass struct instead of multi parameters > > > > virtio: vring_create_virtqueue: pass struct instead of multi > > > > parameters > > > > virtio: vring_new_virtqueue(): pass struct instead of multi parameters > > > > virtio_ring: simplify the parameters of the funcs related to > > > > vring_create/new_virtqueue() > > > > virtio: find_vqs: add new parameter premapped > > > > virtio_ring: export premapped to driver by struct virtqueue > > > > virtio_net: set premapped mode by find_vqs() > > > > virtio_ring: remove api of setting vq premapped > > > > virtio_ring: introduce dma map api for page > > > > virtio_ring: introduce virtqueue_dma_map_sg_attrs > > > > virtio_net: unify the code for recycling the xmit ptr > > > > virtio_net: rename free_old_xmit_skbs to free_old_xmit > > > > virtio_net: sq support premapped mode > > > > > > > > arch/um/drivers/virtio_uml.c | 31 +- > > > > drivers/net/virtio_net.c | 283 ++++++--- > > > > drivers/platform/mellanox/mlxbf-tmfifo.c | 24 +- > > > > drivers/remoteproc/remoteproc_virtio.c | 31 +- > > > > drivers/s390/virtio/virtio_ccw.c | 33 +- > > > > drivers/virtio/virtio_mmio.c | 30 +- > > > > drivers/virtio/virtio_pci_common.c | 59 +- > > > > drivers/virtio/virtio_pci_common.h | 9 +- > > > > drivers/virtio/virtio_pci_legacy.c | 16 +- > > > > drivers/virtio/virtio_pci_modern.c | 38 +- > > > > drivers/virtio/virtio_ring.c | 698 ++++++++++++----------- > > > > drivers/virtio/virtio_vdpa.c | 45 +- > > > > include/linux/virtio.h | 13 +- > > > > include/linux/virtio_config.h | 48 +- > > > > include/linux/virtio_ring.h | 82 +-- > > > > tools/virtio/virtio_test.c | 4 +- > > > > tools/virtio/vringh_test.c | 28 +- > > > > 17 files changed, 847 insertions(+), 625 deletions(-) > > > > > > > > -- > > > > 2.32.0.3.g01195cf9f > > > >
On Thu, 2024-02-29 at 04:34 -0500, Michael S. Tsirkin wrote: > > So the patchset is big, I guess it will take a couple of cycles to > merge gradually. Could also do patches 7, 8, and maybe 9 separately first (which seem reasonable even on their own) and get rid of CC'ing so many people and lists for future iterations of the rest of the patchset :-) johannes
On Thu, 29 Feb 2024 10:41:11 +0100, Johannes Berg <johannes@sipsolutions.net> wrote: > On Thu, 2024-02-29 at 04:34 -0500, Michael S. Tsirkin wrote: > > > > So the patchset is big, I guess it will take a couple of cycles to > > merge gradually. > > Could also do patches 7, 8, and maybe 9 separately first (which seem > reasonable even on their own) and get rid of CC'ing so many people and > lists for future iterations of the rest of the patchset :-) I agree. @Michael How about you? Thanks > > johannes
On Thu, 29 Feb 2024 04:34:20 -0500, "Michael S. Tsirkin" <mst@redhat.com> wrote: > On Thu, Feb 29, 2024 at 05:02:37PM +0800, Xuan Zhuo wrote: > > On Thu, 29 Feb 2024 03:21:14 -0500, "Michael S. Tsirkin" <mst@redhat.com> wrote: > > > On Thu, Feb 29, 2024 at 03:20:25PM +0800, Xuan Zhuo wrote: > > > > As discussed: > > > > http://lore.kernel.org/all/CACGkMEvq0No8QGC46U4mGsMtuD44fD_cfLcPaVmJ3rHYqRZxYg@mail.gmail.com > > > > > > > > If the virtio is premapped mode, the driver should manage the dma info by self. > > > > So the virtio core should not store the dma info. > > > > So we can release the memory used to store the dma info. > > > > > > > > But if the desc_extra has not dma info, we face a new question, > > > > it is hard to get the dma info of the desc with indirect flag. > > > > For split mode, that is easy from desc, but for the packed mode, > > > > it is hard to get the dma info from the desc. And for hardening > > > > the dma unmap is saft, we should store the dma info of indirect > > > > descs. > > > > > > > > So I introduce the "structure the indirect desc table" to > > > > allocate space to store dma info with the desc table. > > > > > > > > On the other side, we mix the descs with indirect flag > > > > with other descs together to share the unmap api. That > > > > is complex. I found if we we distinguish the descs with > > > > VRING_DESC_F_INDIRECT before unmap, thing will be clearer. > > > > > > > > Because of the dma array is allocated in the find_vqs(), > > > > so I introduce a new parameter to find_vqs(). > > > > > > > > Note: > > > > this is on the top of > > > > [PATCH vhost v1] virtio: packed: fix unmap leak for indirect desc table > > > > http://lore.kernel.org/all/20240223071833.26095-1-xuanzhuo@linux.alibaba.com > > > > > > > > Please review. > > > > > > > > Thanks > > > > > > > > v3: > > > > 1. fix the conflict with the vp_modern_create_avq(). > > > > > > Okay but are you going to address huge memory waste all this is causing for > > > - people who never do zero copy > > > - systems where dma unmap is a nop > > > > > > ? > > > > > > You should address all comments when you post a new version, not just > > > what was expedient, or alternatively tag patch as RFC and explain > > > in commit log that you plan to do it later. > > > > > > Do you miss this one? > > http://lore.kernel.org/all/1708997579.5613105-1-xuanzhuo@linux.alibaba.com > > > I did. The answer is that no, you don't get to regress memory usage > for lots of people then fix it up. > So the patchset is big, I guess it will take a couple of cycles to > merge gradually. Hi @Michael So, how about this patch set? I do not think they (dma maintainers) will agree the API dma_can_skip_unmap(). If you think sq wastes too much memory using pre-mapped dma mode, how about we only enable it when xsk is bond? Could you give me some advice? Thanks. > > > I asked you. But I didnot recv your answer. > > > > Thanks. > > > > > > > > > > > v2: > > > > 1. change the dma item of virtio-net, every item have MAX_SKB_FRAGS + 2 > > > > addr + len pairs. > > > > 2. introduce virtnet_sq_free_stats for __free_old_xmit > > > > > > > > v1: > > > > 1. rename transport_vq_config to vq_transport_config > > > > 2. virtio-net set dma meta number to (ring-size + 1)(MAX_SKB_FRGAS +2) > > > > 3. introduce virtqueue_dma_map_sg_attrs > > > > 4. separate vring_create_virtqueue to an independent commit > > > > > > > > > > > > > > > > Xuan Zhuo (19): > > > > virtio_ring: introduce vring_need_unmap_buffer > > > > virtio_ring: packed: remove double check of the unmap ops > > > > virtio_ring: packed: structure the indirect desc table > > > > virtio_ring: split: remove double check of the unmap ops > > > > virtio_ring: split: structure the indirect desc table > > > > virtio_ring: no store dma info when unmap is not needed > > > > virtio: find_vqs: pass struct instead of multi parameters > > > > virtio: vring_create_virtqueue: pass struct instead of multi > > > > parameters > > > > virtio: vring_new_virtqueue(): pass struct instead of multi parameters > > > > virtio_ring: simplify the parameters of the funcs related to > > > > vring_create/new_virtqueue() > > > > virtio: find_vqs: add new parameter premapped > > > > virtio_ring: export premapped to driver by struct virtqueue > > > > virtio_net: set premapped mode by find_vqs() > > > > virtio_ring: remove api of setting vq premapped > > > > virtio_ring: introduce dma map api for page > > > > virtio_ring: introduce virtqueue_dma_map_sg_attrs > > > > virtio_net: unify the code for recycling the xmit ptr > > > > virtio_net: rename free_old_xmit_skbs to free_old_xmit > > > > virtio_net: sq support premapped mode > > > > > > > > arch/um/drivers/virtio_uml.c | 31 +- > > > > drivers/net/virtio_net.c | 283 ++++++--- > > > > drivers/platform/mellanox/mlxbf-tmfifo.c | 24 +- > > > > drivers/remoteproc/remoteproc_virtio.c | 31 +- > > > > drivers/s390/virtio/virtio_ccw.c | 33 +- > > > > drivers/virtio/virtio_mmio.c | 30 +- > > > > drivers/virtio/virtio_pci_common.c | 59 +- > > > > drivers/virtio/virtio_pci_common.h | 9 +- > > > > drivers/virtio/virtio_pci_legacy.c | 16 +- > > > > drivers/virtio/virtio_pci_modern.c | 38 +- > > > > drivers/virtio/virtio_ring.c | 698 ++++++++++++----------- > > > > drivers/virtio/virtio_vdpa.c | 45 +- > > > > include/linux/virtio.h | 13 +- > > > > include/linux/virtio_config.h | 48 +- > > > > include/linux/virtio_ring.h | 82 +-- > > > > tools/virtio/virtio_test.c | 4 +- > > > > tools/virtio/vringh_test.c | 28 +- > > > > 17 files changed, 847 insertions(+), 625 deletions(-) > > > > > > > > -- > > > > 2.32.0.3.g01195cf9f > > > >
On Wed, Mar 6, 2024 at 6:01 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > > On Thu, 29 Feb 2024 04:34:20 -0500, "Michael S. Tsirkin" <mst@redhat.com> wrote: > > On Thu, Feb 29, 2024 at 05:02:37PM +0800, Xuan Zhuo wrote: > > > On Thu, 29 Feb 2024 03:21:14 -0500, "Michael S. Tsirkin" <mst@redhat.com> wrote: > > > > On Thu, Feb 29, 2024 at 03:20:25PM +0800, Xuan Zhuo wrote: > > > > > As discussed: > > > > > http://lore.kernel.org/all/CACGkMEvq0No8QGC46U4mGsMtuD44fD_cfLcPaVmJ3rHYqRZxYg@mail.gmail.com > > > > > > > > > > If the virtio is premapped mode, the driver should manage the dma info by self. > > > > > So the virtio core should not store the dma info. > > > > > So we can release the memory used to store the dma info. > > > > > > > > > > But if the desc_extra has not dma info, we face a new question, > > > > > it is hard to get the dma info of the desc with indirect flag. > > > > > For split mode, that is easy from desc, but for the packed mode, > > > > > it is hard to get the dma info from the desc. And for hardening > > > > > the dma unmap is saft, we should store the dma info of indirect > > > > > descs. > > > > > > > > > > So I introduce the "structure the indirect desc table" to > > > > > allocate space to store dma info with the desc table. > > > > > > > > > > On the other side, we mix the descs with indirect flag > > > > > with other descs together to share the unmap api. That > > > > > is complex. I found if we we distinguish the descs with > > > > > VRING_DESC_F_INDIRECT before unmap, thing will be clearer. > > > > > > > > > > Because of the dma array is allocated in the find_vqs(), > > > > > so I introduce a new parameter to find_vqs(). > > > > > > > > > > Note: > > > > > this is on the top of > > > > > [PATCH vhost v1] virtio: packed: fix unmap leak for indirect desc table > > > > > http://lore.kernel.org/all/20240223071833.26095-1-xuanzhuo@linux.alibaba.com > > > > > > > > > > Please review. > > > > > > > > > > Thanks > > > > > > > > > > v3: > > > > > 1. fix the conflict with the vp_modern_create_avq(). > > > > > > > > Okay but are you going to address huge memory waste all this is causing for > > > > - people who never do zero copy > > > > - systems where dma unmap is a nop > > > > > > > > ? > > > > > > > > You should address all comments when you post a new version, not just > > > > what was expedient, or alternatively tag patch as RFC and explain > > > > in commit log that you plan to do it later. > > > > > > > > > Do you miss this one? > > > http://lore.kernel.org/all/1708997579.5613105-1-xuanzhuo@linux.alibaba.com > > > > > > I did. The answer is that no, you don't get to regress memory usage > > for lots of people then fix it up. > > So the patchset is big, I guess it will take a couple of cycles to > > merge gradually. > > Hi @Michael > > So, how about this patch set? > > I do not think they (dma maintainers) will agree the API dma_can_skip_unmap(). > > If you think sq wastes too much memory using pre-mapped dma mode, how about > we only enable it when xsk is bond? > > Could you give me some advice? I think we have some discussion, one possible solution is: when pre mapping is enabled, virtio core won't store dma metadatas. Then it makes virtio-net align with other NIC. Thanks > > Thanks. > > > > > > > I asked you. But I didnot recv your answer. > > > > > > Thanks. > > > > > > > > > > > > > > > v2: > > > > > 1. change the dma item of virtio-net, every item have MAX_SKB_FRAGS + 2 > > > > > addr + len pairs. > > > > > 2. introduce virtnet_sq_free_stats for __free_old_xmit > > > > > > > > > > v1: > > > > > 1. rename transport_vq_config to vq_transport_config > > > > > 2. virtio-net set dma meta number to (ring-size + 1)(MAX_SKB_FRGAS +2) > > > > > 3. introduce virtqueue_dma_map_sg_attrs > > > > > 4. separate vring_create_virtqueue to an independent commit > > > > > > > > > > > > > > > > > > > > Xuan Zhuo (19): > > > > > virtio_ring: introduce vring_need_unmap_buffer > > > > > virtio_ring: packed: remove double check of the unmap ops > > > > > virtio_ring: packed: structure the indirect desc table > > > > > virtio_ring: split: remove double check of the unmap ops > > > > > virtio_ring: split: structure the indirect desc table > > > > > virtio_ring: no store dma info when unmap is not needed > > > > > virtio: find_vqs: pass struct instead of multi parameters > > > > > virtio: vring_create_virtqueue: pass struct instead of multi > > > > > parameters > > > > > virtio: vring_new_virtqueue(): pass struct instead of multi parameters > > > > > virtio_ring: simplify the parameters of the funcs related to > > > > > vring_create/new_virtqueue() > > > > > virtio: find_vqs: add new parameter premapped > > > > > virtio_ring: export premapped to driver by struct virtqueue > > > > > virtio_net: set premapped mode by find_vqs() > > > > > virtio_ring: remove api of setting vq premapped > > > > > virtio_ring: introduce dma map api for page > > > > > virtio_ring: introduce virtqueue_dma_map_sg_attrs > > > > > virtio_net: unify the code for recycling the xmit ptr > > > > > virtio_net: rename free_old_xmit_skbs to free_old_xmit > > > > > virtio_net: sq support premapped mode > > > > > > > > > > arch/um/drivers/virtio_uml.c | 31 +- > > > > > drivers/net/virtio_net.c | 283 ++++++--- > > > > > drivers/platform/mellanox/mlxbf-tmfifo.c | 24 +- > > > > > drivers/remoteproc/remoteproc_virtio.c | 31 +- > > > > > drivers/s390/virtio/virtio_ccw.c | 33 +- > > > > > drivers/virtio/virtio_mmio.c | 30 +- > > > > > drivers/virtio/virtio_pci_common.c | 59 +- > > > > > drivers/virtio/virtio_pci_common.h | 9 +- > > > > > drivers/virtio/virtio_pci_legacy.c | 16 +- > > > > > drivers/virtio/virtio_pci_modern.c | 38 +- > > > > > drivers/virtio/virtio_ring.c | 698 ++++++++++++----------- > > > > > drivers/virtio/virtio_vdpa.c | 45 +- > > > > > include/linux/virtio.h | 13 +- > > > > > include/linux/virtio_config.h | 48 +- > > > > > include/linux/virtio_ring.h | 82 +-- > > > > > tools/virtio/virtio_test.c | 4 +- > > > > > tools/virtio/vringh_test.c | 28 +- > > > > > 17 files changed, 847 insertions(+), 625 deletions(-) > > > > > > > > > > -- > > > > > 2.32.0.3.g01195cf9f > > > > > > >
On Thu, 7 Mar 2024 13:28:27 +0800, Jason Wang <jasowang@redhat.com> wrote: > On Wed, Mar 6, 2024 at 6:01 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > > > > On Thu, 29 Feb 2024 04:34:20 -0500, "Michael S. Tsirkin" <mst@redhat.com> wrote: > > > On Thu, Feb 29, 2024 at 05:02:37PM +0800, Xuan Zhuo wrote: > > > > On Thu, 29 Feb 2024 03:21:14 -0500, "Michael S. Tsirkin" <mst@redhat.com> wrote: > > > > > On Thu, Feb 29, 2024 at 03:20:25PM +0800, Xuan Zhuo wrote: > > > > > > As discussed: > > > > > > http://lore.kernel.org/all/CACGkMEvq0No8QGC46U4mGsMtuD44fD_cfLcPaVmJ3rHYqRZxYg@mail.gmail.com > > > > > > > > > > > > If the virtio is premapped mode, the driver should manage the dma info by self. > > > > > > So the virtio core should not store the dma info. > > > > > > So we can release the memory used to store the dma info. > > > > > > > > > > > > But if the desc_extra has not dma info, we face a new question, > > > > > > it is hard to get the dma info of the desc with indirect flag. > > > > > > For split mode, that is easy from desc, but for the packed mode, > > > > > > it is hard to get the dma info from the desc. And for hardening > > > > > > the dma unmap is saft, we should store the dma info of indirect > > > > > > descs. > > > > > > > > > > > > So I introduce the "structure the indirect desc table" to > > > > > > allocate space to store dma info with the desc table. > > > > > > > > > > > > On the other side, we mix the descs with indirect flag > > > > > > with other descs together to share the unmap api. That > > > > > > is complex. I found if we we distinguish the descs with > > > > > > VRING_DESC_F_INDIRECT before unmap, thing will be clearer. > > > > > > > > > > > > Because of the dma array is allocated in the find_vqs(), > > > > > > so I introduce a new parameter to find_vqs(). > > > > > > > > > > > > Note: > > > > > > this is on the top of > > > > > > [PATCH vhost v1] virtio: packed: fix unmap leak for indirect desc table > > > > > > http://lore.kernel.org/all/20240223071833.26095-1-xuanzhuo@linux.alibaba.com > > > > > > > > > > > > Please review. > > > > > > > > > > > > Thanks > > > > > > > > > > > > v3: > > > > > > 1. fix the conflict with the vp_modern_create_avq(). > > > > > > > > > > Okay but are you going to address huge memory waste all this is causing for > > > > > - people who never do zero copy > > > > > - systems where dma unmap is a nop > > > > > > > > > > ? > > > > > > > > > > You should address all comments when you post a new version, not just > > > > > what was expedient, or alternatively tag patch as RFC and explain > > > > > in commit log that you plan to do it later. > > > > > > > > > > > > Do you miss this one? > > > > http://lore.kernel.org/all/1708997579.5613105-1-xuanzhuo@linux.alibaba.com > > > > > > > > > I did. The answer is that no, you don't get to regress memory usage > > > for lots of people then fix it up. > > > So the patchset is big, I guess it will take a couple of cycles to > > > merge gradually. > > > > Hi @Michael > > > > So, how about this patch set? > > > > I do not think they (dma maintainers) will agree the API dma_can_skip_unmap(). > > > > If you think sq wastes too much memory using pre-mapped dma mode, how about > > we only enable it when xsk is bond? > > > > Could you give me some advice? > > I think we have some discussion, one possible solution is: > > when pre mapping is enabled, virtio core won't store dma metadatas. > > Then it makes virtio-net align with other NIC. YES. This patch set works as this. But the virtio-net must allocate too much memory to store dma and len. num = queue size * 19 Michael thinks that waste too much memory. http://lore.kernel.org/all/20240225032330-mutt-send-email-mst@kernel.org So we try this: http://lore.kernel.org/all/20240301071918.64631-1-xuanzhuo@linux.alibaba.com But I think that is difficult to be accepted by the DMA maintainers. So I have two advices: 1. virtio-net sq works without indirect. - that more like other NIC - the num of the memory to store the dma info is queue_size 2. The default mode of virtio-net sq is no-premapped - we just switch the mode when binding xsk Thanks. > > Thanks > > > > > Thanks. > > > > > > > > > > > I asked you. But I didnot recv your answer. > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > > > v2: > > > > > > 1. change the dma item of virtio-net, every item have MAX_SKB_FRAGS + 2 > > > > > > addr + len pairs. > > > > > > 2. introduce virtnet_sq_free_stats for __free_old_xmit > > > > > > > > > > > > v1: > > > > > > 1. rename transport_vq_config to vq_transport_config > > > > > > 2. virtio-net set dma meta number to (ring-size + 1)(MAX_SKB_FRGAS +2) > > > > > > 3. introduce virtqueue_dma_map_sg_attrs > > > > > > 4. separate vring_create_virtqueue to an independent commit > > > > > > > > > > > > > > > > > > > > > > > > Xuan Zhuo (19): > > > > > > virtio_ring: introduce vring_need_unmap_buffer > > > > > > virtio_ring: packed: remove double check of the unmap ops > > > > > > virtio_ring: packed: structure the indirect desc table > > > > > > virtio_ring: split: remove double check of the unmap ops > > > > > > virtio_ring: split: structure the indirect desc table > > > > > > virtio_ring: no store dma info when unmap is not needed > > > > > > virtio: find_vqs: pass struct instead of multi parameters > > > > > > virtio: vring_create_virtqueue: pass struct instead of multi > > > > > > parameters > > > > > > virtio: vring_new_virtqueue(): pass struct instead of multi parameters > > > > > > virtio_ring: simplify the parameters of the funcs related to > > > > > > vring_create/new_virtqueue() > > > > > > virtio: find_vqs: add new parameter premapped > > > > > > virtio_ring: export premapped to driver by struct virtqueue > > > > > > virtio_net: set premapped mode by find_vqs() > > > > > > virtio_ring: remove api of setting vq premapped > > > > > > virtio_ring: introduce dma map api for page > > > > > > virtio_ring: introduce virtqueue_dma_map_sg_attrs > > > > > > virtio_net: unify the code for recycling the xmit ptr > > > > > > virtio_net: rename free_old_xmit_skbs to free_old_xmit > > > > > > virtio_net: sq support premapped mode > > > > > > > > > > > > arch/um/drivers/virtio_uml.c | 31 +- > > > > > > drivers/net/virtio_net.c | 283 ++++++--- > > > > > > drivers/platform/mellanox/mlxbf-tmfifo.c | 24 +- > > > > > > drivers/remoteproc/remoteproc_virtio.c | 31 +- > > > > > > drivers/s390/virtio/virtio_ccw.c | 33 +- > > > > > > drivers/virtio/virtio_mmio.c | 30 +- > > > > > > drivers/virtio/virtio_pci_common.c | 59 +- > > > > > > drivers/virtio/virtio_pci_common.h | 9 +- > > > > > > drivers/virtio/virtio_pci_legacy.c | 16 +- > > > > > > drivers/virtio/virtio_pci_modern.c | 38 +- > > > > > > drivers/virtio/virtio_ring.c | 698 ++++++++++++----------- > > > > > > drivers/virtio/virtio_vdpa.c | 45 +- > > > > > > include/linux/virtio.h | 13 +- > > > > > > include/linux/virtio_config.h | 48 +- > > > > > > include/linux/virtio_ring.h | 82 +-- > > > > > > tools/virtio/virtio_test.c | 4 +- > > > > > > tools/virtio/vringh_test.c | 28 +- > > > > > > 17 files changed, 847 insertions(+), 625 deletions(-) > > > > > > > > > > > > -- > > > > > > 2.32.0.3.g01195cf9f > > > > > > > > > > >
On Thu, Mar 7, 2024 at 4:15 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > > On Thu, 7 Mar 2024 13:28:27 +0800, Jason Wang <jasowang@redhat.com> wrote: > > On Wed, Mar 6, 2024 at 6:01 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > > > > > > On Thu, 29 Feb 2024 04:34:20 -0500, "Michael S. Tsirkin" <mst@redhat.com> wrote: > > > > On Thu, Feb 29, 2024 at 05:02:37PM +0800, Xuan Zhuo wrote: > > > > > On Thu, 29 Feb 2024 03:21:14 -0500, "Michael S. Tsirkin" <mst@redhat.com> wrote: > > > > > > On Thu, Feb 29, 2024 at 03:20:25PM +0800, Xuan Zhuo wrote: > > > > > > > As discussed: > > > > > > > http://lore.kernel.org/all/CACGkMEvq0No8QGC46U4mGsMtuD44fD_cfLcPaVmJ3rHYqRZxYg@mail.gmail.com > > > > > > > > > > > > > > If the virtio is premapped mode, the driver should manage the dma info by self. > > > > > > > So the virtio core should not store the dma info. > > > > > > > So we can release the memory used to store the dma info. > > > > > > > > > > > > > > But if the desc_extra has not dma info, we face a new question, > > > > > > > it is hard to get the dma info of the desc with indirect flag. > > > > > > > For split mode, that is easy from desc, but for the packed mode, > > > > > > > it is hard to get the dma info from the desc. And for hardening > > > > > > > the dma unmap is saft, we should store the dma info of indirect > > > > > > > descs. > > > > > > > > > > > > > > So I introduce the "structure the indirect desc table" to > > > > > > > allocate space to store dma info with the desc table. > > > > > > > > > > > > > > On the other side, we mix the descs with indirect flag > > > > > > > with other descs together to share the unmap api. That > > > > > > > is complex. I found if we we distinguish the descs with > > > > > > > VRING_DESC_F_INDIRECT before unmap, thing will be clearer. > > > > > > > > > > > > > > Because of the dma array is allocated in the find_vqs(), > > > > > > > so I introduce a new parameter to find_vqs(). > > > > > > > > > > > > > > Note: > > > > > > > this is on the top of > > > > > > > [PATCH vhost v1] virtio: packed: fix unmap leak for indirect desc table > > > > > > > http://lore.kernel.org/all/20240223071833.26095-1-xuanzhuo@linux.alibaba.com > > > > > > > > > > > > > > Please review. > > > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > v3: > > > > > > > 1. fix the conflict with the vp_modern_create_avq(). > > > > > > > > > > > > Okay but are you going to address huge memory waste all this is causing for > > > > > > - people who never do zero copy > > > > > > - systems where dma unmap is a nop > > > > > > > > > > > > ? > > > > > > > > > > > > You should address all comments when you post a new version, not just > > > > > > what was expedient, or alternatively tag patch as RFC and explain > > > > > > in commit log that you plan to do it later. > > > > > > > > > > > > > > > Do you miss this one? > > > > > http://lore.kernel.org/all/1708997579.5613105-1-xuanzhuo@linux.alibaba.com > > > > > > > > > > > > I did. The answer is that no, you don't get to regress memory usage > > > > for lots of people then fix it up. > > > > So the patchset is big, I guess it will take a couple of cycles to > > > > merge gradually. > > > > > > Hi @Michael > > > > > > So, how about this patch set? > > > > > > I do not think they (dma maintainers) will agree the API dma_can_skip_unmap(). > > > > > > If you think sq wastes too much memory using pre-mapped dma mode, how about > > > we only enable it when xsk is bond? > > > > > > Could you give me some advice? > > > > I think we have some discussion, one possible solution is: > > > > when pre mapping is enabled, virtio core won't store dma metadatas. > > > > Then it makes virtio-net align with other NIC. > > > YES. > > This patch set works as this. > > But the virtio-net must allocate too much memory to store dma and len. > > num = queue size * 19 > > Michael thinks that waste too much memory. > http://lore.kernel.org/all/20240225032330-mutt-send-email-mst@kernel.org > > So we try this: > http://lore.kernel.org/all/20240301071918.64631-1-xuanzhuo@linux.alibaba.com > > But I think that is difficult to be accepted by the DMA maintainers. > > So I have two advices: > > 1. virtio-net sq works without indirect. > - that more like other NIC > - the num of the memory to store the dma info is queue_size This requires benchmarks. > > 2. The default mode of virtio-net sq is no-premapped > - we just switch the mode when binding xsk This could be one step. We can hear from Michael. Thanks > > Thanks. > > > > > > Thanks > > > > > > > > Thanks. > > > > > > > > > > > > > > > I asked you. But I didnot recv your answer. > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > > > > > > > > v2: > > > > > > > 1. change the dma item of virtio-net, every item have MAX_SKB_FRAGS + 2 > > > > > > > addr + len pairs. > > > > > > > 2. introduce virtnet_sq_free_stats for __free_old_xmit > > > > > > > > > > > > > > v1: > > > > > > > 1. rename transport_vq_config to vq_transport_config > > > > > > > 2. virtio-net set dma meta number to (ring-size + 1)(MAX_SKB_FRGAS +2) > > > > > > > 3. introduce virtqueue_dma_map_sg_attrs > > > > > > > 4. separate vring_create_virtqueue to an independent commit > > > > > > > > > > > > > > > > > > > > > > > > > > > > Xuan Zhuo (19): > > > > > > > virtio_ring: introduce vring_need_unmap_buffer > > > > > > > virtio_ring: packed: remove double check of the unmap ops > > > > > > > virtio_ring: packed: structure the indirect desc table > > > > > > > virtio_ring: split: remove double check of the unmap ops > > > > > > > virtio_ring: split: structure the indirect desc table > > > > > > > virtio_ring: no store dma info when unmap is not needed > > > > > > > virtio: find_vqs: pass struct instead of multi parameters > > > > > > > virtio: vring_create_virtqueue: pass struct instead of multi > > > > > > > parameters > > > > > > > virtio: vring_new_virtqueue(): pass struct instead of multi parameters > > > > > > > virtio_ring: simplify the parameters of the funcs related to > > > > > > > vring_create/new_virtqueue() > > > > > > > virtio: find_vqs: add new parameter premapped > > > > > > > virtio_ring: export premapped to driver by struct virtqueue > > > > > > > virtio_net: set premapped mode by find_vqs() > > > > > > > virtio_ring: remove api of setting vq premapped > > > > > > > virtio_ring: introduce dma map api for page > > > > > > > virtio_ring: introduce virtqueue_dma_map_sg_attrs > > > > > > > virtio_net: unify the code for recycling the xmit ptr > > > > > > > virtio_net: rename free_old_xmit_skbs to free_old_xmit > > > > > > > virtio_net: sq support premapped mode > > > > > > > > > > > > > > arch/um/drivers/virtio_uml.c | 31 +- > > > > > > > drivers/net/virtio_net.c | 283 ++++++--- > > > > > > > drivers/platform/mellanox/mlxbf-tmfifo.c | 24 +- > > > > > > > drivers/remoteproc/remoteproc_virtio.c | 31 +- > > > > > > > drivers/s390/virtio/virtio_ccw.c | 33 +- > > > > > > > drivers/virtio/virtio_mmio.c | 30 +- > > > > > > > drivers/virtio/virtio_pci_common.c | 59 +- > > > > > > > drivers/virtio/virtio_pci_common.h | 9 +- > > > > > > > drivers/virtio/virtio_pci_legacy.c | 16 +- > > > > > > > drivers/virtio/virtio_pci_modern.c | 38 +- > > > > > > > drivers/virtio/virtio_ring.c | 698 ++++++++++++----------- > > > > > > > drivers/virtio/virtio_vdpa.c | 45 +- > > > > > > > include/linux/virtio.h | 13 +- > > > > > > > include/linux/virtio_config.h | 48 +- > > > > > > > include/linux/virtio_ring.h | 82 +-- > > > > > > > tools/virtio/virtio_test.c | 4 +- > > > > > > > tools/virtio/vringh_test.c | 28 +- > > > > > > > 17 files changed, 847 insertions(+), 625 deletions(-) > > > > > > > > > > > > > > -- > > > > > > > 2.32.0.3.g01195cf9f > > > > > > > > > > > > > > > >
As discussed: http://lore.kernel.org/all/CACGkMEvq0No8QGC46U4mGsMtuD44fD_cfLcPaVmJ3rHYqRZxYg@mail.gmail.com If the virtio is premapped mode, the driver should manage the dma info by self. So the virtio core should not store the dma info. So we can release the memory used to store the dma info. But if the desc_extra has not dma info, we face a new question, it is hard to get the dma info of the desc with indirect flag. For split mode, that is easy from desc, but for the packed mode, it is hard to get the dma info from the desc. And for hardening the dma unmap is saft, we should store the dma info of indirect descs. So I introduce the "structure the indirect desc table" to allocate space to store dma info with the desc table. On the other side, we mix the descs with indirect flag with other descs together to share the unmap api. That is complex. I found if we we distinguish the descs with VRING_DESC_F_INDIRECT before unmap, thing will be clearer. Because of the dma array is allocated in the find_vqs(), so I introduce a new parameter to find_vqs(). Note: this is on the top of [PATCH vhost v1] virtio: packed: fix unmap leak for indirect desc table http://lore.kernel.org/all/20240223071833.26095-1-xuanzhuo@linux.alibaba.com Please review. Thanks v3: 1. fix the conflict with the vp_modern_create_avq(). v2: 1. change the dma item of virtio-net, every item have MAX_SKB_FRAGS + 2 addr + len pairs. 2. introduce virtnet_sq_free_stats for __free_old_xmit v1: 1. rename transport_vq_config to vq_transport_config 2. virtio-net set dma meta number to (ring-size + 1)(MAX_SKB_FRGAS +2) 3. introduce virtqueue_dma_map_sg_attrs 4. separate vring_create_virtqueue to an independent commit Xuan Zhuo (19): virtio_ring: introduce vring_need_unmap_buffer virtio_ring: packed: remove double check of the unmap ops virtio_ring: packed: structure the indirect desc table virtio_ring: split: remove double check of the unmap ops virtio_ring: split: structure the indirect desc table virtio_ring: no store dma info when unmap is not needed virtio: find_vqs: pass struct instead of multi parameters virtio: vring_create_virtqueue: pass struct instead of multi parameters virtio: vring_new_virtqueue(): pass struct instead of multi parameters virtio_ring: simplify the parameters of the funcs related to vring_create/new_virtqueue() virtio: find_vqs: add new parameter premapped virtio_ring: export premapped to driver by struct virtqueue virtio_net: set premapped mode by find_vqs() virtio_ring: remove api of setting vq premapped virtio_ring: introduce dma map api for page virtio_ring: introduce virtqueue_dma_map_sg_attrs virtio_net: unify the code for recycling the xmit ptr virtio_net: rename free_old_xmit_skbs to free_old_xmit virtio_net: sq support premapped mode arch/um/drivers/virtio_uml.c | 31 +- drivers/net/virtio_net.c | 283 ++++++--- drivers/platform/mellanox/mlxbf-tmfifo.c | 24 +- drivers/remoteproc/remoteproc_virtio.c | 31 +- drivers/s390/virtio/virtio_ccw.c | 33 +- drivers/virtio/virtio_mmio.c | 30 +- drivers/virtio/virtio_pci_common.c | 59 +- drivers/virtio/virtio_pci_common.h | 9 +- drivers/virtio/virtio_pci_legacy.c | 16 +- drivers/virtio/virtio_pci_modern.c | 38 +- drivers/virtio/virtio_ring.c | 698 ++++++++++++----------- drivers/virtio/virtio_vdpa.c | 45 +- include/linux/virtio.h | 13 +- include/linux/virtio_config.h | 48 +- include/linux/virtio_ring.h | 82 +-- tools/virtio/virtio_test.c | 4 +- tools/virtio/vringh_test.c | 28 +- 17 files changed, 847 insertions(+), 625 deletions(-) -- 2.32.0.3.g01195cf9f