mbox series

[vhost,v3,00/19] virtio: drivers maintain dma info for premapped vq

Message ID 20240229072044.77388-1-xuanzhuo@linux.alibaba.com (mailing list archive)
Headers show
Series virtio: drivers maintain dma info for premapped vq | expand

Message

Xuan Zhuo Feb. 29, 2024, 7:20 a.m. UTC
As discussed:
http://lore.kernel.org/all/CACGkMEvq0No8QGC46U4mGsMtuD44fD_cfLcPaVmJ3rHYqRZxYg@mail.gmail.com

If the virtio is premapped mode, the driver should manage the dma info by self.
So the virtio core should not store the dma info.
So we can release the memory used to store the dma info.

But if the desc_extra has not dma info, we face a new question,
it is hard to get the dma info of the desc with indirect flag.
For split mode, that is easy from desc, but for the packed mode,
it is hard to get the dma info from the desc. And for hardening
the dma unmap is saft, we should store the dma info of indirect
descs.

So I introduce the "structure the indirect desc table" to
allocate space to store dma info with the desc table.

On the other side, we mix the descs with indirect flag
with other descs together to share the unmap api. That
is complex. I found if we we distinguish the descs with
VRING_DESC_F_INDIRECT before unmap, thing will be clearer.

Because of the dma array is allocated in the find_vqs(),
so I introduce a new parameter to find_vqs().

Note:
    this is on the top of
        [PATCH vhost v1] virtio: packed: fix unmap leak for indirect desc table
        http://lore.kernel.org/all/20240223071833.26095-1-xuanzhuo@linux.alibaba.com

Please review.

Thanks

v3:
    1. fix the conflict with the vp_modern_create_avq().

v2:
    1. change the dma item of virtio-net, every item have MAX_SKB_FRAGS + 2
        addr + len pairs.
    2. introduce virtnet_sq_free_stats for __free_old_xmit

v1:
    1. rename transport_vq_config to vq_transport_config
    2. virtio-net set dma meta number to (ring-size + 1)(MAX_SKB_FRGAS +2)
    3. introduce virtqueue_dma_map_sg_attrs
    4. separate vring_create_virtqueue to an independent commit



Xuan Zhuo (19):
  virtio_ring: introduce vring_need_unmap_buffer
  virtio_ring: packed: remove double check of the unmap ops
  virtio_ring: packed: structure the indirect desc table
  virtio_ring: split: remove double check of the unmap ops
  virtio_ring: split: structure the indirect desc table
  virtio_ring: no store dma info when unmap is not needed
  virtio: find_vqs: pass struct instead of multi parameters
  virtio: vring_create_virtqueue: pass struct instead of multi
    parameters
  virtio: vring_new_virtqueue(): pass struct instead of multi parameters
  virtio_ring: simplify the parameters of the funcs related to
    vring_create/new_virtqueue()
  virtio: find_vqs: add new parameter premapped
  virtio_ring: export premapped to driver by struct virtqueue
  virtio_net: set premapped mode by find_vqs()
  virtio_ring: remove api of setting vq premapped
  virtio_ring: introduce dma map api for page
  virtio_ring: introduce virtqueue_dma_map_sg_attrs
  virtio_net: unify the code for recycling the xmit ptr
  virtio_net: rename free_old_xmit_skbs to free_old_xmit
  virtio_net: sq support premapped mode

 arch/um/drivers/virtio_uml.c             |  31 +-
 drivers/net/virtio_net.c                 | 283 ++++++---
 drivers/platform/mellanox/mlxbf-tmfifo.c |  24 +-
 drivers/remoteproc/remoteproc_virtio.c   |  31 +-
 drivers/s390/virtio/virtio_ccw.c         |  33 +-
 drivers/virtio/virtio_mmio.c             |  30 +-
 drivers/virtio/virtio_pci_common.c       |  59 +-
 drivers/virtio/virtio_pci_common.h       |   9 +-
 drivers/virtio/virtio_pci_legacy.c       |  16 +-
 drivers/virtio/virtio_pci_modern.c       |  38 +-
 drivers/virtio/virtio_ring.c             | 698 ++++++++++++-----------
 drivers/virtio/virtio_vdpa.c             |  45 +-
 include/linux/virtio.h                   |  13 +-
 include/linux/virtio_config.h            |  48 +-
 include/linux/virtio_ring.h              |  82 +--
 tools/virtio/virtio_test.c               |   4 +-
 tools/virtio/vringh_test.c               |  28 +-
 17 files changed, 847 insertions(+), 625 deletions(-)

--
2.32.0.3.g01195cf9f

Comments

Michael S. Tsirkin Feb. 29, 2024, 8:21 a.m. UTC | #1
On Thu, Feb 29, 2024 at 03:20:25PM +0800, Xuan Zhuo wrote:
> As discussed:
> http://lore.kernel.org/all/CACGkMEvq0No8QGC46U4mGsMtuD44fD_cfLcPaVmJ3rHYqRZxYg@mail.gmail.com
> 
> If the virtio is premapped mode, the driver should manage the dma info by self.
> So the virtio core should not store the dma info.
> So we can release the memory used to store the dma info.
> 
> But if the desc_extra has not dma info, we face a new question,
> it is hard to get the dma info of the desc with indirect flag.
> For split mode, that is easy from desc, but for the packed mode,
> it is hard to get the dma info from the desc. And for hardening
> the dma unmap is saft, we should store the dma info of indirect
> descs.
> 
> So I introduce the "structure the indirect desc table" to
> allocate space to store dma info with the desc table.
> 
> On the other side, we mix the descs with indirect flag
> with other descs together to share the unmap api. That
> is complex. I found if we we distinguish the descs with
> VRING_DESC_F_INDIRECT before unmap, thing will be clearer.
> 
> Because of the dma array is allocated in the find_vqs(),
> so I introduce a new parameter to find_vqs().
> 
> Note:
>     this is on the top of
>         [PATCH vhost v1] virtio: packed: fix unmap leak for indirect desc table
>         http://lore.kernel.org/all/20240223071833.26095-1-xuanzhuo@linux.alibaba.com
> 
> Please review.
> 
> Thanks
> 
> v3:
>     1. fix the conflict with the vp_modern_create_avq().

Okay but are you going to address huge memory waste all this is causing for
- people who never do zero copy
- systems where dma unmap is a nop

?

You should address all comments when you post a new version, not just
what was expedient, or alternatively tag patch as RFC and explain
in commit log that you plan to do it later.

> v2:
>     1. change the dma item of virtio-net, every item have MAX_SKB_FRAGS + 2
>         addr + len pairs.
>     2. introduce virtnet_sq_free_stats for __free_old_xmit
> 
> v1:
>     1. rename transport_vq_config to vq_transport_config
>     2. virtio-net set dma meta number to (ring-size + 1)(MAX_SKB_FRGAS +2)
>     3. introduce virtqueue_dma_map_sg_attrs
>     4. separate vring_create_virtqueue to an independent commit
> 
> 
> 
> Xuan Zhuo (19):
>   virtio_ring: introduce vring_need_unmap_buffer
>   virtio_ring: packed: remove double check of the unmap ops
>   virtio_ring: packed: structure the indirect desc table
>   virtio_ring: split: remove double check of the unmap ops
>   virtio_ring: split: structure the indirect desc table
>   virtio_ring: no store dma info when unmap is not needed
>   virtio: find_vqs: pass struct instead of multi parameters
>   virtio: vring_create_virtqueue: pass struct instead of multi
>     parameters
>   virtio: vring_new_virtqueue(): pass struct instead of multi parameters
>   virtio_ring: simplify the parameters of the funcs related to
>     vring_create/new_virtqueue()
>   virtio: find_vqs: add new parameter premapped
>   virtio_ring: export premapped to driver by struct virtqueue
>   virtio_net: set premapped mode by find_vqs()
>   virtio_ring: remove api of setting vq premapped
>   virtio_ring: introduce dma map api for page
>   virtio_ring: introduce virtqueue_dma_map_sg_attrs
>   virtio_net: unify the code for recycling the xmit ptr
>   virtio_net: rename free_old_xmit_skbs to free_old_xmit
>   virtio_net: sq support premapped mode
> 
>  arch/um/drivers/virtio_uml.c             |  31 +-
>  drivers/net/virtio_net.c                 | 283 ++++++---
>  drivers/platform/mellanox/mlxbf-tmfifo.c |  24 +-
>  drivers/remoteproc/remoteproc_virtio.c   |  31 +-
>  drivers/s390/virtio/virtio_ccw.c         |  33 +-
>  drivers/virtio/virtio_mmio.c             |  30 +-
>  drivers/virtio/virtio_pci_common.c       |  59 +-
>  drivers/virtio/virtio_pci_common.h       |   9 +-
>  drivers/virtio/virtio_pci_legacy.c       |  16 +-
>  drivers/virtio/virtio_pci_modern.c       |  38 +-
>  drivers/virtio/virtio_ring.c             | 698 ++++++++++++-----------
>  drivers/virtio/virtio_vdpa.c             |  45 +-
>  include/linux/virtio.h                   |  13 +-
>  include/linux/virtio_config.h            |  48 +-
>  include/linux/virtio_ring.h              |  82 +--
>  tools/virtio/virtio_test.c               |   4 +-
>  tools/virtio/vringh_test.c               |  28 +-
>  17 files changed, 847 insertions(+), 625 deletions(-)
> 
> --
> 2.32.0.3.g01195cf9f
Xuan Zhuo Feb. 29, 2024, 9:02 a.m. UTC | #2
On Thu, 29 Feb 2024 03:21:14 -0500, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> On Thu, Feb 29, 2024 at 03:20:25PM +0800, Xuan Zhuo wrote:
> > As discussed:
> > http://lore.kernel.org/all/CACGkMEvq0No8QGC46U4mGsMtuD44fD_cfLcPaVmJ3rHYqRZxYg@mail.gmail.com
> >
> > If the virtio is premapped mode, the driver should manage the dma info by self.
> > So the virtio core should not store the dma info.
> > So we can release the memory used to store the dma info.
> >
> > But if the desc_extra has not dma info, we face a new question,
> > it is hard to get the dma info of the desc with indirect flag.
> > For split mode, that is easy from desc, but for the packed mode,
> > it is hard to get the dma info from the desc. And for hardening
> > the dma unmap is saft, we should store the dma info of indirect
> > descs.
> >
> > So I introduce the "structure the indirect desc table" to
> > allocate space to store dma info with the desc table.
> >
> > On the other side, we mix the descs with indirect flag
> > with other descs together to share the unmap api. That
> > is complex. I found if we we distinguish the descs with
> > VRING_DESC_F_INDIRECT before unmap, thing will be clearer.
> >
> > Because of the dma array is allocated in the find_vqs(),
> > so I introduce a new parameter to find_vqs().
> >
> > Note:
> >     this is on the top of
> >         [PATCH vhost v1] virtio: packed: fix unmap leak for indirect desc table
> >         http://lore.kernel.org/all/20240223071833.26095-1-xuanzhuo@linux.alibaba.com
> >
> > Please review.
> >
> > Thanks
> >
> > v3:
> >     1. fix the conflict with the vp_modern_create_avq().
>
> Okay but are you going to address huge memory waste all this is causing for
> - people who never do zero copy
> - systems where dma unmap is a nop
>
> ?
>
> You should address all comments when you post a new version, not just
> what was expedient, or alternatively tag patch as RFC and explain
> in commit log that you plan to do it later.


Do you miss this one?
http://lore.kernel.org/all/1708997579.5613105-1-xuanzhuo@linux.alibaba.com

I asked you. But I didnot recv your answer.

Thanks.


>
> > v2:
> >     1. change the dma item of virtio-net, every item have MAX_SKB_FRAGS + 2
> >         addr + len pairs.
> >     2. introduce virtnet_sq_free_stats for __free_old_xmit
> >
> > v1:
> >     1. rename transport_vq_config to vq_transport_config
> >     2. virtio-net set dma meta number to (ring-size + 1)(MAX_SKB_FRGAS +2)
> >     3. introduce virtqueue_dma_map_sg_attrs
> >     4. separate vring_create_virtqueue to an independent commit
> >
> >
> >
> > Xuan Zhuo (19):
> >   virtio_ring: introduce vring_need_unmap_buffer
> >   virtio_ring: packed: remove double check of the unmap ops
> >   virtio_ring: packed: structure the indirect desc table
> >   virtio_ring: split: remove double check of the unmap ops
> >   virtio_ring: split: structure the indirect desc table
> >   virtio_ring: no store dma info when unmap is not needed
> >   virtio: find_vqs: pass struct instead of multi parameters
> >   virtio: vring_create_virtqueue: pass struct instead of multi
> >     parameters
> >   virtio: vring_new_virtqueue(): pass struct instead of multi parameters
> >   virtio_ring: simplify the parameters of the funcs related to
> >     vring_create/new_virtqueue()
> >   virtio: find_vqs: add new parameter premapped
> >   virtio_ring: export premapped to driver by struct virtqueue
> >   virtio_net: set premapped mode by find_vqs()
> >   virtio_ring: remove api of setting vq premapped
> >   virtio_ring: introduce dma map api for page
> >   virtio_ring: introduce virtqueue_dma_map_sg_attrs
> >   virtio_net: unify the code for recycling the xmit ptr
> >   virtio_net: rename free_old_xmit_skbs to free_old_xmit
> >   virtio_net: sq support premapped mode
> >
> >  arch/um/drivers/virtio_uml.c             |  31 +-
> >  drivers/net/virtio_net.c                 | 283 ++++++---
> >  drivers/platform/mellanox/mlxbf-tmfifo.c |  24 +-
> >  drivers/remoteproc/remoteproc_virtio.c   |  31 +-
> >  drivers/s390/virtio/virtio_ccw.c         |  33 +-
> >  drivers/virtio/virtio_mmio.c             |  30 +-
> >  drivers/virtio/virtio_pci_common.c       |  59 +-
> >  drivers/virtio/virtio_pci_common.h       |   9 +-
> >  drivers/virtio/virtio_pci_legacy.c       |  16 +-
> >  drivers/virtio/virtio_pci_modern.c       |  38 +-
> >  drivers/virtio/virtio_ring.c             | 698 ++++++++++++-----------
> >  drivers/virtio/virtio_vdpa.c             |  45 +-
> >  include/linux/virtio.h                   |  13 +-
> >  include/linux/virtio_config.h            |  48 +-
> >  include/linux/virtio_ring.h              |  82 +--
> >  tools/virtio/virtio_test.c               |   4 +-
> >  tools/virtio/vringh_test.c               |  28 +-
> >  17 files changed, 847 insertions(+), 625 deletions(-)
> >
> > --
> > 2.32.0.3.g01195cf9f
>
Michael S. Tsirkin Feb. 29, 2024, 9:34 a.m. UTC | #3
On Thu, Feb 29, 2024 at 05:02:37PM +0800, Xuan Zhuo wrote:
> On Thu, 29 Feb 2024 03:21:14 -0500, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > On Thu, Feb 29, 2024 at 03:20:25PM +0800, Xuan Zhuo wrote:
> > > As discussed:
> > > http://lore.kernel.org/all/CACGkMEvq0No8QGC46U4mGsMtuD44fD_cfLcPaVmJ3rHYqRZxYg@mail.gmail.com
> > >
> > > If the virtio is premapped mode, the driver should manage the dma info by self.
> > > So the virtio core should not store the dma info.
> > > So we can release the memory used to store the dma info.
> > >
> > > But if the desc_extra has not dma info, we face a new question,
> > > it is hard to get the dma info of the desc with indirect flag.
> > > For split mode, that is easy from desc, but for the packed mode,
> > > it is hard to get the dma info from the desc. And for hardening
> > > the dma unmap is saft, we should store the dma info of indirect
> > > descs.
> > >
> > > So I introduce the "structure the indirect desc table" to
> > > allocate space to store dma info with the desc table.
> > >
> > > On the other side, we mix the descs with indirect flag
> > > with other descs together to share the unmap api. That
> > > is complex. I found if we we distinguish the descs with
> > > VRING_DESC_F_INDIRECT before unmap, thing will be clearer.
> > >
> > > Because of the dma array is allocated in the find_vqs(),
> > > so I introduce a new parameter to find_vqs().
> > >
> > > Note:
> > >     this is on the top of
> > >         [PATCH vhost v1] virtio: packed: fix unmap leak for indirect desc table
> > >         http://lore.kernel.org/all/20240223071833.26095-1-xuanzhuo@linux.alibaba.com
> > >
> > > Please review.
> > >
> > > Thanks
> > >
> > > v3:
> > >     1. fix the conflict with the vp_modern_create_avq().
> >
> > Okay but are you going to address huge memory waste all this is causing for
> > - people who never do zero copy
> > - systems where dma unmap is a nop
> >
> > ?
> >
> > You should address all comments when you post a new version, not just
> > what was expedient, or alternatively tag patch as RFC and explain
> > in commit log that you plan to do it later.
> 
> 
> Do you miss this one?
> http://lore.kernel.org/all/1708997579.5613105-1-xuanzhuo@linux.alibaba.com


I did. The answer is that no, you don't get to regress memory usage
for lots of people then fix it up.
So the patchset is big, I guess it will take a couple of cycles to
merge gradually.

> I asked you. But I didnot recv your answer.
> 
> Thanks.
> 
> 
> >
> > > v2:
> > >     1. change the dma item of virtio-net, every item have MAX_SKB_FRAGS + 2
> > >         addr + len pairs.
> > >     2. introduce virtnet_sq_free_stats for __free_old_xmit
> > >
> > > v1:
> > >     1. rename transport_vq_config to vq_transport_config
> > >     2. virtio-net set dma meta number to (ring-size + 1)(MAX_SKB_FRGAS +2)
> > >     3. introduce virtqueue_dma_map_sg_attrs
> > >     4. separate vring_create_virtqueue to an independent commit
> > >
> > >
> > >
> > > Xuan Zhuo (19):
> > >   virtio_ring: introduce vring_need_unmap_buffer
> > >   virtio_ring: packed: remove double check of the unmap ops
> > >   virtio_ring: packed: structure the indirect desc table
> > >   virtio_ring: split: remove double check of the unmap ops
> > >   virtio_ring: split: structure the indirect desc table
> > >   virtio_ring: no store dma info when unmap is not needed
> > >   virtio: find_vqs: pass struct instead of multi parameters
> > >   virtio: vring_create_virtqueue: pass struct instead of multi
> > >     parameters
> > >   virtio: vring_new_virtqueue(): pass struct instead of multi parameters
> > >   virtio_ring: simplify the parameters of the funcs related to
> > >     vring_create/new_virtqueue()
> > >   virtio: find_vqs: add new parameter premapped
> > >   virtio_ring: export premapped to driver by struct virtqueue
> > >   virtio_net: set premapped mode by find_vqs()
> > >   virtio_ring: remove api of setting vq premapped
> > >   virtio_ring: introduce dma map api for page
> > >   virtio_ring: introduce virtqueue_dma_map_sg_attrs
> > >   virtio_net: unify the code for recycling the xmit ptr
> > >   virtio_net: rename free_old_xmit_skbs to free_old_xmit
> > >   virtio_net: sq support premapped mode
> > >
> > >  arch/um/drivers/virtio_uml.c             |  31 +-
> > >  drivers/net/virtio_net.c                 | 283 ++++++---
> > >  drivers/platform/mellanox/mlxbf-tmfifo.c |  24 +-
> > >  drivers/remoteproc/remoteproc_virtio.c   |  31 +-
> > >  drivers/s390/virtio/virtio_ccw.c         |  33 +-
> > >  drivers/virtio/virtio_mmio.c             |  30 +-
> > >  drivers/virtio/virtio_pci_common.c       |  59 +-
> > >  drivers/virtio/virtio_pci_common.h       |   9 +-
> > >  drivers/virtio/virtio_pci_legacy.c       |  16 +-
> > >  drivers/virtio/virtio_pci_modern.c       |  38 +-
> > >  drivers/virtio/virtio_ring.c             | 698 ++++++++++++-----------
> > >  drivers/virtio/virtio_vdpa.c             |  45 +-
> > >  include/linux/virtio.h                   |  13 +-
> > >  include/linux/virtio_config.h            |  48 +-
> > >  include/linux/virtio_ring.h              |  82 +--
> > >  tools/virtio/virtio_test.c               |   4 +-
> > >  tools/virtio/vringh_test.c               |  28 +-
> > >  17 files changed, 847 insertions(+), 625 deletions(-)
> > >
> > > --
> > > 2.32.0.3.g01195cf9f
> >
Xuan Zhuo Feb. 29, 2024, 9:36 a.m. UTC | #4
On Thu, 29 Feb 2024 04:34:20 -0500, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> On Thu, Feb 29, 2024 at 05:02:37PM +0800, Xuan Zhuo wrote:
> > On Thu, 29 Feb 2024 03:21:14 -0500, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > On Thu, Feb 29, 2024 at 03:20:25PM +0800, Xuan Zhuo wrote:
> > > > As discussed:
> > > > http://lore.kernel.org/all/CACGkMEvq0No8QGC46U4mGsMtuD44fD_cfLcPaVmJ3rHYqRZxYg@mail.gmail.com
> > > >
> > > > If the virtio is premapped mode, the driver should manage the dma info by self.
> > > > So the virtio core should not store the dma info.
> > > > So we can release the memory used to store the dma info.
> > > >
> > > > But if the desc_extra has not dma info, we face a new question,
> > > > it is hard to get the dma info of the desc with indirect flag.
> > > > For split mode, that is easy from desc, but for the packed mode,
> > > > it is hard to get the dma info from the desc. And for hardening
> > > > the dma unmap is saft, we should store the dma info of indirect
> > > > descs.
> > > >
> > > > So I introduce the "structure the indirect desc table" to
> > > > allocate space to store dma info with the desc table.
> > > >
> > > > On the other side, we mix the descs with indirect flag
> > > > with other descs together to share the unmap api. That
> > > > is complex. I found if we we distinguish the descs with
> > > > VRING_DESC_F_INDIRECT before unmap, thing will be clearer.
> > > >
> > > > Because of the dma array is allocated in the find_vqs(),
> > > > so I introduce a new parameter to find_vqs().
> > > >
> > > > Note:
> > > >     this is on the top of
> > > >         [PATCH vhost v1] virtio: packed: fix unmap leak for indirect desc table
> > > >         http://lore.kernel.org/all/20240223071833.26095-1-xuanzhuo@linux.alibaba.com
> > > >
> > > > Please review.
> > > >
> > > > Thanks
> > > >
> > > > v3:
> > > >     1. fix the conflict with the vp_modern_create_avq().
> > >
> > > Okay but are you going to address huge memory waste all this is causing for
> > > - people who never do zero copy
> > > - systems where dma unmap is a nop
> > >
> > > ?
> > >
> > > You should address all comments when you post a new version, not just
> > > what was expedient, or alternatively tag patch as RFC and explain
> > > in commit log that you plan to do it later.
> >
> >
> > Do you miss this one?
> > http://lore.kernel.org/all/1708997579.5613105-1-xuanzhuo@linux.alibaba.com
>
>
> I did. The answer is that no, you don't get to regress memory usage
> for lots of people then fix it up.
> So the patchset is big, I guess it will take a couple of cycles to
> merge gradually.

I see.

But I hope we can review the other part of this patch set.  @Jason @Michael

I don't want us to give up on this plan for some other reason after I've done so
much work.

Thanks.


>
> > I asked you. But I didnot recv your answer.
> >
> > Thanks.
> >
> >
> > >
> > > > v2:
> > > >     1. change the dma item of virtio-net, every item have MAX_SKB_FRAGS + 2
> > > >         addr + len pairs.
> > > >     2. introduce virtnet_sq_free_stats for __free_old_xmit
> > > >
> > > > v1:
> > > >     1. rename transport_vq_config to vq_transport_config
> > > >     2. virtio-net set dma meta number to (ring-size + 1)(MAX_SKB_FRGAS +2)
> > > >     3. introduce virtqueue_dma_map_sg_attrs
> > > >     4. separate vring_create_virtqueue to an independent commit
> > > >
> > > >
> > > >
> > > > Xuan Zhuo (19):
> > > >   virtio_ring: introduce vring_need_unmap_buffer
> > > >   virtio_ring: packed: remove double check of the unmap ops
> > > >   virtio_ring: packed: structure the indirect desc table
> > > >   virtio_ring: split: remove double check of the unmap ops
> > > >   virtio_ring: split: structure the indirect desc table
> > > >   virtio_ring: no store dma info when unmap is not needed
> > > >   virtio: find_vqs: pass struct instead of multi parameters
> > > >   virtio: vring_create_virtqueue: pass struct instead of multi
> > > >     parameters
> > > >   virtio: vring_new_virtqueue(): pass struct instead of multi parameters
> > > >   virtio_ring: simplify the parameters of the funcs related to
> > > >     vring_create/new_virtqueue()
> > > >   virtio: find_vqs: add new parameter premapped
> > > >   virtio_ring: export premapped to driver by struct virtqueue
> > > >   virtio_net: set premapped mode by find_vqs()
> > > >   virtio_ring: remove api of setting vq premapped
> > > >   virtio_ring: introduce dma map api for page
> > > >   virtio_ring: introduce virtqueue_dma_map_sg_attrs
> > > >   virtio_net: unify the code for recycling the xmit ptr
> > > >   virtio_net: rename free_old_xmit_skbs to free_old_xmit
> > > >   virtio_net: sq support premapped mode
> > > >
> > > >  arch/um/drivers/virtio_uml.c             |  31 +-
> > > >  drivers/net/virtio_net.c                 | 283 ++++++---
> > > >  drivers/platform/mellanox/mlxbf-tmfifo.c |  24 +-
> > > >  drivers/remoteproc/remoteproc_virtio.c   |  31 +-
> > > >  drivers/s390/virtio/virtio_ccw.c         |  33 +-
> > > >  drivers/virtio/virtio_mmio.c             |  30 +-
> > > >  drivers/virtio/virtio_pci_common.c       |  59 +-
> > > >  drivers/virtio/virtio_pci_common.h       |   9 +-
> > > >  drivers/virtio/virtio_pci_legacy.c       |  16 +-
> > > >  drivers/virtio/virtio_pci_modern.c       |  38 +-
> > > >  drivers/virtio/virtio_ring.c             | 698 ++++++++++++-----------
> > > >  drivers/virtio/virtio_vdpa.c             |  45 +-
> > > >  include/linux/virtio.h                   |  13 +-
> > > >  include/linux/virtio_config.h            |  48 +-
> > > >  include/linux/virtio_ring.h              |  82 +--
> > > >  tools/virtio/virtio_test.c               |   4 +-
> > > >  tools/virtio/vringh_test.c               |  28 +-
> > > >  17 files changed, 847 insertions(+), 625 deletions(-)
> > > >
> > > > --
> > > > 2.32.0.3.g01195cf9f
> > >
>
Johannes Berg Feb. 29, 2024, 9:41 a.m. UTC | #5
On Thu, 2024-02-29 at 04:34 -0500, Michael S. Tsirkin wrote:
> 
> So the patchset is big, I guess it will take a couple of cycles to
> merge gradually.

Could also do patches 7, 8, and maybe 9 separately first (which seem
reasonable even on their own) and get rid of CC'ing so many people and
lists for future iterations of the rest of the patchset :-)

johannes
Xuan Zhuo Feb. 29, 2024, 9:42 a.m. UTC | #6
On Thu, 29 Feb 2024 10:41:11 +0100, Johannes Berg <johannes@sipsolutions.net> wrote:
> On Thu, 2024-02-29 at 04:34 -0500, Michael S. Tsirkin wrote:
> >
> > So the patchset is big, I guess it will take a couple of cycles to
> > merge gradually.
>
> Could also do patches 7, 8, and maybe 9 separately first (which seem
> reasonable even on their own) and get rid of CC'ing so many people and
> lists for future iterations of the rest of the patchset :-)

I agree.

@Michael How about you?

Thanks


>
> johannes
Xuan Zhuo March 6, 2024, 9:54 a.m. UTC | #7
On Thu, 29 Feb 2024 04:34:20 -0500, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> On Thu, Feb 29, 2024 at 05:02:37PM +0800, Xuan Zhuo wrote:
> > On Thu, 29 Feb 2024 03:21:14 -0500, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > On Thu, Feb 29, 2024 at 03:20:25PM +0800, Xuan Zhuo wrote:
> > > > As discussed:
> > > > http://lore.kernel.org/all/CACGkMEvq0No8QGC46U4mGsMtuD44fD_cfLcPaVmJ3rHYqRZxYg@mail.gmail.com
> > > >
> > > > If the virtio is premapped mode, the driver should manage the dma info by self.
> > > > So the virtio core should not store the dma info.
> > > > So we can release the memory used to store the dma info.
> > > >
> > > > But if the desc_extra has not dma info, we face a new question,
> > > > it is hard to get the dma info of the desc with indirect flag.
> > > > For split mode, that is easy from desc, but for the packed mode,
> > > > it is hard to get the dma info from the desc. And for hardening
> > > > the dma unmap is saft, we should store the dma info of indirect
> > > > descs.
> > > >
> > > > So I introduce the "structure the indirect desc table" to
> > > > allocate space to store dma info with the desc table.
> > > >
> > > > On the other side, we mix the descs with indirect flag
> > > > with other descs together to share the unmap api. That
> > > > is complex. I found if we we distinguish the descs with
> > > > VRING_DESC_F_INDIRECT before unmap, thing will be clearer.
> > > >
> > > > Because of the dma array is allocated in the find_vqs(),
> > > > so I introduce a new parameter to find_vqs().
> > > >
> > > > Note:
> > > >     this is on the top of
> > > >         [PATCH vhost v1] virtio: packed: fix unmap leak for indirect desc table
> > > >         http://lore.kernel.org/all/20240223071833.26095-1-xuanzhuo@linux.alibaba.com
> > > >
> > > > Please review.
> > > >
> > > > Thanks
> > > >
> > > > v3:
> > > >     1. fix the conflict with the vp_modern_create_avq().
> > >
> > > Okay but are you going to address huge memory waste all this is causing for
> > > - people who never do zero copy
> > > - systems where dma unmap is a nop
> > >
> > > ?
> > >
> > > You should address all comments when you post a new version, not just
> > > what was expedient, or alternatively tag patch as RFC and explain
> > > in commit log that you plan to do it later.
> >
> >
> > Do you miss this one?
> > http://lore.kernel.org/all/1708997579.5613105-1-xuanzhuo@linux.alibaba.com
>
>
> I did. The answer is that no, you don't get to regress memory usage
> for lots of people then fix it up.
> So the patchset is big, I guess it will take a couple of cycles to
> merge gradually.

Hi @Michael

So, how about this patch set?

I do not think they (dma maintainers) will agree the API dma_can_skip_unmap().

If you think sq wastes too much memory using pre-mapped dma mode, how about
we only enable it when xsk is bond?

Could you give me some advice?

Thanks.


>
> > I asked you. But I didnot recv your answer.
> >
> > Thanks.
> >
> >
> > >
> > > > v2:
> > > >     1. change the dma item of virtio-net, every item have MAX_SKB_FRAGS + 2
> > > >         addr + len pairs.
> > > >     2. introduce virtnet_sq_free_stats for __free_old_xmit
> > > >
> > > > v1:
> > > >     1. rename transport_vq_config to vq_transport_config
> > > >     2. virtio-net set dma meta number to (ring-size + 1)(MAX_SKB_FRGAS +2)
> > > >     3. introduce virtqueue_dma_map_sg_attrs
> > > >     4. separate vring_create_virtqueue to an independent commit
> > > >
> > > >
> > > >
> > > > Xuan Zhuo (19):
> > > >   virtio_ring: introduce vring_need_unmap_buffer
> > > >   virtio_ring: packed: remove double check of the unmap ops
> > > >   virtio_ring: packed: structure the indirect desc table
> > > >   virtio_ring: split: remove double check of the unmap ops
> > > >   virtio_ring: split: structure the indirect desc table
> > > >   virtio_ring: no store dma info when unmap is not needed
> > > >   virtio: find_vqs: pass struct instead of multi parameters
> > > >   virtio: vring_create_virtqueue: pass struct instead of multi
> > > >     parameters
> > > >   virtio: vring_new_virtqueue(): pass struct instead of multi parameters
> > > >   virtio_ring: simplify the parameters of the funcs related to
> > > >     vring_create/new_virtqueue()
> > > >   virtio: find_vqs: add new parameter premapped
> > > >   virtio_ring: export premapped to driver by struct virtqueue
> > > >   virtio_net: set premapped mode by find_vqs()
> > > >   virtio_ring: remove api of setting vq premapped
> > > >   virtio_ring: introduce dma map api for page
> > > >   virtio_ring: introduce virtqueue_dma_map_sg_attrs
> > > >   virtio_net: unify the code for recycling the xmit ptr
> > > >   virtio_net: rename free_old_xmit_skbs to free_old_xmit
> > > >   virtio_net: sq support premapped mode
> > > >
> > > >  arch/um/drivers/virtio_uml.c             |  31 +-
> > > >  drivers/net/virtio_net.c                 | 283 ++++++---
> > > >  drivers/platform/mellanox/mlxbf-tmfifo.c |  24 +-
> > > >  drivers/remoteproc/remoteproc_virtio.c   |  31 +-
> > > >  drivers/s390/virtio/virtio_ccw.c         |  33 +-
> > > >  drivers/virtio/virtio_mmio.c             |  30 +-
> > > >  drivers/virtio/virtio_pci_common.c       |  59 +-
> > > >  drivers/virtio/virtio_pci_common.h       |   9 +-
> > > >  drivers/virtio/virtio_pci_legacy.c       |  16 +-
> > > >  drivers/virtio/virtio_pci_modern.c       |  38 +-
> > > >  drivers/virtio/virtio_ring.c             | 698 ++++++++++++-----------
> > > >  drivers/virtio/virtio_vdpa.c             |  45 +-
> > > >  include/linux/virtio.h                   |  13 +-
> > > >  include/linux/virtio_config.h            |  48 +-
> > > >  include/linux/virtio_ring.h              |  82 +--
> > > >  tools/virtio/virtio_test.c               |   4 +-
> > > >  tools/virtio/vringh_test.c               |  28 +-
> > > >  17 files changed, 847 insertions(+), 625 deletions(-)
> > > >
> > > > --
> > > > 2.32.0.3.g01195cf9f
> > >
>
Jason Wang March 7, 2024, 5:28 a.m. UTC | #8
On Wed, Mar 6, 2024 at 6:01 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
>
> On Thu, 29 Feb 2024 04:34:20 -0500, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > On Thu, Feb 29, 2024 at 05:02:37PM +0800, Xuan Zhuo wrote:
> > > On Thu, 29 Feb 2024 03:21:14 -0500, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > On Thu, Feb 29, 2024 at 03:20:25PM +0800, Xuan Zhuo wrote:
> > > > > As discussed:
> > > > > http://lore.kernel.org/all/CACGkMEvq0No8QGC46U4mGsMtuD44fD_cfLcPaVmJ3rHYqRZxYg@mail.gmail.com
> > > > >
> > > > > If the virtio is premapped mode, the driver should manage the dma info by self.
> > > > > So the virtio core should not store the dma info.
> > > > > So we can release the memory used to store the dma info.
> > > > >
> > > > > But if the desc_extra has not dma info, we face a new question,
> > > > > it is hard to get the dma info of the desc with indirect flag.
> > > > > For split mode, that is easy from desc, but for the packed mode,
> > > > > it is hard to get the dma info from the desc. And for hardening
> > > > > the dma unmap is saft, we should store the dma info of indirect
> > > > > descs.
> > > > >
> > > > > So I introduce the "structure the indirect desc table" to
> > > > > allocate space to store dma info with the desc table.
> > > > >
> > > > > On the other side, we mix the descs with indirect flag
> > > > > with other descs together to share the unmap api. That
> > > > > is complex. I found if we we distinguish the descs with
> > > > > VRING_DESC_F_INDIRECT before unmap, thing will be clearer.
> > > > >
> > > > > Because of the dma array is allocated in the find_vqs(),
> > > > > so I introduce a new parameter to find_vqs().
> > > > >
> > > > > Note:
> > > > >     this is on the top of
> > > > >         [PATCH vhost v1] virtio: packed: fix unmap leak for indirect desc table
> > > > >         http://lore.kernel.org/all/20240223071833.26095-1-xuanzhuo@linux.alibaba.com
> > > > >
> > > > > Please review.
> > > > >
> > > > > Thanks
> > > > >
> > > > > v3:
> > > > >     1. fix the conflict with the vp_modern_create_avq().
> > > >
> > > > Okay but are you going to address huge memory waste all this is causing for
> > > > - people who never do zero copy
> > > > - systems where dma unmap is a nop
> > > >
> > > > ?
> > > >
> > > > You should address all comments when you post a new version, not just
> > > > what was expedient, or alternatively tag patch as RFC and explain
> > > > in commit log that you plan to do it later.
> > >
> > >
> > > Do you miss this one?
> > > http://lore.kernel.org/all/1708997579.5613105-1-xuanzhuo@linux.alibaba.com
> >
> >
> > I did. The answer is that no, you don't get to regress memory usage
> > for lots of people then fix it up.
> > So the patchset is big, I guess it will take a couple of cycles to
> > merge gradually.
>
> Hi @Michael
>
> So, how about this patch set?
>
> I do not think they (dma maintainers) will agree the API dma_can_skip_unmap().
>
> If you think sq wastes too much memory using pre-mapped dma mode, how about
> we only enable it when xsk is bond?
>
> Could you give me some advice?

I think we have some discussion, one possible solution is:

when pre mapping is enabled, virtio core won't store dma metadatas.

Then it makes virtio-net align with other NIC.

Thanks

>
> Thanks.
>
>
> >
> > > I asked you. But I didnot recv your answer.
> > >
> > > Thanks.
> > >
> > >
> > > >
> > > > > v2:
> > > > >     1. change the dma item of virtio-net, every item have MAX_SKB_FRAGS + 2
> > > > >         addr + len pairs.
> > > > >     2. introduce virtnet_sq_free_stats for __free_old_xmit
> > > > >
> > > > > v1:
> > > > >     1. rename transport_vq_config to vq_transport_config
> > > > >     2. virtio-net set dma meta number to (ring-size + 1)(MAX_SKB_FRGAS +2)
> > > > >     3. introduce virtqueue_dma_map_sg_attrs
> > > > >     4. separate vring_create_virtqueue to an independent commit
> > > > >
> > > > >
> > > > >
> > > > > Xuan Zhuo (19):
> > > > >   virtio_ring: introduce vring_need_unmap_buffer
> > > > >   virtio_ring: packed: remove double check of the unmap ops
> > > > >   virtio_ring: packed: structure the indirect desc table
> > > > >   virtio_ring: split: remove double check of the unmap ops
> > > > >   virtio_ring: split: structure the indirect desc table
> > > > >   virtio_ring: no store dma info when unmap is not needed
> > > > >   virtio: find_vqs: pass struct instead of multi parameters
> > > > >   virtio: vring_create_virtqueue: pass struct instead of multi
> > > > >     parameters
> > > > >   virtio: vring_new_virtqueue(): pass struct instead of multi parameters
> > > > >   virtio_ring: simplify the parameters of the funcs related to
> > > > >     vring_create/new_virtqueue()
> > > > >   virtio: find_vqs: add new parameter premapped
> > > > >   virtio_ring: export premapped to driver by struct virtqueue
> > > > >   virtio_net: set premapped mode by find_vqs()
> > > > >   virtio_ring: remove api of setting vq premapped
> > > > >   virtio_ring: introduce dma map api for page
> > > > >   virtio_ring: introduce virtqueue_dma_map_sg_attrs
> > > > >   virtio_net: unify the code for recycling the xmit ptr
> > > > >   virtio_net: rename free_old_xmit_skbs to free_old_xmit
> > > > >   virtio_net: sq support premapped mode
> > > > >
> > > > >  arch/um/drivers/virtio_uml.c             |  31 +-
> > > > >  drivers/net/virtio_net.c                 | 283 ++++++---
> > > > >  drivers/platform/mellanox/mlxbf-tmfifo.c |  24 +-
> > > > >  drivers/remoteproc/remoteproc_virtio.c   |  31 +-
> > > > >  drivers/s390/virtio/virtio_ccw.c         |  33 +-
> > > > >  drivers/virtio/virtio_mmio.c             |  30 +-
> > > > >  drivers/virtio/virtio_pci_common.c       |  59 +-
> > > > >  drivers/virtio/virtio_pci_common.h       |   9 +-
> > > > >  drivers/virtio/virtio_pci_legacy.c       |  16 +-
> > > > >  drivers/virtio/virtio_pci_modern.c       |  38 +-
> > > > >  drivers/virtio/virtio_ring.c             | 698 ++++++++++++-----------
> > > > >  drivers/virtio/virtio_vdpa.c             |  45 +-
> > > > >  include/linux/virtio.h                   |  13 +-
> > > > >  include/linux/virtio_config.h            |  48 +-
> > > > >  include/linux/virtio_ring.h              |  82 +--
> > > > >  tools/virtio/virtio_test.c               |   4 +-
> > > > >  tools/virtio/vringh_test.c               |  28 +-
> > > > >  17 files changed, 847 insertions(+), 625 deletions(-)
> > > > >
> > > > > --
> > > > > 2.32.0.3.g01195cf9f
> > > >
> >
>
Xuan Zhuo March 7, 2024, 8:06 a.m. UTC | #9
On Thu, 7 Mar 2024 13:28:27 +0800, Jason Wang <jasowang@redhat.com> wrote:
> On Wed, Mar 6, 2024 at 6:01 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> >
> > On Thu, 29 Feb 2024 04:34:20 -0500, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > On Thu, Feb 29, 2024 at 05:02:37PM +0800, Xuan Zhuo wrote:
> > > > On Thu, 29 Feb 2024 03:21:14 -0500, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > On Thu, Feb 29, 2024 at 03:20:25PM +0800, Xuan Zhuo wrote:
> > > > > > As discussed:
> > > > > > http://lore.kernel.org/all/CACGkMEvq0No8QGC46U4mGsMtuD44fD_cfLcPaVmJ3rHYqRZxYg@mail.gmail.com
> > > > > >
> > > > > > If the virtio is premapped mode, the driver should manage the dma info by self.
> > > > > > So the virtio core should not store the dma info.
> > > > > > So we can release the memory used to store the dma info.
> > > > > >
> > > > > > But if the desc_extra has not dma info, we face a new question,
> > > > > > it is hard to get the dma info of the desc with indirect flag.
> > > > > > For split mode, that is easy from desc, but for the packed mode,
> > > > > > it is hard to get the dma info from the desc. And for hardening
> > > > > > the dma unmap is saft, we should store the dma info of indirect
> > > > > > descs.
> > > > > >
> > > > > > So I introduce the "structure the indirect desc table" to
> > > > > > allocate space to store dma info with the desc table.
> > > > > >
> > > > > > On the other side, we mix the descs with indirect flag
> > > > > > with other descs together to share the unmap api. That
> > > > > > is complex. I found if we we distinguish the descs with
> > > > > > VRING_DESC_F_INDIRECT before unmap, thing will be clearer.
> > > > > >
> > > > > > Because of the dma array is allocated in the find_vqs(),
> > > > > > so I introduce a new parameter to find_vqs().
> > > > > >
> > > > > > Note:
> > > > > >     this is on the top of
> > > > > >         [PATCH vhost v1] virtio: packed: fix unmap leak for indirect desc table
> > > > > >         http://lore.kernel.org/all/20240223071833.26095-1-xuanzhuo@linux.alibaba.com
> > > > > >
> > > > > > Please review.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > v3:
> > > > > >     1. fix the conflict with the vp_modern_create_avq().
> > > > >
> > > > > Okay but are you going to address huge memory waste all this is causing for
> > > > > - people who never do zero copy
> > > > > - systems where dma unmap is a nop
> > > > >
> > > > > ?
> > > > >
> > > > > You should address all comments when you post a new version, not just
> > > > > what was expedient, or alternatively tag patch as RFC and explain
> > > > > in commit log that you plan to do it later.
> > > >
> > > >
> > > > Do you miss this one?
> > > > http://lore.kernel.org/all/1708997579.5613105-1-xuanzhuo@linux.alibaba.com
> > >
> > >
> > > I did. The answer is that no, you don't get to regress memory usage
> > > for lots of people then fix it up.
> > > So the patchset is big, I guess it will take a couple of cycles to
> > > merge gradually.
> >
> > Hi @Michael
> >
> > So, how about this patch set?
> >
> > I do not think they (dma maintainers) will agree the API dma_can_skip_unmap().
> >
> > If you think sq wastes too much memory using pre-mapped dma mode, how about
> > we only enable it when xsk is bond?
> >
> > Could you give me some advice?
>
> I think we have some discussion, one possible solution is:
>
> when pre mapping is enabled, virtio core won't store dma metadatas.
>
> Then it makes virtio-net align with other NIC.


YES.

This patch set works as this.

But the virtio-net must allocate too much memory to store dma and len.

num = queue size * 19

Michael thinks that waste too much memory.
	http://lore.kernel.org/all/20240225032330-mutt-send-email-mst@kernel.org

So we try this:
	http://lore.kernel.org/all/20240301071918.64631-1-xuanzhuo@linux.alibaba.com

But I think that is difficult to be accepted by the  DMA maintainers.

So I have two advices:

1. virtio-net sq works without indirect.
	- that more like other NIC
	- the num of the memory to store the dma info is queue_size

2. The default mode of virtio-net sq is no-premapped
	- we just switch the mode when binding xsk

Thanks.


>
> Thanks
>
> >
> > Thanks.
> >
> >
> > >
> > > > I asked you. But I didnot recv your answer.
> > > >
> > > > Thanks.
> > > >
> > > >
> > > > >
> > > > > > v2:
> > > > > >     1. change the dma item of virtio-net, every item have MAX_SKB_FRAGS + 2
> > > > > >         addr + len pairs.
> > > > > >     2. introduce virtnet_sq_free_stats for __free_old_xmit
> > > > > >
> > > > > > v1:
> > > > > >     1. rename transport_vq_config to vq_transport_config
> > > > > >     2. virtio-net set dma meta number to (ring-size + 1)(MAX_SKB_FRGAS +2)
> > > > > >     3. introduce virtqueue_dma_map_sg_attrs
> > > > > >     4. separate vring_create_virtqueue to an independent commit
> > > > > >
> > > > > >
> > > > > >
> > > > > > Xuan Zhuo (19):
> > > > > >   virtio_ring: introduce vring_need_unmap_buffer
> > > > > >   virtio_ring: packed: remove double check of the unmap ops
> > > > > >   virtio_ring: packed: structure the indirect desc table
> > > > > >   virtio_ring: split: remove double check of the unmap ops
> > > > > >   virtio_ring: split: structure the indirect desc table
> > > > > >   virtio_ring: no store dma info when unmap is not needed
> > > > > >   virtio: find_vqs: pass struct instead of multi parameters
> > > > > >   virtio: vring_create_virtqueue: pass struct instead of multi
> > > > > >     parameters
> > > > > >   virtio: vring_new_virtqueue(): pass struct instead of multi parameters
> > > > > >   virtio_ring: simplify the parameters of the funcs related to
> > > > > >     vring_create/new_virtqueue()
> > > > > >   virtio: find_vqs: add new parameter premapped
> > > > > >   virtio_ring: export premapped to driver by struct virtqueue
> > > > > >   virtio_net: set premapped mode by find_vqs()
> > > > > >   virtio_ring: remove api of setting vq premapped
> > > > > >   virtio_ring: introduce dma map api for page
> > > > > >   virtio_ring: introduce virtqueue_dma_map_sg_attrs
> > > > > >   virtio_net: unify the code for recycling the xmit ptr
> > > > > >   virtio_net: rename free_old_xmit_skbs to free_old_xmit
> > > > > >   virtio_net: sq support premapped mode
> > > > > >
> > > > > >  arch/um/drivers/virtio_uml.c             |  31 +-
> > > > > >  drivers/net/virtio_net.c                 | 283 ++++++---
> > > > > >  drivers/platform/mellanox/mlxbf-tmfifo.c |  24 +-
> > > > > >  drivers/remoteproc/remoteproc_virtio.c   |  31 +-
> > > > > >  drivers/s390/virtio/virtio_ccw.c         |  33 +-
> > > > > >  drivers/virtio/virtio_mmio.c             |  30 +-
> > > > > >  drivers/virtio/virtio_pci_common.c       |  59 +-
> > > > > >  drivers/virtio/virtio_pci_common.h       |   9 +-
> > > > > >  drivers/virtio/virtio_pci_legacy.c       |  16 +-
> > > > > >  drivers/virtio/virtio_pci_modern.c       |  38 +-
> > > > > >  drivers/virtio/virtio_ring.c             | 698 ++++++++++++-----------
> > > > > >  drivers/virtio/virtio_vdpa.c             |  45 +-
> > > > > >  include/linux/virtio.h                   |  13 +-
> > > > > >  include/linux/virtio_config.h            |  48 +-
> > > > > >  include/linux/virtio_ring.h              |  82 +--
> > > > > >  tools/virtio/virtio_test.c               |   4 +-
> > > > > >  tools/virtio/vringh_test.c               |  28 +-
> > > > > >  17 files changed, 847 insertions(+), 625 deletions(-)
> > > > > >
> > > > > > --
> > > > > > 2.32.0.3.g01195cf9f
> > > > >
> > >
> >
>
Jason Wang March 8, 2024, 6:03 a.m. UTC | #10
On Thu, Mar 7, 2024 at 4:15 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
>
> On Thu, 7 Mar 2024 13:28:27 +0800, Jason Wang <jasowang@redhat.com> wrote:
> > On Wed, Mar 6, 2024 at 6:01 PM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote:
> > >
> > > On Thu, 29 Feb 2024 04:34:20 -0500, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > On Thu, Feb 29, 2024 at 05:02:37PM +0800, Xuan Zhuo wrote:
> > > > > On Thu, 29 Feb 2024 03:21:14 -0500, "Michael S. Tsirkin" <mst@redhat.com> wrote:
> > > > > > On Thu, Feb 29, 2024 at 03:20:25PM +0800, Xuan Zhuo wrote:
> > > > > > > As discussed:
> > > > > > > http://lore.kernel.org/all/CACGkMEvq0No8QGC46U4mGsMtuD44fD_cfLcPaVmJ3rHYqRZxYg@mail.gmail.com
> > > > > > >
> > > > > > > If the virtio is premapped mode, the driver should manage the dma info by self.
> > > > > > > So the virtio core should not store the dma info.
> > > > > > > So we can release the memory used to store the dma info.
> > > > > > >
> > > > > > > But if the desc_extra has not dma info, we face a new question,
> > > > > > > it is hard to get the dma info of the desc with indirect flag.
> > > > > > > For split mode, that is easy from desc, but for the packed mode,
> > > > > > > it is hard to get the dma info from the desc. And for hardening
> > > > > > > the dma unmap is saft, we should store the dma info of indirect
> > > > > > > descs.
> > > > > > >
> > > > > > > So I introduce the "structure the indirect desc table" to
> > > > > > > allocate space to store dma info with the desc table.
> > > > > > >
> > > > > > > On the other side, we mix the descs with indirect flag
> > > > > > > with other descs together to share the unmap api. That
> > > > > > > is complex. I found if we we distinguish the descs with
> > > > > > > VRING_DESC_F_INDIRECT before unmap, thing will be clearer.
> > > > > > >
> > > > > > > Because of the dma array is allocated in the find_vqs(),
> > > > > > > so I introduce a new parameter to find_vqs().
> > > > > > >
> > > > > > > Note:
> > > > > > >     this is on the top of
> > > > > > >         [PATCH vhost v1] virtio: packed: fix unmap leak for indirect desc table
> > > > > > >         http://lore.kernel.org/all/20240223071833.26095-1-xuanzhuo@linux.alibaba.com
> > > > > > >
> > > > > > > Please review.
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > > v3:
> > > > > > >     1. fix the conflict with the vp_modern_create_avq().
> > > > > >
> > > > > > Okay but are you going to address huge memory waste all this is causing for
> > > > > > - people who never do zero copy
> > > > > > - systems where dma unmap is a nop
> > > > > >
> > > > > > ?
> > > > > >
> > > > > > You should address all comments when you post a new version, not just
> > > > > > what was expedient, or alternatively tag patch as RFC and explain
> > > > > > in commit log that you plan to do it later.
> > > > >
> > > > >
> > > > > Do you miss this one?
> > > > > http://lore.kernel.org/all/1708997579.5613105-1-xuanzhuo@linux.alibaba.com
> > > >
> > > >
> > > > I did. The answer is that no, you don't get to regress memory usage
> > > > for lots of people then fix it up.
> > > > So the patchset is big, I guess it will take a couple of cycles to
> > > > merge gradually.
> > >
> > > Hi @Michael
> > >
> > > So, how about this patch set?
> > >
> > > I do not think they (dma maintainers) will agree the API dma_can_skip_unmap().
> > >
> > > If you think sq wastes too much memory using pre-mapped dma mode, how about
> > > we only enable it when xsk is bond?
> > >
> > > Could you give me some advice?
> >
> > I think we have some discussion, one possible solution is:
> >
> > when pre mapping is enabled, virtio core won't store dma metadatas.
> >
> > Then it makes virtio-net align with other NIC.
>
>
> YES.
>
> This patch set works as this.
>
> But the virtio-net must allocate too much memory to store dma and len.
>
> num = queue size * 19
>
> Michael thinks that waste too much memory.
>         http://lore.kernel.org/all/20240225032330-mutt-send-email-mst@kernel.org
>
> So we try this:
>         http://lore.kernel.org/all/20240301071918.64631-1-xuanzhuo@linux.alibaba.com
>
> But I think that is difficult to be accepted by the  DMA maintainers.
>
> So I have two advices:
>
> 1. virtio-net sq works without indirect.
>         - that more like other NIC
>         - the num of the memory to store the dma info is queue_size

This requires benchmarks.

>
> 2. The default mode of virtio-net sq is no-premapped
>         - we just switch the mode when binding xsk

This could be one step.

We can hear from Michael.

Thanks

>
> Thanks.
>
>
> >
> > Thanks
> >
> > >
> > > Thanks.
> > >
> > >
> > > >
> > > > > I asked you. But I didnot recv your answer.
> > > > >
> > > > > Thanks.
> > > > >
> > > > >
> > > > > >
> > > > > > > v2:
> > > > > > >     1. change the dma item of virtio-net, every item have MAX_SKB_FRAGS + 2
> > > > > > >         addr + len pairs.
> > > > > > >     2. introduce virtnet_sq_free_stats for __free_old_xmit
> > > > > > >
> > > > > > > v1:
> > > > > > >     1. rename transport_vq_config to vq_transport_config
> > > > > > >     2. virtio-net set dma meta number to (ring-size + 1)(MAX_SKB_FRGAS +2)
> > > > > > >     3. introduce virtqueue_dma_map_sg_attrs
> > > > > > >     4. separate vring_create_virtqueue to an independent commit
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Xuan Zhuo (19):
> > > > > > >   virtio_ring: introduce vring_need_unmap_buffer
> > > > > > >   virtio_ring: packed: remove double check of the unmap ops
> > > > > > >   virtio_ring: packed: structure the indirect desc table
> > > > > > >   virtio_ring: split: remove double check of the unmap ops
> > > > > > >   virtio_ring: split: structure the indirect desc table
> > > > > > >   virtio_ring: no store dma info when unmap is not needed
> > > > > > >   virtio: find_vqs: pass struct instead of multi parameters
> > > > > > >   virtio: vring_create_virtqueue: pass struct instead of multi
> > > > > > >     parameters
> > > > > > >   virtio: vring_new_virtqueue(): pass struct instead of multi parameters
> > > > > > >   virtio_ring: simplify the parameters of the funcs related to
> > > > > > >     vring_create/new_virtqueue()
> > > > > > >   virtio: find_vqs: add new parameter premapped
> > > > > > >   virtio_ring: export premapped to driver by struct virtqueue
> > > > > > >   virtio_net: set premapped mode by find_vqs()
> > > > > > >   virtio_ring: remove api of setting vq premapped
> > > > > > >   virtio_ring: introduce dma map api for page
> > > > > > >   virtio_ring: introduce virtqueue_dma_map_sg_attrs
> > > > > > >   virtio_net: unify the code for recycling the xmit ptr
> > > > > > >   virtio_net: rename free_old_xmit_skbs to free_old_xmit
> > > > > > >   virtio_net: sq support premapped mode
> > > > > > >
> > > > > > >  arch/um/drivers/virtio_uml.c             |  31 +-
> > > > > > >  drivers/net/virtio_net.c                 | 283 ++++++---
> > > > > > >  drivers/platform/mellanox/mlxbf-tmfifo.c |  24 +-
> > > > > > >  drivers/remoteproc/remoteproc_virtio.c   |  31 +-
> > > > > > >  drivers/s390/virtio/virtio_ccw.c         |  33 +-
> > > > > > >  drivers/virtio/virtio_mmio.c             |  30 +-
> > > > > > >  drivers/virtio/virtio_pci_common.c       |  59 +-
> > > > > > >  drivers/virtio/virtio_pci_common.h       |   9 +-
> > > > > > >  drivers/virtio/virtio_pci_legacy.c       |  16 +-
> > > > > > >  drivers/virtio/virtio_pci_modern.c       |  38 +-
> > > > > > >  drivers/virtio/virtio_ring.c             | 698 ++++++++++++-----------
> > > > > > >  drivers/virtio/virtio_vdpa.c             |  45 +-
> > > > > > >  include/linux/virtio.h                   |  13 +-
> > > > > > >  include/linux/virtio_config.h            |  48 +-
> > > > > > >  include/linux/virtio_ring.h              |  82 +--
> > > > > > >  tools/virtio/virtio_test.c               |   4 +-
> > > > > > >  tools/virtio/vringh_test.c               |  28 +-
> > > > > > >  17 files changed, 847 insertions(+), 625 deletions(-)
> > > > > > >
> > > > > > > --
> > > > > > > 2.32.0.3.g01195cf9f
> > > > > >
> > > >
> > >
> >
>