| Message ID | 20240312033557.6351-1-xuanzhuo@linux.alibaba.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | virtio: drivers maintain dma info for premapped vq | expand |
On Tue, Mar 12, 2024 at 11:35:47AM +0800, Xuan Zhuo wrote: > As discussed: > > http://lore.kernel.org/all/CACGkMEvq0No8QGC46U4mGsMtuD44fD_cfLcPaVmJ3rHYqRZxYg@mail.gmail.com > > If the virtio is premapped mode, the driver should manage the dma info by self. > So the virtio core should not store the dma info. We can release the memory used > to store the dma info. > > For virtio-net xmit queue, if the virtio-net maintains the dma info, > the virtio-net must allocate too much memory(19 * queue_size for per-queue), so > we do not plan to make the virtio-net to maintain the dma info by default. The > virtio-net xmit queue only maintain the dma info when premapped mode is enable > (such as AF_XDP is enable). This landed when merge window was open already so I'm deferring this to the next merge window, just to be safe. Jason can you review please? > So this patch set try to do: > > 1. make the virtio core to do not store the dma info > - But if the desc_extra has not dma info, we face a new question, > it is hard to get the dma info of the desc with indirect flag. > For split mode, that is easy from desc, but for the packed mode, > it is hard to get the dma info from the desc. And hardening > the dma unmap is safe, we should store the dma info of indirect > descs when the virtio core does not store the bufer dma info. > > So I introduce the "structure the indirect desc table" to > allocate space to store dma info of the desc table. > > +struct vring_split_desc_indir { > + dma_addr_t addr; /* Descriptor Array DMA addr. */ > + u32 len; /* Descriptor Array length. */ > + u32 num; > + struct vring_desc desc[]; > +}; > > The follow patches to this: > * virtio_ring: packed: structure the indirect desc table > * virtio_ring: split: structure the indirect desc table > > - On the other side, in the umap handle, we mix the indirect descs with > other descs. That make things too complex. I found if we we distinguish > the descs with VRING_DESC_F_INDIRECT before unmap, thing will be clearer. > > The follow patches do this. > * virtio_ring: packed: remove double check of the unmap ops > * virtio_ring: split: structure the indirect desc table > > 2. make the virtio core to enable premapped mode by find_vqs() params > - Because the find_vqs() will try to allocate memory for the dma info. > If we set the premapped mode after find_vqs() and release the > dma info, that is odd. > > > Please review. > > Thanks > > v4: > 1. virtio-net xmit queue does not enable premapped mode by default > > v3: > 1. fix the conflict with the vp_modern_create_avq(). > > v2: > 1. change the dma item of virtio-net, every item have MAX_SKB_FRAGS + 2 addr + len pairs. > 2. introduce virtnet_sq_free_stats for __free_old_xmit > > v1: > 1. rename transport_vq_config to vq_transport_config > 2. virtio-net set dma meta number to (ring-size + 1)(MAX_SKB_FRGAS +2) > 3. introduce virtqueue_dma_map_sg_attrs > 4. separate vring_create_virtqueue to an independent commit > > Xuan Zhuo (10): > virtio_ring: introduce vring_need_unmap_buffer > virtio_ring: packed: remove double check of the unmap ops > virtio_ring: packed: structure the indirect desc table > virtio_ring: split: remove double check of the unmap ops > virtio_ring: split: structure the indirect desc table > virtio_ring: no store dma info when unmap is not needed > virtio: find_vqs: add new parameter premapped > virtio_ring: export premapped to driver by struct virtqueue > virtio_net: set premapped mode by find_vqs() > virtio_ring: virtqueue_set_dma_premapped support disable > > drivers/net/virtio_net.c | 57 +++-- > drivers/virtio/virtio_ring.c | 436 +++++++++++++++++++++------------- > include/linux/virtio.h | 3 +- > include/linux/virtio_config.h | 17 +- > 4 files changed, 307 insertions(+), 206 deletions(-) > > -- > 2.32.0.3.g01195cf9f
On Tue, Mar 19, 2024 at 2:56 PM Michael S. Tsirkin <mst@redhat.com> wrote: > > On Tue, Mar 12, 2024 at 11:35:47AM +0800, Xuan Zhuo wrote: > > As discussed: > > > > http://lore.kernel.org/all/CACGkMEvq0No8QGC46U4mGsMtuD44fD_cfLcPaVmJ3rHYqRZxYg@mail.gmail.com > > > > If the virtio is premapped mode, the driver should manage the dma info by self. > > So the virtio core should not store the dma info. We can release the memory used > > to store the dma info. > > > > For virtio-net xmit queue, if the virtio-net maintains the dma info, > > the virtio-net must allocate too much memory(19 * queue_size for per-queue), so > > we do not plan to make the virtio-net to maintain the dma info by default. The > > virtio-net xmit queue only maintain the dma info when premapped mode is enable > > (such as AF_XDP is enable). > > This landed when merge window was open already so I'm deferring this > to the next merge window, just to be safe. Jason can you review please? Will do. Thanks
On Tue, Mar 12, 2024 at 11:36 AM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > > As discussed: > > http://lore.kernel.org/all/CACGkMEvq0No8QGC46U4mGsMtuD44fD_cfLcPaVmJ3rHYqRZxYg@mail.gmail.com > > If the virtio is premapped mode, the driver should manage the dma info by self. > So the virtio core should not store the dma info. We can release the memory used > to store the dma info. > > For virtio-net xmit queue, if the virtio-net maintains the dma info, > the virtio-net must allocate too much memory(19 * queue_size for per-queue), so > we do not plan to make the virtio-net to maintain the dma info by default. The > virtio-net xmit queue only maintain the dma info when premapped mode is enable > (such as AF_XDP is enable). > > So this patch set try to do: > > 1. make the virtio core to do not store the dma info I think you mean "make the virtio core to do not store the dma info when driver can do that" > - But if the desc_extra has not dma info, we face a new question, > it is hard to get the dma info of the desc with indirect flag. I guess you want to avoid allocating desc_extra array, otherwise you won't have this issue. How about keeping that? > For split mode, that is easy from desc, but for the packed mode, > it is hard to get the dma info from the desc. And hardening > the dma unmap is safe, we should store the dma info of indirect > descs when the virtio core does not store the bufer dma info. > > So I introduce the "structure the indirect desc table" to > allocate space to store dma info of the desc table. > > +struct vring_split_desc_indir { > + dma_addr_t addr; /* Descriptor Array DMA addr. */ > + u32 len; /* Descriptor Array length. */ > + u32 num; We can probably just reuse vring_desc_extra here with a known flag (read only for device). > + struct vring_desc desc[]; > +}; > > The follow patches to this: > * virtio_ring: packed: structure the indirect desc table > * virtio_ring: split: structure the indirect desc table > > - On the other side, in the umap handle, we mix the indirect descs with > other descs. That make things too complex. I found if we we distinguish > the descs with VRING_DESC_F_INDIRECT before unmap, thing will be clearer. > > The follow patches do this. > * virtio_ring: packed: remove double check of the unmap ops > * virtio_ring: split: structure the indirect desc table > > 2. make the virtio core to enable premapped mode by find_vqs() params > - Because the find_vqs() will try to allocate memory for the dma info. > If we set the premapped mode after find_vqs() and release the > dma info, that is odd. Thanks > > > Please review. > > Thanks > > v4: > 1. virtio-net xmit queue does not enable premapped mode by default > > v3: > 1. fix the conflict with the vp_modern_create_avq(). > > v2: > 1. change the dma item of virtio-net, every item have MAX_SKB_FRAGS + 2 addr + len pairs. > 2. introduce virtnet_sq_free_stats for __free_old_xmit > > v1: > 1. rename transport_vq_config to vq_transport_config > 2. virtio-net set dma meta number to (ring-size + 1)(MAX_SKB_FRGAS +2) > 3. introduce virtqueue_dma_map_sg_attrs > 4. separate vring_create_virtqueue to an independent commit > > Xuan Zhuo (10): > virtio_ring: introduce vring_need_unmap_buffer > virtio_ring: packed: remove double check of the unmap ops > virtio_ring: packed: structure the indirect desc table > virtio_ring: split: remove double check of the unmap ops > virtio_ring: split: structure the indirect desc table > virtio_ring: no store dma info when unmap is not needed > virtio: find_vqs: add new parameter premapped > virtio_ring: export premapped to driver by struct virtqueue > virtio_net: set premapped mode by find_vqs() > virtio_ring: virtqueue_set_dma_premapped support disable > > drivers/net/virtio_net.c | 57 +++-- > drivers/virtio/virtio_ring.c | 436 +++++++++++++++++++++------------- > include/linux/virtio.h | 3 +- > include/linux/virtio_config.h | 17 +- > 4 files changed, 307 insertions(+), 206 deletions(-) > > -- > 2.32.0.3.g01195cf9f >
On Thu, 21 Mar 2024 12:45:08 +0800, Jason Wang <jasowang@redhat.com> wrote: > On Tue, Mar 12, 2024 at 11:36 AM Xuan Zhuo <xuanzhuo@linux.alibaba.com> wrote: > > > > As discussed: > > > > http://lore.kernel.org/all/CACGkMEvq0No8QGC46U4mGsMtuD44fD_cfLcPaVmJ3rHYqRZxYg@mail.gmail.com > > > > If the virtio is premapped mode, the driver should manage the dma info by self. > > So the virtio core should not store the dma info. We can release the memory used > > to store the dma info. > > > > For virtio-net xmit queue, if the virtio-net maintains the dma info, > > the virtio-net must allocate too much memory(19 * queue_size for per-queue), so > > we do not plan to make the virtio-net to maintain the dma info by default. The > > virtio-net xmit queue only maintain the dma info when premapped mode is enable > > (such as AF_XDP is enable). > > > > So this patch set try to do: > > > > 1. make the virtio core to do not store the dma info > > I think you mean "make the virtio core to do not store the dma info > when driver can do that" YES. > > > - But if the desc_extra has not dma info, we face a new question, > > it is hard to get the dma info of the desc with indirect flag. > > I guess you want to avoid allocating desc_extra array, otherwise you > won't have this issue. > > How about keeping that? This is a way. But when we allocate the indirect desc, we alloc more memory to save that, I think that is a good way. And in the future, we can handen the unmap for the indirect buffer with more memory allocated by once. Thanks. > > > For split mode, that is easy from desc, but for the packed mode, > > it is hard to get the dma info from the desc. And hardening > > the dma unmap is safe, we should store the dma info of indirect > > descs when the virtio core does not store the bufer dma info. > > > > So I introduce the "structure the indirect desc table" to > > allocate space to store dma info of the desc table. > > > > +struct vring_split_desc_indir { > > + dma_addr_t addr; /* Descriptor Array DMA addr. */ > > + u32 len; /* Descriptor Array length. */ > > + u32 num; > > We can probably just reuse vring_desc_extra here with a known flag > (read only for device). > > > + struct vring_desc desc[]; > > +}; > > > > The follow patches to this: > > * virtio_ring: packed: structure the indirect desc table > > * virtio_ring: split: structure the indirect desc table > > > > - On the other side, in the umap handle, we mix the indirect descs with > > other descs. That make things too complex. I found if we we distinguish > > the descs with VRING_DESC_F_INDIRECT before unmap, thing will be clearer. > > > > The follow patches do this. > > * virtio_ring: packed: remove double check of the unmap ops > > * virtio_ring: split: structure the indirect desc table > > > > 2. make the virtio core to enable premapped mode by find_vqs() params > > - Because the find_vqs() will try to allocate memory for the dma info. > > If we set the premapped mode after find_vqs() and release the > > dma info, that is odd. > > Thanks > > > > > > > Please review. > > > > Thanks > > > > v4: > > 1. virtio-net xmit queue does not enable premapped mode by default > > > > v3: > > 1. fix the conflict with the vp_modern_create_avq(). > > > > v2: > > 1. change the dma item of virtio-net, every item have MAX_SKB_FRAGS + 2 addr + len pairs. > > 2. introduce virtnet_sq_free_stats for __free_old_xmit > > > > v1: > > 1. rename transport_vq_config to vq_transport_config > > 2. virtio-net set dma meta number to (ring-size + 1)(MAX_SKB_FRGAS +2) > > 3. introduce virtqueue_dma_map_sg_attrs > > 4. separate vring_create_virtqueue to an independent commit > > > > Xuan Zhuo (10): > > virtio_ring: introduce vring_need_unmap_buffer > > virtio_ring: packed: remove double check of the unmap ops > > virtio_ring: packed: structure the indirect desc table > > virtio_ring: split: remove double check of the unmap ops > > virtio_ring: split: structure the indirect desc table > > virtio_ring: no store dma info when unmap is not needed > > virtio: find_vqs: add new parameter premapped > > virtio_ring: export premapped to driver by struct virtqueue > > virtio_net: set premapped mode by find_vqs() > > virtio_ring: virtqueue_set_dma_premapped support disable > > > > drivers/net/virtio_net.c | 57 +++-- > > drivers/virtio/virtio_ring.c | 436 +++++++++++++++++++++------------- > > include/linux/virtio.h | 3 +- > > include/linux/virtio_config.h | 17 +- > > 4 files changed, 307 insertions(+), 206 deletions(-) > > > > -- > > 2.32.0.3.g01195cf9f > > >
As discussed: http://lore.kernel.org/all/CACGkMEvq0No8QGC46U4mGsMtuD44fD_cfLcPaVmJ3rHYqRZxYg@mail.gmail.com If the virtio is premapped mode, the driver should manage the dma info by self. So the virtio core should not store the dma info. We can release the memory used to store the dma info. For virtio-net xmit queue, if the virtio-net maintains the dma info, the virtio-net must allocate too much memory(19 * queue_size for per-queue), so we do not plan to make the virtio-net to maintain the dma info by default. The virtio-net xmit queue only maintain the dma info when premapped mode is enable (such as AF_XDP is enable). So this patch set try to do: 1. make the virtio core to do not store the dma info - But if the desc_extra has not dma info, we face a new question, it is hard to get the dma info of the desc with indirect flag. For split mode, that is easy from desc, but for the packed mode, it is hard to get the dma info from the desc. And hardening the dma unmap is safe, we should store the dma info of indirect descs when the virtio core does not store the bufer dma info. So I introduce the "structure the indirect desc table" to allocate space to store dma info of the desc table. +struct vring_split_desc_indir { + dma_addr_t addr; /* Descriptor Array DMA addr. */ + u32 len; /* Descriptor Array length. */ + u32 num; + struct vring_desc desc[]; +}; The follow patches to this: * virtio_ring: packed: structure the indirect desc table * virtio_ring: split: structure the indirect desc table - On the other side, in the umap handle, we mix the indirect descs with other descs. That make things too complex. I found if we we distinguish the descs with VRING_DESC_F_INDIRECT before unmap, thing will be clearer. The follow patches do this. * virtio_ring: packed: remove double check of the unmap ops * virtio_ring: split: structure the indirect desc table 2. make the virtio core to enable premapped mode by find_vqs() params - Because the find_vqs() will try to allocate memory for the dma info. If we set the premapped mode after find_vqs() and release the dma info, that is odd. Please review. Thanks v4: 1. virtio-net xmit queue does not enable premapped mode by default v3: 1. fix the conflict with the vp_modern_create_avq(). v2: 1. change the dma item of virtio-net, every item have MAX_SKB_FRAGS + 2 addr + len pairs. 2. introduce virtnet_sq_free_stats for __free_old_xmit v1: 1. rename transport_vq_config to vq_transport_config 2. virtio-net set dma meta number to (ring-size + 1)(MAX_SKB_FRGAS +2) 3. introduce virtqueue_dma_map_sg_attrs 4. separate vring_create_virtqueue to an independent commit Xuan Zhuo (10): virtio_ring: introduce vring_need_unmap_buffer virtio_ring: packed: remove double check of the unmap ops virtio_ring: packed: structure the indirect desc table virtio_ring: split: remove double check of the unmap ops virtio_ring: split: structure the indirect desc table virtio_ring: no store dma info when unmap is not needed virtio: find_vqs: add new parameter premapped virtio_ring: export premapped to driver by struct virtqueue virtio_net: set premapped mode by find_vqs() virtio_ring: virtqueue_set_dma_premapped support disable drivers/net/virtio_net.c | 57 +++-- drivers/virtio/virtio_ring.c | 436 +++++++++++++++++++++------------- include/linux/virtio.h | 3 +- include/linux/virtio_config.h | 17 +- 4 files changed, 307 insertions(+), 206 deletions(-) -- 2.32.0.3.g01195cf9f