| Message ID | 20231219180858.120898-1-dtatulea@nvidia.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | vdpa/mlx5: Add support for resumable vqs | expand |
On Tue, Dec 19, 2023 at 08:08:43PM +0200, Dragos Tatulea wrote: > Add support for resumable vqs in the mlx5_vdpa driver. This is a > firmware feature that can be used for the following benefits: > - Full device .suspend/.resume. > - .set_map doesn't need to destroy and create new vqs anymore just to > update the map. When resumable vqs are supported it is enough to > suspend the vqs, set the new maps, and then resume the vqs. > > The first patch exposes relevant bits for the feature in mlx5_ifc.h. > That means it needs to be applied to the mlx5-vhost tree [0] first. Once > applied there, the change has to be pulled from mlx5-vhost into the > vhost tree and only then the remaining patches can be applied. Same flow > as the vq descriptor mappings patchset [1]. > > The second part implements the vdpa backend feature support to allow > vq state and address changes when the device is in DRIVER_OK state and > suspended. > > The third part adds support for seletively modifying vq parameters. This > is needed to be able to use resumable vqs. > > Then the actual support for resumable vqs is added. > > The last part of the series introduces reference counting for mrs which > is necessary to avoid freeing mkeys too early or leaking them. I lost track. Are you going to send v5 or not? > * Changes in v4: > - Added vdpa backend feature support for changing vq properties in > DRIVER_OK when device is suspended. Added support in the driver as > well. > - Dropped Acked-by for the patches that had the tag mistakenly > added. > > * Changes in v3: > - Faulty version. Please ignore. > > * Changes in v2: > - Added mr refcounting patches. > - Deleted unnecessary patch: "vdpa/mlx5: Split function into locked and > unlocked variants" > - Small print improvement in "Introduce per vq and device resume" > patch. > - Patch 1/7 has been applied to mlx5-vhost branch. > > > Dragos Tatulea (15): > vdpa: Add VHOST_BACKEND_F_CHANGEABLE_VQ_ADDR_IN_SUSPEND flag > vdpa: Add VHOST_BACKEND_F_CHANGEABLE_VQ_STATE_IN_SUSPEND flag > vdpa: Accept VHOST_BACKEND_F_CHANGEABLE_VQ_ADDR_IN_SUSPEND backend > feature > vdpa: Accept VHOST_BACKEND_F_CHANGEABLE_VQ_STATE_IN_SUSPEND backend > feature > vdpa: Track device suspended state > vdpa: Block vq address change in DRIVER_OK unless device supports it > vdpa: Block vq state change in DRIVER_OK unless device supports it > vdpa/mlx5: Expose resumable vq capability > vdpa/mlx5: Allow modifying multiple vq fields in one modify command > vdpa/mlx5: Introduce per vq and device resume > vdpa/mlx5: Mark vq addrs for modification in hw vq > vdpa/mlx5: Mark vq state for modification in hw vq > vdpa/mlx5: Use vq suspend/resume during .set_map > vdpa/mlx5: Introduce reference counting to mrs > vdpa/mlx5: Add mkey leak detection > > drivers/vdpa/mlx5/core/mlx5_vdpa.h | 10 +- > drivers/vdpa/mlx5/core/mr.c | 69 +++++++-- > drivers/vdpa/mlx5/net/mlx5_vnet.c | 218 ++++++++++++++++++++++++++--- > drivers/vhost/vdpa.c | 51 ++++++- > include/linux/mlx5/mlx5_ifc.h | 3 +- > include/linux/mlx5/mlx5_ifc_vdpa.h | 4 + > include/uapi/linux/vhost_types.h | 8 ++ > 7 files changed, 322 insertions(+), 41 deletions(-) > > -- > 2.43.0
On Mon, 2023-12-25 at 09:41 -0500, Michael S. Tsirkin wrote: > On Tue, Dec 19, 2023 at 08:08:43PM +0200, Dragos Tatulea wrote: > > Add support for resumable vqs in the mlx5_vdpa driver. This is a > > firmware feature that can be used for the following benefits: > > - Full device .suspend/.resume. > > - .set_map doesn't need to destroy and create new vqs anymore just to > > update the map. When resumable vqs are supported it is enough to > > suspend the vqs, set the new maps, and then resume the vqs. > > > > The first patch exposes relevant bits for the feature in mlx5_ifc.h. > > That means it needs to be applied to the mlx5-vhost tree [0] first. Once > > applied there, the change has to be pulled from mlx5-vhost into the > > vhost tree and only then the remaining patches can be applied. Same flow > > as the vq descriptor mappings patchset [1]. > > > > The second part implements the vdpa backend feature support to allow > > vq state and address changes when the device is in DRIVER_OK state and > > suspended. > > > > The third part adds support for seletively modifying vq parameters. This > > is needed to be able to use resumable vqs. > > > > Then the actual support for resumable vqs is added. > > > > The last part of the series introduces reference counting for mrs which > > is necessary to avoid freeing mkeys too early or leaking them. > > > I lost track. Are you going to send v5 or not? > I was waiting for your answer if I should send it or not. I suppose this means yes. I will send it in a few minutes. > > * Changes in v4: > > - Added vdpa backend feature support for changing vq properties in > > DRIVER_OK when device is suspended. Added support in the driver as > > well. > > - Dropped Acked-by for the patches that had the tag mistakenly > > added. > > > > * Changes in v3: > > - Faulty version. Please ignore. > > > > * Changes in v2: > > - Added mr refcounting patches. > > - Deleted unnecessary patch: "vdpa/mlx5: Split function into locked and > > unlocked variants" > > - Small print improvement in "Introduce per vq and device resume" > > patch. > > - Patch 1/7 has been applied to mlx5-vhost branch. > > > > > > Dragos Tatulea (15): > > vdpa: Add VHOST_BACKEND_F_CHANGEABLE_VQ_ADDR_IN_SUSPEND flag > > vdpa: Add VHOST_BACKEND_F_CHANGEABLE_VQ_STATE_IN_SUSPEND flag > > vdpa: Accept VHOST_BACKEND_F_CHANGEABLE_VQ_ADDR_IN_SUSPEND backend > > feature > > vdpa: Accept VHOST_BACKEND_F_CHANGEABLE_VQ_STATE_IN_SUSPEND backend > > feature > > vdpa: Track device suspended state > > vdpa: Block vq address change in DRIVER_OK unless device supports it > > vdpa: Block vq state change in DRIVER_OK unless device supports it > > vdpa/mlx5: Expose resumable vq capability > > vdpa/mlx5: Allow modifying multiple vq fields in one modify command > > vdpa/mlx5: Introduce per vq and device resume > > vdpa/mlx5: Mark vq addrs for modification in hw vq > > vdpa/mlx5: Mark vq state for modification in hw vq > > vdpa/mlx5: Use vq suspend/resume during .set_map > > vdpa/mlx5: Introduce reference counting to mrs > > vdpa/mlx5: Add mkey leak detection > > > > drivers/vdpa/mlx5/core/mlx5_vdpa.h | 10 +- > > drivers/vdpa/mlx5/core/mr.c | 69 +++++++-- > > drivers/vdpa/mlx5/net/mlx5_vnet.c | 218 ++++++++++++++++++++++++++--- > > drivers/vhost/vdpa.c | 51 ++++++- > > include/linux/mlx5/mlx5_ifc.h | 3 +- > > include/linux/mlx5/mlx5_ifc_vdpa.h | 4 + > > include/uapi/linux/vhost_types.h | 8 ++ > > 7 files changed, 322 insertions(+), 41 deletions(-) > > > > -- > > 2.43.0 >
Add support for resumable vqs in the mlx5_vdpa driver. This is a firmware feature that can be used for the following benefits: - Full device .suspend/.resume. - .set_map doesn't need to destroy and create new vqs anymore just to update the map. When resumable vqs are supported it is enough to suspend the vqs, set the new maps, and then resume the vqs. The first patch exposes relevant bits for the feature in mlx5_ifc.h. That means it needs to be applied to the mlx5-vhost tree [0] first. Once applied there, the change has to be pulled from mlx5-vhost into the vhost tree and only then the remaining patches can be applied. Same flow as the vq descriptor mappings patchset [1]. The second part implements the vdpa backend feature support to allow vq state and address changes when the device is in DRIVER_OK state and suspended. The third part adds support for seletively modifying vq parameters. This is needed to be able to use resumable vqs. Then the actual support for resumable vqs is added. The last part of the series introduces reference counting for mrs which is necessary to avoid freeing mkeys too early or leaking them. * Changes in v4: - Added vdpa backend feature support for changing vq properties in DRIVER_OK when device is suspended. Added support in the driver as well. - Dropped Acked-by for the patches that had the tag mistakenly added. * Changes in v3: - Faulty version. Please ignore. * Changes in v2: - Added mr refcounting patches. - Deleted unnecessary patch: "vdpa/mlx5: Split function into locked and unlocked variants" - Small print improvement in "Introduce per vq and device resume" patch. - Patch 1/7 has been applied to mlx5-vhost branch. Dragos Tatulea (15): vdpa: Add VHOST_BACKEND_F_CHANGEABLE_VQ_ADDR_IN_SUSPEND flag vdpa: Add VHOST_BACKEND_F_CHANGEABLE_VQ_STATE_IN_SUSPEND flag vdpa: Accept VHOST_BACKEND_F_CHANGEABLE_VQ_ADDR_IN_SUSPEND backend feature vdpa: Accept VHOST_BACKEND_F_CHANGEABLE_VQ_STATE_IN_SUSPEND backend feature vdpa: Track device suspended state vdpa: Block vq address change in DRIVER_OK unless device supports it vdpa: Block vq state change in DRIVER_OK unless device supports it vdpa/mlx5: Expose resumable vq capability vdpa/mlx5: Allow modifying multiple vq fields in one modify command vdpa/mlx5: Introduce per vq and device resume vdpa/mlx5: Mark vq addrs for modification in hw vq vdpa/mlx5: Mark vq state for modification in hw vq vdpa/mlx5: Use vq suspend/resume during .set_map vdpa/mlx5: Introduce reference counting to mrs vdpa/mlx5: Add mkey leak detection drivers/vdpa/mlx5/core/mlx5_vdpa.h | 10 +- drivers/vdpa/mlx5/core/mr.c | 69 +++++++-- drivers/vdpa/mlx5/net/mlx5_vnet.c | 218 ++++++++++++++++++++++++++--- drivers/vhost/vdpa.c | 51 ++++++- include/linux/mlx5/mlx5_ifc.h | 3 +- include/linux/mlx5/mlx5_ifc_vdpa.h | 4 + include/uapi/linux/vhost_types.h | 8 ++ 7 files changed, 322 insertions(+), 41 deletions(-)