| Message ID | 20231201104857.665737-1-dtatulea@nvidia.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | vdpa/mlx5: Add support for resumable vqs | expand |
On Fri, Dec 01, 2023 at 12:48:50PM +0200, Dragos Tatulea wrote: > Add support for resumable vqs in the driver. This is a firmware feature > that can be used for the following benefits: > - Full device .suspend/.resume. > - .set_map doesn't need to destroy and create new vqs anymore just to > update the map. When resumable vqs are supported it is enough to > suspend the vqs, set the new maps, and then resume the vqs. > > The first patch exposes the relevant bits in mlx5_ifc.h. That means it > needs to be applied to the mlx5-vhost tree [0] first. I didn't get this. Why does this need to go through that tree? Is there a dependency on some other commit from that tree? > Once applied > there, the change has to be pulled from mlx5-vhost into the vhost tree > and only then the remaining patches can be applied. Same flow as the vq > descriptor mappings patchset [1]. > > To be able to use resumable vqs properly, support for selectively modifying > vq parameters was needed. This is what the middle part of the series > consists of. > > [0] https://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git/log/?h=mlx5-vhost > [1] https://lore.kernel.org/virtualization/20231018171456.1624030-2-dtatulea@nvidia.com/ > > Dragos Tatulea (7): > vdpa/mlx5: Expose resumable vq capability > vdpa/mlx5: Split function into locked and unlocked variants > vdpa/mlx5: Allow modifying multiple vq fields in one modify command > vdpa/mlx5: Introduce per vq and device resume > vdpa/mlx5: Mark vq addrs for modification in hw vq > vdpa/mlx5: Mark vq state for modification in hw vq > vdpa/mlx5: Use vq suspend/resume during .set_map > > drivers/vdpa/mlx5/core/mr.c | 31 +++--- > drivers/vdpa/mlx5/net/mlx5_vnet.c | 172 +++++++++++++++++++++++++---- > include/linux/mlx5/mlx5_ifc.h | 3 +- > include/linux/mlx5/mlx5_ifc_vdpa.h | 4 + > 4 files changed, 174 insertions(+), 36 deletions(-) > > -- > 2.42.0
On Sat, 2023-12-02 at 15:26 -0500, Michael S. Tsirkin wrote: > On Fri, Dec 01, 2023 at 12:48:50PM +0200, Dragos Tatulea wrote: > > Add support for resumable vqs in the driver. This is a firmware feature > > that can be used for the following benefits: > > - Full device .suspend/.resume. > > - .set_map doesn't need to destroy and create new vqs anymore just to > > update the map. When resumable vqs are supported it is enough to > > suspend the vqs, set the new maps, and then resume the vqs. > > > > The first patch exposes the relevant bits in mlx5_ifc.h. That means it > > needs to be applied to the mlx5-vhost tree [0] first. > > I didn't get this. Why does this need to go through that tree? > Is there a dependency on some other commit from that tree? > To avoid merge issues in Linus's tree in mlx5_ifc.h. The idea is the same as for the "vq descriptor mappings" patchset [1]. Thanks, Dragos > > Once applied > > there, the change has to be pulled from mlx5-vhost into the vhost tree > > and only then the remaining patches can be applied. Same flow as the vq > > descriptor mappings patchset [1]. > > > > To be able to use resumable vqs properly, support for selectively modifying > > vq parameters was needed. This is what the middle part of the series > > consists of. > > > > [0] https://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git/log/?h=mlx5-vhost > > [1] https://lore.kernel.org/virtualization/20231018171456.1624030-2-dtatulea@nvidia.com/ > > > > Dragos Tatulea (7): > > vdpa/mlx5: Expose resumable vq capability > > vdpa/mlx5: Split function into locked and unlocked variants > > vdpa/mlx5: Allow modifying multiple vq fields in one modify command > > vdpa/mlx5: Introduce per vq and device resume > > vdpa/mlx5: Mark vq addrs for modification in hw vq > > vdpa/mlx5: Mark vq state for modification in hw vq > > vdpa/mlx5: Use vq suspend/resume during .set_map > > > > drivers/vdpa/mlx5/core/mr.c | 31 +++--- > > drivers/vdpa/mlx5/net/mlx5_vnet.c | 172 +++++++++++++++++++++++++---- > > include/linux/mlx5/mlx5_ifc.h | 3 +- > > include/linux/mlx5/mlx5_ifc_vdpa.h | 4 + > > 4 files changed, 174 insertions(+), 36 deletions(-) > > > > -- > > 2.42.0 >
On Sun, Dec 03, 2023 at 03:21:01PM +0000, Dragos Tatulea wrote: > On Sat, 2023-12-02 at 15:26 -0500, Michael S. Tsirkin wrote: > > On Fri, Dec 01, 2023 at 12:48:50PM +0200, Dragos Tatulea wrote: > > > Add support for resumable vqs in the driver. This is a firmware feature > > > that can be used for the following benefits: > > > - Full device .suspend/.resume. > > > - .set_map doesn't need to destroy and create new vqs anymore just to > > > update the map. When resumable vqs are supported it is enough to > > > suspend the vqs, set the new maps, and then resume the vqs. > > > > > > The first patch exposes the relevant bits in mlx5_ifc.h. That means it > > > needs to be applied to the mlx5-vhost tree [0] first. > > > > I didn't get this. Why does this need to go through that tree? > > Is there a dependency on some other commit from that tree? > > > To avoid merge issues in Linus's tree in mlx5_ifc.h. The idea is the same as for > the "vq descriptor mappings" patchset [1]. > > Thanks, > Dragos Are there other changes in that area that will cause non-trivial merge conflicts? > > > Once applied > > > there, the change has to be pulled from mlx5-vhost into the vhost tree > > > and only then the remaining patches can be applied. Same flow as the vq > > > descriptor mappings patchset [1]. > > > > > > To be able to use resumable vqs properly, support for selectively modifying > > > vq parameters was needed. This is what the middle part of the series > > > consists of. > > > > > > [0] https://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git/log/?h=mlx5-vhost > > > [1] https://lore.kernel.org/virtualization/20231018171456.1624030-2-dtatulea@nvidia.com/ > > > > > > Dragos Tatulea (7): > > > vdpa/mlx5: Expose resumable vq capability > > > vdpa/mlx5: Split function into locked and unlocked variants > > > vdpa/mlx5: Allow modifying multiple vq fields in one modify command > > > vdpa/mlx5: Introduce per vq and device resume > > > vdpa/mlx5: Mark vq addrs for modification in hw vq > > > vdpa/mlx5: Mark vq state for modification in hw vq > > > vdpa/mlx5: Use vq suspend/resume during .set_map > > > > > > drivers/vdpa/mlx5/core/mr.c | 31 +++--- > > > drivers/vdpa/mlx5/net/mlx5_vnet.c | 172 +++++++++++++++++++++++++---- > > > include/linux/mlx5/mlx5_ifc.h | 3 +- > > > include/linux/mlx5/mlx5_ifc_vdpa.h | 4 + > > > 4 files changed, 174 insertions(+), 36 deletions(-) > > > > > > -- > > > 2.42.0 > > >
On Sun, 2023-12-03 at 11:23 -0500, Michael S. Tsirkin wrote: > On Sun, Dec 03, 2023 at 03:21:01PM +0000, Dragos Tatulea wrote: > > On Sat, 2023-12-02 at 15:26 -0500, Michael S. Tsirkin wrote: > > > On Fri, Dec 01, 2023 at 12:48:50PM +0200, Dragos Tatulea wrote: > > > > Add support for resumable vqs in the driver. This is a firmware feature > > > > that can be used for the following benefits: > > > > - Full device .suspend/.resume. > > > > - .set_map doesn't need to destroy and create new vqs anymore just to > > > > update the map. When resumable vqs are supported it is enough to > > > > suspend the vqs, set the new maps, and then resume the vqs. > > > > > > > > The first patch exposes the relevant bits in mlx5_ifc.h. That means it > > > > needs to be applied to the mlx5-vhost tree [0] first. > > > > > > I didn't get this. Why does this need to go through that tree? > > > Is there a dependency on some other commit from that tree? > > > > > To avoid merge issues in Linus's tree in mlx5_ifc.h. The idea is the same as for > > the "vq descriptor mappings" patchset [1]. > > > > Thanks, > > Dragos > > Are there other changes in that area that will cause non-trivial merge > conflicts? > There are pending changes in mlx5_ifc.h for net-next. I haven't seen any changes around the touched structure but I would prefer not to take any risk. Thanks, Dragos > > > > Once applied > > > > there, the change has to be pulled from mlx5-vhost into the vhost tree > > > > and only then the remaining patches can be applied. Same flow as the vq > > > > descriptor mappings patchset [1]. > > > > > > > > To be able to use resumable vqs properly, support for selectively modifying > > > > vq parameters was needed. This is what the middle part of the series > > > > consists of. > > > > > > > > [0] https://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git/log/?h=mlx5-vhost > > > > [1] https://lore.kernel.org/virtualization/20231018171456.1624030-2-dtatulea@nvidia.com/ > > > > > > > > Dragos Tatulea (7): > > > > vdpa/mlx5: Expose resumable vq capability > > > > vdpa/mlx5: Split function into locked and unlocked variants > > > > vdpa/mlx5: Allow modifying multiple vq fields in one modify command > > > > vdpa/mlx5: Introduce per vq and device resume > > > > vdpa/mlx5: Mark vq addrs for modification in hw vq > > > > vdpa/mlx5: Mark vq state for modification in hw vq > > > > vdpa/mlx5: Use vq suspend/resume during .set_map > > > > > > > > drivers/vdpa/mlx5/core/mr.c | 31 +++--- > > > > drivers/vdpa/mlx5/net/mlx5_vnet.c | 172 +++++++++++++++++++++++++---- > > > > include/linux/mlx5/mlx5_ifc.h | 3 +- > > > > include/linux/mlx5/mlx5_ifc_vdpa.h | 4 + > > > > 4 files changed, 174 insertions(+), 36 deletions(-) > > > > > > > > -- > > > > 2.42.0 > > > > > >
On Mon, Dec 04, 2023 at 08:53:26AM +0000, Dragos Tatulea wrote: > On Sun, 2023-12-03 at 11:23 -0500, Michael S. Tsirkin wrote: > > On Sun, Dec 03, 2023 at 03:21:01PM +0000, Dragos Tatulea wrote: > > > On Sat, 2023-12-02 at 15:26 -0500, Michael S. Tsirkin wrote: > > > > On Fri, Dec 01, 2023 at 12:48:50PM +0200, Dragos Tatulea wrote: > > > > > Add support for resumable vqs in the driver. This is a firmware feature > > > > > that can be used for the following benefits: > > > > > - Full device .suspend/.resume. > > > > > - .set_map doesn't need to destroy and create new vqs anymore just to > > > > > update the map. When resumable vqs are supported it is enough to > > > > > suspend the vqs, set the new maps, and then resume the vqs. > > > > > > > > > > The first patch exposes the relevant bits in mlx5_ifc.h. That means it > > > > > needs to be applied to the mlx5-vhost tree [0] first. > > > > > > > > I didn't get this. Why does this need to go through that tree? > > > > Is there a dependency on some other commit from that tree? > > > > > > > To avoid merge issues in Linus's tree in mlx5_ifc.h. The idea is the same as for > > > the "vq descriptor mappings" patchset [1]. > > > > > > Thanks, > > > Dragos > > > > Are there other changes in that area that will cause non-trivial merge > > conflicts? > > > There are pending changes in mlx5_ifc.h for net-next. I haven't seen any changes > around the touched structure but I would prefer not to take any risk. > > Thanks, > Dragos This is exactly what linux-next is for. > > > > > Once applied > > > > > there, the change has to be pulled from mlx5-vhost into the vhost tree > > > > > and only then the remaining patches can be applied. Same flow as the vq > > > > > descriptor mappings patchset [1]. > > > > > > > > > > To be able to use resumable vqs properly, support for selectively modifying > > > > > vq parameters was needed. This is what the middle part of the series > > > > > consists of. > > > > > > > > > > [0] https://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git/log/?h=mlx5-vhost > > > > > [1] https://lore.kernel.org/virtualization/20231018171456.1624030-2-dtatulea@nvidia.com/ > > > > > > > > > > Dragos Tatulea (7): > > > > > vdpa/mlx5: Expose resumable vq capability > > > > > vdpa/mlx5: Split function into locked and unlocked variants > > > > > vdpa/mlx5: Allow modifying multiple vq fields in one modify command > > > > > vdpa/mlx5: Introduce per vq and device resume > > > > > vdpa/mlx5: Mark vq addrs for modification in hw vq > > > > > vdpa/mlx5: Mark vq state for modification in hw vq > > > > > vdpa/mlx5: Use vq suspend/resume during .set_map > > > > > > > > > > drivers/vdpa/mlx5/core/mr.c | 31 +++--- > > > > > drivers/vdpa/mlx5/net/mlx5_vnet.c | 172 +++++++++++++++++++++++++---- > > > > > include/linux/mlx5/mlx5_ifc.h | 3 +- > > > > > include/linux/mlx5/mlx5_ifc_vdpa.h | 4 + > > > > > 4 files changed, 174 insertions(+), 36 deletions(-) > > > > > > > > > > -- > > > > > 2.42.0 > > > > > > > > > >
On Mon, 2023-12-04 at 03:55 -0500, Michael S. Tsirkin wrote: > On Mon, Dec 04, 2023 at 08:53:26AM +0000, Dragos Tatulea wrote: > > On Sun, 2023-12-03 at 11:23 -0500, Michael S. Tsirkin wrote: > > > On Sun, Dec 03, 2023 at 03:21:01PM +0000, Dragos Tatulea wrote: > > > > On Sat, 2023-12-02 at 15:26 -0500, Michael S. Tsirkin wrote: > > > > > On Fri, Dec 01, 2023 at 12:48:50PM +0200, Dragos Tatulea wrote: > > > > > > Add support for resumable vqs in the driver. This is a firmware feature > > > > > > that can be used for the following benefits: > > > > > > - Full device .suspend/.resume. > > > > > > - .set_map doesn't need to destroy and create new vqs anymore just to > > > > > > update the map. When resumable vqs are supported it is enough to > > > > > > suspend the vqs, set the new maps, and then resume the vqs. > > > > > > > > > > > > The first patch exposes the relevant bits in mlx5_ifc.h. That means it > > > > > > needs to be applied to the mlx5-vhost tree [0] first. > > > > > > > > > > I didn't get this. Why does this need to go through that tree? > > > > > Is there a dependency on some other commit from that tree? > > > > > > > > > To avoid merge issues in Linus's tree in mlx5_ifc.h. The idea is the same as for > > > > the "vq descriptor mappings" patchset [1]. > > > > > > > > Thanks, > > > > Dragos > > > > > > Are there other changes in that area that will cause non-trivial merge > > > conflicts? > > > > > There are pending changes in mlx5_ifc.h for net-next. I haven't seen any changes > > around the touched structure but I would prefer not to take any risk. > > > > Thanks, > > Dragos > > This is exactly what linux-next is for. > Not sure what the suggestion is here. Is it: 1) To post patch 1/7 to net-next? Then we'd have to wait for a few weeks to make sure that it gets into the next tree. or 2) To apply it into the vhost tree directly? Then we run the risk of having merge issues. The "pull from branch" approach for cross subsystem changes was suggested by Linus this merge issue. [0] https://lore.kernel.org/all/CA+55aFxxoO=i7neGBRGW_afHsSZ7K-x6fMO8v-8po3Ls_Ew0Rg@mail.gmail.com/ Thanks, Dragos > > > > > > > Once applied > > > > > > there, the change has to be pulled from mlx5-vhost into the vhost tree > > > > > > and only then the remaining patches can be applied. Same flow as the vq > > > > > > descriptor mappings patchset [1]. > > > > > > > > > > > > To be able to use resumable vqs properly, support for selectively modifying > > > > > > vq parameters was needed. This is what the middle part of the series > > > > > > consists of. > > > > > > > > > > > > [0] https://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git/log/?h=mlx5-vhost > > > > > > [1] https://lore.kernel.org/virtualization/20231018171456.1624030-2-dtatulea@nvidia.com/ > > > > > > > > > > > > Dragos Tatulea (7): > > > > > > vdpa/mlx5: Expose resumable vq capability > > > > > > vdpa/mlx5: Split function into locked and unlocked variants > > > > > > vdpa/mlx5: Allow modifying multiple vq fields in one modify command > > > > > > vdpa/mlx5: Introduce per vq and device resume > > > > > > vdpa/mlx5: Mark vq addrs for modification in hw vq > > > > > > vdpa/mlx5: Mark vq state for modification in hw vq > > > > > > vdpa/mlx5: Use vq suspend/resume during .set_map > > > > > > > > > > > > drivers/vdpa/mlx5/core/mr.c | 31 +++--- > > > > > > drivers/vdpa/mlx5/net/mlx5_vnet.c | 172 +++++++++++++++++++++++++---- > > > > > > include/linux/mlx5/mlx5_ifc.h | 3 +- > > > > > > include/linux/mlx5/mlx5_ifc_vdpa.h | 4 + > > > > > > 4 files changed, 174 insertions(+), 36 deletions(-) > > > > > > > > > > > > -- > > > > > > 2.42.0 > > > > > > > > > > > > > > >
On Mon, Dec 04, 2023 at 09:16:07AM +0000, Dragos Tatulea wrote: > On Mon, 2023-12-04 at 03:55 -0500, Michael S. Tsirkin wrote: > > On Mon, Dec 04, 2023 at 08:53:26AM +0000, Dragos Tatulea wrote: > > > On Sun, 2023-12-03 at 11:23 -0500, Michael S. Tsirkin wrote: > > > > On Sun, Dec 03, 2023 at 03:21:01PM +0000, Dragos Tatulea wrote: > > > > > On Sat, 2023-12-02 at 15:26 -0500, Michael S. Tsirkin wrote: > > > > > > On Fri, Dec 01, 2023 at 12:48:50PM +0200, Dragos Tatulea wrote: > > > > > > > Add support for resumable vqs in the driver. This is a firmware feature > > > > > > > that can be used for the following benefits: > > > > > > > - Full device .suspend/.resume. > > > > > > > - .set_map doesn't need to destroy and create new vqs anymore just to > > > > > > > update the map. When resumable vqs are supported it is enough to > > > > > > > suspend the vqs, set the new maps, and then resume the vqs. > > > > > > > > > > > > > > The first patch exposes the relevant bits in mlx5_ifc.h. That means it > > > > > > > needs to be applied to the mlx5-vhost tree [0] first. > > > > > > > > > > > > I didn't get this. Why does this need to go through that tree? > > > > > > Is there a dependency on some other commit from that tree? > > > > > > > > > > > To avoid merge issues in Linus's tree in mlx5_ifc.h. The idea is the same as for > > > > > the "vq descriptor mappings" patchset [1]. > > > > > > > > > > Thanks, > > > > > Dragos > > > > > > > > Are there other changes in that area that will cause non-trivial merge > > > > conflicts? > > > > > > > There are pending changes in mlx5_ifc.h for net-next. I haven't seen any changes > > > around the touched structure but I would prefer not to take any risk. > > > > > > Thanks, > > > Dragos > > > > This is exactly what linux-next is for. > > > Not sure what the suggestion is here. Is it: > > 1) To post patch 1/7 to net-next? Then we'd have to wait for a few weeks to make > sure that it gets into the next tree. > > or > > 2) To apply it into the vhost tree directly? Then we run the risk of having > merge issues. > > The "pull from branch" approach for cross subsystem changes was suggested by > Linus this merge issue. > > [0] > https://lore.kernel.org/all/CA+55aFxxoO=i7neGBRGW_afHsSZ7K-x6fMO8v-8po3Ls_Ew0Rg@mail.gmail.com/ > > Thanks, > Dragos I will park this in my tree for now so it can get testing in linux next. When it's available in some other tree as well, let me know and I'll figure it out.
On Mon, 2023-12-04 at 06:04 -0500, Michael S. Tsirkin wrote: > On Mon, Dec 04, 2023 at 09:16:07AM +0000, Dragos Tatulea wrote: > > On Mon, 2023-12-04 at 03:55 -0500, Michael S. Tsirkin wrote: > > > On Mon, Dec 04, 2023 at 08:53:26AM +0000, Dragos Tatulea wrote: > > > > On Sun, 2023-12-03 at 11:23 -0500, Michael S. Tsirkin wrote: > > > > > On Sun, Dec 03, 2023 at 03:21:01PM +0000, Dragos Tatulea wrote: > > > > > > On Sat, 2023-12-02 at 15:26 -0500, Michael S. Tsirkin wrote: > > > > > > > On Fri, Dec 01, 2023 at 12:48:50PM +0200, Dragos Tatulea wrote: > > > > > > > > Add support for resumable vqs in the driver. This is a firmware feature > > > > > > > > that can be used for the following benefits: > > > > > > > > - Full device .suspend/.resume. > > > > > > > > - .set_map doesn't need to destroy and create new vqs anymore just to > > > > > > > > update the map. When resumable vqs are supported it is enough to > > > > > > > > suspend the vqs, set the new maps, and then resume the vqs. > > > > > > > > > > > > > > > > The first patch exposes the relevant bits in mlx5_ifc.h. That means it > > > > > > > > needs to be applied to the mlx5-vhost tree [0] first. > > > > > > > > > > > > > > I didn't get this. Why does this need to go through that tree? > > > > > > > Is there a dependency on some other commit from that tree? > > > > > > > > > > > > > To avoid merge issues in Linus's tree in mlx5_ifc.h. The idea is the same as for > > > > > > the "vq descriptor mappings" patchset [1]. > > > > > > > > > > > > Thanks, > > > > > > Dragos > > > > > > > > > > Are there other changes in that area that will cause non-trivial merge > > > > > conflicts? > > > > > > > > > There are pending changes in mlx5_ifc.h for net-next. I haven't seen any changes > > > > around the touched structure but I would prefer not to take any risk. > > > > > > > > Thanks, > > > > Dragos > > > > > > This is exactly what linux-next is for. > > > > > Not sure what the suggestion is here. Is it: > > > > 1) To post patch 1/7 to net-next? Then we'd have to wait for a few weeks to make > > sure that it gets into the next tree. > > > > or > > > > 2) To apply it into the vhost tree directly? Then we run the risk of having > > merge issues. > > > > The "pull from branch" approach for cross subsystem changes was suggested by > > Linus this merge issue. > > > > [0] > > https://lore.kernel.org/all/CA+55aFxxoO=i7neGBRGW_afHsSZ7K-x6fMO8v-8po3Ls_Ew0Rg@mail.gmail.com/ > > > > Thanks, > > Dragos > > I will park this in my tree for now so it can get testing in linux next. > When it's available in some other tree as well, let me know and > I'll figure it out. > Thanks! But don't park it just yet. I would like to send a v2 in the next few days that contains 2 more patches on top. I've sent the v1 early to get reviews for the bulk of the work. Forgot to mention that in the cover letter. My bad, sorry. Thanks, Dragos
On Fri, 01 Dec 2023 12:48:50 +0200, Dragos Tatulea wrote: > Add support for resumable vqs in the driver. This is a firmware feature > that can be used for the following benefits: > - Full device .suspend/.resume. > - .set_map doesn't need to destroy and create new vqs anymore just to > update the map. When resumable vqs are supported it is enough to > suspend the vqs, set the new maps, and then resume the vqs. > > [...] Applied, thanks! [1/7] vdpa/mlx5: Expose resumable vq capability https://git.kernel.org/rdma/rdma/c/b24910e1be0e76 Best regards,