| Message ID | 20230912130132.561193-1-dtatulea@nvidia.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | vdpa: Add support for vq descriptor mappings | expand |
Hi Dragos, Eugenio and Si-Wei My name is Lei Yang, a software Quality Engineer from Red Hat. And always paying attention to improving the live migration downtime issues because there are others QE asked about this problem when I share live migration status recently. Therefore I would like to test it in my environment. Before the testing I want to know if there is an expectation of downtime range based on this series of patches? In addition, QE also can help do a regression test based on this series of patches if there is a requirement. Regards and thanks Lei On Tue, Sep 12, 2023 at 9:04 PM Dragos Tatulea <dtatulea@nvidia.com> wrote: > > This patch series adds support for vq descriptor table mappings which > are used to improve vdpa live migration downtime. The improvement comes > from using smaller mappings which take less time to create and destroy > in hw. > > The first part adds the vdpa core changes from Si-Wei [0]. > > The second part adds support in mlx5_vdpa: > - Refactor the mr code to be able to cleanly add descriptor mappings. > - Add hardware descriptor mr support. > - Properly update iotlb for cvq during ASID switch. > > [0] https://lore.kernel.org/virtualization/1694248959-13369-1-git-send-email-si-wei.liu@oracle.com > > Dragos Tatulea (13): > vdpa/mlx5: Create helper function for dma mappings > vdpa/mlx5: Decouple cvq iotlb handling from hw mapping code > vdpa/mlx5: Take cvq iotlb lock during refresh > vdpa/mlx5: Collapse "dvq" mr add/delete functions > vdpa/mlx5: Rename mr destroy functions > vdpa/mlx5: Allow creation/deletion of any given mr struct > vdpa/mlx5: Move mr mutex out of mr struct > vdpa/mlx5: Improve mr update flow > vdpa/mlx5: Introduce mr for vq descriptor > vdpa/mlx5: Enable hw support for vq descriptor mapping > vdpa/mlx5: Make iotlb helper functions more generic > vdpa/mlx5: Update cvq iotlb mapping on ASID change > Cover letter: vdpa/mlx5: Add support for vq descriptor mappings > > Si-Wei Liu (3): > vdpa: introduce dedicated descriptor group for virtqueue > vhost-vdpa: introduce descriptor group backend feature > vhost-vdpa: uAPI to get dedicated descriptor group id > > drivers/vdpa/mlx5/core/mlx5_vdpa.h | 31 +++-- > drivers/vdpa/mlx5/core/mr.c | 191 ++++++++++++++++------------- > drivers/vdpa/mlx5/core/resources.c | 6 +- > drivers/vdpa/mlx5/net/mlx5_vnet.c | 100 ++++++++++----- > drivers/vhost/vdpa.c | 27 ++++ > include/linux/mlx5/mlx5_ifc.h | 8 +- > include/linux/mlx5/mlx5_ifc_vdpa.h | 7 +- > include/linux/vdpa.h | 11 ++ > include/uapi/linux/vhost.h | 8 ++ > include/uapi/linux/vhost_types.h | 5 + > 10 files changed, 264 insertions(+), 130 deletions(-) > > -- > 2.41.0 >
On Wed, Sep 13, 2023 at 3:03 AM Lei Yang <leiyang@redhat.com> wrote: > > Hi Dragos, Eugenio and Si-Wei > > My name is Lei Yang, a software Quality Engineer from Red Hat. And > always paying attention to improving the live migration downtime > issues because there are others QE asked about this problem when I > share live migration status recently. Therefore I would like to test > it in my environment. Before the testing I want to know if there is an > expectation of downtime range based on this series of patches? In > addition, QE also can help do a regression test based on this series > of patches if there is a requirement. > Hi Lei, Thanks for offering the testing bandwidth! I think we can only do regression tests here, as the userland part is still not sent to qemu. > Regards and thanks > Lei > > > On Tue, Sep 12, 2023 at 9:04 PM Dragos Tatulea <dtatulea@nvidia.com> wrote: > > > > This patch series adds support for vq descriptor table mappings which > > are used to improve vdpa live migration downtime. The improvement comes > > from using smaller mappings which take less time to create and destroy > > in hw. > > > > The first part adds the vdpa core changes from Si-Wei [0]. > > > > The second part adds support in mlx5_vdpa: > > - Refactor the mr code to be able to cleanly add descriptor mappings. > > - Add hardware descriptor mr support. > > - Properly update iotlb for cvq during ASID switch. > > > > [0] https://lore.kernel.org/virtualization/1694248959-13369-1-git-send-email-si-wei.liu@oracle.com > > > > Dragos Tatulea (13): > > vdpa/mlx5: Create helper function for dma mappings > > vdpa/mlx5: Decouple cvq iotlb handling from hw mapping code > > vdpa/mlx5: Take cvq iotlb lock during refresh > > vdpa/mlx5: Collapse "dvq" mr add/delete functions > > vdpa/mlx5: Rename mr destroy functions > > vdpa/mlx5: Allow creation/deletion of any given mr struct > > vdpa/mlx5: Move mr mutex out of mr struct > > vdpa/mlx5: Improve mr update flow > > vdpa/mlx5: Introduce mr for vq descriptor > > vdpa/mlx5: Enable hw support for vq descriptor mapping > > vdpa/mlx5: Make iotlb helper functions more generic > > vdpa/mlx5: Update cvq iotlb mapping on ASID change > > Cover letter: vdpa/mlx5: Add support for vq descriptor mappings > > > > Si-Wei Liu (3): > > vdpa: introduce dedicated descriptor group for virtqueue > > vhost-vdpa: introduce descriptor group backend feature > > vhost-vdpa: uAPI to get dedicated descriptor group id > > > > drivers/vdpa/mlx5/core/mlx5_vdpa.h | 31 +++-- > > drivers/vdpa/mlx5/core/mr.c | 191 ++++++++++++++++------------- > > drivers/vdpa/mlx5/core/resources.c | 6 +- > > drivers/vdpa/mlx5/net/mlx5_vnet.c | 100 ++++++++++----- > > drivers/vhost/vdpa.c | 27 ++++ > > include/linux/mlx5/mlx5_ifc.h | 8 +- > > include/linux/mlx5/mlx5_ifc_vdpa.h | 7 +- > > include/linux/vdpa.h | 11 ++ > > include/uapi/linux/vhost.h | 8 ++ > > include/uapi/linux/vhost_types.h | 5 + > > 10 files changed, 264 insertions(+), 130 deletions(-) > > > > -- > > 2.41.0 > > >
On 9/13/2023 9:08 AM, Eugenio Perez Martin wrote: > On Wed, Sep 13, 2023 at 3:03 AM Lei Yang <leiyang@redhat.com> wrote: >> Hi Dragos, Eugenio and Si-Wei >> >> My name is Lei Yang, a software Quality Engineer from Red Hat. And >> always paying attention to improving the live migration downtime >> issues because there are others QE asked about this problem when I >> share live migration status recently. Therefore I would like to test >> it in my environment. Before the testing I want to know if there is an >> expectation of downtime range based on this series of patches? In >> addition, QE also can help do a regression test based on this series >> of patches if there is a requirement. >> > Hi Lei, > > Thanks for offering the testing bandwidth! > > I think we can only do regression tests here, as the userland part is > still not sent to qemu. Right. Regression for now, even QEMU has it, to exercise the relevant feature it would need a supporting firmware that is not yet available for now. Just stay tuned. thanks for your patience, -Siwei > >> Regards and thanks >> Lei >> >> >> On Tue, Sep 12, 2023 at 9:04 PM Dragos Tatulea <dtatulea@nvidia.com> wrote: >>> This patch series adds support for vq descriptor table mappings which >>> are used to improve vdpa live migration downtime. The improvement comes >>> from using smaller mappings which take less time to create and destroy >>> in hw. >>> >>> The first part adds the vdpa core changes from Si-Wei [0]. >>> >>> The second part adds support in mlx5_vdpa: >>> - Refactor the mr code to be able to cleanly add descriptor mappings. >>> - Add hardware descriptor mr support. >>> - Properly update iotlb for cvq during ASID switch. >>> >>> [0] https://lore.kernel.org/virtualization/1694248959-13369-1-git-send-email-si-wei.liu@oracle.com >>> >>> Dragos Tatulea (13): >>> vdpa/mlx5: Create helper function for dma mappings >>> vdpa/mlx5: Decouple cvq iotlb handling from hw mapping code >>> vdpa/mlx5: Take cvq iotlb lock during refresh >>> vdpa/mlx5: Collapse "dvq" mr add/delete functions >>> vdpa/mlx5: Rename mr destroy functions >>> vdpa/mlx5: Allow creation/deletion of any given mr struct >>> vdpa/mlx5: Move mr mutex out of mr struct >>> vdpa/mlx5: Improve mr update flow >>> vdpa/mlx5: Introduce mr for vq descriptor >>> vdpa/mlx5: Enable hw support for vq descriptor mapping >>> vdpa/mlx5: Make iotlb helper functions more generic >>> vdpa/mlx5: Update cvq iotlb mapping on ASID change >>> Cover letter: vdpa/mlx5: Add support for vq descriptor mappings >>> >>> Si-Wei Liu (3): >>> vdpa: introduce dedicated descriptor group for virtqueue >>> vhost-vdpa: introduce descriptor group backend feature >>> vhost-vdpa: uAPI to get dedicated descriptor group id >>> >>> drivers/vdpa/mlx5/core/mlx5_vdpa.h | 31 +++-- >>> drivers/vdpa/mlx5/core/mr.c | 191 ++++++++++++++++------------- >>> drivers/vdpa/mlx5/core/resources.c | 6 +- >>> drivers/vdpa/mlx5/net/mlx5_vnet.c | 100 ++++++++++----- >>> drivers/vhost/vdpa.c | 27 ++++ >>> include/linux/mlx5/mlx5_ifc.h | 8 +- >>> include/linux/mlx5/mlx5_ifc_vdpa.h | 7 +- >>> include/linux/vdpa.h | 11 ++ >>> include/uapi/linux/vhost.h | 8 ++ >>> include/uapi/linux/vhost_types.h | 5 + >>> 10 files changed, 264 insertions(+), 130 deletions(-) >>> >>> -- >>> 2.41.0 >>>
QE tested this series with regression testing on real nic, there are no new issues. Tested-by: Lei Yang <leiyang@redhat.com> On Fri, Sep 15, 2023 at 2:34 PM Si-Wei Liu <si-wei.liu@oracle.com> wrote: > > > > On 9/13/2023 9:08 AM, Eugenio Perez Martin wrote: > > On Wed, Sep 13, 2023 at 3:03 AM Lei Yang <leiyang@redhat.com> wrote: > >> Hi Dragos, Eugenio and Si-Wei > >> > >> My name is Lei Yang, a software Quality Engineer from Red Hat. And > >> always paying attention to improving the live migration downtime > >> issues because there are others QE asked about this problem when I > >> share live migration status recently. Therefore I would like to test > >> it in my environment. Before the testing I want to know if there is an > >> expectation of downtime range based on this series of patches? In > >> addition, QE also can help do a regression test based on this series > >> of patches if there is a requirement. > >> > > Hi Lei, > > > > Thanks for offering the testing bandwidth! > > > > I think we can only do regression tests here, as the userland part is > > still not sent to qemu. > Right. Regression for now, even QEMU has it, to exercise the relevant > feature it would need a supporting firmware that is not yet available > for now. Just stay tuned. > > thanks for your patience, > -Siwei > > > >> Regards and thanks > >> Lei > >> > >> > >> On Tue, Sep 12, 2023 at 9:04 PM Dragos Tatulea <dtatulea@nvidia.com> wrote: > >>> This patch series adds support for vq descriptor table mappings which > >>> are used to improve vdpa live migration downtime. The improvement comes > >>> from using smaller mappings which take less time to create and destroy > >>> in hw. > >>> > >>> The first part adds the vdpa core changes from Si-Wei [0]. > >>> > >>> The second part adds support in mlx5_vdpa: > >>> - Refactor the mr code to be able to cleanly add descriptor mappings. > >>> - Add hardware descriptor mr support. > >>> - Properly update iotlb for cvq during ASID switch. > >>> > >>> [0] https://lore.kernel.org/virtualization/1694248959-13369-1-git-send-email-si-wei.liu@oracle.com > >>> > >>> Dragos Tatulea (13): > >>> vdpa/mlx5: Create helper function for dma mappings > >>> vdpa/mlx5: Decouple cvq iotlb handling from hw mapping code > >>> vdpa/mlx5: Take cvq iotlb lock during refresh > >>> vdpa/mlx5: Collapse "dvq" mr add/delete functions > >>> vdpa/mlx5: Rename mr destroy functions > >>> vdpa/mlx5: Allow creation/deletion of any given mr struct > >>> vdpa/mlx5: Move mr mutex out of mr struct > >>> vdpa/mlx5: Improve mr update flow > >>> vdpa/mlx5: Introduce mr for vq descriptor > >>> vdpa/mlx5: Enable hw support for vq descriptor mapping > >>> vdpa/mlx5: Make iotlb helper functions more generic > >>> vdpa/mlx5: Update cvq iotlb mapping on ASID change > >>> Cover letter: vdpa/mlx5: Add support for vq descriptor mappings > >>> > >>> Si-Wei Liu (3): > >>> vdpa: introduce dedicated descriptor group for virtqueue > >>> vhost-vdpa: introduce descriptor group backend feature > >>> vhost-vdpa: uAPI to get dedicated descriptor group id > >>> > >>> drivers/vdpa/mlx5/core/mlx5_vdpa.h | 31 +++-- > >>> drivers/vdpa/mlx5/core/mr.c | 191 ++++++++++++++++------------- > >>> drivers/vdpa/mlx5/core/resources.c | 6 +- > >>> drivers/vdpa/mlx5/net/mlx5_vnet.c | 100 ++++++++++----- > >>> drivers/vhost/vdpa.c | 27 ++++ > >>> include/linux/mlx5/mlx5_ifc.h | 8 +- > >>> include/linux/mlx5/mlx5_ifc_vdpa.h | 7 +- > >>> include/linux/vdpa.h | 11 ++ > >>> include/uapi/linux/vhost.h | 8 ++ > >>> include/uapi/linux/vhost_types.h | 5 + > >>> 10 files changed, 264 insertions(+), 130 deletions(-) > >>> > >>> -- > >>> 2.41.0 > >>> >
On Tue, 2023-09-12 at 16:01 +0300, Dragos Tatulea wrote: > This patch series adds support for vq descriptor table mappings which > are used to improve vdpa live migration downtime. The improvement comes > from using smaller mappings which take less time to create and destroy > in hw. > Gentle ping. Note that I will have to send a v2. The changes in mlx5_ifc.h will need to be merged first separately into the mlx5-next branch [0] and then pulled from there when the series is applied. [0] https://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git/log/?h=mlx5-next Thanks, Dragos > The first part adds the vdpa core changes from Si-Wei [0]. > > The second part adds support in mlx5_vdpa: > - Refactor the mr code to be able to cleanly add descriptor mappings. > - Add hardware descriptor mr support. > - Properly update iotlb for cvq during ASID switch. > > [0] > https://lore.kernel.org/virtualization/1694248959-13369-1-git-send-email-si-wei.liu@oracle.com > > Dragos Tatulea (13): > vdpa/mlx5: Create helper function for dma mappings > vdpa/mlx5: Decouple cvq iotlb handling from hw mapping code > vdpa/mlx5: Take cvq iotlb lock during refresh > vdpa/mlx5: Collapse "dvq" mr add/delete functions > vdpa/mlx5: Rename mr destroy functions > vdpa/mlx5: Allow creation/deletion of any given mr struct > vdpa/mlx5: Move mr mutex out of mr struct > vdpa/mlx5: Improve mr update flow > vdpa/mlx5: Introduce mr for vq descriptor > vdpa/mlx5: Enable hw support for vq descriptor mapping > vdpa/mlx5: Make iotlb helper functions more generic > vdpa/mlx5: Update cvq iotlb mapping on ASID change > Cover letter: vdpa/mlx5: Add support for vq descriptor mappings > > Si-Wei Liu (3): > vdpa: introduce dedicated descriptor group for virtqueue > vhost-vdpa: introduce descriptor group backend feature > vhost-vdpa: uAPI to get dedicated descriptor group id > > drivers/vdpa/mlx5/core/mlx5_vdpa.h | 31 +++-- > drivers/vdpa/mlx5/core/mr.c | 191 ++++++++++++++++------------- > drivers/vdpa/mlx5/core/resources.c | 6 +- > drivers/vdpa/mlx5/net/mlx5_vnet.c | 100 ++++++++++----- > drivers/vhost/vdpa.c | 27 ++++ > include/linux/mlx5/mlx5_ifc.h | 8 +- > include/linux/mlx5/mlx5_ifc_vdpa.h | 7 +- > include/linux/vdpa.h | 11 ++ > include/uapi/linux/vhost.h | 8 ++ > include/uapi/linux/vhost_types.h | 5 + > 10 files changed, 264 insertions(+), 130 deletions(-) >
On 9/25/2023 12:59 AM, Dragos Tatulea wrote: > On Tue, 2023-09-12 at 16:01 +0300, Dragos Tatulea wrote: >> This patch series adds support for vq descriptor table mappings which >> are used to improve vdpa live migration downtime. The improvement comes >> from using smaller mappings which take less time to create and destroy >> in hw. >> > Gentle ping. > > Note that I will have to send a v2. The changes in mlx5_ifc.h will need to be > merged first separately into the mlx5-next branch [0] and then pulled from there > when the series is applied. This separation is unnecessary, as historically the virtio emulation portion of the update to mlx5_ifc.h often had to go through the vhost tree. See commits 1892a3d425bf and e13cd45d352d. Especially the additions from this series (mainly desc group mkey) have nothing to do with any networking or NIC driver feature. -Siwei > > [0] > https://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git/log/?h=mlx5-next > > Thanks, > Dragos > >> The first part adds the vdpa core changes from Si-Wei [0]. >> >> The second part adds support in mlx5_vdpa: >> - Refactor the mr code to be able to cleanly add descriptor mappings. >> - Add hardware descriptor mr support. >> - Properly update iotlb for cvq during ASID switch. >> >> [0] >> https://lore.kernel.org/virtualization/1694248959-13369-1-git-send-email-si-wei.liu@oracle.com >> >> Dragos Tatulea (13): >> vdpa/mlx5: Create helper function for dma mappings >> vdpa/mlx5: Decouple cvq iotlb handling from hw mapping code >> vdpa/mlx5: Take cvq iotlb lock during refresh >> vdpa/mlx5: Collapse "dvq" mr add/delete functions >> vdpa/mlx5: Rename mr destroy functions >> vdpa/mlx5: Allow creation/deletion of any given mr struct >> vdpa/mlx5: Move mr mutex out of mr struct >> vdpa/mlx5: Improve mr update flow >> vdpa/mlx5: Introduce mr for vq descriptor >> vdpa/mlx5: Enable hw support for vq descriptor mapping >> vdpa/mlx5: Make iotlb helper functions more generic >> vdpa/mlx5: Update cvq iotlb mapping on ASID change >> Cover letter: vdpa/mlx5: Add support for vq descriptor mappings >> >> Si-Wei Liu (3): >> vdpa: introduce dedicated descriptor group for virtqueue >> vhost-vdpa: introduce descriptor group backend feature >> vhost-vdpa: uAPI to get dedicated descriptor group id >> >> drivers/vdpa/mlx5/core/mlx5_vdpa.h | 31 +++-- >> drivers/vdpa/mlx5/core/mr.c | 191 ++++++++++++++++------------- >> drivers/vdpa/mlx5/core/resources.c | 6 +- >> drivers/vdpa/mlx5/net/mlx5_vnet.c | 100 ++++++++++----- >> drivers/vhost/vdpa.c | 27 ++++ >> include/linux/mlx5/mlx5_ifc.h | 8 +- >> include/linux/mlx5/mlx5_ifc_vdpa.h | 7 +- >> include/linux/vdpa.h | 11 ++ >> include/uapi/linux/vhost.h | 8 ++ >> include/uapi/linux/vhost_types.h | 5 + >> 10 files changed, 264 insertions(+), 130 deletions(-) >>
On Tue, 2023-09-26 at 05:22 -0700, Si-Wei Liu wrote: > > > On 9/25/2023 12:59 AM, Dragos Tatulea wrote: > > On Tue, 2023-09-12 at 16:01 +0300, Dragos Tatulea wrote: > > > This patch series adds support for vq descriptor table mappings which > > > are used to improve vdpa live migration downtime. The improvement comes > > > from using smaller mappings which take less time to create and destroy > > > in hw. > > > > > Gentle ping. > > > > Note that I will have to send a v2. The changes in mlx5_ifc.h will need to > > be > > merged first separately into the mlx5-next branch [0] and then pulled from > > there > > when the series is applied. > This separation is unnecessary, as historically the virtio emulation > portion of the update to mlx5_ifc.h often had to go through the vhost > tree. See commits 1892a3d425bf and e13cd45d352d. Especially the > additions from this series (mainly desc group mkey) have nothing to do > with any networking or NIC driver feature. > The reason for doing that is to avoid conflicts in Linus's tree on the mlx5_ifc.h file. Thanks, Dragos > -Siwei > > > > > [0] > > https://git.kernel.org/pub/scm/linux/kernel/git/mellanox/linux.git/log/?h=mlx5-next > > > > Thanks, > > Dragos > > > > > The first part adds the vdpa core changes from Si-Wei [0]. > > > > > > The second part adds support in mlx5_vdpa: > > > - Refactor the mr code to be able to cleanly add descriptor mappings. > > > - Add hardware descriptor mr support. > > > - Properly update iotlb for cvq during ASID switch. > > > > > > [0] > > > https://lore.kernel.org/virtualization/1694248959-13369-1-git-send-email-si-wei.liu@oracle.com > > > > > > Dragos Tatulea (13): > > > vdpa/mlx5: Create helper function for dma mappings > > > vdpa/mlx5: Decouple cvq iotlb handling from hw mapping code > > > vdpa/mlx5: Take cvq iotlb lock during refresh > > > vdpa/mlx5: Collapse "dvq" mr add/delete functions > > > vdpa/mlx5: Rename mr destroy functions > > > vdpa/mlx5: Allow creation/deletion of any given mr struct > > > vdpa/mlx5: Move mr mutex out of mr struct > > > vdpa/mlx5: Improve mr update flow > > > vdpa/mlx5: Introduce mr for vq descriptor > > > vdpa/mlx5: Enable hw support for vq descriptor mapping > > > vdpa/mlx5: Make iotlb helper functions more generic > > > vdpa/mlx5: Update cvq iotlb mapping on ASID change > > > Cover letter: vdpa/mlx5: Add support for vq descriptor mappings > > > > > > Si-Wei Liu (3): > > > vdpa: introduce dedicated descriptor group for virtqueue > > > vhost-vdpa: introduce descriptor group backend feature > > > vhost-vdpa: uAPI to get dedicated descriptor group id > > > > > > drivers/vdpa/mlx5/core/mlx5_vdpa.h | 31 +++-- > > > drivers/vdpa/mlx5/core/mr.c | 191 ++++++++++++++++------------- > > > drivers/vdpa/mlx5/core/resources.c | 6 +- > > > drivers/vdpa/mlx5/net/mlx5_vnet.c | 100 ++++++++++----- > > > drivers/vhost/vdpa.c | 27 ++++ > > > include/linux/mlx5/mlx5_ifc.h | 8 +- > > > include/linux/mlx5/mlx5_ifc_vdpa.h | 7 +- > > > include/linux/vdpa.h | 11 ++ > > > include/uapi/linux/vhost.h | 8 ++ > > > include/uapi/linux/vhost_types.h | 5 + > > > 10 files changed, 264 insertions(+), 130 deletions(-) > > > >