| Message ID | 20210310094106.2191-1-jiangkunkun@huawei.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | vfio/migration: Support manual clear vfio dirty log | expand |
kindly ping, Any comments and reviews are welcome.
> From: Kunkun Jiang > Sent: Wednesday, March 10, 2021 5:41 PM > > Hi all, > > In the past, we clear dirty log immediately after sync dirty log to > userspace. This may cause redundant dirty handling if userspace > handles dirty log iteratively: > > After vfio clears dirty log, new dirty log starts to generate. These > new dirty log will be reported to userspace even if they are generated > before userspace handles the same dirty page. > > Since a new dirty log tracking method for vfio based on iommu hwdbm[1] > has been introduced in the kernel and added a new capability named > VFIO_DIRTY_LOG_MANUAL_CLEAR, we can eliminate some redundant dirty > handling by supporting it. Is there any performance data showing the benefit of this new method? > > This series include patches as below: > Patch 1: > - updated the linux-headers/linux/vfio.h from kernel side > > Patch 2: > - introduced 'struct VFIODMARange' to describe a range of the given DMA > mapping and with respect to a VFIO_IOMMU_MAP_DMA operation > > Patch 3: > - implemented the operation to manual clear vfio dirty log, which can > eliminate some redundant dirty handling > > Thanks, > Kunkun Jiang > > [1] https://lore.kernel.org/linux-iommu/20210310090614.26668-1- > zhukeqian1@huawei.com/T/#mb168c9738ecd3d8794e2da14f970545d5820f > 863 > > Zenghui Yu (3): > linux-headers: update against 5.12-rc2 and "vfio log clear" series > vfio: Maintain DMA mapping range for the container > vfio/migration: Support VFIO_IOMMU_DIRTY_PAGES_FLAG_CLEAR_BITMAP > > hw/vfio/common.c | 207 ++++++++++++++++++++++++++++++++-- > include/hw/vfio/vfio-common.h | 10 ++ > linux-headers/linux/vfio.h | 55 ++++++++- > 3 files changed, 264 insertions(+), 8 deletions(-) > > -- > 2.23.0 >
Hi Kevin, On 2021/3/18 14:28, Tian, Kevin wrote: >> From: Kunkun Jiang >> Sent: Wednesday, March 10, 2021 5:41 PM >> >> Hi all, >> >> In the past, we clear dirty log immediately after sync dirty log to >> userspace. This may cause redundant dirty handling if userspace >> handles dirty log iteratively: >> >> After vfio clears dirty log, new dirty log starts to generate. These >> new dirty log will be reported to userspace even if they are generated >> before userspace handles the same dirty page. >> >> Since a new dirty log tracking method for vfio based on iommu hwdbm[1] >> has been introduced in the kernel and added a new capability named >> VFIO_DIRTY_LOG_MANUAL_CLEAR, we can eliminate some redundant dirty >> handling by supporting it. > Is there any performance data showing the benefit of this new method? > Current dirty log tracking method for VFIO: [1] All pages marked dirty if not all iommu_groups have pinned_scope [2] pinned pages by various vendor drivers if all iommu_groups have pinned scope Both methods are coarse-grained and can not determine which pages are really dirty. Each round may mark the pages that are not really dirty as dirty and send them to the destination. ( It might be better if the range of the pinned_scope was smaller. ) This will result in a waste of resources. HWDBM is short for Hardware Dirty Bit Management. (e.g. smmuv3 HTTU, Hardware Translation Table Update) About SMMU HTTU: HTTU is a feature of ARM SMMUv3, it can update access flag or/and dirty state of the TTD (Translation Table Descriptor) by hardware. With HTTU, stage1 TTD is classified into 3 types: DBM bit AP[2](readonly bit) 1. writable_clean 1 1 2. writable_dirty 1 0 3. readonly 0 1 If HTTU_HD (manage dirty state) is enabled, smmu can change TTD from writable_clean to writable_dirty. Then software can scan TTD to sync dirty state into dirty bitmap. With this feature, we can track the dirty log of DMA continuously and precisely. The capability of VFIO_DIRTY_LOG_MANUAL_CLEAR is similar to that on the KVM side. We add this new log_clear() interface only to split the old log_sync() into two separated procedures: - use log_sync() to collect the collection only, and, - use log_clear() to clear the dirty bitmap. If you're interested in this new method, you can take a look at our set of patches. [1] https://lore.kernel.org/linux-iommu/20210310090614.26668-1-zhukeqian1@huawei.com/ Best regards, Kunkun Jiang >> This series include patches as below: >> Patch 1: >> - updated the linux-headers/linux/vfio.h from kernel side >> >> Patch 2: >> - introduced 'struct VFIODMARange' to describe a range of the given DMA >> mapping and with respect to a VFIO_IOMMU_MAP_DMA operation >> >> Patch 3: >> - implemented the operation to manual clear vfio dirty log, which can >> eliminate some redundant dirty handling >> >> Thanks, >> Kunkun Jiang >> >> [1] https://lore.kernel.org/linux-iommu/20210310090614.26668-1- >> zhukeqian1@huawei.com/T/#mb168c9738ecd3d8794e2da14f970545d5820f >> 863 >> >> Zenghui Yu (3): >> linux-headers: update against 5.12-rc2 and "vfio log clear" series >> vfio: Maintain DMA mapping range for the container >> vfio/migration: Support VFIO_IOMMU_DIRTY_PAGES_FLAG_CLEAR_BITMAP >> >> hw/vfio/common.c | 207 ++++++++++++++++++++++++++++++++-- >> include/hw/vfio/vfio-common.h | 10 ++ >> linux-headers/linux/vfio.h | 55 ++++++++- >> 3 files changed, 264 insertions(+), 8 deletions(-) >> >> -- >> 2.23.0 >> > > .
> From: Kunkun Jiang <jiangkunkun@huawei.com> > Sent: Thursday, March 18, 2021 3:59 PM > > Hi Kevin, > > On 2021/3/18 14:28, Tian, Kevin wrote: > >> From: Kunkun Jiang > >> Sent: Wednesday, March 10, 2021 5:41 PM > >> > >> Hi all, > >> > >> In the past, we clear dirty log immediately after sync dirty log to > >> userspace. This may cause redundant dirty handling if userspace > >> handles dirty log iteratively: > >> > >> After vfio clears dirty log, new dirty log starts to generate. These > >> new dirty log will be reported to userspace even if they are generated > >> before userspace handles the same dirty page. > >> > >> Since a new dirty log tracking method for vfio based on iommu hwdbm[1] > >> has been introduced in the kernel and added a new capability named > >> VFIO_DIRTY_LOG_MANUAL_CLEAR, we can eliminate some redundant > dirty > >> handling by supporting it. > > Is there any performance data showing the benefit of this new method? > > > Current dirty log tracking method for VFIO: > [1] All pages marked dirty if not all iommu_groups have pinned_scope > [2] pinned pages by various vendor drivers if all iommu_groups have > pinned scope > > Both methods are coarse-grained and can not determine which pages are > really dirty. Each round may mark the pages that are not really dirty as > dirty > and send them to the destination. ( It might be better if the range of the > pinned_scope was smaller. ) This will result in a waste of resources. > > HWDBM is short for Hardware Dirty Bit Management. > (e.g. smmuv3 HTTU, Hardware Translation Table Update) > > About SMMU HTTU: > HTTU is a feature of ARM SMMUv3, it can update access flag or/and dirty > state of the TTD (Translation Table Descriptor) by hardware. > > With HTTU, stage1 TTD is classified into 3 types: > DBM bit AP[2](readonly bit) > 1. writable_clean 1 1 > 2. writable_dirty 1 0 > 3. readonly 0 1 > > If HTTU_HD (manage dirty state) is enabled, smmu can change TTD from > writable_clean to writable_dirty. Then software can scan TTD to sync dirty > state into dirty bitmap. With this feature, we can track the dirty log of > DMA continuously and precisely. > > The capability of VFIO_DIRTY_LOG_MANUAL_CLEAR is similar to that on > the KVM side. We add this new log_clear() interface only to split the old > log_sync() into two separated procedures: > > - use log_sync() to collect the collection only, and, > - use log_clear() to clear the dirty bitmap. > > If you're interested in this new method, you can take a look at our set of > patches. > [1] > https://lore.kernel.org/linux-iommu/20210310090614.26668-1- > zhukeqian1@huawei.com/ > I know what you are doing. Intel is also working on VT-d dirty bit support based on above link. What I'm curious is the actual performance gain with this optimization. KVM doing that is one good reference, but IOMMU has different characteristics (e.g. longer invalidation latency) compared to CPU MMU. It's always good to understand what a so-called optimization can actually optimize in a context different from where it's originally proved.
Hi Kevin, On 2021/3/18 17:04, Tian, Kevin wrote: >> From: Kunkun Jiang <jiangkunkun@huawei.com> >> Sent: Thursday, March 18, 2021 3:59 PM >> >> Hi Kevin, >> >> On 2021/3/18 14:28, Tian, Kevin wrote: >>>> From: Kunkun Jiang >>>> Sent: Wednesday, March 10, 2021 5:41 PM >>>> >>>> Hi all, >>>> >>>> In the past, we clear dirty log immediately after sync dirty log to >>>> userspace. This may cause redundant dirty handling if userspace >>>> handles dirty log iteratively: >>>> >>>> After vfio clears dirty log, new dirty log starts to generate. These >>>> new dirty log will be reported to userspace even if they are generated >>>> before userspace handles the same dirty page. >>>> >>>> Since a new dirty log tracking method for vfio based on iommu hwdbm[1] >>>> has been introduced in the kernel and added a new capability named >>>> VFIO_DIRTY_LOG_MANUAL_CLEAR, we can eliminate some redundant >> dirty >>>> handling by supporting it. >>> Is there any performance data showing the benefit of this new method? >>> >> Current dirty log tracking method for VFIO: >> [1] All pages marked dirty if not all iommu_groups have pinned_scope >> [2] pinned pages by various vendor drivers if all iommu_groups have >> pinned scope >> >> Both methods are coarse-grained and can not determine which pages are >> really dirty. Each round may mark the pages that are not really dirty as >> dirty >> and send them to the destination. ( It might be better if the range of the >> pinned_scope was smaller. ) This will result in a waste of resources. >> >> HWDBM is short for Hardware Dirty Bit Management. >> (e.g. smmuv3 HTTU, Hardware Translation Table Update) >> >> About SMMU HTTU: >> HTTU is a feature of ARM SMMUv3, it can update access flag or/and dirty >> state of the TTD (Translation Table Descriptor) by hardware. >> >> With HTTU, stage1 TTD is classified into 3 types: >> DBM bit AP[2](readonly bit) >> 1. writable_clean 1 1 >> 2. writable_dirty 1 0 >> 3. readonly 0 1 >> >> If HTTU_HD (manage dirty state) is enabled, smmu can change TTD from >> writable_clean to writable_dirty. Then software can scan TTD to sync dirty >> state into dirty bitmap. With this feature, we can track the dirty log of >> DMA continuously and precisely. >> >> The capability of VFIO_DIRTY_LOG_MANUAL_CLEAR is similar to that on >> the KVM side. We add this new log_clear() interface only to split the old >> log_sync() into two separated procedures: >> >> - use log_sync() to collect the collection only, and, >> - use log_clear() to clear the dirty bitmap. >> >> If you're interested in this new method, you can take a look at our set of >> patches. >> [1] >> https://lore.kernel.org/linux-iommu/20210310090614.26668-1- >> zhukeqian1@huawei.com/ >> > I know what you are doing. Intel is also working on VT-d dirty bit support > based on above link. What I'm curious is the actual performance gain > with this optimization. KVM doing that is one good reference, but IOMMU > has different characteristics (e.g. longer invalidation latency) compared to > CPU MMU. It's always good to understand what a so-called optimization > can actually optimize in a context different from where it's originally proved.
> From: Kunkun Jiang <jiangkunkun@huawei.com> > Sent: Thursday, March 18, 2021 8:29 PM > > Hi Kevin, > > On 2021/3/18 17:04, Tian, Kevin wrote: > >> From: Kunkun Jiang <jiangkunkun@huawei.com> > >> Sent: Thursday, March 18, 2021 3:59 PM > >> > >> Hi Kevin, > >> > >> On 2021/3/18 14:28, Tian, Kevin wrote: > >>>> From: Kunkun Jiang > >>>> Sent: Wednesday, March 10, 2021 5:41 PM > >>>> > >>>> Hi all, > >>>> > >>>> In the past, we clear dirty log immediately after sync dirty log to > >>>> userspace. This may cause redundant dirty handling if userspace > >>>> handles dirty log iteratively: > >>>> > >>>> After vfio clears dirty log, new dirty log starts to generate. These > >>>> new dirty log will be reported to userspace even if they are generated > >>>> before userspace handles the same dirty page. > >>>> > >>>> Since a new dirty log tracking method for vfio based on iommu > hwdbm[1] > >>>> has been introduced in the kernel and added a new capability named > >>>> VFIO_DIRTY_LOG_MANUAL_CLEAR, we can eliminate some redundant > >> dirty > >>>> handling by supporting it. > >>> Is there any performance data showing the benefit of this new method? > >>> > >> Current dirty log tracking method for VFIO: > >> [1] All pages marked dirty if not all iommu_groups have pinned_scope > >> [2] pinned pages by various vendor drivers if all iommu_groups have > >> pinned scope > >> > >> Both methods are coarse-grained and can not determine which pages are > >> really dirty. Each round may mark the pages that are not really dirty as > >> dirty > >> and send them to the destination. ( It might be better if the range of the > >> pinned_scope was smaller. ) This will result in a waste of resources. > >> > >> HWDBM is short for Hardware Dirty Bit Management. > >> (e.g. smmuv3 HTTU, Hardware Translation Table Update) > >> > >> About SMMU HTTU: > >> HTTU is a feature of ARM SMMUv3, it can update access flag or/and dirty > >> state of the TTD (Translation Table Descriptor) by hardware. > >> > >> With HTTU, stage1 TTD is classified into 3 types: > >> DBM bit AP[2](readonly bit) > >> 1. writable_clean 1 1 > >> 2. writable_dirty 1 0 > >> 3. readonly 0 1 > >> > >> If HTTU_HD (manage dirty state) is enabled, smmu can change TTD from > >> writable_clean to writable_dirty. Then software can scan TTD to sync dirty > >> state into dirty bitmap. With this feature, we can track the dirty log of > >> DMA continuously and precisely. > >> > >> The capability of VFIO_DIRTY_LOG_MANUAL_CLEAR is similar to that on > >> the KVM side. We add this new log_clear() interface only to split the old > >> log_sync() into two separated procedures: > >> > >> - use log_sync() to collect the collection only, and, > >> - use log_clear() to clear the dirty bitmap. > >> > >> If you're interested in this new method, you can take a look at our set of > >> patches. > >> [1] > >> https://lore.kernel.org/linux-iommu/20210310090614.26668-1- > >> zhukeqian1@huawei.com/ > >> > > I know what you are doing. Intel is also working on VT-d dirty bit support > > based on above link. What I'm curious is the actual performance gain > > with this optimization. KVM doing that is one good reference, but IOMMU > > has different characteristics (e.g. longer invalidation latency) compared to > > CPU MMU. It's always good to understand what a so-called optimization > > can actually optimize in a context different from where it's originally > proved.
On 2021/3/18 20:36, Tian, Kevin wrote: >> From: Kunkun Jiang <jiangkunkun@huawei.com> >> Sent: Thursday, March 18, 2021 8:29 PM >> >> Hi Kevin, >> >> On 2021/3/18 17:04, Tian, Kevin wrote: >>>> From: Kunkun Jiang <jiangkunkun@huawei.com> >>>> Sent: Thursday, March 18, 2021 3:59 PM >>>> >>>> Hi Kevin, >>>> >>>> On 2021/3/18 14:28, Tian, Kevin wrote: >>>>>> From: Kunkun Jiang >>>>>> Sent: Wednesday, March 10, 2021 5:41 PM >>>>>> >>>>>> Hi all, >>>>>> >>>>>> In the past, we clear dirty log immediately after sync dirty log to >>>>>> userspace. This may cause redundant dirty handling if userspace >>>>>> handles dirty log iteratively: >>>>>> >>>>>> After vfio clears dirty log, new dirty log starts to generate. These >>>>>> new dirty log will be reported to userspace even if they are generated >>>>>> before userspace handles the same dirty page. >>>>>> >>>>>> Since a new dirty log tracking method for vfio based on iommu >> hwdbm[1] >>>>>> has been introduced in the kernel and added a new capability named >>>>>> VFIO_DIRTY_LOG_MANUAL_CLEAR, we can eliminate some redundant >>>> dirty >>>>>> handling by supporting it. >>>>> Is there any performance data showing the benefit of this new method? >>>>> >>>> Current dirty log tracking method for VFIO: >>>> [1] All pages marked dirty if not all iommu_groups have pinned_scope >>>> [2] pinned pages by various vendor drivers if all iommu_groups have >>>> pinned scope >>>> >>>> Both methods are coarse-grained and can not determine which pages are >>>> really dirty. Each round may mark the pages that are not really dirty as >>>> dirty >>>> and send them to the destination. ( It might be better if the range of the >>>> pinned_scope was smaller. ) This will result in a waste of resources. >>>> >>>> HWDBM is short for Hardware Dirty Bit Management. >>>> (e.g. smmuv3 HTTU, Hardware Translation Table Update) >>>> >>>> About SMMU HTTU: >>>> HTTU is a feature of ARM SMMUv3, it can update access flag or/and dirty >>>> state of the TTD (Translation Table Descriptor) by hardware. >>>> >>>> With HTTU, stage1 TTD is classified into 3 types: >>>> DBM bit AP[2](readonly bit) >>>> 1. writable_clean 1 1 >>>> 2. writable_dirty 1 0 >>>> 3. readonly 0 1 >>>> >>>> If HTTU_HD (manage dirty state) is enabled, smmu can change TTD from >>>> writable_clean to writable_dirty. Then software can scan TTD to sync dirty >>>> state into dirty bitmap. With this feature, we can track the dirty log of >>>> DMA continuously and precisely. >>>> >>>> The capability of VFIO_DIRTY_LOG_MANUAL_CLEAR is similar to that on >>>> the KVM side. We add this new log_clear() interface only to split the old >>>> log_sync() into two separated procedures: >>>> >>>> - use log_sync() to collect the collection only, and, >>>> - use log_clear() to clear the dirty bitmap. >>>> >>>> If you're interested in this new method, you can take a look at our set of >>>> patches. >>>> [1] >>>> https://lore.kernel.org/linux-iommu/20210310090614.26668-1- >>>> zhukeqian1@huawei.com/ >>>> >>> I know what you are doing. Intel is also working on VT-d dirty bit support >>> based on above link. What I'm curious is the actual performance gain >>> with this optimization. KVM doing that is one good reference, but IOMMU >>> has different characteristics (e.g. longer invalidation latency) compared to >>> CPU MMU. It's always good to understand what a so-called optimization >>> can actually optimize in a context different from where it's originally >> proved.
On 2021/3/18 20:36, Tian, Kevin wrote: >> From: Kunkun Jiang <jiangkunkun@huawei.com> >> Sent: Thursday, March 18, 2021 8:29 PM >> >> Hi Kevin, >> >> On 2021/3/18 17:04, Tian, Kevin wrote: >>>> From: Kunkun Jiang <jiangkunkun@huawei.com> >>>> Sent: Thursday, March 18, 2021 3:59 PM >>>> >>>> Hi Kevin, >>>> >>>> On 2021/3/18 14:28, Tian, Kevin wrote: >>>>>> From: Kunkun Jiang >>>>>> Sent: Wednesday, March 10, 2021 5:41 PM >>>>>> >>>>>> Hi all, >>>>>> >>>>>> In the past, we clear dirty log immediately after sync dirty log to >>>>>> userspace. This may cause redundant dirty handling if userspace >>>>>> handles dirty log iteratively: >>>>>> >>>>>> After vfio clears dirty log, new dirty log starts to generate. These >>>>>> new dirty log will be reported to userspace even if they are generated >>>>>> before userspace handles the same dirty page. >>>>>> >>>>>> Since a new dirty log tracking method for vfio based on iommu >> hwdbm[1] >>>>>> has been introduced in the kernel and added a new capability named >>>>>> VFIO_DIRTY_LOG_MANUAL_CLEAR, we can eliminate some redundant >>>> dirty >>>>>> handling by supporting it. >>>>> Is there any performance data showing the benefit of this new method? >>>>> >>>> Current dirty log tracking method for VFIO: >>>> [1] All pages marked dirty if not all iommu_groups have pinned_scope >>>> [2] pinned pages by various vendor drivers if all iommu_groups have >>>> pinned scope >>>> >>>> Both methods are coarse-grained and can not determine which pages are >>>> really dirty. Each round may mark the pages that are not really dirty as >>>> dirty >>>> and send them to the destination. ( It might be better if the range of the >>>> pinned_scope was smaller. ) This will result in a waste of resources. >>>> >>>> HWDBM is short for Hardware Dirty Bit Management. >>>> (e.g. smmuv3 HTTU, Hardware Translation Table Update) >>>> >>>> About SMMU HTTU: >>>> HTTU is a feature of ARM SMMUv3, it can update access flag or/and dirty >>>> state of the TTD (Translation Table Descriptor) by hardware. >>>> >>>> With HTTU, stage1 TTD is classified into 3 types: >>>> DBM bit AP[2](readonly bit) >>>> 1. writable_clean 1 1 >>>> 2. writable_dirty 1 0 >>>> 3. readonly 0 1 >>>> >>>> If HTTU_HD (manage dirty state) is enabled, smmu can change TTD from >>>> writable_clean to writable_dirty. Then software can scan TTD to sync dirty >>>> state into dirty bitmap. With this feature, we can track the dirty log of >>>> DMA continuously and precisely. >>>> >>>> The capability of VFIO_DIRTY_LOG_MANUAL_CLEAR is similar to that on >>>> the KVM side. We add this new log_clear() interface only to split the old >>>> log_sync() into two separated procedures: >>>> >>>> - use log_sync() to collect the collection only, and, >>>> - use log_clear() to clear the dirty bitmap. >>>> >>>> If you're interested in this new method, you can take a look at our set of >>>> patches. >>>> [1] >>>> https://lore.kernel.org/linux-iommu/20210310090614.26668-1- >>>> zhukeqian1@huawei.com/ >>>> >>> I know what you are doing. Intel is also working on VT-d dirty bit support >>> based on above link. What I'm curious is the actual performance gain >>> with this optimization. KVM doing that is one good reference, but IOMMU >>> has different characteristics (e.g. longer invalidation latency) compared to >>> CPU MMU. It's always good to understand what a so-called optimization >>> can actually optimize in a context different from where it's originally >> proved.