| Message ID | 20210923115840.17813-13-yong.wu@mediatek.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | MT8195 IOMMU SUPPORT | expand |
Hi This patch is needed in order to update the tlb when a device is powered on. Could you send this patch alone without the whole series so it get accepted easier? I can resend the patch on your behalf if you want. Thanks, Dafna On 23.09.21 14:58, Yong Wu wrote: > Prepare for 2 HWs that sharing pgtable in different power-domains. > > When there are 2 M4U HWs, it may has problem in the flush_range in which > we get the pm_status via the m4u dev, BUT that function don't reflect the > real power-domain status of the HW since there may be other HW also use > that power-domain. > > The function dma_alloc_attrs help allocate the iommu buffer which > need the corresponding power domain since tlb flush is needed when > preparing iova. BUT this function only is for allocating buffer, > we have no good reason to request the user always call pm_runtime_get > before calling dma_alloc_xxx. Therefore, we add a tlb_flush_all > in the pm_runtime_resume to make sure the tlb always is clean. > > Another solution is always call pm_runtime_get in the tlb_flush_range. > This will trigger pm runtime resume/backup so often when the iommu > power is not active at some time(means user don't call pm_runtime_get > before calling dma_alloc_xxx), This may cause the performance drop. > thus we don't use this. > > In other case, the iommu's power should always be active via device > link with smi. > > The previous SoC don't have PM except mt8192. the mt8192 IOMMU is display's > power-domain which nearly always is enabled. thus no need fix tags here. > Prepare for mt8195. > > Signed-off-by: Yong Wu <yong.wu@mediatek.com> > --- > drivers/iommu/mtk_iommu.c | 11 +++++++++++ > 1 file changed, 11 insertions(+) > > diff --git a/drivers/iommu/mtk_iommu.c b/drivers/iommu/mtk_iommu.c > index 44cf5547d084..e9e94944ed91 100644 > --- a/drivers/iommu/mtk_iommu.c > +++ b/drivers/iommu/mtk_iommu.c > @@ -984,6 +984,17 @@ static int __maybe_unused mtk_iommu_runtime_resume(struct device *dev) > return ret; > } > > + /* > + * Users may allocate dma buffer before they call pm_runtime_get, then > + * it will lack the necessary tlb flush. > + * > + * We have no good reason to request the users always call dma_alloc_xx > + * after pm_runtime_get_sync. > + * > + * Thus, Make sure the tlb always is clean after each PM resume. > + */ > + mtk_iommu_tlb_do_flush_all(data); > + > /* > * Uppon first resume, only enable the clk and return, since the values of the > * registers are not yet set. >
On Tue, 2021-11-09 at 14:21 +0200, Dafna Hirschfeld wrote: > Hi > This patch is needed in order to update the tlb when a device is > powered on. > Could you send this patch alone without the whole series so it get > accepted easier? Which SoC are you testing on? In previous SoC, the IOMMU HW don't have power-domain, and we have a "has_pm"[1] in the tlb function for that case. The "has_pm" should be always 0 for the previous SoC like mt8173, it should always tlb synchronize. thus, Could you help share more about your issue? In which case it lack the necessary tlb operation. At least, We need confirm if it needs a "Fixes" tags if sending this patch alone. Thanks. [1] https://elixir.bootlin.com/linux/v5.15/source/drivers/iommu/mtk_iommu.c#L236 > I can resend the patch on your behalf if you want. > > Thanks, > Dafna > > On 23.09.21 14:58, Yong Wu wrote: > > Prepare for 2 HWs that sharing pgtable in different power-domains. > > > > When there are 2 M4U HWs, it may has problem in the flush_range in > > which > > we get the pm_status via the m4u dev, BUT that function don't > > reflect the > > real power-domain status of the HW since there may be other HW also > > use > > that power-domain. > > > > The function dma_alloc_attrs help allocate the iommu buffer which > > need the corresponding power domain since tlb flush is needed when > > preparing iova. BUT this function only is for allocating buffer, > > we have no good reason to request the user always call > > pm_runtime_get > > before calling dma_alloc_xxx. Therefore, we add a tlb_flush_all > > in the pm_runtime_resume to make sure the tlb always is clean. > > > > Another solution is always call pm_runtime_get in the > > tlb_flush_range. > > This will trigger pm runtime resume/backup so often when the iommu > > power is not active at some time(means user don't call > > pm_runtime_get > > before calling dma_alloc_xxx), This may cause the performance drop. > > thus we don't use this. > > > > In other case, the iommu's power should always be active via device > > link with smi. > > > > The previous SoC don't have PM except mt8192. the mt8192 IOMMU is > > display's > > power-domain which nearly always is enabled. thus no need fix tags > > here. > > Prepare for mt8195. > > > > Signed-off-by: Yong Wu <yong.wu@mediatek.com> > > --- > > drivers/iommu/mtk_iommu.c | 11 +++++++++++ > > 1 file changed, 11 insertions(+) > > > > diff --git a/drivers/iommu/mtk_iommu.c b/drivers/iommu/mtk_iommu.c > > index 44cf5547d084..e9e94944ed91 100644 > > --- a/drivers/iommu/mtk_iommu.c > > +++ b/drivers/iommu/mtk_iommu.c > > @@ -984,6 +984,17 @@ static int __maybe_unused > > mtk_iommu_runtime_resume(struct device *dev) > > return ret; > > } > > > > + /* > > + * Users may allocate dma buffer before they call > > pm_runtime_get, then > > + * it will lack the necessary tlb flush. > > + * > > + * We have no good reason to request the users always call > > dma_alloc_xx > > + * after pm_runtime_get_sync. > > + * > > + * Thus, Make sure the tlb always is clean after each PM > > resume. > > + */ > > + mtk_iommu_tlb_do_flush_all(data); > > + > > /* > > * Uppon first resume, only enable the clk and return, since > > the values of the > > * registers are not yet set. > >
On 10.11.21 04:20, Yong Wu wrote: > On Tue, 2021-11-09 at 14:21 +0200, Dafna Hirschfeld wrote: >> Hi >> This patch is needed in order to update the tlb when a device is >> powered on. >> Could you send this patch alone without the whole series so it get >> accepted easier? > > Which SoC are you testing on? In previous SoC, the IOMMU HW don't have > power-domain, and we have a "has_pm"[1] in the tlb function for that > case. The "has_pm" should be always 0 for the previous SoC like mt8173, > it should always tlb synchronize. > > thus, Could you help share more about your issue? In which case it lack > the necessary tlb operation. At least, We need confirm if it needs a > "Fixes" tags if sending this patch alone. Hi, I work with the mtk-vcodec driver on mt8173. As you wrote, the iommu doesn't have a power-domain and so when allocating buffers before the device is powered on, there is the warning "Partial TLB flush timed out, falling back to full flush" flooding the log buf. Sebastian Reichel suggested to remove the 'if(has_pm)' check to avoid this warning, and avoid flushing the tlb if the device is off: [1] http://ix.io/3Eyr This fixes the warning, but then the tlb is not flushed in sync, Therefore the tlb should be flushed when the device is resumed. So the two patches (the one suggested in the link [1] and this patch) should be sent together as a 2-patch series. Thanks, Dafna > > Thanks. > > [1] > https://elixir.bootlin.com/linux/v5.15/source/drivers/iommu/mtk_iommu.c#L236 > >> I can resend the patch on your behalf if you want. >> >> Thanks, >> Dafna >> >> On 23.09.21 14:58, Yong Wu wrote: >>> Prepare for 2 HWs that sharing pgtable in different power-domains. >>> >>> When there are 2 M4U HWs, it may has problem in the flush_range in >>> which >>> we get the pm_status via the m4u dev, BUT that function don't >>> reflect the >>> real power-domain status of the HW since there may be other HW also >>> use >>> that power-domain. >>> >>> The function dma_alloc_attrs help allocate the iommu buffer which >>> need the corresponding power domain since tlb flush is needed when >>> preparing iova. BUT this function only is for allocating buffer, >>> we have no good reason to request the user always call >>> pm_runtime_get >>> before calling dma_alloc_xxx. Therefore, we add a tlb_flush_all >>> in the pm_runtime_resume to make sure the tlb always is clean. >>> >>> Another solution is always call pm_runtime_get in the >>> tlb_flush_range. >>> This will trigger pm runtime resume/backup so often when the iommu >>> power is not active at some time(means user don't call >>> pm_runtime_get >>> before calling dma_alloc_xxx), This may cause the performance drop. >>> thus we don't use this. >>> >>> In other case, the iommu's power should always be active via device >>> link with smi. >>> >>> The previous SoC don't have PM except mt8192. the mt8192 IOMMU is >>> display's >>> power-domain which nearly always is enabled. thus no need fix tags >>> here. >>> Prepare for mt8195. >>> >>> Signed-off-by: Yong Wu <yong.wu@mediatek.com> >>> --- >>> drivers/iommu/mtk_iommu.c | 11 +++++++++++ >>> 1 file changed, 11 insertions(+) >>> >>> diff --git a/drivers/iommu/mtk_iommu.c b/drivers/iommu/mtk_iommu.c >>> index 44cf5547d084..e9e94944ed91 100644 >>> --- a/drivers/iommu/mtk_iommu.c >>> +++ b/drivers/iommu/mtk_iommu.c >>> @@ -984,6 +984,17 @@ static int __maybe_unused >>> mtk_iommu_runtime_resume(struct device *dev) >>> return ret; >>> } >>> >>> + /* >>> + * Users may allocate dma buffer before they call >>> pm_runtime_get, then >>> + * it will lack the necessary tlb flush. >>> + * >>> + * We have no good reason to request the users always call >>> dma_alloc_xx >>> + * after pm_runtime_get_sync. >>> + * >>> + * Thus, Make sure the tlb always is clean after each PM >>> resume. >>> + */ >>> + mtk_iommu_tlb_do_flush_all(data); >>> + >>> /* >>> * Uppon first resume, only enable the clk and return, since >>> the values of the >>> * registers are not yet set. >>>
On Wed, 2021-11-10 at 07:29 +0200, Dafna Hirschfeld wrote: > > On 10.11.21 04:20, Yong Wu wrote: > > On Tue, 2021-11-09 at 14:21 +0200, Dafna Hirschfeld wrote: > > > Hi > > > This patch is needed in order to update the tlb when a device is > > > powered on. > > > Could you send this patch alone without the whole series so it > > > get > > > accepted easier? > > > > Which SoC are you testing on? In previous SoC, the IOMMU HW don't > > have > > power-domain, and we have a "has_pm"[1] in the tlb function for > > that > > case. The "has_pm" should be always 0 for the previous SoC like > > mt8173, > > it should always tlb synchronize. > > > > thus, Could you help share more about your issue? In which case it > > lack > > the necessary tlb operation. At least, We need confirm if it needs > > a > > "Fixes" tags if sending this patch alone. > > Hi, > I work with the mtk-vcodec driver on mt8173. As you wrote, the iommu > doesn't > have a power-domain and so when allocating buffers before the device > is powered > on, there is the warning > "Partial TLB flush timed out, falling back to full flush" > flooding the log buf. oh. Thanks very much for your information. Get it now. This issue should be introduced by the: b34ea31fe013 ("iommu/mediatek: Always enable the clk on resume") tlb failed due to the bclk is not enabled. Could you help try that after reverting this? > > Sebastian Reichel suggested to remove the 'if(has_pm)' check to avoid > this warning, > and avoid flushing the tlb if the device is off: > > [1] http://ix.io/3Eyr > > This fixes the warning, but then the tlb is not flushed in sync, > Therefore the tlb should be flushed when the device is resumed. > > So the two patches (the one suggested in the link [1] and this patch) > should be sent together as a 2-patch series. then this is reasonable. You could help this into a new patchset if you are free(add Fixes tag). Thanks. > > Thanks, > Dafna > > > > > Thanks. > > > > [1] > > https://elixir.bootlin.com/linux/v5.15/source/drivers/iommu/mtk_iommu.c#L236 > > > > > I can resend the patch on your behalf if you want. > > > > > > Thanks, > > > Dafna > > > > > > On 23.09.21 14:58, Yong Wu wrote: > > > > Prepare for 2 HWs that sharing pgtable in different power- > > > > domains. > > > > > > > > When there are 2 M4U HWs, it may has problem in the flush_range > > > > in > > > > which > > > > we get the pm_status via the m4u dev, BUT that function don't > > > > reflect the > > > > real power-domain status of the HW since there may be other HW > > > > also > > > > use > > > > that power-domain. > > > > > > > > The function dma_alloc_attrs help allocate the iommu buffer > > > > which > > > > need the corresponding power domain since tlb flush is needed > > > > when > > > > preparing iova. BUT this function only is for allocating > > > > buffer, > > > > we have no good reason to request the user always call > > > > pm_runtime_get > > > > before calling dma_alloc_xxx. Therefore, we add a tlb_flush_all > > > > in the pm_runtime_resume to make sure the tlb always is clean. > > > > > > > > Another solution is always call pm_runtime_get in the > > > > tlb_flush_range. > > > > This will trigger pm runtime resume/backup so often when the > > > > iommu > > > > power is not active at some time(means user don't call > > > > pm_runtime_get > > > > before calling dma_alloc_xxx), This may cause the performance > > > > drop. > > > > thus we don't use this. > > > > > > > > In other case, the iommu's power should always be active via > > > > device > > > > link with smi. > > > > > > > > The previous SoC don't have PM except mt8192. the mt8192 IOMMU > > > > is > > > > display's > > > > power-domain which nearly always is enabled. thus no need fix > > > > tags > > > > here. > > > > Prepare for mt8195. > > > > > > > > Signed-off-by: Yong Wu <yong.wu@mediatek.com> > > > > --- > > > > drivers/iommu/mtk_iommu.c | 11 +++++++++++ > > > > 1 file changed, 11 insertions(+) > > > > > > > > diff --git a/drivers/iommu/mtk_iommu.c > > > > b/drivers/iommu/mtk_iommu.c > > > > index 44cf5547d084..e9e94944ed91 100644 > > > > --- a/drivers/iommu/mtk_iommu.c > > > > +++ b/drivers/iommu/mtk_iommu.c > > > > @@ -984,6 +984,17 @@ static int __maybe_unused > > > > mtk_iommu_runtime_resume(struct device *dev) > > > > return ret; > > > > } > > > > > > > > + /* > > > > + * Users may allocate dma buffer before they call > > > > pm_runtime_get, then > > > > + * it will lack the necessary tlb flush. > > > > + * > > > > + * We have no good reason to request the users always > > > > call > > > > dma_alloc_xx > > > > + * after pm_runtime_get_sync. > > > > + * > > > > + * Thus, Make sure the tlb always is clean after each > > > > PM > > > > resume. > > > > + */ > > > > + mtk_iommu_tlb_do_flush_all(data); > > > > + > > > > /* > > > > * Uppon first resume, only enable the clk and return, > > > > since > > > > the values of the > > > > * registers are not yet set. > > > >
Hi Dafna, On Wed, 2021-11-10 at 15:50 +0800, Yong Wu wrote: > On Wed, 2021-11-10 at 07:29 +0200, Dafna Hirschfeld wrote: > > > > On 10.11.21 04:20, Yong Wu wrote: > > > On Tue, 2021-11-09 at 14:21 +0200, Dafna Hirschfeld wrote: > > > > Hi > > > > This patch is needed in order to update the tlb when a device > > > > is > > > > powered on. > > > > Could you send this patch alone without the whole series so it > > > > get > > > > accepted easier? > > > > > > Which SoC are you testing on? In previous SoC, the IOMMU HW don't > > > have > > > power-domain, and we have a "has_pm"[1] in the tlb function for > > > that > > > case. The "has_pm" should be always 0 for the previous SoC like > > > mt8173, > > > it should always tlb synchronize. > > > > > > thus, Could you help share more about your issue? In which case > > > it > > > lack > > > the necessary tlb operation. At least, We need confirm if it > > > needs > > > a > > > "Fixes" tags if sending this patch alone. > > > > Hi, > > I work with the mtk-vcodec driver on mt8173. As you wrote, the > > iommu > > doesn't > > have a power-domain and so when allocating buffers before the > > device > > is powered > > on, there is the warning > > "Partial TLB flush timed out, falling back to full flush" > > flooding the log buf. > > oh. Thanks very much for your information. Get it now. > > This issue should be introduced by the: > > b34ea31fe013 ("iommu/mediatek: Always enable the clk on resume") > > tlb failed due to the bclk is not enabled. Could you help try that > after reverting this? > > > > > Sebastian Reichel suggested to remove the 'if(has_pm)' check to > > avoid > > this warning, > > and avoid flushing the tlb if the device is off: > > > > [1] http://ix.io/3Eyr > > > > This fixes the warning, but then the tlb is not flushed in sync, > > Therefore the tlb should be flushed when the device is resumed. > > > > So the two patches (the one suggested in the link [1] and this > > patch) > > should be sent together as a 2-patch series. > > then this is reasonable. You could help this into a new patchset if > you > are free(add Fixes tag). > > Thanks. > > > > > Thanks, > > Dafna > > > > > > > > Thanks. > > > > > > [1] > > > > > https://elixir.bootlin.com/linux/v5.15/source/drivers/iommu/mtk_iommu.c#L236 > > > > > > > I can resend the patch on your behalf if you want. What's your plan here? If you send the two as a independent patchset on v5.16, I will rebase yours. If you have no time for this, I could help this. Thanks. > > > > > > > > Thanks, > > > > Dafna > > > > > > > > On 23.09.21 14:58, Yong Wu wrote: > > > > > Prepare for 2 HWs that sharing pgtable in different power- > > > > > domains. > > > > > > > > > > When there are 2 M4U HWs, it may has problem in the > > > > > flush_range > > > > > in > > > > > which > > > > > we get the pm_status via the m4u dev, BUT that function don't > > > > > reflect the > > > > > real power-domain status of the HW since there may be other > > > > > HW > > > > > also > > > > > use > > > > > that power-domain. > > > > > > > > > > The function dma_alloc_attrs help allocate the iommu buffer > > > > > which > > > > > need the corresponding power domain since tlb flush is needed > > > > > when > > > > > preparing iova. BUT this function only is for allocating > > > > > buffer, > > > > > we have no good reason to request the user always call > > > > > pm_runtime_get > > > > > before calling dma_alloc_xxx. Therefore, we add a > > > > > tlb_flush_all > > > > > in the pm_runtime_resume to make sure the tlb always is > > > > > clean. > > > > > > > > > > Another solution is always call pm_runtime_get in the > > > > > tlb_flush_range. > > > > > This will trigger pm runtime resume/backup so often when the > > > > > iommu > > > > > power is not active at some time(means user don't call > > > > > pm_runtime_get > > > > > before calling dma_alloc_xxx), This may cause the performance > > > > > drop. > > > > > thus we don't use this. > > > > > > > > > > In other case, the iommu's power should always be active via > > > > > device > > > > > link with smi. > > > > > > > > > > The previous SoC don't have PM except mt8192. the mt8192 > > > > > IOMMU > > > > > is > > > > > display's > > > > > power-domain which nearly always is enabled. thus no need fix > > > > > tags > > > > > here. > > > > > Prepare for mt8195. > > > > > > > > > > Signed-off-by: Yong Wu <yong.wu@mediatek.com> > > > > > --- > > > > > drivers/iommu/mtk_iommu.c | 11 +++++++++++ > > > > > 1 file changed, 11 insertions(+) > > > > > > > > > > diff --git a/drivers/iommu/mtk_iommu.c > > > > > b/drivers/iommu/mtk_iommu.c > > > > > index 44cf5547d084..e9e94944ed91 100644 > > > > > --- a/drivers/iommu/mtk_iommu.c > > > > > +++ b/drivers/iommu/mtk_iommu.c > > > > > @@ -984,6 +984,17 @@ static int __maybe_unused > > > > > mtk_iommu_runtime_resume(struct device *dev) > > > > > return ret; > > > > > } > > > > > > > > > > + /* > > > > > + * Users may allocate dma buffer before they call > > > > > pm_runtime_get, then > > > > > + * it will lack the necessary tlb flush. > > > > > + * > > > > > + * We have no good reason to request the users always > > > > > call > > > > > dma_alloc_xx > > > > > + * after pm_runtime_get_sync. > > > > > + * > > > > > + * Thus, Make sure the tlb always is clean after each > > > > > PM > > > > > resume. > > > > > + */ > > > > > + mtk_iommu_tlb_do_flush_all(data); > > > > > + > > > > > /* > > > > > * Uppon first resume, only enable the clk and return, > > > > > since > > > > > the values of the > > > > > * registers are not yet set. > > > > > > > _______________________________________________ > iommu mailing list > iommu@lists.linux-foundation.org > https://lists.linuxfoundation.org/mailman/listinfo/iommu
On 22.11.21 09:05, Yong Wu wrote: > Hi Dafna, > > On Wed, 2021-11-10 at 15:50 +0800, Yong Wu wrote: >> On Wed, 2021-11-10 at 07:29 +0200, Dafna Hirschfeld wrote: >>> >>> On 10.11.21 04:20, Yong Wu wrote: >>>> On Tue, 2021-11-09 at 14:21 +0200, Dafna Hirschfeld wrote: >>>>> Hi >>>>> This patch is needed in order to update the tlb when a device >>>>> is >>>>> powered on. >>>>> Could you send this patch alone without the whole series so it >>>>> get >>>>> accepted easier? >>>> >>>> Which SoC are you testing on? In previous SoC, the IOMMU HW don't >>>> have >>>> power-domain, and we have a "has_pm"[1] in the tlb function for >>>> that >>>> case. The "has_pm" should be always 0 for the previous SoC like >>>> mt8173, >>>> it should always tlb synchronize. >>>> >>>> thus, Could you help share more about your issue? In which case >>>> it >>>> lack >>>> the necessary tlb operation. At least, We need confirm if it >>>> needs >>>> a >>>> "Fixes" tags if sending this patch alone. >>> >>> Hi, >>> I work with the mtk-vcodec driver on mt8173. As you wrote, the >>> iommu >>> doesn't >>> have a power-domain and so when allocating buffers before the >>> device >>> is powered >>> on, there is the warning >>> "Partial TLB flush timed out, falling back to full flush" >>> flooding the log buf. >> >> oh. Thanks very much for your information. Get it now. >> >> This issue should be introduced by the: >> >> b34ea31fe013 ("iommu/mediatek: Always enable the clk on resume") >> >> tlb failed due to the bclk is not enabled. Could you help try that >> after reverting this? >> >>> >>> Sebastian Reichel suggested to remove the 'if(has_pm)' check to >>> avoid >>> this warning, >>> and avoid flushing the tlb if the device is off: >>> >>> [1] http://ix.io/3Eyr >>> >>> This fixes the warning, but then the tlb is not flushed in sync, >>> Therefore the tlb should be flushed when the device is resumed. >>> >>> So the two patches (the one suggested in the link [1] and this >>> patch) >>> should be sent together as a 2-patch series. >> >> then this is reasonable. You could help this into a new patchset if >> you >> are free(add Fixes tag). >> >> Thanks. >> >>> >>> Thanks, >>> Dafna >>> >>>> >>>> Thanks. >>>> >>>> [1] >>>> >> >> > https://elixir.bootlin.com/linux/v5.15/source/drivers/iommu/mtk_iommu.c#L236 >>>> >>>>> I can resend the patch on your behalf if you want. > > What's your plan here? > > If you send the two as a independent patchset on v5.16, I will rebase > yours. > If you have no time for this, I could help this. Hi, just sent it in the patchset ` iommu/mediatek: fix tlb flush logic` Thanks, Dafna > > Thanks. > >>>>> >>>>> Thanks, >>>>> Dafna >>>>> >>>>> On 23.09.21 14:58, Yong Wu wrote: >>>>>> Prepare for 2 HWs that sharing pgtable in different power- >>>>>> domains. >>>>>> >>>>>> When there are 2 M4U HWs, it may has problem in the >>>>>> flush_range >>>>>> in >>>>>> which >>>>>> we get the pm_status via the m4u dev, BUT that function don't >>>>>> reflect the >>>>>> real power-domain status of the HW since there may be other >>>>>> HW >>>>>> also >>>>>> use >>>>>> that power-domain. >>>>>> >>>>>> The function dma_alloc_attrs help allocate the iommu buffer >>>>>> which >>>>>> need the corresponding power domain since tlb flush is needed >>>>>> when >>>>>> preparing iova. BUT this function only is for allocating >>>>>> buffer, >>>>>> we have no good reason to request the user always call >>>>>> pm_runtime_get >>>>>> before calling dma_alloc_xxx. Therefore, we add a >>>>>> tlb_flush_all >>>>>> in the pm_runtime_resume to make sure the tlb always is >>>>>> clean. >>>>>> >>>>>> Another solution is always call pm_runtime_get in the >>>>>> tlb_flush_range. >>>>>> This will trigger pm runtime resume/backup so often when the >>>>>> iommu >>>>>> power is not active at some time(means user don't call >>>>>> pm_runtime_get >>>>>> before calling dma_alloc_xxx), This may cause the performance >>>>>> drop. >>>>>> thus we don't use this. >>>>>> >>>>>> In other case, the iommu's power should always be active via >>>>>> device >>>>>> link with smi. >>>>>> >>>>>> The previous SoC don't have PM except mt8192. the mt8192 >>>>>> IOMMU >>>>>> is >>>>>> display's >>>>>> power-domain which nearly always is enabled. thus no need fix >>>>>> tags >>>>>> here. >>>>>> Prepare for mt8195. >>>>>> >>>>>> Signed-off-by: Yong Wu <yong.wu@mediatek.com> >>>>>> --- >>>>>> drivers/iommu/mtk_iommu.c | 11 +++++++++++ >>>>>> 1 file changed, 11 insertions(+) >>>>>> >>>>>> diff --git a/drivers/iommu/mtk_iommu.c >>>>>> b/drivers/iommu/mtk_iommu.c >>>>>> index 44cf5547d084..e9e94944ed91 100644 >>>>>> --- a/drivers/iommu/mtk_iommu.c >>>>>> +++ b/drivers/iommu/mtk_iommu.c >>>>>> @@ -984,6 +984,17 @@ static int __maybe_unused >>>>>> mtk_iommu_runtime_resume(struct device *dev) >>>>>> return ret; >>>>>> } >>>>>> >>>>>> + /* >>>>>> + * Users may allocate dma buffer before they call >>>>>> pm_runtime_get, then >>>>>> + * it will lack the necessary tlb flush. >>>>>> + * >>>>>> + * We have no good reason to request the users always >>>>>> call >>>>>> dma_alloc_xx >>>>>> + * after pm_runtime_get_sync. >>>>>> + * >>>>>> + * Thus, Make sure the tlb always is clean after each >>>>>> PM >>>>>> resume. >>>>>> + */ >>>>>> + mtk_iommu_tlb_do_flush_all(data); >>>>>> + >>>>>> /* >>>>>> * Uppon first resume, only enable the clk and return, >>>>>> since >>>>>> the values of the >>>>>> * registers are not yet set. >>>>>> >> >> _______________________________________________ >> iommu mailing list >> iommu@lists.linux-foundation.org >> https://lists.linuxfoundation.org/mailman/listinfo/iommu
On 10.11.21 09:50, Yong Wu wrote: > On Wed, 2021-11-10 at 07:29 +0200, Dafna Hirschfeld wrote: >> >> On 10.11.21 04:20, Yong Wu wrote: >>> On Tue, 2021-11-09 at 14:21 +0200, Dafna Hirschfeld wrote: >>>> Hi >>>> This patch is needed in order to update the tlb when a device is >>>> powered on. >>>> Could you send this patch alone without the whole series so it >>>> get >>>> accepted easier? >>> >>> Which SoC are you testing on? In previous SoC, the IOMMU HW don't >>> have >>> power-domain, and we have a "has_pm"[1] in the tlb function for >>> that >>> case. The "has_pm" should be always 0 for the previous SoC like >>> mt8173, >>> it should always tlb synchronize. >>> >>> thus, Could you help share more about your issue? In which case it >>> lack >>> the necessary tlb operation. At least, We need confirm if it needs >>> a >>> "Fixes" tags if sending this patch alone. >> >> Hi, >> I work with the mtk-vcodec driver on mt8173. As you wrote, the iommu >> doesn't >> have a power-domain and so when allocating buffers before the device >> is powered >> on, there is the warning >> "Partial TLB flush timed out, falling back to full flush" >> flooding the log buf. > > oh. Thanks very much for your information. Get it now. > > This issue should be introduced by the: > > b34ea31fe013 ("iommu/mediatek: Always enable the clk on resume") Hi, reverting this commit didn't solve those warnings, I think this is because in the function mtk_iommu_attach_device the first call to pm_runtime_resume_and_get does not turn the clks on since m4u_dom is not yet initialize. And then mtk_iommu_attach_device calls pm_runtime_put right after mtk_iommu_hw_init is called (where the clks are turned on) thanks, Dafna > > tlb failed due to the bclk is not enabled. Could you help try that > after reverting this? > >> >> Sebastian Reichel suggested to remove the 'if(has_pm)' check to avoid >> this warning, >> and avoid flushing the tlb if the device is off: >> >> [1] http://ix.io/3Eyr >> >> This fixes the warning, but then the tlb is not flushed in sync, >> Therefore the tlb should be flushed when the device is resumed. >> >> So the two patches (the one suggested in the link [1] and this patch) >> should be sent together as a 2-patch series. > > then this is reasonable. You could help this into a new patchset if you > are free(add Fixes tag). > > Thanks. > >> >> Thanks, >> Dafna >> >>> >>> Thanks. >>> >>> [1] >>> > https://elixir.bootlin.com/linux/v5.15/source/drivers/iommu/mtk_iommu.c#L236 >>> >>>> I can resend the patch on your behalf if you want. >>>> >>>> Thanks, >>>> Dafna >>>> >>>> On 23.09.21 14:58, Yong Wu wrote: >>>>> Prepare for 2 HWs that sharing pgtable in different power- >>>>> domains. >>>>> >>>>> When there are 2 M4U HWs, it may has problem in the flush_range >>>>> in >>>>> which >>>>> we get the pm_status via the m4u dev, BUT that function don't >>>>> reflect the >>>>> real power-domain status of the HW since there may be other HW >>>>> also >>>>> use >>>>> that power-domain. >>>>> >>>>> The function dma_alloc_attrs help allocate the iommu buffer >>>>> which >>>>> need the corresponding power domain since tlb flush is needed >>>>> when >>>>> preparing iova. BUT this function only is for allocating >>>>> buffer, >>>>> we have no good reason to request the user always call >>>>> pm_runtime_get >>>>> before calling dma_alloc_xxx. Therefore, we add a tlb_flush_all >>>>> in the pm_runtime_resume to make sure the tlb always is clean. >>>>> >>>>> Another solution is always call pm_runtime_get in the >>>>> tlb_flush_range. >>>>> This will trigger pm runtime resume/backup so often when the >>>>> iommu >>>>> power is not active at some time(means user don't call >>>>> pm_runtime_get >>>>> before calling dma_alloc_xxx), This may cause the performance >>>>> drop. >>>>> thus we don't use this. >>>>> >>>>> In other case, the iommu's power should always be active via >>>>> device >>>>> link with smi. >>>>> >>>>> The previous SoC don't have PM except mt8192. the mt8192 IOMMU >>>>> is >>>>> display's >>>>> power-domain which nearly always is enabled. thus no need fix >>>>> tags >>>>> here. >>>>> Prepare for mt8195. >>>>> >>>>> Signed-off-by: Yong Wu <yong.wu@mediatek.com> >>>>> --- >>>>> drivers/iommu/mtk_iommu.c | 11 +++++++++++ >>>>> 1 file changed, 11 insertions(+) >>>>> >>>>> diff --git a/drivers/iommu/mtk_iommu.c >>>>> b/drivers/iommu/mtk_iommu.c >>>>> index 44cf5547d084..e9e94944ed91 100644 >>>>> --- a/drivers/iommu/mtk_iommu.c >>>>> +++ b/drivers/iommu/mtk_iommu.c >>>>> @@ -984,6 +984,17 @@ static int __maybe_unused >>>>> mtk_iommu_runtime_resume(struct device *dev) >>>>> return ret; >>>>> } >>>>> >>>>> + /* >>>>> + * Users may allocate dma buffer before they call >>>>> pm_runtime_get, then >>>>> + * it will lack the necessary tlb flush. >>>>> + * >>>>> + * We have no good reason to request the users always >>>>> call >>>>> dma_alloc_xx >>>>> + * after pm_runtime_get_sync. >>>>> + * >>>>> + * Thus, Make sure the tlb always is clean after each >>>>> PM >>>>> resume. >>>>> + */ >>>>> + mtk_iommu_tlb_do_flush_all(data); >>>>> + >>>>> /* >>>>> * Uppon first resume, only enable the clk and return, >>>>> since >>>>> the values of the >>>>> * registers are not yet set. >>>>>
On Sat, 2021-11-27 at 12:11 +0200, Dafna Hirschfeld wrote: > > On 10.11.21 09:50, Yong Wu wrote: > > On Wed, 2021-11-10 at 07:29 +0200, Dafna Hirschfeld wrote: > > > > > > On 10.11.21 04:20, Yong Wu wrote: > > > > On Tue, 2021-11-09 at 14:21 +0200, Dafna Hirschfeld wrote: > > > > > Hi > > > > > This patch is needed in order to update the tlb when a device > > > > > is > > > > > powered on. > > > > > Could you send this patch alone without the whole series so > > > > > it > > > > > get > > > > > accepted easier? > > > > > > > > Which SoC are you testing on? In previous SoC, the IOMMU HW > > > > don't > > > > have > > > > power-domain, and we have a "has_pm"[1] in the tlb function for > > > > that > > > > case. The "has_pm" should be always 0 for the previous SoC like > > > > mt8173, > > > > it should always tlb synchronize. > > > > > > > > thus, Could you help share more about your issue? In which case > > > > it > > > > lack > > > > the necessary tlb operation. At least, We need confirm if it > > > > needs > > > > a > > > > "Fixes" tags if sending this patch alone. > > > > > > Hi, > > > I work with the mtk-vcodec driver on mt8173. As you wrote, the > > > iommu > > > doesn't > > > have a power-domain and so when allocating buffers before the > > > device > > > is powered > > > on, there is the warning > > > "Partial TLB flush timed out, falling back to full flush" > > > flooding the log buf. > > > > oh. Thanks very much for your information. Get it now. > > > > This issue should be introduced by the: > > > > b34ea31fe013 ("iommu/mediatek: Always enable the clk on resume") > > Hi, reverting this commit didn't solve those warnings, > I think this is because in the function mtk_iommu_attach_device > the first call to pm_runtime_resume_and_get does not turn the clks on > since m4u_dom is not yet initialize. And then mtk_iommu_attach_device > calls pm_runtime_put right after mtk_iommu_hw_init is called > (where the clks are turned on) oh. Right. this is also related with the patch of "Add power-domain operation". The current problem is that there is a redundant log of "Partial TLB flush timed out" in mt8173. We need fix this issue firstly. Are you going to prepare the patches again? If not, I could help this. You could help confirm them if you are free. Thanks. > > thanks, > Dafna > > > > > > tlb failed due to the bclk is not enabled. Could you help try that > > after reverting this? > > > > > > > > Sebastian Reichel suggested to remove the 'if(has_pm)' check to > > > avoid > > > this warning, > > > and avoid flushing the tlb if the device is off: > > > > > > [1] http://ix.io/3Eyr > > > > > > This fixes the warning, but then the tlb is not flushed in sync, > > > Therefore the tlb should be flushed when the device is resumed. > > > > > > So the two patches (the one suggested in the link [1] and this > > > patch) > > > should be sent together as a 2-patch series. > > > > then this is reasonable. You could help this into a new patchset if > > you > > are free(add Fixes tag). > > > > Thanks. > > > > > > > > Thanks, > > > Dafna > > >
On 30.11.21 09:39, Yong Wu wrote: > On Sat, 2021-11-27 at 12:11 +0200, Dafna Hirschfeld wrote: >> >> On 10.11.21 09:50, Yong Wu wrote: >>> On Wed, 2021-11-10 at 07:29 +0200, Dafna Hirschfeld wrote: >>>> >>>> On 10.11.21 04:20, Yong Wu wrote: >>>>> On Tue, 2021-11-09 at 14:21 +0200, Dafna Hirschfeld wrote: >>>>>> Hi >>>>>> This patch is needed in order to update the tlb when a device >>>>>> is >>>>>> powered on. >>>>>> Could you send this patch alone without the whole series so >>>>>> it >>>>>> get >>>>>> accepted easier? >>>>> >>>>> Which SoC are you testing on? In previous SoC, the IOMMU HW >>>>> don't >>>>> have >>>>> power-domain, and we have a "has_pm"[1] in the tlb function for >>>>> that >>>>> case. The "has_pm" should be always 0 for the previous SoC like >>>>> mt8173, >>>>> it should always tlb synchronize. >>>>> >>>>> thus, Could you help share more about your issue? In which case >>>>> it >>>>> lack >>>>> the necessary tlb operation. At least, We need confirm if it >>>>> needs >>>>> a >>>>> "Fixes" tags if sending this patch alone. >>>> >>>> Hi, >>>> I work with the mtk-vcodec driver on mt8173. As you wrote, the >>>> iommu >>>> doesn't >>>> have a power-domain and so when allocating buffers before the >>>> device >>>> is powered >>>> on, there is the warning >>>> "Partial TLB flush timed out, falling back to full flush" >>>> flooding the log buf. >>> >>> oh. Thanks very much for your information. Get it now. >>> >>> This issue should be introduced by the: >>> >>> b34ea31fe013 ("iommu/mediatek: Always enable the clk on resume") >> >> Hi, reverting this commit didn't solve those warnings, >> I think this is because in the function mtk_iommu_attach_device >> the first call to pm_runtime_resume_and_get does not turn the clks on >> since m4u_dom is not yet initialize. And then mtk_iommu_attach_device >> calls pm_runtime_put right after mtk_iommu_hw_init is called >> (where the clks are turned on) > > oh. Right. this is also related with the patch of "Add power-domain > operation". > > The current problem is that there is a redundant log of "Partial TLB > flush timed out" in mt8173. We need fix this issue firstly. Are you > going to prepare the patches again? If not, I could help this. You > could help confirm them if you are free. Hi, I already sent a patch last week: https://lore.kernel.org/linux-iommu/afb46ad6ca9477a2bf71233858406caa6ccb1588.camel@mediatek.com/T/ could you please review it? Thanks, Dafna > > Thanks. > >> >> thanks, >> Dafna >> >> >>> >>> tlb failed due to the bclk is not enabled. Could you help try that >>> after reverting this? >>> >>>> >>>> Sebastian Reichel suggested to remove the 'if(has_pm)' check to >>>> avoid >>>> this warning, >>>> and avoid flushing the tlb if the device is off: >>>> >>>> [1] http://ix.io/3Eyr >>>> >>>> This fixes the warning, but then the tlb is not flushed in sync, >>>> Therefore the tlb should be flushed when the device is resumed. >>>> >>>> So the two patches (the one suggested in the link [1] and this >>>> patch) >>>> should be sent together as a 2-patch series. >>> >>> then this is reasonable. You could help this into a new patchset if >>> you >>> are free(add Fixes tag). >>> >>> Thanks. >>> >>>> >>>> Thanks, >>>> Dafna >>>> >
On Tue, 2021-11-30 at 13:33 +0200, Dafna Hirschfeld wrote: > > On 30.11.21 09:39, Yong Wu wrote: > > On Sat, 2021-11-27 at 12:11 +0200, Dafna Hirschfeld wrote: > > > > > > On 10.11.21 09:50, Yong Wu wrote: > > > > On Wed, 2021-11-10 at 07:29 +0200, Dafna Hirschfeld wrote: > > > > > > > > > > On 10.11.21 04:20, Yong Wu wrote: > > > > > > On Tue, 2021-11-09 at 14:21 +0200, Dafna Hirschfeld wrote: > > > > > > > Hi > > > > > > > This patch is needed in order to update the tlb when a > > > > > > > device > > > > > > > is > > > > > > > powered on. > > > > > > > Could you send this patch alone without the whole series > > > > > > > so > > > > > > > it > > > > > > > get > > > > > > > accepted easier? > > > > > > > > > > > > Which SoC are you testing on? In previous SoC, the IOMMU HW > > > > > > don't > > > > > > have > > > > > > power-domain, and we have a "has_pm"[1] in the tlb function > > > > > > for > > > > > > that > > > > > > case. The "has_pm" should be always 0 for the previous SoC > > > > > > like > > > > > > mt8173, > > > > > > it should always tlb synchronize. > > > > > > > > > > > > thus, Could you help share more about your issue? In which > > > > > > case > > > > > > it > > > > > > lack > > > > > > the necessary tlb operation. At least, We need confirm if > > > > > > it > > > > > > needs > > > > > > a > > > > > > "Fixes" tags if sending this patch alone. > > > > > > > > > > Hi, > > > > > I work with the mtk-vcodec driver on mt8173. As you wrote, > > > > > the > > > > > iommu > > > > > doesn't > > > > > have a power-domain and so when allocating buffers before the > > > > > device > > > > > is powered > > > > > on, there is the warning > > > > > "Partial TLB flush timed out, falling back to full flush" > > > > > flooding the log buf. > > > > > > > > oh. Thanks very much for your information. Get it now. > > > > > > > > This issue should be introduced by the: > > > > > > > > b34ea31fe013 ("iommu/mediatek: Always enable the clk on > > > > resume") > > > > > > Hi, reverting this commit didn't solve those warnings, > > > I think this is because in the function mtk_iommu_attach_device > > > the first call to pm_runtime_resume_and_get does not turn the > > > clks on > > > since m4u_dom is not yet initialize. And then > > > mtk_iommu_attach_device > > > calls pm_runtime_put right after mtk_iommu_hw_init is called > > > (where the clks are turned on) > > > > oh. Right. this is also related with the patch of "Add power-domain > > operation". > > > > The current problem is that there is a redundant log of "Partial > > TLB > > flush timed out" in mt8173. We need fix this issue firstly. Are you > > going to prepare the patches again? If not, I could help this. You > > could help confirm them if you are free. > > Hi, > I already sent a patch last week: > https://lore.kernel.org/linux-iommu/afb46ad6ca9477a2bf71233858406caa6ccb1588.camel@mediatek.com/T/ > could you please review it? I have replied them before you send this reply. Have you got them? https://lore.kernel.org/linux-iommu/afb46ad6ca9477a2bf71233858406caa6ccb1588.camel@mediatek.com/ https://lore.kernel.org/linux-iommu/6abef78f6447c626b737fd35688f421c29871f43.camel@mediatek.com/ Thanks. > > Thanks, > Dafna > > > > > > > Thanks. > > > > > > > > thanks, > > > Dafna > > > > > > > > > > > > > > tlb failed due to the bclk is not enabled. Could you help try > > > > that > > > > after reverting this? > > > > > > > > > > > > > > Sebastian Reichel suggested to remove the 'if(has_pm)' check > > > > > to > > > > > avoid > > > > > this warning, > > > > > and avoid flushing the tlb if the device is off: > > > > > > > > > > [1] http://ix.io/3Eyr > > > > > > > > > > This fixes the warning, but then the tlb is not flushed in > > > > > sync, > > > > > Therefore the tlb should be flushed when the device is > > > > > resumed. > > > > > > > > > > So the two patches (the one suggested in the link [1] and > > > > > this > > > > > patch) > > > > > should be sent together as a 2-patch series. > > > > > > > > then this is reasonable. You could help this into a new > > > > patchset if > > > > you > > > > are free(add Fixes tag). > > > > > > > > Thanks. > > > > > > > > > > > > > > Thanks, > > > > > Dafna > > > > >
Il 23/09/21 13:58, Yong Wu ha scritto: > Prepare for 2 HWs that sharing pgtable in different power-domains. > > When there are 2 M4U HWs, it may has problem in the flush_range in which > we get the pm_status via the m4u dev, BUT that function don't reflect the > real power-domain status of the HW since there may be other HW also use > that power-domain. > > The function dma_alloc_attrs help allocate the iommu buffer which > need the corresponding power domain since tlb flush is needed when > preparing iova. BUT this function only is for allocating buffer, > we have no good reason to request the user always call pm_runtime_get > before calling dma_alloc_xxx. Therefore, we add a tlb_flush_all > in the pm_runtime_resume to make sure the tlb always is clean. > > Another solution is always call pm_runtime_get in the tlb_flush_range. > This will trigger pm runtime resume/backup so often when the iommu > power is not active at some time(means user don't call pm_runtime_get > before calling dma_alloc_xxx), This may cause the performance drop. > thus we don't use this. > > In other case, the iommu's power should always be active via device > link with smi. > > The previous SoC don't have PM except mt8192. the mt8192 IOMMU is display's > power-domain which nearly always is enabled. thus no need fix tags here. > Prepare for mt8195. > > Signed-off-by: Yong Wu <yong.wu@mediatek.com> Reviewed-by: AngeloGioacchino Del Regno <angelogioacchino.delregno@collabora.com>
diff --git a/drivers/iommu/mtk_iommu.c b/drivers/iommu/mtk_iommu.c index 44cf5547d084..e9e94944ed91 100644 --- a/drivers/iommu/mtk_iommu.c +++ b/drivers/iommu/mtk_iommu.c @@ -984,6 +984,17 @@ static int __maybe_unused mtk_iommu_runtime_resume(struct device *dev) return ret; } + /* + * Users may allocate dma buffer before they call pm_runtime_get, then + * it will lack the necessary tlb flush. + * + * We have no good reason to request the users always call dma_alloc_xx + * after pm_runtime_get_sync. + * + * Thus, Make sure the tlb always is clean after each PM resume. + */ + mtk_iommu_tlb_do_flush_all(data); + /* * Uppon first resume, only enable the clk and return, since the values of the * registers are not yet set.
Prepare for 2 HWs that sharing pgtable in different power-domains. When there are 2 M4U HWs, it may has problem in the flush_range in which we get the pm_status via the m4u dev, BUT that function don't reflect the real power-domain status of the HW since there may be other HW also use that power-domain. The function dma_alloc_attrs help allocate the iommu buffer which need the corresponding power domain since tlb flush is needed when preparing iova. BUT this function only is for allocating buffer, we have no good reason to request the user always call pm_runtime_get before calling dma_alloc_xxx. Therefore, we add a tlb_flush_all in the pm_runtime_resume to make sure the tlb always is clean. Another solution is always call pm_runtime_get in the tlb_flush_range. This will trigger pm runtime resume/backup so often when the iommu power is not active at some time(means user don't call pm_runtime_get before calling dma_alloc_xxx), This may cause the performance drop. thus we don't use this. In other case, the iommu's power should always be active via device link with smi. The previous SoC don't have PM except mt8192. the mt8192 IOMMU is display's power-domain which nearly always is enabled. thus no need fix tags here. Prepare for mt8195. Signed-off-by: Yong Wu <yong.wu@mediatek.com> --- drivers/iommu/mtk_iommu.c | 11 +++++++++++ 1 file changed, 11 insertions(+)