| Message ID | 20200604112040.22144-3-wsa+renesas@sang-engineering.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | renesas_sdhi: fix hang when SCC loses its clock | expand |
Hi Wolfram-san, Thank you for the patch! > From: Wolfram Sang, Sent: Thursday, June 4, 2020 8:21 PM > > Tuning procedure switches to lower frequencies but that will turn the > SCC off and accessing its register then will hang. So, flag when we are > tuning and keep the current setup of the external clock if we are doing > so. Note that we still switch to the lower frequency because of the > internal divider. We just make sure to not modify the external clock. > This patch depends on a MMC core patch calling the downgrade function > earlier. > > Signed-off-by: Wolfram Sang <wsa+renesas@sang-engineering.com> > --- > drivers/mmc/host/renesas_sdhi.h | 1 + > drivers/mmc/host/renesas_sdhi_core.c | 25 ++++++++++++++++++++++--- > 2 files changed, 23 insertions(+), 3 deletions(-) > > diff --git a/drivers/mmc/host/renesas_sdhi.h b/drivers/mmc/host/renesas_sdhi.h > index 14c64caefc64..58a8c9133ba4 100644 > --- a/drivers/mmc/host/renesas_sdhi.h > +++ b/drivers/mmc/host/renesas_sdhi.h > @@ -59,6 +59,7 @@ struct renesas_sdhi { > u32 scc_tappos; > u32 scc_tappos_hs400; > bool doing_tune; > + bool keep_scc_freq; > > /* Tuning values: 1 for success, 0 for failure */ > DECLARE_BITMAP(taps, BITS_PER_LONG); > diff --git a/drivers/mmc/host/renesas_sdhi_core.c b/drivers/mmc/host/renesas_sdhi_core.c > index 15e21894bd44..589a59fb70eb 100644 > --- a/drivers/mmc/host/renesas_sdhi_core.c > +++ b/drivers/mmc/host/renesas_sdhi_core.c > @@ -117,8 +117,12 @@ static unsigned int renesas_sdhi_clk_update(struct tmio_mmc_host *host, > unsigned int freq, diff, best_freq = 0, diff_min = ~0; > int i; > > - /* tested only on R-Car Gen2+ currently; may work for others */ > - if (!(host->pdata->flags & TMIO_MMC_MIN_RCAR2)) > + /* > + * We simply return the current rate if a) we are not on a R-Car Gen2+ > + * SoC (may work for others, but untested) or b) if the SCC needs its > + * clock during tuning, so we don't change the external clock setup. > + */ > + if (!(host->pdata->flags & TMIO_MMC_MIN_RCAR2) || priv->keep_scc_freq) > return clk_get_rate(priv->clk); > > /* > @@ -323,6 +327,9 @@ static void renesas_sdhi_hs400_complete(struct mmc_host *mmc) > u32 bad_taps = priv->quirks ? priv->quirks->hs400_bad_taps : 0; > bool use_4tap = priv->quirks && priv->quirks->hs400_4taps; > > + /* Tuning done, no special handling for SCC clock needed anymore */ > + priv->keep_scc_freq = false; > + Setting keep_scc_freq to false is only here. But, I'm thinking we should set it in some error paths like below somehow too: - error paths before hs400_complete() in mmc_select_hs400(). - error path of mmc_execute_tuning() in mmc_retune(). Best regards, Yoshihiro Shimoda
> > + /* Tuning done, no special handling for SCC clock needed anymore */ > > + priv->keep_scc_freq = false; > > + > > Setting keep_scc_freq to false is only here. But, I'm thinking > we should set it in some error paths like below somehow too: > - error paths before hs400_complete() in mmc_select_hs400(). > - error path of mmc_execute_tuning() in mmc_retune(). Hmm, I guess you are right. That would kind of spoil my approach taken here. Maybe we need another flag in the core like 'doing_tune' to supplement 'doing_retune', so or driver knows when any kind of tuning is going on?
Hi Wolfram-san, > From: Wolfram Sang, Sent: Tuesday, June 9, 2020 6:27 AM > > > > + /* Tuning done, no special handling for SCC clock needed anymore */ > > > + priv->keep_scc_freq = false; > > > + > > > > Setting keep_scc_freq to false is only here. But, I'm thinking > > we should set it in some error paths like below somehow too: > > - error paths before hs400_complete() in mmc_select_hs400(). > > - error path of mmc_execute_tuning() in mmc_retune(). > > Hmm, I guess you are right. That would kind of spoil my approach taken > here. Maybe we need another flag in the core like 'doing_tune' to > supplement 'doing_retune', so or driver knows when any kind of tuning is > going on? Adding such a new flag is better, I think. Best regards, Yoshihiro Shimoda
Hi Shimoda-san,tftp 0x58000000 r8a77965-salvator-xs.dtb; tftp 0x50000000 Image-m3n-wsa; booti 0x50000000 - 0x58000000 > > > > + /* Tuning done, no special handling for SCC clock needed anymore */ > > > > + priv->keep_scc_freq = false; > > > > + > > > > > > Setting keep_scc_freq to false is only here. But, I'm thinking > > > we should set it in some error paths like below somehow too: > > > - error paths before hs400_complete() in mmc_select_hs400(). > > > - error path of mmc_execute_tuning() in mmc_retune(). > > > > Hmm, I guess you are right. That would kind of spoil my approach taken > > here. Maybe we need another flag in the core like 'doing_tune' to > > supplement 'doing_retune', so or driver knows when any kind of tuning is > > going on? > > Adding such a new flag is better, I think. So, I added a flag to the MMC core and I think it should work. However, I can't test it currently because, sadly, the issue disappeared again :( I even can't reproduce the issue with the same codebase and config which I used when I was working last time on it. And back then, the issue was happening. I am at a loss currently what really triggers this hang. I added some code to enforce reading something from the SCC with the hclk disabled. However, that reading works fine today here, no hang. So, it seems that keeping hclk enabled will fix the hang. However, it doesn't look like it will hang just when we allow to disable it. Seems something else is part of the equation, too... I kept trying to figure this out for the last two days, but no success so far. Will keep you updated. Thanks, Wolfram
Hi Wolfram-san, > From: Wolfram Sang, Sent: Friday, August 14, 2020 4:15 PM > > > > > > + /* Tuning done, no special handling for SCC clock needed anymore */ > > > > > + priv->keep_scc_freq = false; > > > > > + > > > > > > > > Setting keep_scc_freq to false is only here. But, I'm thinking > > > > we should set it in some error paths like below somehow too: > > > > - error paths before hs400_complete() in mmc_select_hs400(). > > > > - error path of mmc_execute_tuning() in mmc_retune(). > > > > > > Hmm, I guess you are right. That would kind of spoil my approach taken > > > here. Maybe we need another flag in the core like 'doing_tune' to > > > supplement 'doing_retune', so or driver knows when any kind of tuning is > > > going on? > > > > Adding such a new flag is better, I think. > > So, I added a flag to the MMC core and I think it should work. However, > I can't test it currently because, sadly, the issue disappeared again :( I got a report from a colleague about this issue. According to the report, this issue is related to retuning. When retuning happens, the mmc core calls mmc_hs400_to_hs200() and then mmc_hs400_to_hs200() will set the clock as 52MHz at first. So, it's possible to cause the issue. It's difficult to cause retuning in normal situation. But, according to the report, if we add a code which the sdhi driver reports an error at the first CMD18 once, we can cause retuning and then the issue happens. Best regards, Yoshihiro Shimoda
Hi Shimoda-san, > I got a report from a colleague about this issue. According to the report, > this issue is related to retuning. When retuning happens, the mmc core > calls mmc_hs400_to_hs200() and then mmc_hs400_to_hs200() will set the clock > as 52MHz at first. So, it's possible to cause the issue. > > It's difficult to cause retuning in normal situation. But, according to > the report, if we add a code which the sdhi driver reports an error > at the first CMD18 once, we can cause retuning and then the issue happens. I took the liberty of a different approach because I wanted to reproduce the issue when doing the initial tuning and not a retune. Because my new series adds (and checks) a flag for doing_initial_tune, so I really wanted to excercise this code path. This is a real problem, too, because I saw this with my boards earlier back then. And halleluja, today I saw it again, once. I switched to my H3-ES2.0 board which I haven't used for weeks. And when booting that for the first time, I got a failure including logs. Later boots just went fine. And because of the logs, I could finally inject an error which will reproducibly cause the boot to hang because of a stalled SCC. Tada, here is the patch: From: Wolfram Sang <wsa+renesas@sang-engineering.com> Subject: [PATCH] GOLD: simulate stalled SCC Geez, this took ages to find... --- drivers/mmc/core/mmc.c | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/mmc/core/mmc.c b/drivers/mmc/core/mmc.c index 216bd1aed373..6b3056437b37 100644 --- a/drivers/mmc/core/mmc.c +++ b/drivers/mmc/core/mmc.c @@ -1218,6 +1218,7 @@ static int mmc_select_hs400(struct mmc_card *card) host->ops->hs400_complete(host); err = mmc_switch_status(card, true); + err = -EILSEQ; if (err) goto out_err; Interestingly, the other mmc_switch_status() in mmc_select_hs400() was not stalling the SCC. Anyhow, after this failute, the MMC core switches back to 300kHz and the SCC clock is off but for some reason SCC is still accessed. I will investigate why. The good news is that my new patch set fixes the hang as expected. The board will continue to boot so we probably want to have this series. However, I have the feeling that this SCC access which hangs the board might be a bug because of an unintended code path. I mean, this is also one reason why the bug triggers so rarely these days. We have been fixing a lot of things and the SCC is only accessed when it should be accessed. We will see. I also need to test other boards, too. So much for now, I hope I can report more later. Happy hacking and kind regards, Wolfram
> not stalling the SCC. Anyhow, after this failute, the MMC core switches > back to 300kHz and the SCC clock is off but for some reason SCC is still > accessed. I will investigate why. The good news is that my new patch set > fixes the hang as expected. The board will continue to boot so we > probably want to have this series. However, I have the feeling that this > SCC access which hangs the board might be a bug because of an unintended > code path. I mean, this is also one reason why the bug triggers so > rarely these days. We have been fixing a lot of things and the SCC is > only accessed when it should be accessed. We will see. I also need to > test other boards, too. Some more good news: I can reproduce the issue now not only with H3-ES2.0 but also with my M3-N. Interesting news: The hang comes from a code path I would have not expected. It is not because of accessing an SCC register, it is this line from renesas_sdhi_set_clock() which causes the issue: 186 sd_ctrl_write16(host, CTL_SD_CARD_CLK_CTL, clk & CLK_CTL_DIV_MASK); I mean I can guess that the clock setting has something to do with the SCC, but I can't see the direct connection with the documentation I have. I will stop that research here and will prepare now my series to leave the SCC clock enabled as long as some tuning is in progress.
diff --git a/drivers/mmc/host/renesas_sdhi.h b/drivers/mmc/host/renesas_sdhi.h index 14c64caefc64..58a8c9133ba4 100644 --- a/drivers/mmc/host/renesas_sdhi.h +++ b/drivers/mmc/host/renesas_sdhi.h @@ -59,6 +59,7 @@ struct renesas_sdhi { u32 scc_tappos; u32 scc_tappos_hs400; bool doing_tune; + bool keep_scc_freq; /* Tuning values: 1 for success, 0 for failure */ DECLARE_BITMAP(taps, BITS_PER_LONG); diff --git a/drivers/mmc/host/renesas_sdhi_core.c b/drivers/mmc/host/renesas_sdhi_core.c index 15e21894bd44..589a59fb70eb 100644 --- a/drivers/mmc/host/renesas_sdhi_core.c +++ b/drivers/mmc/host/renesas_sdhi_core.c @@ -117,8 +117,12 @@ static unsigned int renesas_sdhi_clk_update(struct tmio_mmc_host *host, unsigned int freq, diff, best_freq = 0, diff_min = ~0; int i; - /* tested only on R-Car Gen2+ currently; may work for others */ - if (!(host->pdata->flags & TMIO_MMC_MIN_RCAR2)) + /* + * We simply return the current rate if a) we are not on a R-Car Gen2+ + * SoC (may work for others, but untested) or b) if the SCC needs its + * clock during tuning, so we don't change the external clock setup. + */ + if (!(host->pdata->flags & TMIO_MMC_MIN_RCAR2) || priv->keep_scc_freq) return clk_get_rate(priv->clk); /* @@ -323,6 +327,9 @@ static void renesas_sdhi_hs400_complete(struct mmc_host *mmc) u32 bad_taps = priv->quirks ? priv->quirks->hs400_bad_taps : 0; bool use_4tap = priv->quirks && priv->quirks->hs400_4taps; + /* Tuning done, no special handling for SCC clock needed anymore */ + priv->keep_scc_freq = false; + sd_ctrl_write16(host, CTL_SD_CARD_CLK_CTL, ~CLK_CTL_SCLKEN & sd_ctrl_read16(host, CTL_SD_CARD_CLK_CTL)); @@ -401,6 +408,14 @@ static void renesas_sdhi_disable_scc(struct mmc_host *mmc) sd_ctrl_write16(host, CTL_SD_CARD_CLK_CTL, CLK_CTL_SCLKEN | sd_ctrl_read16(host, CTL_SD_CARD_CLK_CTL)); + + /* + * During retune, we still access SCC registers, so keep clock on. + * We have it here again because for retuning it is too late when we + * only enable this flag when HS400 tuning gets initialized. + */ + if (mmc_doing_retune(mmc)) + priv->keep_scc_freq = true; } static void renesas_sdhi_reset_hs400_mode(struct tmio_mmc_host *host, @@ -427,8 +442,12 @@ static void renesas_sdhi_reset_hs400_mode(struct tmio_mmc_host *host, static int renesas_sdhi_prepare_hs400_tuning(struct mmc_host *mmc, struct mmc_ios *ios) { struct tmio_mmc_host *host = mmc_priv(mmc); + struct renesas_sdhi *priv = host_to_priv(host); + + /* During tuning, we still access SCC registers, so keep clock on */ + priv->keep_scc_freq = true; + renesas_sdhi_reset_hs400_mode(host, priv); - renesas_sdhi_reset_hs400_mode(host, host_to_priv(host)); return 0; }
Tuning procedure switches to lower frequencies but that will turn the SCC off and accessing its register then will hang. So, flag when we are tuning and keep the current setup of the external clock if we are doing so. Note that we still switch to the lower frequency because of the internal divider. We just make sure to not modify the external clock. This patch depends on a MMC core patch calling the downgrade function earlier. Signed-off-by: Wolfram Sang <wsa+renesas@sang-engineering.com> --- drivers/mmc/host/renesas_sdhi.h | 1 + drivers/mmc/host/renesas_sdhi_core.c | 25 ++++++++++++++++++++++--- 2 files changed, 23 insertions(+), 3 deletions(-)