| Message ID | 1374129572-6079-1-git-send-email-michal.kazior@tieto.com (mailing list archive) |
|---|---|
| State | Not Applicable, archived |
| Headers | show |
Michal Kazior <michal.kazior@tieto.com> writes: > There was a slight race during PCI shutdown. Since > interrupts weren't really stopped (only Copy > Engine interrupts were disabled through device hw > registers) it was possible for a firmware > indication (crash) interrupt to come in after > tasklets were synced/killed. This would cause > memory corruption and a panic in most cases. It > was also possible for interrupt to come before CE > was initialized during device probing. > > Interrupts are required for BMI phase so they are enabled as soon as > power_up() is called but are freed upon both power_down() and stop() > so there's asymmetry here. As by design stop() cannot be followed by > start() it is okay. Both power_down() and stop() should be merged > later on to avoid confusion. Why are the interrupts freed both in power_down() and stop()? I don't get that. What if we call disable_irq() in power_down() instead? > Before this can be really properly fixed var/hw > init code split is necessary. > > Signed-off-by: Michal Kazior <michal.kazior@tieto.com> > --- > > Please note: this is based on my (still under > review at the time of posting) previous patchests: > device setup refactor and recovery. > > I'm posting this before those patchsets are merged > so anyone interested in testing this fix (I can't > reproduce the problem on my setup) can give it a > try. This was reported by Ben, right? So this sould have a Reported-by line attributing him. > @@ -1783,16 +1792,24 @@ static int ath10k_pci_hif_power_up(struct ath10k *ar) > return 0; > > err_ce: > + /* XXX: Until var/hw init is split it's impossible to fix the ordering > + * here so we must call stop_intr() here too to prevent interrupts after > + * CE is teared down. It's okay to double call the stop_intr() > */ "FIXME:" > exit: > + ar_pci->intr_started = ret == 0; A bit too clever for the sake of readibility for my taste, but I guess it's ok. > --- a/drivers/net/wireless/ath/ath10k/pci.h > +++ b/drivers/net/wireless/ath/ath10k/pci.h > @@ -198,6 +198,7 @@ struct ath10k_pci { > * interrupts. > */ > int num_msi_intrs; > + bool intr_started; Adding a new state variable makes me worried. I really would prefer a solution which would not require that. Also if we call request_irq() in ath10k_pci_probe() we should also call free_irq() in ath10k_pci_remove() for symmetry. Just doing a temporary hack will most likely stay forever :)
On 30 July 2013 20:35, Kalle Valo <kvalo@qca.qualcomm.com> wrote: > Michal Kazior <michal.kazior@tieto.com> writes: > >> There was a slight race during PCI shutdown. Since >> interrupts weren't really stopped (only Copy >> Engine interrupts were disabled through device hw >> registers) it was possible for a firmware >> indication (crash) interrupt to come in after >> tasklets were synced/killed. This would cause >> memory corruption and a panic in most cases. It >> was also possible for interrupt to come before CE >> was initialized during device probing. >> >> Interrupts are required for BMI phase so they are enabled as soon as >> power_up() is called but are freed upon both power_down() and stop() >> so there's asymmetry here. As by design stop() cannot be followed by >> start() it is okay. Both power_down() and stop() should be merged >> later on to avoid confusion. > > Why are the interrupts freed both in power_down() and stop()? I don't > get that. > > What if we call disable_irq() in power_down() instead? power_down() must call free_irq(), because power_up() calls request_irq() (if you want the symmetry). If anything, the stop() should call disable_irq(), but wouldn't that mean start() should call enable_irq()? But than, irqs are needed before start().. I did think about disable_irq() but AFAIR you need to enable_irq() later on (so either way you need to keep track of the irq state or you'll get a ton of WARN_ONs from the system). I'll double check that and report back later >> Before this can be really properly fixed var/hw >> init code split is necessary. >> >> Signed-off-by: Michal Kazior <michal.kazior@tieto.com> >> --- >> >> Please note: this is based on my (still under >> review at the time of posting) previous patchests: >> device setup refactor and recovery. >> >> I'm posting this before those patchsets are merged >> so anyone interested in testing this fix (I can't >> reproduce the problem on my setup) can give it a >> try. > > This was reported by Ben, right? So this sould have a Reported-by line > attributing him. Yes. I'll fix that, provided we get through the review with the patch :) >> @@ -1783,16 +1792,24 @@ static int ath10k_pci_hif_power_up(struct ath10k *ar) >> return 0; >> >> err_ce: >> + /* XXX: Until var/hw init is split it's impossible to fix the ordering >> + * here so we must call stop_intr() here too to prevent interrupts after >> + * CE is teared down. It's okay to double call the stop_intr() >> */ > > "FIXME:" Ok. >> exit: >> + ar_pci->intr_started = ret == 0; > > A bit too clever for the sake of readibility for my taste, but I guess > it's ok. > >> --- a/drivers/net/wireless/ath/ath10k/pci.h >> +++ b/drivers/net/wireless/ath/ath10k/pci.h >> @@ -198,6 +198,7 @@ struct ath10k_pci { >> * interrupts. >> */ >> int num_msi_intrs; >> + bool intr_started; > > Adding a new state variable makes me worried. I really would prefer a > solution which would not require that. I know that. That's why I mentioned in the commit log that it is more of a workaround than a real fix. Me, I don't like this either but a real fix requires a lot of rework from what I can tell. This bug can be triggered more easily now apparently after recovery patches went in. I'm not experiencing it but I get reports of rare panics when a machine is left idle for a very long time with interfaces brought down. > Also if we call request_irq() in ath10k_pci_probe() we should also call > free_irq() in ath10k_pci_remove() for symmetry. Just doing a temporary > hack will most likely stay forever :) With the patch interrupts are temporarily enabled&disabled for probe_fw() during pci_probe() and are then not requested until mac80211 start(). Pozdrawiam / Best regards, Micha? Kazior. -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On 31 July 2013 07:50, Michal Kazior <michal.kazior@tieto.com> wrote: > On 30 July 2013 20:35, Kalle Valo <kvalo@qca.qualcomm.com> wrote: >> Michal Kazior <michal.kazior@tieto.com> writes: >> >>> There was a slight race during PCI shutdown. Since >>> interrupts weren't really stopped (only Copy >>> Engine interrupts were disabled through device hw >>> registers) it was possible for a firmware >>> indication (crash) interrupt to come in after >>> tasklets were synced/killed. This would cause >>> memory corruption and a panic in most cases. It >>> was also possible for interrupt to come before CE >>> was initialized during device probing. >>> >>> Interrupts are required for BMI phase so they are enabled as soon as >>> power_up() is called but are freed upon both power_down() and stop() >>> so there's asymmetry here. As by design stop() cannot be followed by >>> start() it is okay. Both power_down() and stop() should be merged >>> later on to avoid confusion. >> >> Why are the interrupts freed both in power_down() and stop()? I don't >> get that. >> >> What if we call disable_irq() in power_down() instead? > > power_down() must call free_irq(), because power_up() calls > request_irq() (if you want the symmetry). If anything, the stop() > should call disable_irq(), but wouldn't that mean start() should call > enable_irq()? But than, irqs are needed before start().. > > I did think about disable_irq() but AFAIR you need to enable_irq() > later on (so either way you need to keep track of the irq state or > you'll get a ton of WARN_ONs from the system). I'll double check that > and report back later enable/disable_irq must be balanced as well. There are two cases of power cycle: * power_up, power_down * power_up, start, stop, power_down If irq setup is moved from pci_probe/remove to power_up/power_down, then stop() still needs irqs to be halted - either disable_irq, or free_irq. In the latter case power_down must be prepared and not issue free_irq again. If irq setup remains in pci_probe/remove then both stop() and power_down() need irqs to be halted too. Same issue applies. If stop/power_down is merged than the whole problem is solved. This seems like the sane solution to the whole problem but requires some refactoring to be done first. Pozdrawiam / Best regards, Micha? Kazior. -- To unsubscribe from this list: send the line "unsubscribe linux-wireless" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/drivers/net/wireless/ath/ath10k/pci.c b/drivers/net/wireless/ath/ath10k/pci.c index c71b488..e814151 100644 --- a/drivers/net/wireless/ath/ath10k/pci.c +++ b/drivers/net/wireless/ath/ath10k/pci.c @@ -56,6 +56,8 @@ static void ath10k_pci_rx_pipe_cleanup(struct hif_ce_pipe_info *pipe_info); static void ath10k_pci_stop_ce(struct ath10k *ar); static void ath10k_pci_device_reset(struct ath10k *ar); static int ath10k_pci_reset_target(struct ath10k *ar); +static int ath10k_pci_start_intr(struct ath10k *ar); +static void ath10k_pci_stop_intr(struct ath10k *ar); static const struct ce_attr host_ce_config_wlan[] = { /* host->target HTC control and raw streams */ @@ -827,6 +829,7 @@ static void ath10k_pci_stop_ce(struct ath10k *ar) int i; ath10k_ce_disable_interrupts(ar); + ath10k_pci_stop_intr(ar); /* Cancel the pending tasklet */ tasklet_kill(&ar_pci->intr_tq); @@ -1742,6 +1745,12 @@ static int ath10k_pci_hif_power_up(struct ath10k *ar) { int ret; + ret = ath10k_pci_start_intr(ar); + if (ret) { + ath10k_err("could not start interrupt handling (%d)\n", ret); + goto err; + } + /* * Bring the target up cleanly. * @@ -1756,7 +1765,7 @@ static int ath10k_pci_hif_power_up(struct ath10k *ar) ret = ath10k_pci_reset_target(ar); if (ret) - goto err; + goto err_intr; if (ath10k_target_ps) { ath10k_dbg(ATH10K_DBG_PCI, "on-chip power save enabled\n"); @@ -1783,16 +1792,24 @@ static int ath10k_pci_hif_power_up(struct ath10k *ar) return 0; err_ce: + /* XXX: Until var/hw init is split it's impossible to fix the ordering + * here so we must call stop_intr() here too to prevent interrupts after + * CE is teared down. It's okay to double call the stop_intr() */ + ath10k_pci_stop_intr(ar); ath10k_pci_ce_deinit(ar); err_ps: if (!ath10k_target_ps) ath10k_do_pci_sleep(ar); +err_intr: + ath10k_pci_stop_intr(ar); err: return ret; } static void ath10k_pci_hif_power_down(struct ath10k *ar) { + ath10k_pci_stop_intr(ar); + ath10k_pci_ce_deinit(ar); if (!ath10k_target_ps) ath10k_do_pci_sleep(ar); @@ -2119,6 +2136,7 @@ static int ath10k_pci_start_intr(struct ath10k *ar) ret = ath10k_pci_start_intr_legacy(ar); exit: + ar_pci->intr_started = ret == 0; ar_pci->num_msi_intrs = num; ar_pci->ce_count = CE_COUNT; return ret; @@ -2129,6 +2147,9 @@ static void ath10k_pci_stop_intr(struct ath10k *ar) struct ath10k_pci *ar_pci = ath10k_pci_priv(ar); int i; + if (ar_pci->intr_started == false) + return; + /* There's at least one interrupt irregardless whether its legacy INTR * or MSI or MSI-X */ for (i = 0; i < max(1, ar_pci->num_msi_intrs); i++) @@ -2136,6 +2157,8 @@ static void ath10k_pci_stop_intr(struct ath10k *ar) if (ar_pci->num_msi_intrs > 0) pci_disable_msi(ar_pci->pdev); + + ar_pci->intr_started = false; } static int ath10k_pci_reset_target(struct ath10k *ar) @@ -2358,22 +2381,14 @@ static int ath10k_pci_probe(struct pci_dev *pdev, ar_pci->cacheline_sz = dma_get_cache_alignment(); - ret = ath10k_pci_start_intr(ar); - if (ret) { - ath10k_err("could not start interrupt handling (%d)\n", ret); - goto err_iomap; - } - ret = ath10k_core_register(ar); if (ret) { ath10k_err("could not register driver core (%d)\n", ret); - goto err_intr; + goto err_iomap; } return 0; -err_intr: - ath10k_pci_stop_intr(ar); err_iomap: pci_iounmap(pdev, mem); err_master: @@ -2410,7 +2425,6 @@ static void ath10k_pci_remove(struct pci_dev *pdev) tasklet_kill(&ar_pci->msi_fw_err); ath10k_core_unregister(ar); - ath10k_pci_stop_intr(ar); pci_set_drvdata(pdev, NULL); pci_iounmap(pdev, ar_pci->mem); diff --git a/drivers/net/wireless/ath/ath10k/pci.h b/drivers/net/wireless/ath/ath10k/pci.h index d3a2e6c..5eb628f 100644 --- a/drivers/net/wireless/ath/ath10k/pci.h +++ b/drivers/net/wireless/ath/ath10k/pci.h @@ -198,6 +198,7 @@ struct ath10k_pci { * interrupts. */ int num_msi_intrs; + bool intr_started; struct tasklet_struct intr_tq; struct tasklet_struct msi_fw_err;
There was a slight race during PCI shutdown. Since interrupts weren't really stopped (only Copy Engine interrupts were disabled through device hw registers) it was possible for a firmware indication (crash) interrupt to come in after tasklets were synced/killed. This would cause memory corruption and a panic in most cases. It was also possible for interrupt to come before CE was initialized during device probing. Interrupts are required for BMI phase so they are enabled as soon as power_up() is called but are freed upon both power_down() and stop() so there's asymmetry here. As by design stop() cannot be followed by start() it is okay. Both power_down() and stop() should be merged later on to avoid confusion. Before this can be really properly fixed var/hw init code split is necessary. Signed-off-by: Michal Kazior <michal.kazior@tieto.com> --- Please note: this is based on my (still under review at the time of posting) previous patchests: device setup refactor and recovery. I'm posting this before those patchsets are merged so anyone interested in testing this fix (I can't reproduce the problem on my setup) can give it a try. drivers/net/wireless/ath/ath10k/pci.c | 36 +++++++++++++++++++++++---------- drivers/net/wireless/ath/ath10k/pci.h | 1 + 2 files changed, 26 insertions(+), 11 deletions(-)