diff mbox

[15/15] tty: serial: 8250: omap: add dma support

Message ID 20140828225454.GM16006@atomide.com (mailing list archive)
State New, archived
Headers show

Commit Message

Tony Lindgren Aug. 28, 2014, 10:54 p.m. UTC
* Sebastian Andrzej Siewior <bigeasy@linutronix.de> [140828 12:37]:
> On 08/28/2014 06:46 PM, Tony Lindgren wrote:
> > 
> > Sounds like there should be some way to clear that state.. I wonder
> > if omap-serial.c had something before it's DMA support was removed?
> 
> Its DMA was removed? Like there was DMA support?

Yeah see commit 494574304711a333386e7dd5fd3ebbc3b7024994...
 
> > I'd assume when the UART is powered down by runtime PM it's state
> > is completetely reset and we could restore the non-DMA state?
> 
> I tried that by canceling the RX-DMA request and removing the DMA-enable
> bits from UART but it didn't help. Then I noticed that once that DMA en
> bit is set, the UART won't do any idle.
> 
> > Maybe post your current patches and a test patch to try to toggle
> > the DMA on and off?
> 
> Oh. I pushed my dirty tree to
>  git://git.breakpoint.cc/bigeasy/linux.git uart_v8_wip
> 
> The top most commit does not setup dma at all and adds commented out
> code how to enable DMA via SCR or FCR register. With this I hit
> core-off. Once _one_ of the modes are enabled, it doesn't work anymore.
> 
> I will try to address review comments tomorrow and hopefully post a v8
> based on -rc2. The same goes for your patch (which I will try tomorrow).

OK thanks, I'm seeing the same issue as you. And the idlest registers
don't show any blockers. Looking at the errata docs, seems like
"Usage Note 2.7" in sprz318f.pdf says:

 Details When configured for DMA operations using smartidle mode (SYSC[4:3].IDLEMODE =
 0x2), the UART module will not acknowledge incoming idle requests. As a consequence,
 it can prevent L4 from going to idle.
 When there are additional expected transfers, the UART should be placed in force-idle
 mode.

So I've added also Paul to Cc, he may have better suggestions for the
hwmod flags to use. The experimental patch below seems to allow idling
for me, care to give it a try?

Regards,

Tony

8< --------------------
From: Tony Lindgren <tony@atomide.com>
Date: Thu, 28 Aug 2014 15:41:21 -0700
Subject: [PATCH] ARM: OMAP3: Use force-idle for UARTs because of DMA errata

In sprz318f.pdf Usage Note 2.7 says that UARTs cannot acknowledge
idle requests in smartidle mode when configure for DMA operations.
This prevents L4 from going idle. Sol et's use force-idle mode
instead.

--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Comments

Sebastian Andrzej Siewior Aug. 29, 2014, 9:32 a.m. UTC | #1
On 08/29/2014 12:54 AM, Tony Lindgren wrote:
> * Sebastian Andrzej Siewior <bigeasy@linutronix.de> [140828 12:37]:
>> On 08/28/2014 06:46 PM, Tony Lindgren wrote:
>>>
>>> Sounds like there should be some way to clear that state.. I wonder
>>> if omap-serial.c had something before it's DMA support was removed?
>>
>> Its DMA was removed? Like there was DMA support?
> 
> Yeah see commit 494574304711a333386e7dd5fd3ebbc3b7024994...

Interesting. I've been browsing that file and checking other trees and
never noticed that it was there at some point. I only saw the pieces
which looked it was there.

> OK thanks, I'm seeing the same issue as you. And the idlest registers
> don't show any blockers. Looking at the errata docs, seems like
> "Usage Note 2.7" in sprz318f.pdf says:
> 
>  Details When configured for DMA operations using smartidle mode (SYSC[4:3].IDLEMODE =
>  0x2), the UART module will not acknowledge incoming idle requests. As a consequence,
>  it can prevent L4 from going to idle.
>  When there are additional expected transfers, the UART should be placed in force-idle
>  mode.

Ehm. So I haven't found an errata document for omap3630, there is
nothing like that in that one I have for am335x or dra7. The document
you mentioned is for AM3715. Interesting…

> So I've added also Paul to Cc, he may have better suggestions for the
> hwmod flags to use. The experimental patch below seems to allow idling
> for me, care to give it a try?

Yep, this one works. And I see DMA transfers (for RX side) after the
core hit idle so it seems to look good :)

> 
> Regards,
> 
> Tony

Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Felipe Balbi Aug. 29, 2014, 3:55 p.m. UTC | #2
On Fri, Aug 29, 2014 at 11:32:36AM +0200, Sebastian Andrzej Siewior wrote:
> On 08/29/2014 12:54 AM, Tony Lindgren wrote:
> > * Sebastian Andrzej Siewior <bigeasy@linutronix.de> [140828 12:37]:
> >> On 08/28/2014 06:46 PM, Tony Lindgren wrote:
> >>>
> >>> Sounds like there should be some way to clear that state.. I wonder
> >>> if omap-serial.c had something before it's DMA support was removed?
> >>
> >> Its DMA was removed? Like there was DMA support?
> > 
> > Yeah see commit 494574304711a333386e7dd5fd3ebbc3b7024994...
> 
> Interesting. I've been browsing that file and checking other trees and
> never noticed that it was there at some point. I only saw the pieces
> which looked it was there.

it was known to be broken and unused. There was no way to even enable
that code.
Tony Lindgren Aug. 29, 2014, 4:12 p.m. UTC | #3
* Sebastian Andrzej Siewior <bigeasy@linutronix.de> [140829 02:32]:
> On 08/29/2014 12:54 AM, Tony Lindgren wrote:
> > OK thanks, I'm seeing the same issue as you. And the idlest registers
> > don't show any blockers. Looking at the errata docs, seems like
> > "Usage Note 2.7" in sprz318f.pdf says:
> > 
> >  Details When configured for DMA operations using smartidle mode (SYSC[4:3].IDLEMODE =
> >  0x2), the UART module will not acknowledge incoming idle requests. As a consequence,
> >  it can prevent L4 from going to idle.
> >  When there are additional expected transfers, the UART should be placed in force-idle
> >  mode.
> 
> Ehm. So I haven't found an errata document for omap3630, there is
> nothing like that in that one I have for am335x or dra7. The document
> you mentioned is for AM3715. Interesting…

Yes I would not be suprised if that same bug is in all of them though..
 
> > So I've added also Paul to Cc, he may have better suggestions for the
> > hwmod flags to use. The experimental patch below seems to allow idling
> > for me, care to give it a try?
> 
> Yep, this one works. And I see DMA transfers (for RX side) after the
> core hit idle so it seems to look good :)

OK great :)

Looks like the paste bug is there for sure, doing off idle and pasting
240 characters to the console can hang the UART RX after few attempts,
and pasting 16 charactes won't show up at all if the system is idling.
So you may want to play with that too a bit :)

Regards,

Tony
--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Sebastian Andrzej Siewior Aug. 29, 2014, 4:31 p.m. UTC | #4
On 08/29/2014 06:12 PM, Tony Lindgren wrote:
> Looks like the paste bug is there for sure, doing off idle and pasting
> 240 characters to the console can hang the UART RX after few attempts,
> and pasting 16 charactes won't show up at all if the system is idling.
> So you may want to play with that too a bit :)

Oh. perfect. Please note that we the threshold moved from 16 to 48.
However I see that usually we lose a lot of characters before the uart
wakes up and does its job. Usually I see a couple characters but
sometimes is see broken characters which suggests that the clock was
not (yet) perfect. And I managed not get get it woken once.
So let me look at this once I am through with the review responses…

> 
> Regards,
> 
> Tony
> 
Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Sebastian Andrzej Siewior Sept. 1, 2014, 5:47 p.m. UTC | #5
On 08/29/2014 06:12 PM, Tony Lindgren wrote:
> Looks like the paste bug is there for sure, doing off idle and pasting
> 240 characters to the console can hang the UART RX after few attempts,
> and pasting 16 charactes won't show up at all if the system is idling.
> So you may want to play with that too a bit :)

One character wakes it up. After that you can send 16, 64 and you see
them. Right away. No delay.

If you send "a lot" data in one-go it takes approx 152 characters until
the first one is displayed properly at 115200,8N1. That is approx 13ms.
Could it take that long to get up and be ready?

Comparing it with serial-omap I see the same thing: I takes approx the
same amount of data until the first one is displayed. After a lot of
"long" writes which wake the chip up from idle I manage to freeze both,
the serial-omap driver and mine driver.

One thing that is probably a dumb idea is that printk in
omap_8250_mdr1_errataset().
Would it be possible that when I hit a printk in the resume path that I
may deadlock and box will freeze?

> 
> Regards,
> 
> Tony
> 

Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Sebastian Reichel Sept. 2, 2014, 3:05 a.m. UTC | #6
Hi,

On Mon, Sep 01, 2014 at 07:47:53PM +0200, Sebastian Andrzej Siewior wrote:
> On 08/29/2014 06:12 PM, Tony Lindgren wrote:
> > Looks like the paste bug is there for sure, doing off idle and pasting
> > 240 characters to the console can hang the UART RX after few attempts,
> > and pasting 16 charactes won't show up at all if the system is idling.
> > So you may want to play with that too a bit :)
> 
> One character wakes it up. After that you can send 16, 64 and you see
> them. Right away. No delay.
> 
> If you send "a lot" data in one-go it takes approx 152 characters until
> the first one is displayed properly at 115200,8N1. That is approx 13ms.
> Could it take that long to get up and be ready?

I noticed the same behaviour when I tested the runtime PM stuff on
my N900 with the existing serial-omap driver and I also assumed,
that the chip needs that long to get up.

> Comparing it with serial-omap I see the same thing: I takes approx the
> same amount of data until the first one is displayed. After a lot of
> "long" writes which wake the chip up from idle I manage to freeze both,
> the serial-omap driver and mine driver.
> 
> One thing that is probably a dumb idea is that printk in
> omap_8250_mdr1_errataset().
> Would it be possible that when I hit a printk in the resume path that I
> may deadlock and box will freeze?

-- Sebastian
Tony Lindgren Sept. 2, 2014, 4:55 p.m. UTC | #7
* Sebastian Reichel <sre@kernel.org> [140901 20:06]:
> Hi,
> 
> On Mon, Sep 01, 2014 at 07:47:53PM +0200, Sebastian Andrzej Siewior wrote:
> > On 08/29/2014 06:12 PM, Tony Lindgren wrote:
> > > Looks like the paste bug is there for sure, doing off idle and pasting
> > > 240 characters to the console can hang the UART RX after few attempts,
> > > and pasting 16 charactes won't show up at all if the system is idling.
> > > So you may want to play with that too a bit :)
> > 
> > One character wakes it up. After that you can send 16, 64 and you see
> > them. Right away. No delay.
> > 
> > If you send "a lot" data in one-go it takes approx 152 characters until
> > the first one is displayed properly at 115200,8N1. That is approx 13ms.
> > Could it take that long to get up and be ready?
> 
> I noticed the same behaviour when I tested the runtime PM stuff on
> my N900 with the existing serial-omap driver and I also assumed,
> that the chip needs that long to get up.

OK yeah I've confirmed that serial-omap also won't do anything with the
pasted data until woken up. It takes some time to get things powered up
again, there's nothing we can do beyond having the autoidle disabled by
default.
 
> > Comparing it with serial-omap I see the same thing: I takes approx the
> > same amount of data until the first one is displayed. After a lot of
> > "long" writes which wake the chip up from idle I manage to freeze both,
> > the serial-omap driver and mine driver.

Hmm I have not seen the RX hang with serial-omap when pasting data to
the console to wake it up.
 
> > One thing that is probably a dumb idea is that printk in
> > omap_8250_mdr1_errataset().
> > Would it be possible that when I hit a printk in the resume path that I
> > may deadlock and box will freeze?

I guess yeah. You may want to use pstore as console to debug that. You
need to use this patch to prevent pstore from hanging:

https://lkml.org/lkml/2013/4/9/831

Then enable:

CONFIG_PSTORE=y
CONFIG_PSTORE_CONSOLE=y
CONFIG_PSTORE_RAM=y
CONFIG_FUNCTION_TRACER=y
CONFIG_DEBUG_FS=y
CONFIG_PSTORE_FTRACE=y

Then at kernel cmdline, specify something like this:

memmap=2M$0x88000000 ramoops.mem_address=0x88000000 ramoops.mem_size=0x200000 ramoops.record_size=0x40000

And leave out console=ttyS line and spin up a getty on the serial
line.

After booting, you should just need to do:

# mount -t pstore - /sys/fs/pstore

And then you see dmesg in /sys/fs/pstore. To debug hangs, set up the
PMIC watchdog and do not set up omap watchdog, and you should see the
last dmesg in /sys/fs/pstore after a warm reset.

Regards,

Tony
--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Sebastian Andrzej Siewior Sept. 2, 2014, 6:39 p.m. UTC | #8
On 09/01/2014 07:47 PM, Sebastian Andrzej Siewior wrote:
> Comparing it with serial-omap I see the same thing: I takes approx the
> same amount of data until the first one is displayed. After a lot of
> "long" writes which wake the chip up from idle I manage to freeze both,
> the serial-omap driver and mine driver.

So after some testing:
- it happens with omap-serial as well. Especially after disabling the
  LED trigger for both LEDs.

- it seemed that disabling the MDR1 check whether or not we lost
  context made the problem appear less often but it was a trick. Even
  with restoring the context each time I see the same problem.

- it seems to be easier to trigger with the LED trigger switched off.
  However sometimes it works for 10 minutes, sometimes it triggers
  after one.

- I see to face two kind of "deaths":
  - the LED still goes on and off and the uart just does not respond
    even if I tell the button print something on the screen (the button
    also changes the frequency of the LED so I know that the button is
    doing something).
    Also from dumping the content of /proc/interrupts it seems that a
    wake up is made, the uart should have restored the registers.

  - one where the system is dead and the LED does not blink anymore.
    Also my button is dead.

- disabling DMA makes the problem not go away.

- mdelay(25) in omap8250_lost_context() is long enough to drop the 403
  bytes I send in my testcase. That means I see only "good" characters.
  With this the box remained alive for 2h. However the uart died anyway.

>>
>> Regards,
>>
>> Tony
>>
> 

Sebastian

--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Tony Lindgren Sept. 2, 2014, 8:15 p.m. UTC | #9
* Sebastian Andrzej Siewior <bigeasy@linutronix.de> [140902 11:40]:
> On 09/01/2014 07:47 PM, Sebastian Andrzej Siewior wrote:
> > Comparing it with serial-omap I see the same thing: I takes approx the
> > same amount of data until the first one is displayed. After a lot of
> > "long" writes which wake the chip up from idle I manage to freeze both,
> > the serial-omap driver and mine driver.
> 
> So after some testing:
> - it happens with omap-serial as well. Especially after disabling the
>   LED trigger for both LEDs.
> 
> - it seemed that disabling the MDR1 check whether or not we lost
>   context made the problem appear less often but it was a trick. Even
>   with restoring the context each time I see the same problem.
> 
> - it seems to be easier to trigger with the LED trigger switched off.
>   However sometimes it works for 10 minutes, sometimes it triggers
>   after one.
> 
> - I see to face two kind of "deaths":
>   - the LED still goes on and off and the uart just does not respond
>     even if I tell the button print something on the screen (the button
>     also changes the frequency of the LED so I know that the button is
>     doing something).
>     Also from dumping the content of /proc/interrupts it seems that a
>     wake up is made, the uart should have restored the registers.

OK yeah this is the case I was seeing too. So do you just set the
LED triggers to none in sysfs to make it easier to reproduce?

>   - one where the system is dead and the LED does not blink anymore.
>     Also my button is dead.

This I don't think I've seen. This could also be the errata issue on
your earlier rev beagleboard-xm with off-idle.
 
> - disabling DMA makes the problem not go away.

OK
 
> - mdelay(25) in omap8250_lost_context() is long enough to drop the 403
>   bytes I send in my testcase. That means I see only "good" characters.
>   With this the box remained alive for 2h. However the uart died anyway.

I wonder if doing some TX on the uart restores it? So maybe try

$ while [ 1 ]; do date; sleep 10 done &

To have it occasionally print something in the background.

Regards,

Tony

--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Sebastian Reichel Sept. 2, 2014, 8:38 p.m. UTC | #10
On Tue, Sep 02, 2014 at 01:15:37PM -0700, Tony Lindgren wrote:
> * Sebastian Andrzej Siewior <bigeasy@linutronix.de> [140902 11:40]:
> > On 09/01/2014 07:47 PM, Sebastian Andrzej Siewior wrote:
> > > Comparing it with serial-omap I see the same thing: I takes approx the
> > > same amount of data until the first one is displayed. After a lot of
> > > "long" writes which wake the chip up from idle I manage to freeze both,
> > > the serial-omap driver and mine driver.
> > 
> > So after some testing:
> > - it happens with omap-serial as well. Especially after disabling the
> >   LED trigger for both LEDs.
> > 
> > - it seemed that disabling the MDR1 check whether or not we lost
> >   context made the problem appear less often but it was a trick. Even
> >   with restoring the context each time I see the same problem.
> > 
> > - it seems to be easier to trigger with the LED trigger switched off.
> >   However sometimes it works for 10 minutes, sometimes it triggers
> >   after one.
> > 
> > - I see to face two kind of "deaths":
> >   - the LED still goes on and off and the uart just does not respond
> >     even if I tell the button print something on the screen (the button
> >     also changes the frequency of the LED so I know that the button is
> >     doing something).
> >     Also from dumping the content of /proc/interrupts it seems that a
> >     wake up is made, the uart should have restored the registers.
> 
> OK yeah this is the case I was seeing too. So do you just set the
> LED triggers to none in sysfs to make it easier to reproduce?
> 
> >   - one where the system is dead and the LED does not blink anymore.
> >     Also my button is dead.
> 
> This I don't think I've seen. This could also be the errata issue on
> your earlier rev beagleboard-xm with off-idle.
>  
> > - disabling DMA makes the problem not go away.
> 
> OK
>  
> > - mdelay(25) in omap8250_lost_context() is long enough to drop the 403
> >   bytes I send in my testcase. That means I see only "good" characters.
> >   With this the box remained alive for 2h. However the uart died anyway.
> 
> I wonder if doing some TX on the uart restores it? So maybe try
> 
> $ while [ 1 ]; do date; sleep 10 done &
> 
> To have it occasionally print something in the background.

If there is a way to detect the hangup you may try to recover by
resetting the serial device using omap_hwmod_reset().

-- Sebastian
Sebastian Andrzej Siewior Sept. 3, 2014, 4:46 p.m. UTC | #11
On 09/02/2014 10:15 PM, Tony Lindgren wrote:
>> - I see to face two kind of "deaths":
>>   - the LED still goes on and off and the uart just does not respond
>>     even if I tell the button print something on the screen (the button
>>     also changes the frequency of the LED so I know that the button is
>>     doing something).
>>     Also from dumping the content of /proc/interrupts it seems that a
>>     wake up is made, the uart should have restored the registers.
> 
> OK yeah this is the case I was seeing too. So do you just set the
> LED triggers to none in sysfs to make it easier to reproduce?

Yes.

>>   - one where the system is dead and the LED does not blink anymore.
>>     Also my button is dead.
> 
> This I don't think I've seen. This could also be the errata issue on
> your earlier rev beagleboard-xm with off-idle.

might be.

Your pstore hint gave me something. I tried that earlier but somehow
assumed that dram content was killed on init. But the content is even
there are pressing the reset button :)

However, I was able to capture the case where the LED was not blinking:
The IIR register says 0xc6 (=> line status error). That is okay. At the
same time LSR register says 0xe0. This is not okay. It means that there
is some kind of error and at least one error bit is set in this
register which is not the case. Also those bits are cleared on read
which does not happen here. And we loop forever so the LED does blink
anymore.

The RX-count register says that it is empty which sense because bit 0
is not set (in LSR). However I can read multiple times from the RX FIFO
until I get the "unhandled bus access" error which usually happens
right away if the empty FIFO is read on omap3 HW. In the last test I
mange to read 91 times before the crash. I hoped that this FIFO read
would make the interrupt go away but it did not.

The HW seems to be in a strange state. It might be either the errata
or something else. I even took the resume routine from omap-serial in
case I did something wrong. In my last test it worked for 10minues
before the interrupt storm came.

This is probably the same thing I see on the omap-serial driver where I
got from pstore:

[   32.659271] random: nonblocking pool is initialized
[  212.170623] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s!
[swapper:0]

So I *guess* the interrupt routine is looping. This is problem one, no
idea what is going on (the register status captured on 8250-omap makes
no sense).

Problem two, where the UART does not wakeup:
What I observed is that sometimes the UART does not wake up properly
i.e. it does not write anything on the console, even where it should. I
can't tell if the read is working properly, the write does not.
From my capture I see that the resume routine was running and the
register should have been written. That means the UART should be up and
running but nothing happens.
It often works again after the system comes out of resume again (i.e.
RPM suspens and resumes the UART). So it is okay on the next wakeup. Or
the wakeup after next.
From the script:

| while ((1))
| do
|
|         echo -n 409-chars >/dev/ttyUSB0
|
|         sleep 1
|         a=$(date)
|         echo -e "\n#$a" >/dev/ttyUSB0
|         echo $a
|         sleep 13;
| done

I see that sometimes one or two sequential timestamps are missing. And
the it continues like nothing happened.

> Tony

Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Tony Lindgren Sept. 3, 2014, 5:48 p.m. UTC | #12
* Sebastian Andrzej Siewior <bigeasy@linutronix.de> [140903 09:46]:
> On 09/02/2014 10:15 PM, Tony Lindgren wrote:
> >> - I see to face two kind of "deaths":
> >>   - the LED still goes on and off and the uart just does not respond
> >>     even if I tell the button print something on the screen (the button
> >>     also changes the frequency of the LED so I know that the button is
> >>     doing something).
> >>     Also from dumping the content of /proc/interrupts it seems that a
> >>     wake up is made, the uart should have restored the registers.
> > 
> > OK yeah this is the case I was seeing too. So do you just set the
> > LED triggers to none in sysfs to make it easier to reproduce?
> 
> Yes.
> 
> >>   - one where the system is dead and the LED does not blink anymore.
> >>     Also my button is dead.
> > 
> > This I don't think I've seen. This could also be the errata issue on
> > your earlier rev beagleboard-xm with off-idle.
> 
> might be.
> 
> Your pstore hint gave me something. I tried that earlier but somehow
> assumed that dram content was killed on init. But the content is even
> there are pressing the reset button :)

Yeah pstore is very nice for debugging mystery hangs :)
 
> However, I was able to capture the case where the LED was not blinking:
> The IIR register says 0xc6 (=> line status error). That is okay. At the
> same time LSR register says 0xe0. This is not okay. It means that there
> is some kind of error and at least one error bit is set in this
> register which is not the case. Also those bits are cleared on read
> which does not happen here. And we loop forever so the LED does blink
> anymore.

OK
 
> The RX-count register says that it is empty which sense because bit 0
> is not set (in LSR). However I can read multiple times from the RX FIFO
> until I get the "unhandled bus access" error which usually happens
> right away if the empty FIFO is read on omap3 HW. In the last test I
> mange to read 91 times before the crash. I hoped that this FIFO read
> would make the interrupt go away but it did not.
> 
> The HW seems to be in a strange state. It might be either the errata
> or something else. I even took the resume routine from omap-serial in
> case I did something wrong. In my last test it worked for 10minues
> before the interrupt storm came.
> 
> This is probably the same thing I see on the omap-serial driver where I
> got from pstore:
> 
> [   32.659271] random: nonblocking pool is initialized
> [  212.170623] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s!
> [swapper:0]
> 
> So I *guess* the interrupt routine is looping. This is problem one, no
> idea what is going on (the register status captured on 8250-omap makes
> no sense).

See recent commit cc824534d4fe, and try commenting out the check for
HWMOD_FORCE_MSTANDBY in omap_hwmod.c so _reconfigure_io_chain() is always
called. If that changes something, we at least have some idea.

It could be also the wake-up interrupt looping. So you may also want to
try adding some printks (pstore only) into omap_prcm_irq_handler() and
omap3xxx_prm_clear_mod_irqs() as that's handling the wake-up event
interrupts.
 
> Problem two, where the UART does not wakeup:
> What I observed is that sometimes the UART does not wake up properly
> i.e. it does not write anything on the console, even where it should. I
> can't tell if the read is working properly, the write does not.
> From my capture I see that the resume routine was running and the
> register should have been written. That means the UART should be up and
> running but nothing happens.

This seems also be hinting to something needing _reconfigure_io_chain()
to be called along the lines of commit cc824534d4fe.

> It often works again after the system comes out of resume again (i.e.
> RPM suspens and resumes the UART). So it is okay on the next wakeup. Or
> the wakeup after next.
> From the script:
> 
> | while ((1))
> | do
> |
> |         echo -n 409-chars >/dev/ttyUSB0
> |
> |         sleep 1
> |         a=$(date)
> |         echo -e "\n#$a" >/dev/ttyUSB0
> |         echo $a
> |         sleep 13;
> | done
> 
> I see that sometimes one or two sequential timestamps are missing. And
> the it continues like nothing happened.

OK. At least it's starting to now sound that the bugs are pretty much
the same with 8250 and serial-omap :)

Regards,

Tony
--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Sebastian Andrzej Siewior Sept. 4, 2014, 1:44 p.m. UTC | #13
On 09/03/2014 07:48 PM, Tony Lindgren wrote:

I will check this upper part of this email soon…

> OK. At least it's starting to now sound that the bugs are pretty much
> the same with 8250 and serial-omap :)

Do you see a reason for not posting the entire series again since now I
am bug compatible in the pm-runtime part? We have -rc3 and I would like
to get this merge the series I have now and this seems to be the only
problem I am aware of.

> 
> Regards,
> 
> Tony

Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Tony Lindgren Sept. 4, 2014, 2:52 p.m. UTC | #14
* Sebastian Andrzej Siewior <bigeasy@linutronix.de> [140904 06:46]:
> On 09/03/2014 07:48 PM, Tony Lindgren wrote:
> 
> I will check this upper part of this email soon…
> 
> > OK. At least it's starting to now sound that the bugs are pretty much
> > the same with 8250 and serial-omap :)
> 
> Do you see a reason for not posting the entire series again since now I
> am bug compatible in the pm-runtime part? We have -rc3 and I would like
> to get this merge the series I have now and this seems to be the only
> problem I am aware of.

Yeah makes sense to me. Thanks for ensuring the bug
compability :)

Regards,

Tony
--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Sebastian Andrzej Siewior Sept. 4, 2014, 2:56 p.m. UTC | #15
On 09/04/2014 04:52 PM, Tony Lindgren wrote:
> Yeah makes sense to me. Thanks for ensuring the bug
> compability :)

Thanks. What about the ttyOx vs ttySx. Do you want me to add kernel
line fixup somewhere arch/arm or we ignore this for now? Greg didn't
say anything…

> 
> Regards,
> 
> Tony
> 

Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Tony Lindgren Sept. 4, 2014, 4:25 p.m. UTC | #16
* Sebastian Andrzej Siewior <bigeasy@linutronix.de> [140904 07:56]:
> On 09/04/2014 04:52 PM, Tony Lindgren wrote:
> > Yeah makes sense to me. Thanks for ensuring the bug
> > compability :)
> 
> Thanks. What about the ttyOx vs ttySx. Do you want me to add kernel
> line fixup somewhere arch/arm or we ignore this for now? Greg didn't
> say anything…

I suggest we have both drivers just available first and then attempt
to deal with the removal of most of omap-serial later on.

If both drivers are present, we should try to ensure 8250 gets probed
first which I believe it already does based on the Makefile order.

Once the 8250 based driver is known to work, it might make sense to
remove most of the functions in omap-serial and have it use the 8250
functions too to remove duplicate code.

That might be already enough and no proxying is needed and we get
rid of the duplicate code :)

Regards,

Tony
--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
diff mbox

Patch

--- a/arch/arm/mach-omap2/omap_hwmod_3xxx_data.c
+++ b/arch/arm/mach-omap2/omap_hwmod_3xxx_data.c
@@ -490,7 +490,9 @@  static struct omap_hwmod omap3xxx_uart1_hwmod = {
 	.mpu_irqs	= omap2_uart1_mpu_irqs,
 	.sdma_reqs	= omap2_uart1_sdma_reqs,
 	.main_clk	= "uart1_fck",
-	.flags		= DEBUG_TI81XXUART1_FLAGS | HWMOD_SWSUP_SIDLE_ACT,
+	.flags		= DEBUG_TI81XXUART1_FLAGS |
+				HWMOD_NO_OCP_AUTOIDLE | HWMOD_SWSUP_SIDLE |
+				HWMOD_FORCE_MSTANDBY,
 	.prcm		= {
 		.omap2 = {
 			.module_offs = CORE_MOD,
@@ -509,7 +511,9 @@  static struct omap_hwmod omap3xxx_uart2_hwmod = {
 	.mpu_irqs	= omap2_uart2_mpu_irqs,
 	.sdma_reqs	= omap2_uart2_sdma_reqs,
 	.main_clk	= "uart2_fck",
-	.flags		= DEBUG_TI81XXUART2_FLAGS | HWMOD_SWSUP_SIDLE_ACT,
+	.flags		= DEBUG_TI81XXUART2_FLAGS |
+				HWMOD_NO_OCP_AUTOIDLE | HWMOD_SWSUP_SIDLE |
+				HWMOD_FORCE_MSTANDBY,
 	.prcm		= {
 		.omap2 = {
 			.module_offs = CORE_MOD,
@@ -529,7 +533,8 @@  static struct omap_hwmod omap3xxx_uart3_hwmod = {
 	.sdma_reqs	= omap2_uart3_sdma_reqs,
 	.main_clk	= "uart3_fck",
 	.flags		= DEBUG_OMAP3UART3_FLAGS | DEBUG_TI81XXUART3_FLAGS |
-				HWMOD_SWSUP_SIDLE_ACT,
+				HWMOD_NO_OCP_AUTOIDLE | HWMOD_SWSUP_SIDLE |
+				HWMOD_FORCE_MSTANDBY,
 	.prcm		= {
 		.omap2 = {
 			.module_offs = OMAP3430_PER_MOD,
@@ -559,7 +564,9 @@  static struct omap_hwmod omap36xx_uart4_hwmod = {
 	.mpu_irqs	= uart4_mpu_irqs,
 	.sdma_reqs	= uart4_sdma_reqs,
 	.main_clk	= "uart4_fck",
-	.flags		= DEBUG_OMAP3UART4_FLAGS | HWMOD_SWSUP_SIDLE_ACT,
+	.flags		= DEBUG_OMAP3UART4_FLAGS |
+				HWMOD_NO_OCP_AUTOIDLE | HWMOD_SWSUP_SIDLE |
+				HWMOD_FORCE_MSTANDBY,
 	.prcm		= {
 		.omap2 = {
 			.module_offs = OMAP3430_PER_MOD,