[0/3] scripts/qemu-gdb: Make coroutine dumps to work with coredumps

Message

Peter Xu Dec. 11, 2024, 8:17 p.m. UTC

Coroutines are used in many cases in block layers. It's also used in live
migration when on destination side, and it'll be handy to diagnose crashes
within a coroutine when we want to also know what other coroutines are
doing.

This series adds initial support for that, not pretty but it should start
working.  Since we can't use the trick to modify registers on the fly in
non-live gdb sessions, we do manual unwinds.

Thanks,

Peter Xu (3):
  scripts/qemu-gdb: Always do full stack dump for python errors
  scripts/qemu-gdb: Simplify fs_base fetching for coroutines
  scripts/qemu-gdb: Support coroutine dumps in coredumps

 scripts/qemu-gdb.py          |  2 +
 scripts/qemugdb/coroutine.py | 73 ++++++++++++++++++++++--------------
 2 files changed, 47 insertions(+), 28 deletions(-)

Comments

Fabiano Rosas Dec. 11, 2024, 8:25 p.m. UTC | #1

Peter Xu <[e-mail]> writes:

> Coroutines are used in many cases in block layers. It's also used in live
> migration when on destination side, and it'll be handy to diagnose crashes
> within a coroutine when we want to also know what other coroutines are
> doing.

Not sure if you've seen this message on the list:

https://lore.kernel.org/r/[e-mail]

Peter Xu Dec. 11, 2024, 9:39 p.m. UTC | #2

On Wed, Dec 11, 2024 at 05:25:10PM -0300, Fabiano Rosas wrote:
> Peter Xu <[e-mail]> writes:
> 
> > Coroutines are used in many cases in block layers. It's also used in live
> > migration when on destination side, and it'll be handy to diagnose crashes
> > within a coroutine when we want to also know what other coroutines are
> > doing.
> 
> Not sure if you've seen this message on the list:
> 
> https://lore.kernel.org/r/[e-mail]

No I didn't.  I only started looking at this because I got a bug a few days
ago that I need to look at the main coroutine where dest crashed, then
Stefan told me this script and also told me it only works with live session.
Ideally I'll need coredump debug-ability, then I figured it isn't too hard.

I saw that it didn't yet land gdb, and it's much more involved even if it
could be more generic.  Not sure how the block developers think, personally
I prefer this much smaller change because it works on old systems, where I
can easily install gdb with package managers.

Kevin Wolf Dec. 12, 2024, 10:28 a.m. UTC | #3

Am 11.12.2024 um 22:39 hat Peter Xu geschrieben:
> On Wed, Dec 11, 2024 at 05:25:10PM -0300, Fabiano Rosas wrote:
> > Peter Xu <[e-mail]> writes:
> > 
> > > Coroutines are used in many cases in block layers. It's also used in live
> > > migration when on destination side, and it'll be handy to diagnose crashes
> > > within a coroutine when we want to also know what other coroutines are
> > > doing.
> > 
> > Not sure if you've seen this message on the list:
> > 
> > https://lore.kernel.org/r/[e-mail]

Let me add Niall to this thread.

> No I didn't.  I only started looking at this because I got a bug a few days
> ago that I need to look at the main coroutine where dest crashed, then
> Stefan told me this script and also told me it only works with live session.
> Ideally I'll need coredump debug-ability, then I figured it isn't too hard.

I agree this is useful at least in the current state of things, and
possibly in the future for QEMUs built without support for new gdb
magic.

> I saw that it didn't yet land gdb, and it's much more involved even if it
> could be more generic.  Not sure how the block developers think, personally
> I prefer this much smaller change because it works on old systems, where I
> can easily install gdb with package managers.

I don't think this is a question of preferring one or the other! Your
simple change works today with existing QEMU builds and has fewer
requirements, but it's also more limited in functionality.

The gdb changes look really nice at the first sight, debugging
coroutines as if they were threads is exactly what you want in most
cases. So I'd love to see this. Of course, someone would have to first
understand what needs to be done in detail and then implement the QEMU
side of it once the gdb side is merged.

Kevin

Peter Xu Dec. 12, 2024, 2:45 p.m. UTC | #4

On Thu, Dec 12, 2024 at 11:28:52AM +0100, Kevin Wolf wrote:
> Am 11.12.2024 um 22:39 hat Peter Xu geschrieben:
> > On Wed, Dec 11, 2024 at 05:25:10PM -0300, Fabiano Rosas wrote:
> > > Peter Xu <[e-mail]> writes:
> > > 
> > > > Coroutines are used in many cases in block layers. It's also used in live
> > > > migration when on destination side, and it'll be handy to diagnose crashes
> > > > within a coroutine when we want to also know what other coroutines are
> > > > doing.
> > > 
> > > Not sure if you've seen this message on the list:
> > > 
> > > https://lore.kernel.org/r/[e-mail]
> 
> Let me add Niall to this thread.
> 
> > No I didn't.  I only started looking at this because I got a bug a few days
> > ago that I need to look at the main coroutine where dest crashed, then
> > Stefan told me this script and also told me it only works with live session.
> > Ideally I'll need coredump debug-ability, then I figured it isn't too hard.
> 
> I agree this is useful at least in the current state of things, and
> possibly in the future for QEMUs built without support for new gdb
> magic.
> 
> > I saw that it didn't yet land gdb, and it's much more involved even if it
> > could be more generic.  Not sure how the block developers think, personally
> > I prefer this much smaller change because it works on old systems, where I
> > can easily install gdb with package managers.
> 
> I don't think this is a question of preferring one or the other! Your
> simple change works today with existing QEMU builds and has fewer
> requirements, but it's also more limited in functionality.

Ah, definitely..

I had no intention to mean that we should only merge one, but more of pure
thinking what I would like to use in the near future, even if the solution
landed at some point in both gdb and qemu.

I'll still need to look at the old systems that miss the generic solution,
so I'd still prefer using the scripts (which currently satisfies most of my
needs..).

> 
> The gdb changes look really nice at the first sight, debugging
> coroutines as if they were threads is exactly what you want in most
> cases. So I'd love to see this. Of course, someone would have to first
> understand what needs to be done in detail and then implement the QEMU
> side of it once the gdb side is merged.

Yes, I wish that could happen some day, that by default gdb on top of QEMU
can dump coroutines.

Thanks,