mbox series

[v3,0/8,RfC] fix tracing for modules

Message ID 20210121125028.3247190-1-kraxel@redhat.com (mailing list archive)
Headers show
Series fix tracing for modules | expand

Message

Gerd Hoffmann Jan. 21, 2021, 12:50 p.m. UTC
First version that actually works.  Only qxl covered for this RfC,
other modules will follow once the basics are hashed out.

v3:
 - handle initialization of modular tracepoints.

TODO:
Enabling modular tracepoints via -trace cmd line doesn't work yet.
Guess we need to store the list somewhere for later re-processing.
Error handling is tricky, specifically the "tracepoint doesn't exist"
error.  Suggestions / ideas are welcome.

More context:
  https://bugzilla.redhat.com/show_bug.cgi?id=1898700
  https://bugzilla.redhat.com/show_bug.cgi?id=1869642

take care,
  Gerd

Gerd Hoffmann (8):
  meson: add trace_events_config[]
  meson: move up hw subdir (specifically before trace subdir)
  meson: add module_trace & module_trace_src
  meson: move qxl trace events to separate file
  trace: iter init tweaks
  trace: add trace_event_iter_init_group
  trace/simple: pass iter to st_write_event_mapping
  trace/simple: add st_init_group

 trace/control.h             | 30 ++++++++++++++---
 trace/simple.h              |  1 +
 hw/display/qxl-render.c     |  1 +
 hw/display/qxl.c            |  1 +
 monitor/misc.c              |  4 +--
 trace/control-target.c      |  2 +-
 trace/control.c             | 39 ++++++++++++++++-----
 trace/qmp.c                 |  6 ++--
 trace/simple.c              | 22 +++++++++---
 hw/display/meson.build      |  5 +++
 hw/display/trace-events     | 67 -------------------------------------
 hw/display/trace-events-qxl | 66 ++++++++++++++++++++++++++++++++++++
 meson.build                 |  7 ++--
 trace/meson.build           | 37 +++++++++++++++-----
 14 files changed, 187 insertions(+), 101 deletions(-)
 create mode 100644 hw/display/trace-events-qxl

Comments

Stefan Hajnoczi Feb. 3, 2021, 4:32 p.m. UTC | #1
On Thu, Jan 21, 2021 at 01:50:20PM +0100, Gerd Hoffmann wrote:
> First version that actually works.  Only qxl covered for this RfC,
> other modules will follow once the basics are hashed out.
> 
> v3:
>  - handle initialization of modular tracepoints.

Cool, this looks promising!

> TODO:
> Enabling modular tracepoints via -trace cmd line doesn't work yet.
> Guess we need to store the list somewhere for later re-processing.
> Error handling is tricky, specifically the "tracepoint doesn't exist"
> error.  Suggestions / ideas are welcome.

Two ideas:

Global trace event name list
----------------------------
Build *some* global information about all trace events, including
modules, into the main QEMU binary. For example, generate an array of
all trace event names so QEMU can always print an error if a
non-existent trace event name is used. (This is similar to the
trace-events-all file, which is a global list of all trace events.)

Module name prefixes
--------------------
Allow an optional module/group prefix like qxl:my_trace_event. When the
user says:

  --trace qxl:my_trace_event

QEMU knows that this trace event belongs to the "qxl" module/group. It
will not attempt to load it until the qxl module registers itself.

If "my_trace_event" doesn't exist in the qxl module:
1. If the qxl module is not loaded we don't hit an error. Nevermind.
2. When the qxl module is loaded pending events are resolved and an
   error is printed.
Gerd Hoffmann Feb. 22, 2021, 3:13 p.m. UTC | #2
Hi,

> > TODO:
> > Enabling modular tracepoints via -trace cmd line doesn't work yet.
> > Guess we need to store the list somewhere for later re-processing.
> > Error handling is tricky, specifically the "tracepoint doesn't exist"
> > error.  Suggestions / ideas are welcome.
> 
> Two ideas:
> 
> Global trace event name list
> ----------------------------
> Build *some* global information about all trace events, including
> modules, into the main QEMU binary. For example, generate an array of
> all trace event names so QEMU can always print an error if a
> non-existent trace event name is used. (This is similar to the
> trace-events-all file, which is a global list of all trace events.)
> 
> Module name prefixes
> --------------------
> Allow an optional module/group prefix like qxl:my_trace_event. When the
> user says:
> 
>   --trace qxl:my_trace_event
> 
> QEMU knows that this trace event belongs to the "qxl" module/group. It
> will not attempt to load it until the qxl module registers itself.
> 
> If "my_trace_event" doesn't exist in the qxl module:
> 1. If the qxl module is not loaded we don't hit an error. Nevermind.
> 2. When the qxl module is loaded pending events are resolved and an
>    error is printed.

Finally found the time to look at this again... 

So, we already have a "group".  Which is basically the sub-directory of
the trace-events file right now, and it seems to be mostly a build system
thing.  We get many small lists instead of one huge, but there seems to
be no other effect.  We could change that though, by giving each group
an (optional?) prefix.

There also is a probe prefix, apparently used by dtrace only.  Not sure
how to deal with that.  It prefix is qemu-<target-type>-<target-name>.
Giving qemu modules its own dtrace prefix looks sensible to me.  That
would probably something like "qemu-module-<name>".

take care,
  Gerd
Stefan Hajnoczi March 22, 2021, 11:36 a.m. UTC | #3
On Mon, Feb 22, 2021 at 04:13:32PM +0100, Gerd Hoffmann wrote:
>   Hi,
> 
> > > TODO:
> > > Enabling modular tracepoints via -trace cmd line doesn't work yet.
> > > Guess we need to store the list somewhere for later re-processing.
> > > Error handling is tricky, specifically the "tracepoint doesn't exist"
> > > error.  Suggestions / ideas are welcome.
> > 
> > Two ideas:
> > 
> > Global trace event name list
> > ----------------------------
> > Build *some* global information about all trace events, including
> > modules, into the main QEMU binary. For example, generate an array of
> > all trace event names so QEMU can always print an error if a
> > non-existent trace event name is used. (This is similar to the
> > trace-events-all file, which is a global list of all trace events.)
> > 
> > Module name prefixes
> > --------------------
> > Allow an optional module/group prefix like qxl:my_trace_event. When the
> > user says:
> > 
> >   --trace qxl:my_trace_event
> > 
> > QEMU knows that this trace event belongs to the "qxl" module/group. It
> > will not attempt to load it until the qxl module registers itself.
> > 
> > If "my_trace_event" doesn't exist in the qxl module:
> > 1. If the qxl module is not loaded we don't hit an error. Nevermind.
> > 2. When the qxl module is loaded pending events are resolved and an
> >    error is printed.
> 
> Finally found the time to look at this again... 
> 
> So, we already have a "group".  Which is basically the sub-directory of
> the trace-events file right now, and it seems to be mostly a build system
> thing.  We get many small lists instead of one huge, but there seems to
> be no other effect.  We could change that though, by giving each group
> an (optional?) prefix.

Yes. This reminds me of an idea that was mentioned at the beginning of
this effort: maybe QEMU modules should always have their own directory
in the source tree instead of being alongside other source files that
are built into the main binary.

> There also is a probe prefix, apparently used by dtrace only.  Not sure
> how to deal with that.  It prefix is qemu-<target-type>-<target-name>.
> Giving qemu modules its own dtrace prefix looks sensible to me.  That
> would probably something like "qemu-module-<name>".

I think the DTrace prefix needs to be the same as the executable name,
but I'm not sure. I also don't know how that extends to shared libraries
like QEMU modules. I'm afraid you would need to investigate the DTrace
prefix.

Stefan
Daniel P. Berrangé March 22, 2021, 11:46 a.m. UTC | #4
On Mon, Mar 22, 2021 at 11:36:39AM +0000, Stefan Hajnoczi wrote:
> On Mon, Feb 22, 2021 at 04:13:32PM +0100, Gerd Hoffmann wrote:
> >   Hi,
> > 
> > > > TODO:
> > > > Enabling modular tracepoints via -trace cmd line doesn't work yet.
> > > > Guess we need to store the list somewhere for later re-processing.
> > > > Error handling is tricky, specifically the "tracepoint doesn't exist"
> > > > error.  Suggestions / ideas are welcome.
> > > 
> > > Two ideas:
> > > 
> > > Global trace event name list
> > > ----------------------------
> > > Build *some* global information about all trace events, including
> > > modules, into the main QEMU binary. For example, generate an array of
> > > all trace event names so QEMU can always print an error if a
> > > non-existent trace event name is used. (This is similar to the
> > > trace-events-all file, which is a global list of all trace events.)
> > > 
> > > Module name prefixes
> > > --------------------
> > > Allow an optional module/group prefix like qxl:my_trace_event. When the
> > > user says:
> > > 
> > >   --trace qxl:my_trace_event
> > > 
> > > QEMU knows that this trace event belongs to the "qxl" module/group. It
> > > will not attempt to load it until the qxl module registers itself.
> > > 
> > > If "my_trace_event" doesn't exist in the qxl module:
> > > 1. If the qxl module is not loaded we don't hit an error. Nevermind.
> > > 2. When the qxl module is loaded pending events are resolved and an
> > >    error is printed.
> > 
> > Finally found the time to look at this again... 
> > 
> > So, we already have a "group".  Which is basically the sub-directory of
> > the trace-events file right now, and it seems to be mostly a build system
> > thing.  We get many small lists instead of one huge, but there seems to
> > be no other effect.  We could change that though, by giving each group
> > an (optional?) prefix.
> 
> Yes. This reminds me of an idea that was mentioned at the beginning of
> this effort: maybe QEMU modules should always have their own directory
> in the source tree instead of being alongside other source files that
> are built into the main binary.

This implies that each time we modularize something, we have to move
its source code. It is possible,  but it wouldn't be by preferred
choice.

> > There also is a probe prefix, apparently used by dtrace only.  Not sure
> > how to deal with that.  It prefix is qemu-<target-type>-<target-name>.
> > Giving qemu modules its own dtrace prefix looks sensible to me.  That
> > would probably something like "qemu-module-<name>".
> 
> I think the DTrace prefix needs to be the same as the executable name,
> but I'm not sure. I also don't know how that extends to shared libraries
> like QEMU modules. I'm afraid you would need to investigate the DTrace
> prefix.

I'm not aware of any requirement for dtrace prefix to match the
executable name. We don't even have an executable called "qemu"
so we're not matching even today.


Regards,
Daniel
Daniel P. Berrangé March 22, 2021, 12:03 p.m. UTC | #5
On Mon, Feb 22, 2021 at 04:13:32PM +0100, Gerd Hoffmann wrote:
>   Hi,
> 
> > > TODO:
> > > Enabling modular tracepoints via -trace cmd line doesn't work yet.
> > > Guess we need to store the list somewhere for later re-processing.
> > > Error handling is tricky, specifically the "tracepoint doesn't exist"
> > > error.  Suggestions / ideas are welcome.
> > 
> > Two ideas:
> > 
> > Global trace event name list
> > ----------------------------
> > Build *some* global information about all trace events, including
> > modules, into the main QEMU binary. For example, generate an array of
> > all trace event names so QEMU can always print an error if a
> > non-existent trace event name is used. (This is similar to the
> > trace-events-all file, which is a global list of all trace events.)
> > 
> > Module name prefixes
> > --------------------
> > Allow an optional module/group prefix like qxl:my_trace_event. When the
> > user says:
> > 
> >   --trace qxl:my_trace_event
> > 
> > QEMU knows that this trace event belongs to the "qxl" module/group. It
> > will not attempt to load it until the qxl module registers itself.
> > 
> > If "my_trace_event" doesn't exist in the qxl module:
> > 1. If the qxl module is not loaded we don't hit an error. Nevermind.
> > 2. When the qxl module is loaded pending events are resolved and an
> >    error is printed.
> 
> Finally found the time to look at this again... 
> 
> So, we already have a "group".  Which is basically the sub-directory of
> the trace-events file right now, and it seems to be mostly a build system
> thing.  We get many small lists instead of one huge, but there seems to
> be no other effect.  We could change that though, by giving each group
> an (optional?) prefix.
> 
> There also is a probe prefix, apparently used by dtrace only.  Not sure
> how to deal with that.  It prefix is qemu-<target-type>-<target-name>.
> Giving qemu modules its own dtrace prefix looks sensible to me.  That
> would probably something like "qemu-module-<name>".

The prefix is used by the systemtap backend. It lets us define
friendly probes, scoped for each QEMU emulator target.

eg a trace point "dma_map_wait" gets mapped to probes in many
.stp files, once per target, because we need to match based on
the executable path:

  probe qemu.system.x86_64.dma_map_wait = process("/usr/libexec/qemu-system-x86_64").mark("dma_map_wait")
  probe qemu.system.x86_64.dma_map_wait = process("/usr/libexec/qemu-system-ppc64").mark("dma_map_wait")
  probe qemu.system.x86_64.dma_map_wait = process("/usr/libexec/qemu-system-aarch64").mark("dma_map_wait")

there is nothing here that reqiures us to change the prefix for
a module - we can carry on using the same systemtap probe name
whether modules are used or not.


Currently we kind of unofficially have a convention that the name
of a trace point should reflect its functional area. So all QXL
related probes have a name prefix "qxl_".

We could make that explicit, by having the trace-events files
support a group, IOW instead of

  qxl_destroy_primary(int qid) "%d"
  qxl_enter_vga_mode(int qid) "%d"
  qxl_exit_vga_mode(int qid) "%d"

We could say that a dot separates the group from the probe
name, and thus have


  qxl.destroy_primary(int qid) "%d"
  qxl.enter_vga_mode(int qid) "%d"
  qxl.exit_vga_mode(int qid) "%d"

this would be backwards compatible, as we can turn the "." back into
a "_" for all existing trace backends, except stp.

NB this is all tagential to use of modules at a build system /runtime
level. I think this explicit grouping of probes would make sense for
everything, to make the naming convention explicit instead of implicit.

Regards,
Daniel
Gerd Hoffmann March 26, 2021, 12:47 p.m. UTC | #6
Hi,

> eg a trace point "dma_map_wait" gets mapped to probes in many
> .stp files, once per target, because we need to match based on
> the executable path:
> 
>   probe qemu.system.x86_64.dma_map_wait = process("/usr/libexec/qemu-system-x86_64").mark("dma_map_wait")
>   probe qemu.system.x86_64.dma_map_wait = process("/usr/libexec/qemu-system-ppc64").mark("dma_map_wait")

Probe qemu.system.ppc64.dma_map_wait = ...

Can I trace qemu started from build directory?
Seems scripts/qemu-trace-stap doesn't support that.

>   qxl.destroy_primary(int qid) "%d"
>   qxl.enter_vga_mode(int qid) "%d"
>   qxl.exit_vga_mode(int qid) "%d"
> 
> this would be backwards compatible, as we can turn the "." back into
> a "_" for all existing trace backends, except stp.

Hmm, not sure I like this inconsistency.  I think names should be the
same no matter which trace backend is used.

take care,
  Gerd
Daniel P. Berrangé March 29, 2021, 9:23 a.m. UTC | #7
On Fri, Mar 26, 2021 at 01:47:00PM +0100, Gerd Hoffmann wrote:
>   Hi,
> 
> > eg a trace point "dma_map_wait" gets mapped to probes in many
> > .stp files, once per target, because we need to match based on
> > the executable path:
> > 
> >   probe qemu.system.x86_64.dma_map_wait = process("/usr/libexec/qemu-system-x86_64").mark("dma_map_wait")
> >   probe qemu.system.x86_64.dma_map_wait = process("/usr/libexec/qemu-system-ppc64").mark("dma_map_wait")
> 
> Probe qemu.system.ppc64.dma_map_wait = ...
> 
> Can I trace qemu started from build directory?
> Seems scripts/qemu-trace-stap doesn't support that.

We should really generate extra equiv .stp files just for running from
the build.

> >   qxl.destroy_primary(int qid) "%d"
> >   qxl.enter_vga_mode(int qid) "%d"
> >   qxl.exit_vga_mode(int qid) "%d"
> > 
> > this would be backwards compatible, as we can turn the "." back into
> > a "_" for all existing trace backends, except stp.
> 
> Hmm, not sure I like this inconsistency.  I think names should be the
> same no matter which trace backend is used.

Regards,
Daniel
Gerd Hoffmann March 29, 2021, 9:48 a.m. UTC | #8
On Mon, Mar 29, 2021 at 10:23:42AM +0100, Daniel P. Berrangé wrote:
> On Fri, Mar 26, 2021 at 01:47:00PM +0100, Gerd Hoffmann wrote:
> >   Hi,
> > 
> > > eg a trace point "dma_map_wait" gets mapped to probes in many
> > > .stp files, once per target, because we need to match based on
> > > the executable path:
> > > 
> > >   probe qemu.system.x86_64.dma_map_wait = process("/usr/libexec/qemu-system-x86_64").mark("dma_map_wait")
> > >   probe qemu.system.x86_64.dma_map_wait = process("/usr/libexec/qemu-system-ppc64").mark("dma_map_wait")
> > 
> > Probe qemu.system.ppc64.dma_map_wait = ...
> > 
> > Can I trace qemu started from build directory?
> > Seems scripts/qemu-trace-stap doesn't support that.
> 
> We should really generate extra equiv .stp files just for running from
> the build.

Well, "make install" with --prefix=$HOME/qemu-install fixed that for the time
being.

Now I have this:

kraxel@sirius ~/qemu-install/bin# sudo ./qemu-trace-stap -v run ./qemu-system-x86_64 "qxl_soft_reset"
Using tapset dir '/home/kraxel/qemu-install/share/systemtap/tapset' for binary './qemu-system-x86_64'
Compiling script 'probe qemu.system.x86_64.log.qxl_soft_reset {}'
semantic error: unresolved function pid: identifier 'pid' at /home/kraxel/qemu-install/share/systemtap/tapset/qemu-system-x86_64-log.stp:5451:41
        source:     printf("%d@%d qxl_soft_reset %d\n", pid(), gettimeofday_ns(), qid)
                                                        ^

Pass 2: analysis failed.  [man error::pass2]

Any clue why pid() isn't known?

thanks,
  Gerd
Daniel P. Berrangé March 29, 2021, 10:02 a.m. UTC | #9
On Mon, Mar 29, 2021 at 11:48:18AM +0200, Gerd Hoffmann wrote:
> On Mon, Mar 29, 2021 at 10:23:42AM +0100, Daniel P. Berrangé wrote:
> > On Fri, Mar 26, 2021 at 01:47:00PM +0100, Gerd Hoffmann wrote:
> > >   Hi,
> > > 
> > > > eg a trace point "dma_map_wait" gets mapped to probes in many
> > > > .stp files, once per target, because we need to match based on
> > > > the executable path:
> > > > 
> > > >   probe qemu.system.x86_64.dma_map_wait = process("/usr/libexec/qemu-system-x86_64").mark("dma_map_wait")
> > > >   probe qemu.system.x86_64.dma_map_wait = process("/usr/libexec/qemu-system-ppc64").mark("dma_map_wait")
> > > 
> > > Probe qemu.system.ppc64.dma_map_wait = ...
> > > 
> > > Can I trace qemu started from build directory?
> > > Seems scripts/qemu-trace-stap doesn't support that.
> > 
> > We should really generate extra equiv .stp files just for running from
> > the build.
> 
> Well, "make install" with --prefix=$HOME/qemu-install fixed that for the time
> being.
> 
> Now I have this:
> 
> kraxel@sirius ~/qemu-install/bin# sudo ./qemu-trace-stap -v run ./qemu-system-x86_64 "qxl_soft_reset"
> Using tapset dir '/home/kraxel/qemu-install/share/systemtap/tapset' for binary './qemu-system-x86_64'
> Compiling script 'probe qemu.system.x86_64.log.qxl_soft_reset {}'
> semantic error: unresolved function pid: identifier 'pid' at /home/kraxel/qemu-install/share/systemtap/tapset/qemu-system-x86_64-log.stp:5451:41
>         source:     printf("%d@%d qxl_soft_reset %d\n", pid(), gettimeofday_ns(), qid)
>                                                         ^
> 
> Pass 2: analysis failed.  [man error::pass2]
> 
> Any clue why pid() isn't known?

Hmm, strange, makes me think we have a bug causing it to not pull in
global functions.

Regards,
Daniel
Stefan Hajnoczi March 29, 2021, 4:32 p.m. UTC | #10
On Mon, Mar 22, 2021 at 11:46:55AM +0000, Daniel P. Berrangé wrote:
> On Mon, Mar 22, 2021 at 11:36:39AM +0000, Stefan Hajnoczi wrote:
> > On Mon, Feb 22, 2021 at 04:13:32PM +0100, Gerd Hoffmann wrote:
> > >   Hi,
> > > 
> > > > > TODO:
> > > > > Enabling modular tracepoints via -trace cmd line doesn't work yet.
> > > > > Guess we need to store the list somewhere for later re-processing.
> > > > > Error handling is tricky, specifically the "tracepoint doesn't exist"
> > > > > error.  Suggestions / ideas are welcome.
> > > > 
> > > > Two ideas:
> > > > 
> > > > Global trace event name list
> > > > ----------------------------
> > > > Build *some* global information about all trace events, including
> > > > modules, into the main QEMU binary. For example, generate an array of
> > > > all trace event names so QEMU can always print an error if a
> > > > non-existent trace event name is used. (This is similar to the
> > > > trace-events-all file, which is a global list of all trace events.)
> > > > 
> > > > Module name prefixes
> > > > --------------------
> > > > Allow an optional module/group prefix like qxl:my_trace_event. When the
> > > > user says:
> > > > 
> > > >   --trace qxl:my_trace_event
> > > > 
> > > > QEMU knows that this trace event belongs to the "qxl" module/group. It
> > > > will not attempt to load it until the qxl module registers itself.
> > > > 
> > > > If "my_trace_event" doesn't exist in the qxl module:
> > > > 1. If the qxl module is not loaded we don't hit an error. Nevermind.
> > > > 2. When the qxl module is loaded pending events are resolved and an
> > > >    error is printed.
> > > 
> > > Finally found the time to look at this again... 
> > > 
> > > So, we already have a "group".  Which is basically the sub-directory of
> > > the trace-events file right now, and it seems to be mostly a build system
> > > thing.  We get many small lists instead of one huge, but there seems to
> > > be no other effect.  We could change that though, by giving each group
> > > an (optional?) prefix.
> > 
> > Yes. This reminds me of an idea that was mentioned at the beginning of
> > this effort: maybe QEMU modules should always have their own directory
> > in the source tree instead of being alongside other source files that
> > are built into the main binary.
> 
> This implies that each time we modularize something, we have to move
> its source code. It is possible,  but it wouldn't be by preferred
> choice.

If it bothers you then it probably bothers others too. Let's leave it.

> > > There also is a probe prefix, apparently used by dtrace only.  Not sure
> > > how to deal with that.  It prefix is qemu-<target-type>-<target-name>.
> > > Giving qemu modules its own dtrace prefix looks sensible to me.  That
> > > would probably something like "qemu-module-<name>".
> > 
> > I think the DTrace prefix needs to be the same as the executable name,
> > but I'm not sure. I also don't know how that extends to shared libraries
> > like QEMU modules. I'm afraid you would need to investigate the DTrace
> > prefix.
> 
> I'm not aware of any requirement for dtrace prefix to match the
> executable name. We don't even have an executable called "qemu"
> so we're not matching even today.

Great, thanks for the other email reply where you showed how the
SystemTap tapsetsuse the prefix!

Stefan
Gerd Hoffmann March 31, 2021, 10:55 a.m. UTC | #11
Hi,

> > Well, "make install" with --prefix=$HOME/qemu-install fixed that for the time
> > being.
> > 
> > Now I have this:
> > 
> > kraxel@sirius ~/qemu-install/bin# sudo ./qemu-trace-stap -v run ./qemu-system-x86_64 "qxl_soft_reset"
> > Using tapset dir '/home/kraxel/qemu-install/share/systemtap/tapset' for binary './qemu-system-x86_64'
> > Compiling script 'probe qemu.system.x86_64.log.qxl_soft_reset {}'
> > semantic error: unresolved function pid: identifier 'pid' at /home/kraxel/qemu-install/share/systemtap/tapset/qemu-system-x86_64-log.stp:5451:41
> >         source:     printf("%d@%d qxl_soft_reset %d\n", pid(), gettimeofday_ns(), qid)
> >                                                         ^
> > 
> > Pass 2: analysis failed.  [man error::pass2]
> > 
> > Any clue why pid() isn't known?
> 
> Hmm, strange, makes me think we have a bug causing it to not pull in
> global functions.

Hmm.  5.1.0 fails the same way.  5.2.0 fails in a different way.  Seems
we had temporary breakage in 5.2.0 (was that the module thing which
needed some workaround?).  Given 5.1.0 fails too I suspect this is a
systemtap change (/me runs fedora 33).  Googling didn't found much,
other than indicating pid() is used frequently in examples and
tutorials.

stap-prep asked for kernel-debuginfo, but installing that (huge package
with 3.5G(!) installed size) hasn't changed the situation.  Guess I only
actually need that if I want trace the kernel not userspace?

take care,
  Gerd
Gerd Hoffmann April 9, 2021, 9:10 a.m. UTC | #12
Hi,

> > semantic error: unresolved function pid: identifier 'pid' at /home/kraxel/qemu-install/share/systemtap/tapset/qemu-system-x86_64-log.stp:5451:41
> >         source:     printf("%d@%d qxl_soft_reset %d\n", pid(), gettimeofday_ns(), qid)

> Hmm, strange, makes me think we have a bug causing it to not pull in
> global functions.

Yep.  qemu-trace-stap sets SYSTEMTAP_TAPSET to
/home/kraxel/qemu-install/share/systemtap/tapset.  Which makes stap use
that exclusively and ignore /usr/share/systemtap/tapset.  Which in turn
makes pid() disappear because that apparently is defined somewhere in
those tapsets.

IOW: It's broken for any $prefix != "/usr".

Using stap -I /home/kraxel/qemu-install/share/systemtap/tapset seems to
work fine, I guess qemu-trace-stap should use that instead ...

take care,
  Gerd
Gerd Hoffmann April 9, 2021, 1:12 p.m. UTC | #13
Hi,

> eg a trace point "dma_map_wait" gets mapped to probes in many
> .stp files, once per target, because we need to match based on
> the executable path:
> 
>   probe qemu.system.x86_64.dma_map_wait = process("/usr/libexec/qemu-system-x86_64").mark("dma_map_wait")

So, that changes with modules, we need the module name now, i.e.

    probe qemu.system.x86_64.qxl_soft_reset = \
	process("/home/kraxel/qemu-install/lib/qemu/hw-display-qxl.so").mark("qxl_soft_reset")

We could repeat that in every qemu-system-$arch.stp file.

We could also have one stp file per module, with probes like this:

    probe qemu.modules.qxl_soft_reset = \
        process("/home/kraxel/qemu-install/lib/qemu/hw-display-qxl.so").mark("qxl_soft_reset")

The later looks like a better fit to me, but has the drawback that the
tracepoints have different names in modular and non-modular builds ...

take care,
  Gerd
Daniel P. Berrangé April 9, 2021, 1:17 p.m. UTC | #14
On Fri, Apr 09, 2021 at 03:12:45PM +0200, Gerd Hoffmann wrote:
>   Hi,
> 
> > eg a trace point "dma_map_wait" gets mapped to probes in many
> > .stp files, once per target, because we need to match based on
> > the executable path:
> > 
> >   probe qemu.system.x86_64.dma_map_wait = process("/usr/libexec/qemu-system-x86_64").mark("dma_map_wait")
> 
> So, that changes with modules, we need the module name now, i.e.
> 
>     probe qemu.system.x86_64.qxl_soft_reset = \
> 	process("/home/kraxel/qemu-install/lib/qemu/hw-display-qxl.so").mark("qxl_soft_reset")
> 
> We could repeat that in every qemu-system-$arch.stp file.

This would have the surprise the 'qemu.system.x86_64.qxl_soft_reset'
probes will fire even for qemu-system-ppc64 / qemu-system-xxxxx etc
because we've not restricted the scope as the original probe did.

If we can't fix that, then we must use the second option to avoid
the surprise IMHO

> We could also have one stp file per module, with probes like this:
> 
>     probe qemu.modules.qxl_soft_reset = \
>         process("/home/kraxel/qemu-install/lib/qemu/hw-display-qxl.so").mark("qxl_soft_reset")
> 
> The later looks like a better fit to me, but has the drawback that the
> tracepoints have different names in modular and non-modular builds ...

Regards,
Daniel
Gerd Hoffmann April 12, 2021, 1:07 p.m. UTC | #15
On Fri, Apr 09, 2021 at 02:17:13PM +0100, Daniel P. Berrangé wrote:
> On Fri, Apr 09, 2021 at 03:12:45PM +0200, Gerd Hoffmann wrote:
> >   Hi,
> > 
> > > eg a trace point "dma_map_wait" gets mapped to probes in many
> > > .stp files, once per target, because we need to match based on
> > > the executable path:
> > > 
> > >   probe qemu.system.x86_64.dma_map_wait = process("/usr/libexec/qemu-system-x86_64").mark("dma_map_wait")
> > 
> > So, that changes with modules, we need the module name now, i.e.
> > 
> >     probe qemu.system.x86_64.qxl_soft_reset = \
> > 	process("/home/kraxel/qemu-install/lib/qemu/hw-display-qxl.so").mark("qxl_soft_reset")
> > 
> > We could repeat that in every qemu-system-$arch.stp file.
> 
> This would have the surprise the 'qemu.system.x86_64.qxl_soft_reset'
> probes will fire even for qemu-system-ppc64 / qemu-system-xxxxx etc
> because we've not restricted the scope as the original probe did.

Oh, right.

> If we can't fix that, then we must use the second option to avoid
> the surprise IMHO

Yep.  Got that working.  Only problem is qemu-trace-stap is broken now
and it seems there is no easy way out.  Right now qemu-trace-stap can
simply work with a constant prefix, with that change the prefix can be
either qemu.system.$arch or qemu.system.modules and I suspect there is
no way around listing tracepoints to figure the correct name ...

take care,
  Gerd

PS: https://git.kraxel.org/cgit/qemu/log/?h=sirius/trace-modules
Daniel P. Berrangé April 15, 2021, 1:10 p.m. UTC | #16
On Mon, Apr 12, 2021 at 03:07:59PM +0200, Gerd Hoffmann wrote:
> On Fri, Apr 09, 2021 at 02:17:13PM +0100, Daniel P. Berrangé wrote:
> > On Fri, Apr 09, 2021 at 03:12:45PM +0200, Gerd Hoffmann wrote:
> > >   Hi,
> > > 
> > > > eg a trace point "dma_map_wait" gets mapped to probes in many
> > > > .stp files, once per target, because we need to match based on
> > > > the executable path:
> > > > 
> > > >   probe qemu.system.x86_64.dma_map_wait = process("/usr/libexec/qemu-system-x86_64").mark("dma_map_wait")
> > > 
> > > So, that changes with modules, we need the module name now, i.e.
> > > 
> > >     probe qemu.system.x86_64.qxl_soft_reset = \
> > > 	process("/home/kraxel/qemu-install/lib/qemu/hw-display-qxl.so").mark("qxl_soft_reset")
> > > 
> > > We could repeat that in every qemu-system-$arch.stp file.
> > 
> > This would have the surprise the 'qemu.system.x86_64.qxl_soft_reset'
> > probes will fire even for qemu-system-ppc64 / qemu-system-xxxxx etc
> > because we've not restricted the scope as the original probe did.
> 
> Oh, right.
> 
> > If we can't fix that, then we must use the second option to avoid
> > the surprise IMHO
> 
> Yep.  Got that working.  Only problem is qemu-trace-stap is broken now
> and it seems there is no easy way out.  Right now qemu-trace-stap can
> simply work with a constant prefix, with that change the prefix can be
> either qemu.system.$arch or qemu.system.modules and I suspect there is
> no way around listing tracepoints to figure the correct name ...

Maybe just don't change the probe names, and declare that people have
to tell stap to filter by pid if you want exact matches

Regards,
Daniel