| Message ID | 1685610013-33478-1-git-send-email-akaher@vmware.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | tracing: introducing eventfs | expand |
> On 01-Jun-2023, at 2:30 PM, Ajay Kaher <akaher@vmware.com> wrote: > > Events Tracing infrastructure contains lot of files, directories > (internally in terms of inodes, dentries). And ends up by consuming > memory in MBs. We can have multiple events of Events Tracing, which > further requires more memory. > > Instead of creating inodes/dentries, eventfs could keep meta-data and > skip the creation of inodes/dentries. As and when require, eventfs will > create the inodes/dentries only for required files/directories. > Also eventfs would delete the inodes/dentries once no more requires > but preserve the meta data. > > Tracing events took ~9MB, with this approach it took ~4.5MB > for ~10K files/dir. > Steve, I have used nested rw-semaphore for eventfs locking (same as in cifs). As per Amit Nadav, this has to be revisited/reviewed. Please have a look and share your thoughts. -Ajay
> On 01-Jun-2023, at 2:30 PM, Ajay Kaher <akaher@vmware.com> wrote: > > Events Tracing infrastructure contains lot of files, directories > (internally in terms of inodes, dentries). And ends up by consuming > memory in MBs. We can have multiple events of Events Tracing, which > further requires more memory. > > Instead of creating inodes/dentries, eventfs could keep meta-data and > skip the creation of inodes/dentries. As and when require, eventfs will > create the inodes/dentries only for required files/directories. > Also eventfs would delete the inodes/dentries once no more requires > but preserve the meta data. > > Tracing events took ~9MB, with this approach it took ~4.5MB > for ~10K files/dir. > > v2: > Patch 01: new patch:'Require all trace events to have a TRACE_SYSTEM' > Patch 02: moved from v1 1/9 > Patch 03: moved from v1 2/9 > As suggested by Zheng Yejian, introduced eventfs_prepare_ef() > helper function to add files or directories to eventfs > fix WARNING reported by kernel test robot in v1 8/9 > Patch 04: moved from v1 3/9 > used eventfs_prepare_ef() to add files > fix WARNING reported by kernel test robot in v1 8/9 > Patch 05: moved from v1 4/9 > fix compiling warning reported by kernel test robot in v1 4/9 > Patch 06: moved from v1 5/9 > Patch 07: moved from v1 6/9 > Patch 08: moved from v1 7/9 > Patch 09: moved from v1 8/9 > rebased because of v3 01/10 > Patch 10: moved from v1 9/9 > > v1: > Patch 1: add header file > Patch 2: resolved kernel test robot issues > protecting eventfs lists using nested eventfs_rwsem > Patch 3: protecting eventfs lists using nested eventfs_rwsem > Patch 4: improve events cleanup code to fix crashes > Patch 5: resolved kernel test robot issues > removed d_instantiate_anon() calls > Patch 6: resolved kernel test robot issues > fix kprobe test in eventfs_root_lookup() > protecting eventfs lists using nested eventfs_rwsem > Patch 7: remove header file > Patch 8: pass eventfs_rwsem as argument to eventfs functions > called eventfs_remove_events_dir() instead of tracefs_remove() > from event_trace_del_tracer() > Patch 9: new patch to fix kprobe test case > > fs/tracefs/Makefile | 1 + > fs/tracefs/event_inode.c | 757 ++++++++++++++++++ > fs/tracefs/inode.c | 124 ++- > fs/tracefs/internal.h | 25 + > include/linux/trace_events.h | 1 + > include/linux/tracefs.h | 49 ++ > kernel/trace/trace.h | 3 +- > kernel/trace/trace_events.c | 78 +- > .../ftrace/test.d/kprobe/kprobe_args_char.tc | 4 +- > .../test.d/kprobe/kprobe_args_string.tc | 4 +- > 10 files changed, 994 insertions(+), 52 deletions(-) > create mode 100644 fs/tracefs/event_inode.c > create mode 100644 fs/tracefs/internal.h > Hi Steve, below is the ftracetest results with v3 of eventfs: root@photon-6 [ ~/linux-6.3-rc5/tools/testing/selftests/ftrace ]# ./ftracetest === Ftrace unit tests === [1] Basic trace file check [PASS] [2] Basic test for tracers [PASS] [3] Basic trace clock test [PASS] [4] Basic event tracing check [PASS] [5] Change the ringbuffer size [PASS] [6] Snapshot and tracing setting [UNSUPPORTED] [7] trace_pipe and trace_marker [PASS] [8] Test ftrace direct functions against tracers [UNRESOLVED] [9] Test ftrace direct functions against kprobes [UNRESOLVED] [10] Generic dynamic event - add/remove eprobe events [PASS] [11] Generic dynamic event - add/remove kprobe events [PASS] [12] Generic dynamic event - add/remove synthetic events [UNSUPPORTED] [13] Generic dynamic event - selective clear (compatibility) [UNSUPPORTED] [14] Event probe event parser error log check [PASS] [15] Generic dynamic event - generic clear event [UNSUPPORTED] [16] Generic dynamic event - check if duplicate events are caught [PASS] [17] event tracing - enable/disable with event level files [PASS] [18] event tracing - restricts events based on pid notrace filtering [PASS] [19] event tracing - restricts events based on pid [PASS] [20] event tracing - enable/disable with subsystem level files [PASS] [21] event tracing - enable/disable with top level files [PASS] [22] Test trace_printk from module [UNRESOLVED] [23] event filter function - test event filtering on functions [FAIL] [24] ftrace - function graph filters with stack tracer [UNSUPPORTED] [25] ftrace - function graph filters [PASS] [26] ftrace - function trace with cpumask [PASS] [27] ftrace - test for function event triggers [PASS] [28] ftrace - function glob filters [PASS] [29] ftrace - function pid notrace filters [PASS] [30] ftrace - function pid filters [PASS] [31] ftrace - stacktrace filter command [PASS] [32] ftrace - function trace on module [UNRESOLVED] [33] ftrace - function profiler with function tracing [UNSUPPORTED] [34] ftrace - function profiling [UNSUPPORTED] [35] ftrace - test reading of set_ftrace_filter [PASS] [36] ftrace - Max stack tracer [UNSUPPORTED] [37] ftrace - test for function traceon/off triggers [PASS] [38] ftrace - test tracing error log support [PASS] [39] Test creation and deletion of trace instances while setting an event [PASS] [40] Test creation and deletion of trace instances [PASS] [41] Kprobe dynamic event - adding and removing [PASS] [42] Kprobe dynamic event - busy event check [PASS] [43] Kprobe event char type argument [PASS] [44] Kprobe event with comm arguments [PASS] [45] Kprobe event string type argument [PASS] [46] Kprobe event symbol argument [PASS] [47] Kprobe event argument syntax [PASS] [48] Kprobe dynamic event with arguments [PASS] [49] Kprobes event arguments with types [PASS] [50] Kprobe event user-memory access [PASS] [51] Kprobe event auto/manual naming [PASS] [52] Kprobe dynamic event with function tracer [UNSUPPORTED] [53] Kprobe dynamic event - probing module [UNRESOLVED] [54] Create/delete multiprobe on kprobe event [PASS] [55] Kprobe event parser error log check [PASS] [56] Kretprobe dynamic event with arguments [PASS] [57] Kretprobe dynamic event with maxactive [PASS] [58] Kretprobe %return suffix test [PASS] [59] Register/unregister many kprobe events [PASS] [60] Kprobe events - probe points [FAIL] [61] Kprobe profile [PASS] [62] Uprobe event parser error log check [PASS] [63] test for the preemptirqsoff tracer [UNSUPPORTED] [64] Meta-selftest: Checkbashisms [UNRESOLVED] [65] Test wakeup RT tracer [UNSUPPORTED] [66] Test wakeup tracer [UNSUPPORTED] [67] event trigger - test inter-event histogram trigger expected fail actions [UNSUPPORTED] [68] event trigger - test field variable support [UNSUPPORTED] [69] event trigger - test inter-event combined histogram trigger [UNSUPPORTED] [70] event trigger - test multiple actions on hist trigger [UNSUPPORTED] [71] event trigger - test inter-event histogram trigger onchange action [UNSUPPORTED] [72] event trigger - test inter-event histogram trigger onmatch action [UNSUPPORTED] [73] event trigger - test inter-event histogram trigger onmatch-onmax action [UNSUPPORTED] [74] event trigger - test inter-event histogram trigger onmax action [UNSUPPORTED] [75] event trigger - test inter-event histogram trigger snapshot action [UNSUPPORTED] [76] event trigger - test inter-event histogram trigger eprobe on synthetic event [UNSUPPORTED] [77] event trigger - test synthetic event create remove [UNSUPPORTED] [78] event trigger - test inter-event histogram trigger trace action with dynamic string param [UNSUPPORTED] [79] event trigger - test inter-event histogram trigger trace action with dynamic string param [UNSUPPORTED] [80] event trigger - test synthetic_events syntax parser errors [UNSUPPORTED] [81] event trigger - test synthetic_events syntax parser [UNSUPPORTED] [82] event trigger - test inter-event histogram trigger trace action [UNSUPPORTED] [83] event trigger - test event enable/disable trigger [PASS] [84] event trigger - test trigger filter [PASS] [85] event trigger - test histogram expression parsing [UNSUPPORTED] [86] event trigger - test histogram modifiers [UNSUPPORTED] [87] event trigger - test histogram parser errors [UNSUPPORTED] [88] event trigger - test histogram trigger [UNSUPPORTED] [89] event trigger - test multiple histogram triggers [UNSUPPORTED] [90] event trigger - test snapshot-trigger [UNSUPPORTED] [91] event trigger - test stacktrace-trigger [PASS] [92] trace_marker trigger - test histogram trigger [UNSUPPORTED] [93] trace_marker trigger - test snapshot trigger [UNSUPPORTED] [94] trace_marker trigger - test histogram with synthetic event against kernel event [UNSUPPORTED] [95] trace_marker trigger - test histogram with synthetic event [UNSUPPORTED] [96] event trigger - test traceon/off trigger [PASS] [97] (instance) Basic test for tracers [PASS] [98] (instance) Basic trace clock test [PASS] [99] (instance) Change the ringbuffer size [PASS] [100] (instance) Snapshot and tracing setting [UNSUPPORTED] [101] (instance) trace_pipe and trace_marker [PASS] [102] (instance) event tracing - enable/disable with event level files [PASS] [103] (instance) event tracing - restricts events based on pid notrace filtering [PASS] [104] (instance) event tracing - restricts events based on pid [PASS] [105] (instance) event tracing - enable/disable with subsystem level files [PASS] [106] (instance) event filter function - test event filtering on functions [FAIL] [107] (instance) ftrace - test for function event triggers [PASS] [108] (instance) ftrace - function pid notrace filters [PASS] [109] (instance) ftrace - function pid filters [PASS] [110] (instance) ftrace - stacktrace filter command [PASS] [111] (instance) ftrace - test for function traceon/off triggers [PASS] [112] (instance) event trigger - test event enable/disable trigger [PASS] [113] (instance) event trigger - test trigger filter [PASS] [114] (instance) event trigger - test histogram modifiers [UNSUPPORTED] [115] (instance) event trigger - test histogram trigger [UNSUPPORTED] [116] (instance) event trigger - test multiple histogram triggers [UNSUPPORTED] [117] (instance) trace_marker trigger - test histogram trigger [UNSUPPORTED] [118] (instance) trace_marker trigger - test snapshot trigger [UNSUPPORTED] # of passed: 65 # of failed: 3 # of unresolved: 6 # of untested: 0 # of unsupported: 44 # of xfailed: 0 # of undefined(test bug): 0 These results are same with/without eventfs. -Ajay
On Mon, 19 Jun 2023 05:38:25 +0000 Ajay Kaher <akaher@vmware.com> wrote: > # of passed: 65 > # of failed: 3 Unrelated to your patches, but have you checked why these fail? Do you have the latest tests running on the latest kernel? > # of unresolved: 6 > # of untested: 0 > # of unsupported: 44 > # of xfailed: 0 > # of undefined(test bug): 0 > > These results are same with/without eventfs. I'm hoping to look at these patches this week. Thanks! -- Steve
> On 20-Jun-2023, at 8:32 PM, Steven Rostedt <rostedt@goodmis.org> wrote: > > !! External Email > > On Mon, 19 Jun 2023 05:38:25 +0000 > Ajay Kaher <akaher@vmware.com> wrote: > >> # of passed: 65 >> # of failed: 3 > > Unrelated to your patches, but have you checked why these fail? Do you have > the latest tests running on the latest kernel? > Failed test got passed after enabling /proc/kallsyms, using: echo 0 > /proc/sys/kernel/kptr_restrict Following is the report of ftracetest on Linux v6.4.0-rc7 (with/without eventfs): # of passed: 68 # of failed: 0 # of unresolved: 6 # of untested: 0 # of unsupported: 45 # of xfailed: 0 # of undefined(test bug): 0 If lockdep is enabled getting same warning as reported by 'kernel test robot' for v3 09/10: https://lore.kernel.org/all/1686640004-47546-1-git-send-email-akaher@vmware.com/ >> # of unresolved: 6 >> # of untested: 0 >> # of unsupported: 44 >> # of xfailed: 0 >> # of undefined(test bug): 0 >> >> These results are same with/without eventfs. > > I'm hoping to look at these patches this week. Yes, please. Thanks. -Ajay
On Wed, 21 Jun 2023 11:42:24 +0000 Ajay Kaher <akaher@vmware.com> wrote: > > > > On 20-Jun-2023, at 8:32 PM, Steven Rostedt <rostedt@goodmis.org> wrote: > > > > !! External Email > > > > On Mon, 19 Jun 2023 05:38:25 +0000 > > Ajay Kaher <akaher@vmware.com> wrote: > > > >> # of passed: 65 > >> # of failed: 3 > > > > Unrelated to your patches, but have you checked why these fail? Do you have > > the latest tests running on the latest kernel? > > > > Failed test got passed after enabling /proc/kallsyms, using: > echo 0 > /proc/sys/kernel/kptr_restrict Oh, interesting. It should be 'unresolved' (skipped) if that happens. [23] event filter function - test event filtering on functions [FAIL] [60] Kprobe events - probe points [FAIL] [106] (instance) event filter function - test event filtering on functions [FAIL] OK, let me see. Thanks for reporting! > Following is the report of ftracetest on Linux v6.4.0-rc7 (with/without eventfs): > # of passed: 68 > # of failed: 0 > # of unresolved: 6 > # of untested: 0 > # of unsupported: 45 > # of xfailed: 0 > # of undefined(test bug): 0 > > If lockdep is enabled getting same warning as reported by 'kernel test robot' for v3 09/10: > https://lore.kernel.org/all/1686640004-47546-1-git-send-email-akaher@vmware.com/ > > >> # of unresolved: 6 > >> # of untested: 0 > >> # of unsupported: 44 > >> # of xfailed: 0 > >> # of undefined(test bug): 0 > >> > >> These results are same with/without eventfs. > > > > I'm hoping to look at these patches this week. > > Yes, please. Thanks. > > -Ajay >
Events Tracing infrastructure contains lot of files, directories (internally in terms of inodes, dentries). And ends up by consuming memory in MBs. We can have multiple events of Events Tracing, which further requires more memory. Instead of creating inodes/dentries, eventfs could keep meta-data and skip the creation of inodes/dentries. As and when require, eventfs will create the inodes/dentries only for required files/directories. Also eventfs would delete the inodes/dentries once no more requires but preserve the meta data. Tracing events took ~9MB, with this approach it took ~4.5MB for ~10K files/dir. v2: Patch 01: new patch:'Require all trace events to have a TRACE_SYSTEM' Patch 02: moved from v1 1/9 Patch 03: moved from v1 2/9 As suggested by Zheng Yejian, introduced eventfs_prepare_ef() helper function to add files or directories to eventfs fix WARNING reported by kernel test robot in v1 8/9 Patch 04: moved from v1 3/9 used eventfs_prepare_ef() to add files fix WARNING reported by kernel test robot in v1 8/9 Patch 05: moved from v1 4/9 fix compiling warning reported by kernel test robot in v1 4/9 Patch 06: moved from v1 5/9 Patch 07: moved from v1 6/9 Patch 08: moved from v1 7/9 Patch 09: moved from v1 8/9 rebased because of v3 01/10 Patch 10: moved from v1 9/9 v1: Patch 1: add header file Patch 2: resolved kernel test robot issues protecting eventfs lists using nested eventfs_rwsem Patch 3: protecting eventfs lists using nested eventfs_rwsem Patch 4: improve events cleanup code to fix crashes Patch 5: resolved kernel test robot issues removed d_instantiate_anon() calls Patch 6: resolved kernel test robot issues fix kprobe test in eventfs_root_lookup() protecting eventfs lists using nested eventfs_rwsem Patch 7: remove header file Patch 8: pass eventfs_rwsem as argument to eventfs functions called eventfs_remove_events_dir() instead of tracefs_remove() from event_trace_del_tracer() Patch 9: new patch to fix kprobe test case fs/tracefs/Makefile | 1 + fs/tracefs/event_inode.c | 757 ++++++++++++++++++ fs/tracefs/inode.c | 124 ++- fs/tracefs/internal.h | 25 + include/linux/trace_events.h | 1 + include/linux/tracefs.h | 49 ++ kernel/trace/trace.h | 3 +- kernel/trace/trace_events.c | 78 +- .../ftrace/test.d/kprobe/kprobe_args_char.tc | 4 +- .../test.d/kprobe/kprobe_args_string.tc | 4 +- 10 files changed, 994 insertions(+), 52 deletions(-) create mode 100644 fs/tracefs/event_inode.c create mode 100644 fs/tracefs/internal.h