[bpf-next,v6,00/14] selftests/bpf: Add parallelism to test_progs

Message ID	20211006185619.364369-1-fallentree@fb.com (mailing list archive)
Headers	show Return-Path: <bpf-owner@kernel.org> From: Yucong Sun <fallentree@fb.com> To: <bpf@vger.kernel.org> CC: <andrii@kernel.org>, <sunyucong@gmail.com>, Yucong Sun <fallentree@fb.com> Subject: [PATCH bpf-next v6 00/14] selftests/bpf: Add parallelism to test_progs Date: Wed, 6 Oct 2021 11:56:05 -0700 Message-ID: <20211006185619.364369-1-fallentree@fb.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain Precedence: bulk
Series	selftests/bpf: Add parallelism to test_progs \| expand [bpf-next,v6,00/14] selftests/bpf: Add parallelism to test_progs [bpf-next,v6,01/14] selftests/bpf: Add parallelism to test_progs [bpf-next,v6,02/14] selftests/bpf: Allow some tests to be executed in sequence [bpf-next,v6,03/14] selftests/bpf: disable perf rate limiting when running tests. [bpf-next,v6,04/14] selftests/bpf: add per worker cgroup suffix [bpf-next,v6,05/14] selftests/bpf: adding read_perf_max_sample_freq() helper [bpf-next,v6,06/14] selftests/bpf: fix race condition in enable_stats [bpf-next,v6,07/14] selftests/bpf: make cgroup_v1v2 use its own port [bpf-next,v6,08/14] selftests/bpf: adding a namespace reset for tc_redirect [bpf-next,v6,09/14] selftests/bpf: Make uprobe tests use different attach functions. [bpf-next,v6,10/14] selftests/bpf: adding pid filtering for atomics test [bpf-next,v6,11/14] selftests/bpf: adding random delay for send_signal test [bpf-next,v6,12/14] selftests/bpf: Fix pid check in fexit_sleep test [bpf-next,v6,13/14] selftests/bpf: increase loop count for perf_branches [bpf-next,v6,14/14] selfetest/bpf: make some tests serial

Yucong Sun Oct. 6, 2021, 6:56 p.m. UTC

This patch series adds "-j" parelell execution to test_progs, with "--debug" to
display server/worker communications. Also, some Tests that often fails in
parallel are marked as serial test, and it will run in sequence after parallel
execution is done.

This patch series also adds a error summary after all tests execution finished.

V6 -> V5:
  * adding error summary logic for non parallel mode too.
  * changed how serial tests are implemented, use main process instead of worker 0.
  * fixed a dozen broken test when running in parallel.

V5 -> V4:
  * change to SOCK_SEQPACKET for close notification.
  * move all debug output to "--debug" mode
  * output log as test finish, and all error logs again after summary line.
  * variable naming / style changes
  * adds serial_test_name() to replace serial test lists.


Yucong Sun (14):
  selftests/bpf: Add parallelism to test_progs
  selftests/bpf: Allow some tests to be executed in sequence
  selftests/bpf: disable perf rate limiting when running tests.
  selftests/bpf: add per worker cgroup suffix
  selftests/bpf: adding read_perf_max_sample_freq() helper
  selftests/bpf: fix race condition in enable_stats
  selftests/bpf: make cgroup_v1v2 use its own port
  selftests/bpf: adding a namespace reset for tc_redirect
  selftests/bpf: Make uprobe tests use different attach functions.
  selftests/bpf: adding pid filtering for atomics test
  selftests/bpf: adding random delay for send_signal test
  selftests/bpf: Fix pid check in fexit_sleep test
  selftests/bpf: increase loop count for perf_branches
  selfetest/bpf: make some tests serial

 tools/testing/selftests/bpf/cgroup_helpers.c  |   6 +-
 tools/testing/selftests/bpf/cgroup_helpers.h  |   2 +-
 .../selftests/bpf/prog_tests/atomics.c        |   1 +
 .../selftests/bpf/prog_tests/attach_probe.c   |   8 +-
 .../selftests/bpf/prog_tests/bpf_cookie.c     |  10 +-
 .../bpf/prog_tests/bpf_iter_setsockopt.c      |   2 +-
 .../selftests/bpf/prog_tests/bpf_obj_id.c     |   2 +-
 .../bpf/prog_tests/cg_storage_multi.c         |   2 +-
 .../bpf/prog_tests/cgroup_attach_autodetach.c |   2 +-
 .../bpf/prog_tests/cgroup_attach_multi.c      |   2 +-
 .../bpf/prog_tests/cgroup_attach_override.c   |   2 +-
 .../selftests/bpf/prog_tests/cgroup_link.c    |   2 +-
 .../selftests/bpf/prog_tests/cgroup_v1v2.c    |   2 +-
 .../selftests/bpf/prog_tests/check_mtu.c      |   2 +-
 .../selftests/bpf/prog_tests/fexit_bpf2bpf.c  |   3 +-
 .../prog_tests/flow_dissector_load_bytes.c    |   2 +-
 .../bpf/prog_tests/flow_dissector_reattach.c  |   2 +-
 .../bpf/prog_tests/get_branch_snapshot.c      |   2 +-
 .../selftests/bpf/prog_tests/kfree_skb.c      |   3 +-
 .../bpf/prog_tests/migrate_reuseport.c        |   2 +-
 .../selftests/bpf/prog_tests/modify_return.c  |   3 +-
 .../bpf/prog_tests/ns_current_pid_tgid.c      |   3 +-
 .../selftests/bpf/prog_tests/perf_branches.c  |  10 +-
 .../selftests/bpf/prog_tests/perf_buffer.c    |   2 +-
 .../selftests/bpf/prog_tests/perf_link.c      |   5 +-
 .../selftests/bpf/prog_tests/probe_user.c     |   3 +-
 .../bpf/prog_tests/raw_tp_writable_test_run.c |   3 +-
 .../bpf/prog_tests/select_reuseport.c         |   2 +-
 .../selftests/bpf/prog_tests/send_signal.c    |   6 +-
 .../bpf/prog_tests/send_signal_sched_switch.c |   3 +-
 .../bpf/prog_tests/sk_storage_tracing.c       |   2 +-
 .../selftests/bpf/prog_tests/snprintf_btf.c   |   2 +-
 .../selftests/bpf/prog_tests/sock_fields.c    |   2 +-
 .../selftests/bpf/prog_tests/sockmap_listen.c |   2 +-
 .../bpf/prog_tests/stacktrace_build_id_nmi.c  |  19 +-
 .../selftests/bpf/prog_tests/task_pt_regs.c   |   8 +-
 .../selftests/bpf/prog_tests/tc_redirect.c    |  14 +
 .../testing/selftests/bpf/prog_tests/timer.c  |   3 +-
 .../selftests/bpf/prog_tests/timer_mim.c      |   2 +-
 .../bpf/prog_tests/tp_attach_query.c          |   2 +-
 .../selftests/bpf/prog_tests/trace_printk.c   |   2 +-
 .../selftests/bpf/prog_tests/trace_vprintk.c  |   2 +-
 .../bpf/prog_tests/trampoline_count.c         |   3 +-
 .../selftests/bpf/prog_tests/xdp_attach.c     |   2 +-
 .../selftests/bpf/prog_tests/xdp_bonding.c    |   2 +-
 .../bpf/prog_tests/xdp_cpumap_attach.c        |   2 +-
 .../bpf/prog_tests/xdp_devmap_attach.c        |   2 +-
 .../selftests/bpf/prog_tests/xdp_info.c       |   2 +-
 .../selftests/bpf/prog_tests/xdp_link.c       |   2 +-
 tools/testing/selftests/bpf/progs/atomics.c   |  16 +
 .../selftests/bpf/progs/connect4_dropper.c    |   2 +-
 .../testing/selftests/bpf/progs/fexit_sleep.c |   4 +-
 .../selftests/bpf/progs/test_enable_stats.c   |   2 +-
 tools/testing/selftests/bpf/test_progs.c      | 671 +++++++++++++++++-
 tools/testing/selftests/bpf/test_progs.h      |  37 +-
 55 files changed, 790 insertions(+), 116 deletions(-)

Andrii Nakryiko Oct. 8, 2021, 10:26 p.m. UTC | #1

On Wed, Oct 6, 2021 at 11:56 AM Yucong Sun <fallentree@fb.com> wrote:
>
> This patch series adds "-j" parelell execution to test_progs, with "--debug" to
> display server/worker communications. Also, some Tests that often fails in
> parallel are marked as serial test, and it will run in sequence after parallel
> execution is done.
>
> This patch series also adds a error summary after all tests execution finished.
>

Huge milestone, good job! Applied most patches to bpf-next. See
comments below in respective patches.

We'll need to iterate on improving the stability of parallel mode, but
this is a great start. I've dropped a bunch of "fix up" patches where
I didn't feel confident yet about the approach. We should discuss it
independently from the parallelization changes in this patch set. See
some more thoughts below, but overall:

time sudo ./test_progs -j
...
Summary: 181/977 PASSED, 3 SKIPPED, 0 FAILED

real    0m36.949s
user    0m4.546s
sys     0m30.872s

VS

$ time sudo ./test_progs
...
Summary: 181/977 PASSED, 3 SKIPPED, 0 FAILED

real    1m3.031s
user    0m4.157s
sys     0m28.820s

2x speed up and the gap will just grow over time as we add more tests.
And that's also with bpf_verif_scale as is, which we should break up
into individual tests to parallelize them.

So few things worth mentioning:

1. To focus future efforts on parallelizing existing tests, we should
probably emit how long did the test take.

2. We are losing subtest progress when running in parallel mode. That
sucks. While it's not easy to parallelize subtests, it's easy to send
separate logs for each subtest and display them as they come. Let's do
that?

3. Parallel execution times are not consistent, once I got 30 seconds
(which is 8 seconds faster than sequential, I excluded
bpf_verif_scale), other times it was 45 seconds and more than 1
minute. Not sure what's going on there, but this doesn't look right.

4. A bunch of tests still fail from time to time (see examples below).
What's even scarier that once I got the "failed to determine
tracepoint perf event ID" message, subsequent sequential executions
kept failing. I don't see what selftest could have done to cause this,
so this is concerning and seems to point to the kernel.
/sys/kernel/debug and /sys/kernel/tracing directories were empty at
this point. cc Steven, is there any situation when tracefs can become
"defunct"?

#84 ns_current_pid_tgid:FAIL
test_current_pid_tgid:PASS:skel_open_load 0 nsec
test_current_pid_tgid:PASS:stat 0 nsec
libbpf: failed to determine tracepoint 'syscalls/sys_enter_nanosleep'
perf event ID: No such file or directory
libbpf: prog 'handler': failed to create tracepoint
'syscalls/sys_enter_nanosleep' perf event: No such file or directory
libbpf: failed to auto-attach program 'handler': -2
test_current_pid_tgid:FAIL:skel_attach skeleton attach failed: -2
#84/1 ns_current_pid_tgid/ns_current_pid_tgid_root_ns:FAIL
test_ns_current_pid_tgid_new_ns:PASS:clone 0 nsec
test_ns_current_pid_tgid_new_ns:PASS:waitpid 0 nsec
test_ns_current_pid_tgid_new_ns:FAIL:newns_pidtgid failed#84/2
ns_current_pid_tgid/ns_current_pid_tgid_new_ns:FAIL

#88 perf_buffer:FAIL
serial_test_perf_buffer:PASS:nr_cpus 0 nsec
serial_test_perf_buffer:PASS:nr_on_cpus 0 nsec
serial_test_perf_buffer:PASS:skel_load 0 nsec
libbpf: failed to determine tracepoint 'raw_syscalls/sys_enter' perf
event ID: No such file or directory
libbpf: prog 'handle_sys_enter': failed to create tracepoint
'raw_syscalls/sys_enter' perf event: No such file or directory
libbpf: failed to auto-attach program 'handle_sys_enter': -2
serial_test_perf_buffer:FAIL:attach_kprobe err -2

#110 send_signal_sched_switch:FAIL
serial_test_send_signal_sched_switch:PASS:skel_open_and_load 0 nsec
libbpf: failed to determine tracepoint 'syscalls/sys_enter_nanosleep'
perf event ID: No such file or directory
libbpf: prog 'send_signal_tp': failed to create tracepoint
'syscalls/sys_enter_nanosleep' perf event: No such file or directory
libbpf: failed to auto-attach program 'send_signal_tp': -2
serial_test_send_signal_sched_switch:FAIL:skel_attach skeleton attach failed

#161 tp_attach_query:FAIL
serial_test_tp_attach_query:FAIL:open err -1 errno 2

#163 trace_printk:FAIL
serial_test_trace_printk:PASS:trace_printk__open 0 nsec
serial_test_trace_printk:PASS:skel->rodata->fmt[0] 0 nsec
serial_test_trace_printk:PASS:trace_printk__load 0 nsec
serial_test_trace_printk:PASS:trace_printk__attach 0 nsec
serial_test_trace_printk:FAIL:fopen(TRACEBUF) unexpected error: -2

#164 trace_vprintk:FAIL
serial_test_trace_vprintk:PASS:trace_vprintk__open_and_load 0 nsec
serial_test_trace_vprintk:PASS:trace_vprintk__attach 0 nsec
serial_test_trace_vprintk:FAIL:fopen(TRACEBUF) unexpected error: -2

#46 fexit_stress:FAIL
test_fexit_stress:PASS:find_vmlinux_btf_id 0 nsec
test_fexit_stress:PASS:fexit loaded 0 nsec
test_fexit_stress:PASS:fexit attach failed 0 nsec
test_fexit_stress:PASS:fexit loaded 0 nsec

...

test_fexit_stress:PASS:fexit loaded 0 nsec
test_fexit_stress:PASS:fexit attach failed 0 nsec
test_fexit_stress:PASS:fexit loaded 0 nsec
test_fexit_stress:FAIL:fexit attach failed prog 37 failed: -7 err 7



> V6 -> V5:
>   * adding error summary logic for non parallel mode too.
>   * changed how serial tests are implemented, use main process instead of worker 0.
>   * fixed a dozen broken test when running in parallel.
>
> V5 -> V4:
>   * change to SOCK_SEQPACKET for close notification.
>   * move all debug output to "--debug" mode
>   * output log as test finish, and all error logs again after summary line.
>   * variable naming / style changes
>   * adds serial_test_name() to replace serial test lists.
>
>
> Yucong Sun (14):
>   selftests/bpf: Add parallelism to test_progs
>   selftests/bpf: Allow some tests to be executed in sequence
>   selftests/bpf: disable perf rate limiting when running tests.
>   selftests/bpf: add per worker cgroup suffix
>   selftests/bpf: adding read_perf_max_sample_freq() helper
>   selftests/bpf: fix race condition in enable_stats
>   selftests/bpf: make cgroup_v1v2 use its own port
>   selftests/bpf: adding a namespace reset for tc_redirect
>   selftests/bpf: Make uprobe tests use different attach functions.
>   selftests/bpf: adding pid filtering for atomics test
>   selftests/bpf: adding random delay for send_signal test
>   selftests/bpf: Fix pid check in fexit_sleep test
>   selftests/bpf: increase loop count for perf_branches
>   selfetest/bpf: make some tests serial
>
>  tools/testing/selftests/bpf/cgroup_helpers.c  |   6 +-
>  tools/testing/selftests/bpf/cgroup_helpers.h  |   2 +-
>  .../selftests/bpf/prog_tests/atomics.c        |   1 +
>  .../selftests/bpf/prog_tests/attach_probe.c   |   8 +-
>  .../selftests/bpf/prog_tests/bpf_cookie.c     |  10 +-
>  .../bpf/prog_tests/bpf_iter_setsockopt.c      |   2 +-
>  .../selftests/bpf/prog_tests/bpf_obj_id.c     |   2 +-
>  .../bpf/prog_tests/cg_storage_multi.c         |   2 +-
>  .../bpf/prog_tests/cgroup_attach_autodetach.c |   2 +-
>  .../bpf/prog_tests/cgroup_attach_multi.c      |   2 +-
>  .../bpf/prog_tests/cgroup_attach_override.c   |   2 +-
>  .../selftests/bpf/prog_tests/cgroup_link.c    |   2 +-
>  .../selftests/bpf/prog_tests/cgroup_v1v2.c    |   2 +-
>  .../selftests/bpf/prog_tests/check_mtu.c      |   2 +-
>  .../selftests/bpf/prog_tests/fexit_bpf2bpf.c  |   3 +-
>  .../prog_tests/flow_dissector_load_bytes.c    |   2 +-
>  .../bpf/prog_tests/flow_dissector_reattach.c  |   2 +-
>  .../bpf/prog_tests/get_branch_snapshot.c      |   2 +-
>  .../selftests/bpf/prog_tests/kfree_skb.c      |   3 +-
>  .../bpf/prog_tests/migrate_reuseport.c        |   2 +-
>  .../selftests/bpf/prog_tests/modify_return.c  |   3 +-
>  .../bpf/prog_tests/ns_current_pid_tgid.c      |   3 +-
>  .../selftests/bpf/prog_tests/perf_branches.c  |  10 +-
>  .../selftests/bpf/prog_tests/perf_buffer.c    |   2 +-
>  .../selftests/bpf/prog_tests/perf_link.c      |   5 +-
>  .../selftests/bpf/prog_tests/probe_user.c     |   3 +-
>  .../bpf/prog_tests/raw_tp_writable_test_run.c |   3 +-
>  .../bpf/prog_tests/select_reuseport.c         |   2 +-
>  .../selftests/bpf/prog_tests/send_signal.c    |   6 +-
>  .../bpf/prog_tests/send_signal_sched_switch.c |   3 +-
>  .../bpf/prog_tests/sk_storage_tracing.c       |   2 +-
>  .../selftests/bpf/prog_tests/snprintf_btf.c   |   2 +-
>  .../selftests/bpf/prog_tests/sock_fields.c    |   2 +-
>  .../selftests/bpf/prog_tests/sockmap_listen.c |   2 +-
>  .../bpf/prog_tests/stacktrace_build_id_nmi.c  |  19 +-
>  .../selftests/bpf/prog_tests/task_pt_regs.c   |   8 +-
>  .../selftests/bpf/prog_tests/tc_redirect.c    |  14 +
>  .../testing/selftests/bpf/prog_tests/timer.c  |   3 +-
>  .../selftests/bpf/prog_tests/timer_mim.c      |   2 +-
>  .../bpf/prog_tests/tp_attach_query.c          |   2 +-
>  .../selftests/bpf/prog_tests/trace_printk.c   |   2 +-
>  .../selftests/bpf/prog_tests/trace_vprintk.c  |   2 +-
>  .../bpf/prog_tests/trampoline_count.c         |   3 +-
>  .../selftests/bpf/prog_tests/xdp_attach.c     |   2 +-
>  .../selftests/bpf/prog_tests/xdp_bonding.c    |   2 +-
>  .../bpf/prog_tests/xdp_cpumap_attach.c        |   2 +-
>  .../bpf/prog_tests/xdp_devmap_attach.c        |   2 +-
>  .../selftests/bpf/prog_tests/xdp_info.c       |   2 +-
>  .../selftests/bpf/prog_tests/xdp_link.c       |   2 +-
>  tools/testing/selftests/bpf/progs/atomics.c   |  16 +
>  .../selftests/bpf/progs/connect4_dropper.c    |   2 +-
>  .../testing/selftests/bpf/progs/fexit_sleep.c |   4 +-
>  .../selftests/bpf/progs/test_enable_stats.c   |   2 +-
>  tools/testing/selftests/bpf/test_progs.c      | 671 +++++++++++++++++-
>  tools/testing/selftests/bpf/test_progs.h      |  37 +-
>  55 files changed, 790 insertions(+), 116 deletions(-)
>
> --
> 2.30.2
>

Andrii Nakryiko Oct. 8, 2021, 10:42 p.m. UTC | #2

On Fri, Oct 8, 2021 at 3:26 PM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Wed, Oct 6, 2021 at 11:56 AM Yucong Sun <fallentree@fb.com> wrote:
> >
> > This patch series adds "-j" parelell execution to test_progs, with "--debug" to
> > display server/worker communications. Also, some Tests that often fails in
> > parallel are marked as serial test, and it will run in sequence after parallel
> > execution is done.
> >
> > This patch series also adds a error summary after all tests execution finished.
> >
>
> Huge milestone, good job! Applied most patches to bpf-next. See
> comments below in respective patches.
>
> We'll need to iterate on improving the stability of parallel mode, but
> this is a great start. I've dropped a bunch of "fix up" patches where
> I didn't feel confident yet about the approach. We should discuss it
> independently from the parallelization changes in this patch set. See
> some more thoughts below, but overall:
>
> time sudo ./test_progs -j
> ...
> Summary: 181/977 PASSED, 3 SKIPPED, 0 FAILED
>
> real    0m36.949s
> user    0m4.546s
> sys     0m30.872s
>
> VS
>
> $ time sudo ./test_progs
> ...
> Summary: 181/977 PASSED, 3 SKIPPED, 0 FAILED
>
> real    1m3.031s
> user    0m4.157s
> sys     0m28.820s
>
> 2x speed up and the gap will just grow over time as we add more tests.
> And that's also with bpf_verif_scale as is, which we should break up
> into individual tests to parallelize them.
>
> So few things worth mentioning:
>
> 1. To focus future efforts on parallelizing existing tests, we should
> probably emit how long did the test take.
>
> 2. We are losing subtest progress when running in parallel mode. That
> sucks. While it's not easy to parallelize subtests, it's easy to send
> separate logs for each subtest and display them as they come. Let's do
> that?
>
> 3. Parallel execution times are not consistent, once I got 30 seconds
> (which is 8 seconds faster than sequential, I excluded
> bpf_verif_scale), other times it was 45 seconds and more than 1
> minute. Not sure what's going on there, but this doesn't look right.
>
> 4. A bunch of tests still fail from time to time (see examples below).
> What's even scarier that once I got the "failed to determine
> tracepoint perf event ID" message, subsequent sequential executions
> kept failing. I don't see what selftest could have done to cause this,
> so this is concerning and seems to point to the kernel.
> /sys/kernel/debug and /sys/kernel/tracing directories were empty at
> this point. cc Steven, is there any situation when tracefs can become
> "defunct"?
>

Forgot to actually cc Steven, oops. Steven, I've run into the problem
when running a few selftests that do uprobe/kprobe attachment. At some
point, they started complaining that files like
/sys/kernel/debug/tracing/events/syscalls/sys_enter_nanosleep/id don't
exist. And this condition persisted. When I checked
/sys/kernel/debug/tracing in QEMU, it was empty. Is this a known
problem?

> #84 ns_current_pid_tgid:FAIL
> test_current_pid_tgid:PASS:skel_open_load 0 nsec
> test_current_pid_tgid:PASS:stat 0 nsec
> libbpf: failed to determine tracepoint 'syscalls/sys_enter_nanosleep'
> perf event ID: No such file or directory
> libbpf: prog 'handler': failed to create tracepoint
> 'syscalls/sys_enter_nanosleep' perf event: No such file or directory
> libbpf: failed to auto-attach program 'handler': -2
> test_current_pid_tgid:FAIL:skel_attach skeleton attach failed: -2
> #84/1 ns_current_pid_tgid/ns_current_pid_tgid_root_ns:FAIL
> test_ns_current_pid_tgid_new_ns:PASS:clone 0 nsec
> test_ns_current_pid_tgid_new_ns:PASS:waitpid 0 nsec
> test_ns_current_pid_tgid_new_ns:FAIL:newns_pidtgid failed#84/2
> ns_current_pid_tgid/ns_current_pid_tgid_new_ns:FAIL
>
> #88 perf_buffer:FAIL
> serial_test_perf_buffer:PASS:nr_cpus 0 nsec
> serial_test_perf_buffer:PASS:nr_on_cpus 0 nsec
> serial_test_perf_buffer:PASS:skel_load 0 nsec
> libbpf: failed to determine tracepoint 'raw_syscalls/sys_enter' perf
> event ID: No such file or directory
> libbpf: prog 'handle_sys_enter': failed to create tracepoint
> 'raw_syscalls/sys_enter' perf event: No such file or directory
> libbpf: failed to auto-attach program 'handle_sys_enter': -2
> serial_test_perf_buffer:FAIL:attach_kprobe err -2
>
> #110 send_signal_sched_switch:FAIL
> serial_test_send_signal_sched_switch:PASS:skel_open_and_load 0 nsec
> libbpf: failed to determine tracepoint 'syscalls/sys_enter_nanosleep'
> perf event ID: No such file or directory
> libbpf: prog 'send_signal_tp': failed to create tracepoint
> 'syscalls/sys_enter_nanosleep' perf event: No such file or directory
> libbpf: failed to auto-attach program 'send_signal_tp': -2
> serial_test_send_signal_sched_switch:FAIL:skel_attach skeleton attach failed
>
> #161 tp_attach_query:FAIL
> serial_test_tp_attach_query:FAIL:open err -1 errno 2
>
> #163 trace_printk:FAIL
> serial_test_trace_printk:PASS:trace_printk__open 0 nsec
> serial_test_trace_printk:PASS:skel->rodata->fmt[0] 0 nsec
> serial_test_trace_printk:PASS:trace_printk__load 0 nsec
> serial_test_trace_printk:PASS:trace_printk__attach 0 nsec
> serial_test_trace_printk:FAIL:fopen(TRACEBUF) unexpected error: -2
>
> #164 trace_vprintk:FAIL
> serial_test_trace_vprintk:PASS:trace_vprintk__open_and_load 0 nsec
> serial_test_trace_vprintk:PASS:trace_vprintk__attach 0 nsec
> serial_test_trace_vprintk:FAIL:fopen(TRACEBUF) unexpected error: -2
>
> #46 fexit_stress:FAIL
> test_fexit_stress:PASS:find_vmlinux_btf_id 0 nsec
> test_fexit_stress:PASS:fexit loaded 0 nsec
> test_fexit_stress:PASS:fexit attach failed 0 nsec
> test_fexit_stress:PASS:fexit loaded 0 nsec
>
> ...
>
> test_fexit_stress:PASS:fexit loaded 0 nsec
> test_fexit_stress:PASS:fexit attach failed 0 nsec
> test_fexit_stress:PASS:fexit loaded 0 nsec
> test_fexit_stress:FAIL:fexit attach failed prog 37 failed: -7 err 7
>
>
>
> > V6 -> V5:
> >   * adding error summary logic for non parallel mode too.
> >   * changed how serial tests are implemented, use main process instead of worker 0.
> >   * fixed a dozen broken test when running in parallel.
> >
> > V5 -> V4:
> >   * change to SOCK_SEQPACKET for close notification.
> >   * move all debug output to "--debug" mode
> >   * output log as test finish, and all error logs again after summary line.
> >   * variable naming / style changes
> >   * adds serial_test_name() to replace serial test lists.
> >
> >
> > Yucong Sun (14):
> >   selftests/bpf: Add parallelism to test_progs
> >   selftests/bpf: Allow some tests to be executed in sequence
> >   selftests/bpf: disable perf rate limiting when running tests.
> >   selftests/bpf: add per worker cgroup suffix
> >   selftests/bpf: adding read_perf_max_sample_freq() helper
> >   selftests/bpf: fix race condition in enable_stats
> >   selftests/bpf: make cgroup_v1v2 use its own port
> >   selftests/bpf: adding a namespace reset for tc_redirect
> >   selftests/bpf: Make uprobe tests use different attach functions.
> >   selftests/bpf: adding pid filtering for atomics test
> >   selftests/bpf: adding random delay for send_signal test
> >   selftests/bpf: Fix pid check in fexit_sleep test
> >   selftests/bpf: increase loop count for perf_branches
> >   selfetest/bpf: make some tests serial
> >
> >  tools/testing/selftests/bpf/cgroup_helpers.c  |   6 +-
> >  tools/testing/selftests/bpf/cgroup_helpers.h  |   2 +-
> >  .../selftests/bpf/prog_tests/atomics.c        |   1 +
> >  .../selftests/bpf/prog_tests/attach_probe.c   |   8 +-
> >  .../selftests/bpf/prog_tests/bpf_cookie.c     |  10 +-
> >  .../bpf/prog_tests/bpf_iter_setsockopt.c      |   2 +-
> >  .../selftests/bpf/prog_tests/bpf_obj_id.c     |   2 +-
> >  .../bpf/prog_tests/cg_storage_multi.c         |   2 +-
> >  .../bpf/prog_tests/cgroup_attach_autodetach.c |   2 +-
> >  .../bpf/prog_tests/cgroup_attach_multi.c      |   2 +-
> >  .../bpf/prog_tests/cgroup_attach_override.c   |   2 +-
> >  .../selftests/bpf/prog_tests/cgroup_link.c    |   2 +-
> >  .../selftests/bpf/prog_tests/cgroup_v1v2.c    |   2 +-
> >  .../selftests/bpf/prog_tests/check_mtu.c      |   2 +-
> >  .../selftests/bpf/prog_tests/fexit_bpf2bpf.c  |   3 +-
> >  .../prog_tests/flow_dissector_load_bytes.c    |   2 +-
> >  .../bpf/prog_tests/flow_dissector_reattach.c  |   2 +-
> >  .../bpf/prog_tests/get_branch_snapshot.c      |   2 +-
> >  .../selftests/bpf/prog_tests/kfree_skb.c      |   3 +-
> >  .../bpf/prog_tests/migrate_reuseport.c        |   2 +-
> >  .../selftests/bpf/prog_tests/modify_return.c  |   3 +-
> >  .../bpf/prog_tests/ns_current_pid_tgid.c      |   3 +-
> >  .../selftests/bpf/prog_tests/perf_branches.c  |  10 +-
> >  .../selftests/bpf/prog_tests/perf_buffer.c    |   2 +-
> >  .../selftests/bpf/prog_tests/perf_link.c      |   5 +-
> >  .../selftests/bpf/prog_tests/probe_user.c     |   3 +-
> >  .../bpf/prog_tests/raw_tp_writable_test_run.c |   3 +-
> >  .../bpf/prog_tests/select_reuseport.c         |   2 +-
> >  .../selftests/bpf/prog_tests/send_signal.c    |   6 +-
> >  .../bpf/prog_tests/send_signal_sched_switch.c |   3 +-
> >  .../bpf/prog_tests/sk_storage_tracing.c       |   2 +-
> >  .../selftests/bpf/prog_tests/snprintf_btf.c   |   2 +-
> >  .../selftests/bpf/prog_tests/sock_fields.c    |   2 +-
> >  .../selftests/bpf/prog_tests/sockmap_listen.c |   2 +-
> >  .../bpf/prog_tests/stacktrace_build_id_nmi.c  |  19 +-
> >  .../selftests/bpf/prog_tests/task_pt_regs.c   |   8 +-
> >  .../selftests/bpf/prog_tests/tc_redirect.c    |  14 +
> >  .../testing/selftests/bpf/prog_tests/timer.c  |   3 +-
> >  .../selftests/bpf/prog_tests/timer_mim.c      |   2 +-
> >  .../bpf/prog_tests/tp_attach_query.c          |   2 +-
> >  .../selftests/bpf/prog_tests/trace_printk.c   |   2 +-
> >  .../selftests/bpf/prog_tests/trace_vprintk.c  |   2 +-
> >  .../bpf/prog_tests/trampoline_count.c         |   3 +-
> >  .../selftests/bpf/prog_tests/xdp_attach.c     |   2 +-
> >  .../selftests/bpf/prog_tests/xdp_bonding.c    |   2 +-
> >  .../bpf/prog_tests/xdp_cpumap_attach.c        |   2 +-
> >  .../bpf/prog_tests/xdp_devmap_attach.c        |   2 +-
> >  .../selftests/bpf/prog_tests/xdp_info.c       |   2 +-
> >  .../selftests/bpf/prog_tests/xdp_link.c       |   2 +-
> >  tools/testing/selftests/bpf/progs/atomics.c   |  16 +
> >  .../selftests/bpf/progs/connect4_dropper.c    |   2 +-
> >  .../testing/selftests/bpf/progs/fexit_sleep.c |   4 +-
> >  .../selftests/bpf/progs/test_enable_stats.c   |   2 +-
> >  tools/testing/selftests/bpf/test_progs.c      | 671 +++++++++++++++++-
> >  tools/testing/selftests/bpf/test_progs.h      |  37 +-
> >  55 files changed, 790 insertions(+), 116 deletions(-)
> >
> > --
> > 2.30.2
> >

Steven Rostedt Oct. 8, 2021, 11:37 p.m. UTC | #3

On Fri, 8 Oct 2021 15:42:31 -0700
Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote:

> orgot to actually cc Steven, oops. Steven, I've run into the problem
> when running a few selftests that do uprobe/kprobe attachment. At some
> point, they started complaining that files like
> /sys/kernel/debug/tracing/events/syscalls/sys_enter_nanosleep/id don't
> exist. And this condition persisted. When I checked
> /sys/kernel/debug/tracing in QEMU, it was empty. Is this a known
> problem?

The tracefs directory should automatically be mounted in the debugfs
"tracing" directory when debugfs is mounted.

Does /sys/kernel/tracing exist?

-- Steve

Andrii Nakryiko Oct. 9, 2021, 3:20 a.m. UTC | #4

On Fri, Oct 8, 2021 at 4:37 PM Steven Rostedt <rostedt@goodmis.org> wrote:
>
> On Fri, 8 Oct 2021 15:42:31 -0700
> Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote:
>
> > orgot to actually cc Steven, oops. Steven, I've run into the problem
> > when running a few selftests that do uprobe/kprobe attachment. At some
> > point, they started complaining that files like
> > /sys/kernel/debug/tracing/events/syscalls/sys_enter_nanosleep/id don't
> > exist. And this condition persisted. When I checked
> > /sys/kernel/debug/tracing in QEMU, it was empty. Is this a known
> > problem?
>
> The tracefs directory should automatically be mounted in the debugfs
> "tracing" directory when debugfs is mounted.

Yeah, we do that for our testing environment and it works reliably.
This time we started running a bunch of tests completely in parallel,
and in one case I ran into this situation that caused "permanent
damage". Unfortunately I haven't checked if debugfs is still mounted
or not.

Yucong mentioned that one of the selftests (tc_redirect.c) is
manipulating /sys mount, so I wonder if this could have caused some
consequences like this... Jussi, any idea?

>
> Does /sys/kernel/tracing exist?

Yeah, it was there, but was empty.

>
> -- Steve

Steven Rostedt Oct. 9, 2021, 3:28 a.m. UTC | #5

On Fri, 8 Oct 2021 20:20:38 -0700
Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote:

> > Does /sys/kernel/tracing exist?  
> 
> Yeah, it was there, but was empty.

And would be filled with: mount -t tracefs nodev /sys/kernel/tracing

-- Steve

[bpf-next,v6,00/14] selftests/bpf: Add parallelism to test_progs

Message

Comments