| Message ID | 20230421174234.2391278-1-eddyz87@gmail.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Second set of verifier/*.c migrated to inline assembly | expand |
Hello: This series was applied to bpf/bpf-next.git (master) by Alexei Starovoitov <ast@kernel.org>: On Fri, 21 Apr 2023 20:42:10 +0300 you wrote: > This is a follow up for RFC [1]. It migrates a second batch of 23 > verifier/*.c tests to inline assembly and use of ./test_progs for > actual execution. Link to the first batch is [2]. > > The migration is done by a python script (see [3]) with minimal manual > adjustments. > > [...] Here is the summary with links: - [bpf-next,01/24] selftests/bpf: Add notion of auxiliary programs for test_loader https://git.kernel.org/bpf/bpf-next/c/63bb645b9da3 - [bpf-next,02/24] selftests/bpf: verifier/bounds converted to inline assembly https://git.kernel.org/bpf/bpf-next/c/c92336559ac0 - [bpf-next,03/24] selftests/bpf: verifier/bpf_get_stack converted to inline assembly https://git.kernel.org/bpf/bpf-next/c/965a3f913e72 - [bpf-next,04/24] selftests/bpf: verifier/btf_ctx_access converted to inline assembly https://git.kernel.org/bpf/bpf-next/c/37467c79e16a - [bpf-next,05/24] selftests/bpf: verifier/ctx converted to inline assembly https://git.kernel.org/bpf/bpf-next/c/fcd36964f22b - [bpf-next,06/24] selftests/bpf: verifier/d_path converted to inline assembly https://git.kernel.org/bpf/bpf-next/c/608028024384 - [bpf-next,07/24] selftests/bpf: verifier/direct_packet_access converted to inline assembly https://git.kernel.org/bpf/bpf-next/c/0a372c9c0812 - [bpf-next,08/24] selftests/bpf: verifier/jeq_infer_not_null converted to inline assembly https://git.kernel.org/bpf/bpf-next/c/a5828e3154d1 - [bpf-next,09/24] selftests/bpf: verifier/loops1 converted to inline assembly https://git.kernel.org/bpf/bpf-next/c/a6fc14dc5e8d - [bpf-next,10/24] selftests/bpf: verifier/lwt converted to inline assembly https://git.kernel.org/bpf/bpf-next/c/b427ca576f83 - [bpf-next,11/24] selftests/bpf: verifier/map_in_map converted to inline assembly https://git.kernel.org/bpf/bpf-next/c/4a400ef9ba41 - [bpf-next,12/24] selftests/bpf: verifier/map_ptr_mixing converted to inline assembly https://git.kernel.org/bpf/bpf-next/c/aee1779f0dec - [bpf-next,13/24] selftests/bpf: verifier/precise converted to inline assembly (no matching commit) - [bpf-next,14/24] selftests/bpf: verifier/prevent_map_lookup converted to inline assembly (no matching commit) - [bpf-next,15/24] selftests/bpf: verifier/ref_tracking converted to inline assembly https://git.kernel.org/bpf/bpf-next/c/8be632795996 - [bpf-next,16/24] selftests/bpf: verifier/regalloc converted to inline assembly https://git.kernel.org/bpf/bpf-next/c/16a42573c253 - [bpf-next,17/24] selftests/bpf: verifier/runtime_jit converted to inline assembly https://git.kernel.org/bpf/bpf-next/c/65222842ca04 - [bpf-next,18/24] selftests/bpf: verifier/search_pruning converted to inline assembly https://git.kernel.org/bpf/bpf-next/c/034d9ad25db3 - [bpf-next,19/24] selftests/bpf: verifier/sock converted to inline assembly https://git.kernel.org/bpf/bpf-next/c/426fc0e3fce2 - [bpf-next,20/24] selftests/bpf: verifier/spin_lock converted to inline assembly https://git.kernel.org/bpf/bpf-next/c/f323a81806bd - [bpf-next,21/24] selftests/bpf: verifier/subreg converted to inline assembly https://git.kernel.org/bpf/bpf-next/c/81d1d6dd4037 - [bpf-next,22/24] selftests/bpf: verifier/unpriv converted to inline assembly https://git.kernel.org/bpf/bpf-next/c/82887c2568e4 - [bpf-next,23/24] selftests/bpf: verifier/value_illegal_alu converted to inline assembly https://git.kernel.org/bpf/bpf-next/c/efe25a330b10 - [bpf-next,24/24] selftests/bpf: verifier/value_ptr_arith converted to inline assembly https://git.kernel.org/bpf/bpf-next/c/4db10a8243df You are awesome, thank you!
On Fri, Apr 21, 2023 at 10:42 AM Eduard Zingerman <eddyz87@gmail.com> wrote: > > This is a follow up for RFC [1]. It migrates a second batch of 23 > verifier/*.c tests to inline assembly and use of ./test_progs for > actual execution. Link to the first batch is [2]. > > The migration is done by a python script (see [3]) with minimal manual > adjustments. All makes sense to me. Took 22 out of 24 patches. The 13 and 14 had conflicts. Also there is a precision fix in bpf tree. So we're going to wait for bpf/bpf-next to converge during the merge window next week, then add another precision test as asm and then regenerate conversion of the precision tests. Not sure why 14 was conflicting.
On Fri, 2023-04-21 at 19:40 +0000, patchwork-bot+netdevbpf@kernel.org wrote: > Hello: > > This series was applied to bpf/bpf-next.git (master) > by Alexei Starovoitov <ast@kernel.org>: Hi Alexei, Thank you for merging these changes! I've noticed that email from the bot does not list commit hashes for patches #13,14 (precise, prevent_map_lookup). And these commits are indeed not in git [1]. Is this intentional? Thanks, Eduard [1] https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git/log/ > > On Fri, 21 Apr 2023 20:42:10 +0300 you wrote: > > This is a follow up for RFC [1]. It migrates a second batch of 23 > > verifier/*.c tests to inline assembly and use of ./test_progs for > > actual execution. Link to the first batch is [2]. > > > > The migration is done by a python script (see [3]) with minimal manual > > adjustments. > > > > [...] > > Here is the summary with links: > - [bpf-next,01/24] selftests/bpf: Add notion of auxiliary programs for test_loader > https://git.kernel.org/bpf/bpf-next/c/63bb645b9da3 > - [bpf-next,02/24] selftests/bpf: verifier/bounds converted to inline assembly > https://git.kernel.org/bpf/bpf-next/c/c92336559ac0 > - [bpf-next,03/24] selftests/bpf: verifier/bpf_get_stack converted to inline assembly > https://git.kernel.org/bpf/bpf-next/c/965a3f913e72 > - [bpf-next,04/24] selftests/bpf: verifier/btf_ctx_access converted to inline assembly > https://git.kernel.org/bpf/bpf-next/c/37467c79e16a > - [bpf-next,05/24] selftests/bpf: verifier/ctx converted to inline assembly > https://git.kernel.org/bpf/bpf-next/c/fcd36964f22b > - [bpf-next,06/24] selftests/bpf: verifier/d_path converted to inline assembly > https://git.kernel.org/bpf/bpf-next/c/608028024384 > - [bpf-next,07/24] selftests/bpf: verifier/direct_packet_access converted to inline assembly > https://git.kernel.org/bpf/bpf-next/c/0a372c9c0812 > - [bpf-next,08/24] selftests/bpf: verifier/jeq_infer_not_null converted to inline assembly > https://git.kernel.org/bpf/bpf-next/c/a5828e3154d1 > - [bpf-next,09/24] selftests/bpf: verifier/loops1 converted to inline assembly > https://git.kernel.org/bpf/bpf-next/c/a6fc14dc5e8d > - [bpf-next,10/24] selftests/bpf: verifier/lwt converted to inline assembly > https://git.kernel.org/bpf/bpf-next/c/b427ca576f83 > - [bpf-next,11/24] selftests/bpf: verifier/map_in_map converted to inline assembly > https://git.kernel.org/bpf/bpf-next/c/4a400ef9ba41 > - [bpf-next,12/24] selftests/bpf: verifier/map_ptr_mixing converted to inline assembly > https://git.kernel.org/bpf/bpf-next/c/aee1779f0dec > - [bpf-next,13/24] selftests/bpf: verifier/precise converted to inline assembly > (no matching commit) > - [bpf-next,14/24] selftests/bpf: verifier/prevent_map_lookup converted to inline assembly > (no matching commit) > - [bpf-next,15/24] selftests/bpf: verifier/ref_tracking converted to inline assembly > https://git.kernel.org/bpf/bpf-next/c/8be632795996 > - [bpf-next,16/24] selftests/bpf: verifier/regalloc converted to inline assembly > https://git.kernel.org/bpf/bpf-next/c/16a42573c253 > - [bpf-next,17/24] selftests/bpf: verifier/runtime_jit converted to inline assembly > https://git.kernel.org/bpf/bpf-next/c/65222842ca04 > - [bpf-next,18/24] selftests/bpf: verifier/search_pruning converted to inline assembly > https://git.kernel.org/bpf/bpf-next/c/034d9ad25db3 > - [bpf-next,19/24] selftests/bpf: verifier/sock converted to inline assembly > https://git.kernel.org/bpf/bpf-next/c/426fc0e3fce2 > - [bpf-next,20/24] selftests/bpf: verifier/spin_lock converted to inline assembly > https://git.kernel.org/bpf/bpf-next/c/f323a81806bd > - [bpf-next,21/24] selftests/bpf: verifier/subreg converted to inline assembly > https://git.kernel.org/bpf/bpf-next/c/81d1d6dd4037 > - [bpf-next,22/24] selftests/bpf: verifier/unpriv converted to inline assembly > https://git.kernel.org/bpf/bpf-next/c/82887c2568e4 > - [bpf-next,23/24] selftests/bpf: verifier/value_illegal_alu converted to inline assembly > https://git.kernel.org/bpf/bpf-next/c/efe25a330b10 > - [bpf-next,24/24] selftests/bpf: verifier/value_ptr_arith converted to inline assembly > https://git.kernel.org/bpf/bpf-next/c/4db10a8243df > > You are awesome, thank you!
On Fri, Apr 21, 2023 at 12:49 PM Eduard Zingerman <eddyz87@gmail.com> wrote: > > On Fri, 2023-04-21 at 19:40 +0000, patchwork-bot+netdevbpf@kernel.org wrote: > > Hello: > > > > This series was applied to bpf/bpf-next.git (master) > > by Alexei Starovoitov <ast@kernel.org>: > > Hi Alexei, > > Thank you for merging these changes! > > I've noticed that email from the bot does not list > commit hashes for patches #13,14 (precise, prevent_map_lookup). > And these commits are indeed not in git [1]. > Is this intentional? Yes. See other reply.
On Fri, 2023-04-21 at 12:48 -0700, Alexei Starovoitov wrote: > On Fri, Apr 21, 2023 at 10:42 AM Eduard Zingerman <eddyz87@gmail.com> wrote: > > > > This is a follow up for RFC [1]. It migrates a second batch of 23 > > verifier/*.c tests to inline assembly and use of ./test_progs for > > actual execution. Link to the first batch is [2]. > > > > The migration is done by a python script (see [3]) with minimal manual > > adjustments. > > All makes sense to me. > Took 22 out of 24 patches. > The 13 and 14 had conflicts. > Also there is a precision fix in bpf tree. > So we're going to wait for bpf/bpf-next to converge during > the merge window next week, then add another precision test as asm > and then regenerate conversion of the precision tests. > Not sure why 14 was conflicting. Oh, understood, thank you. The #14 does not apply because it also starts from 'p', so the hunk below does not match: --- tools/testing/selftests/bpf/prog_tests/verifier.c +++ tools/testing/selftests/bpf/prog_tests/verifier.c @@ -41,6 +41,7 @@ #include "verifier_masking.skel.h" #include "verifier_meta_access.skel.h" #include "verifier_precise.skel.h" +#include "verifier_prevent_map_lookup.skel.h" #include "verifier_raw_stack.skel.h" #include "verifier_raw_tp_writable.skel.h" #include "verifier_reg_equal.skel.h" I will resend it shortly.
This is a follow up for RFC [1]. It migrates a second batch of 23 verifier/*.c tests to inline assembly and use of ./test_progs for actual execution. Link to the first batch is [2]. The migration is done by a python script (see [3]) with minimal manual adjustments. Each migrated verifier/xxx.c file is mapped to progs/verifier_xxx.c plus an entry in the prog_tests/verifier.c. One patch per each file. The first patch in a series adds a notion of auxiliary programs to test_loader. This is necessary to support tests that use BPF_MAP_TYPE_PROG_ARRAY maps. Programs marked as auxiliary are always loaded but are not treated as a separate tests. The main differences compared to the previous batch: - Tests that use pseudo call instruction are migrated, called functions are translated as below: static __naked __noinline __attribute__((used)) void bounded_recursion__1(void) { asm volatile (...); } Pseudo call in the inline assembly looks as follows: __naked void bounded_recursion(void) { asm volatile (" \ r1 = 0; \ call bounded_recursion__1; \ exit; \ " ::: __clobber_all); } Interestingly enough, callee declaration does not have to precede caller definition when used from inline assembly. - Tests that specify .expected_attach_type are migrated using corresponding SEC annotations. For example, the following test specification: { "reference tracking: ...", .insns = { ... }, .prog_type = BPF_PROG_TYPE_LSM, .kfunc = "bpf", .expected_attach_type = BPF_LSM_MAC, .flags = BPF_F_SLEEPABLE, ... }, Becomes: SEC("lsm.s/bpf") __description("reference tracking: ...") __success __naked void acquire_release_user_key_reference(void) { ... } - Tests that use BPF_MAP_TYPE_PROG_ARRAY are migrated, the definitions of map and dummy programs that populate it are repeated in each test. There `__imm_insn` macro had to be used in a few tests because of the limitations of clang BPF inline assembler: - For BPF_ST_MEM instruction in verifier_precise.c and verifier_unpriv.c; - For BPF_LD_IND with three arguments in verifier_ref_tracking.c. Migrated tests could be selected for execution using the following filter: ./test_progs -a verifier_* While reviewing the changes I noticed the following irregularities in the original tests: - verifier_sock.c: Tests "bpf_sk_select_reuseport(ctx, sockhash, &key, flags)" and "bpf_sk_select_reuseport(ctx, sockmap, &key, flags)" have identical bodies. - unpriv.c: Despite the name of the file, 12 tests defined in it have program types that are not considered for unprivileged execution by test_verifier (e.g. BPF_PROG_TYPE_TRACEPOINT). Modifications were applied for the following tests: - loop1.c: Some of the tests have .retval field specified, but program type is BPF_PROG_TYPE_TRACEPOINT, which does not support BPF_PROG_TEST_RUN command, for these tests - either program type is changed to "xdp", which supports BPF_PROG_TEST_RUN; - or retval tag is removed, if test run result is not actually predictable (e.g. depends on bpf_get_prandom_u32()). - unpriv.c: This file is split in two parts: - progs/verifier_unpriv.c - progs/verifier_unpriv_perf.c First part requires inclusion of filter.h, second part requires inclusion of vmlinux.h. - value_ptr_arith.c: "sanitation: alu with different scalars 2" and "sanitation: alu with different scalars 3" are modified to avoid retval "-EINVAL * 2" which cannot be encoded as a type tag. Additional details are in the relevant commit messages. [1] RFC https://lore.kernel.org/bpf/20230123145148.2791939-1-eddyz87@gmail.com/ [2] First batch of migrated tests https://lore.kernel.org/bpf/20230325025524.144043-1-eddyz87@gmail.com/ [3] Migration tool https://github.com/eddyz87/verifier-tests-migrator Eduard Zingerman (24): selftests/bpf: Add notion of auxiliary programs for test_loader selftests/bpf: verifier/bounds converted to inline assembly selftests/bpf: verifier/bpf_get_stack converted to inline assembly selftests/bpf: verifier/btf_ctx_access converted to inline assembly selftests/bpf: verifier/ctx converted to inline assembly selftests/bpf: verifier/d_path converted to inline assembly selftests/bpf: verifier/direct_packet_access converted to inline assembly selftests/bpf: verifier/jeq_infer_not_null converted to inline assembly selftests/bpf: verifier/loops1 converted to inline assembly selftests/bpf: verifier/lwt converted to inline assembly selftests/bpf: verifier/map_in_map converted to inline assembly selftests/bpf: verifier/map_ptr_mixing converted to inline assembly selftests/bpf: verifier/precise converted to inline assembly selftests/bpf: verifier/prevent_map_lookup converted to inline assembly selftests/bpf: verifier/ref_tracking converted to inline assembly selftests/bpf: verifier/regalloc converted to inline assembly selftests/bpf: verifier/runtime_jit converted to inline assembly selftests/bpf: verifier/search_pruning converted to inline assembly selftests/bpf: verifier/sock converted to inline assembly selftests/bpf: verifier/spin_lock converted to inline assembly selftests/bpf: verifier/subreg converted to inline assembly selftests/bpf: verifier/unpriv converted to inline assembly selftests/bpf: verifier/value_illegal_alu converted to inline assembly selftests/bpf: verifier/value_ptr_arith converted to inline assembly .../selftests/bpf/prog_tests/verifier.c | 80 +- tools/testing/selftests/bpf/progs/bpf_misc.h | 6 + .../selftests/bpf/progs/verifier_bounds.c | 1076 ++++++++++++ .../bpf/progs/verifier_bpf_get_stack.c | 124 ++ .../bpf/progs/verifier_btf_ctx_access.c | 32 + .../selftests/bpf/progs/verifier_ctx.c | 221 +++ .../selftests/bpf/progs/verifier_d_path.c | 48 + .../bpf/progs/verifier_direct_packet_access.c | 803 +++++++++ .../bpf/progs/verifier_jeq_infer_not_null.c | 213 +++ .../selftests/bpf/progs/verifier_loops1.c | 259 +++ .../selftests/bpf/progs/verifier_lwt.c | 234 +++ .../selftests/bpf/progs/verifier_map_in_map.c | 142 ++ .../bpf/progs/verifier_map_ptr_mixing.c | 265 +++ .../selftests/bpf/progs/verifier_precise.c | 269 +++ .../bpf/progs/verifier_prevent_map_lookup.c | 65 + .../bpf/progs/verifier_ref_tracking.c | 1495 +++++++++++++++++ .../selftests/bpf/progs/verifier_regalloc.c | 364 ++++ .../bpf/progs/verifier_runtime_jit.c | 360 ++++ .../bpf/progs/verifier_search_pruning.c | 339 ++++ .../selftests/bpf/progs/verifier_sock.c | 980 +++++++++++ .../selftests/bpf/progs/verifier_spin_lock.c | 533 ++++++ .../selftests/bpf/progs/verifier_subreg.c | 673 ++++++++ .../selftests/bpf/progs/verifier_unpriv.c | 726 ++++++++ .../bpf/progs/verifier_unpriv_perf.c | 34 + .../bpf/progs/verifier_value_illegal_alu.c | 149 ++ .../bpf/progs/verifier_value_ptr_arith.c | 1423 ++++++++++++++++ tools/testing/selftests/bpf/test_loader.c | 89 +- tools/testing/selftests/bpf/verifier/bounds.c | 884 ---------- .../selftests/bpf/verifier/bpf_get_stack.c | 87 - .../selftests/bpf/verifier/btf_ctx_access.c | 25 - tools/testing/selftests/bpf/verifier/ctx.c | 186 -- tools/testing/selftests/bpf/verifier/d_path.c | 37 - .../bpf/verifier/direct_packet_access.c | 710 -------- .../bpf/verifier/jeq_infer_not_null.c | 174 -- tools/testing/selftests/bpf/verifier/loops1.c | 206 --- tools/testing/selftests/bpf/verifier/lwt.c | 189 --- .../selftests/bpf/verifier/map_in_map.c | 96 -- .../selftests/bpf/verifier/map_ptr_mixing.c | 100 -- .../testing/selftests/bpf/verifier/precise.c | 219 --- .../bpf/verifier/prevent_map_lookup.c | 29 - .../selftests/bpf/verifier/ref_tracking.c | 1082 ------------ .../testing/selftests/bpf/verifier/regalloc.c | 277 --- .../selftests/bpf/verifier/runtime_jit.c | 231 --- .../selftests/bpf/verifier/search_pruning.c | 266 --- tools/testing/selftests/bpf/verifier/sock.c | 706 -------- .../selftests/bpf/verifier/spin_lock.c | 447 ----- tools/testing/selftests/bpf/verifier/subreg.c | 533 ------ tools/testing/selftests/bpf/verifier/unpriv.c | 562 ------- .../bpf/verifier/value_illegal_alu.c | 95 -- .../selftests/bpf/verifier/value_ptr_arith.c | 1140 ------------- 50 files changed, 10974 insertions(+), 8309 deletions(-) create mode 100644 tools/testing/selftests/bpf/progs/verifier_bounds.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_bpf_get_stack.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_btf_ctx_access.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_ctx.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_d_path.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_direct_packet_access.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_jeq_infer_not_null.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_loops1.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_lwt.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_map_in_map.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_map_ptr_mixing.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_precise.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_prevent_map_lookup.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_ref_tracking.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_regalloc.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_runtime_jit.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_search_pruning.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_sock.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_spin_lock.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_subreg.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_unpriv.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_unpriv_perf.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_value_illegal_alu.c create mode 100644 tools/testing/selftests/bpf/progs/verifier_value_ptr_arith.c delete mode 100644 tools/testing/selftests/bpf/verifier/bounds.c delete mode 100644 tools/testing/selftests/bpf/verifier/bpf_get_stack.c delete mode 100644 tools/testing/selftests/bpf/verifier/btf_ctx_access.c delete mode 100644 tools/testing/selftests/bpf/verifier/ctx.c delete mode 100644 tools/testing/selftests/bpf/verifier/d_path.c delete mode 100644 tools/testing/selftests/bpf/verifier/direct_packet_access.c delete mode 100644 tools/testing/selftests/bpf/verifier/jeq_infer_not_null.c delete mode 100644 tools/testing/selftests/bpf/verifier/loops1.c delete mode 100644 tools/testing/selftests/bpf/verifier/lwt.c delete mode 100644 tools/testing/selftests/bpf/verifier/map_in_map.c delete mode 100644 tools/testing/selftests/bpf/verifier/map_ptr_mixing.c delete mode 100644 tools/testing/selftests/bpf/verifier/precise.c delete mode 100644 tools/testing/selftests/bpf/verifier/prevent_map_lookup.c delete mode 100644 tools/testing/selftests/bpf/verifier/ref_tracking.c delete mode 100644 tools/testing/selftests/bpf/verifier/regalloc.c delete mode 100644 tools/testing/selftests/bpf/verifier/runtime_jit.c delete mode 100644 tools/testing/selftests/bpf/verifier/search_pruning.c delete mode 100644 tools/testing/selftests/bpf/verifier/sock.c delete mode 100644 tools/testing/selftests/bpf/verifier/spin_lock.c delete mode 100644 tools/testing/selftests/bpf/verifier/subreg.c delete mode 100644 tools/testing/selftests/bpf/verifier/unpriv.c delete mode 100644 tools/testing/selftests/bpf/verifier/value_illegal_alu.c delete mode 100644 tools/testing/selftests/bpf/verifier/value_ptr_arith.c