[v4,bpf-next,0/9] bpf: Add gen_epilogue to bpf_verifier_ops

Message

Martin KaFai Lau Aug. 27, 2024, 7:48 p.m. UTC

From: Martin KaFai Lau <[e-mail]>

This set allows the subsystem to patch codes before BPF_EXIT.
The verifier ops, .gen_epilogue, is added for this purpose.
One of the use case will be in the bpf qdisc, the bpf qdisc
subsystem can ensure the skb->dev is in the correct value.
The bpf qdisc subsystem can either inline fixing it in the
epilogue or call another kernel function to handle it (e.g. drop)
in the epilogue. Another use case could be in bpf_tcp_ca.c to
enforce snd_cwnd has valid value (e.g. positive value).

v4:
 * Fixed a bug in the memcpy in patch 3
   The size in the memcpy should be
   epilogue_cnt * sizeof(*epilogue_buf)
v3:
 * Moved epilogue_buf[16] to env.
   Patch 1 is added to move the existing insn_buf[16] to env.
 * Fixed a case that the bpf prog has a BPF_JMP that goes back
   to the first instruction of the main prog.
   The jump back to 1st insn case also applies to the prologue.
   Patch 2 is added to handle it.
 * If the bpf main prog has multiple BPF_EXIT, use a BPF_JA
   to goto the earlier patched epilogue.
   Note that there are (BPF_JMP32 | BPF_JA) vs (BPF_JMP | BPF_JA)
   details in the patch 3 commit message.
 * There are subtle changes in patch 3, so I reset the Reviewed-by.
 * Added patch 8 and patch 9 to cover the changes in patch 2 and patch 3.
 * Dropped the kfunc call from pro/epilogue and its selftests.

v2:
 * Remove the RFC tag. Keep the ordering at where .gen_epilogue is
   called in the verifier relative to the check_max_stack_depth().
   This will be consistent with the other extra stack_depth
   usage like optimize_bpf_loop().
 * Use __xlated check provided by the test_loader to
   check the patched instructions after gen_pro/epilogue (Eduard).
 * Added Patch 3 by Eduard (Thanks!).

Eduard Zingerman (1):
  selftests/bpf: attach struct_ops maps before test prog runs

Martin KaFai Lau (8):
  bpf: Move insn_buf[16] to bpf_verifier_env
  bpf: Adjust BPF_JMP that jumps to the 1st insn of the prologue
  bpf: Add gen_epilogue to bpf_verifier_ops
  bpf: Export bpf_base_func_proto
  selftests/bpf: Test gen_prologue and gen_epilogue
  selftests/bpf: Add tailcall epilogue test
  selftests/bpf: A pro/epilogue test when the main prog jumps back to
    the 1st insn
  selftests/bpf: Test epilogue patching when the main prog has multiple
    BPF_EXIT

 include/linux/bpf.h                           |   2 +
 include/linux/bpf_verifier.h                  |   4 +
 kernel/bpf/helpers.c                          |   1 +
 kernel/bpf/verifier.c                         |  73 +++++--
 .../selftests/bpf/bpf_testmod/bpf_testmod.c   | 190 ++++++++++++++++++
 .../selftests/bpf/bpf_testmod/bpf_testmod.h   |  11 +
 .../bpf/bpf_testmod/bpf_testmod_kfunc.h       |   6 +
 .../selftests/bpf/prog_tests/pro_epilogue.c   |  54 +++++
 .../selftests/bpf/progs/epilogue_exit.c       |  78 +++++++
 .../selftests/bpf/progs/epilogue_tailcall.c   |  58 ++++++
 .../bpf/progs/pro_epilogue_goto_start.c       | 149 ++++++++++++++
 .../selftests/bpf/progs/pro_epilogue_kfunc.c  | 156 ++++++++++++++
 .../bpf/progs/pro_epilogue_subprog.c          | 125 ++++++++++++
 tools/testing/selftests/bpf/test_loader.c     |  27 +++
 14 files changed, 922 insertions(+), 12 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/pro_epilogue.c
 create mode 100644 tools/testing/selftests/bpf/progs/epilogue_exit.c
 create mode 100644 tools/testing/selftests/bpf/progs/epilogue_tailcall.c
 create mode 100644 tools/testing/selftests/bpf/progs/pro_epilogue_goto_start.c
 create mode 100644 tools/testing/selftests/bpf/progs/pro_epilogue_kfunc.c
 create mode 100644 tools/testing/selftests/bpf/progs/pro_epilogue_subprog.c

Comments

Eduard Zingerman Aug. 29, 2024, 2:01 a.m. UTC | #1

On Tue, 2024-08-27 at 12:48 -0700, Martin KaFai Lau wrote:

[...]

> @@ -19705,6 +19705,9 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env)
>  		}
>  	}
>  
> +	if (delta)
> +		WARN_ON(adjust_jmp_off(env->prog, 0, delta, delta));
> +

I double checked my old calculations and agree that this adjustment is necessary.
Alexei's suggestion to use delta == skip_cnt sounds reasonable.


>  	if (bpf_prog_is_offloaded(env->prog->aux))
>  		return 0;
>  

[...]