[RFC,bpf-next,0/8] bpf: support BTF kind metadata to separate

Message

Alan Maguire May 31, 2023, 8:19 p.m. UTC

BTF kind metadata provides information to parse BTF kinds.
By separating parsing BTF from using all the information
it provides, we allow BTF to encode new features even if
they cannot be used.  This is helpful in particular for
cases where newer tools for BTF generation run on an
older kernel; BTF kinds may be present that the kernel
cannot yet use, but at least it can parse the BTF
provided.  Meanwhile userspace tools with newer libbpf
may be able to use the newer information.

The intent is to support encoding of kind metadata
optionally so that tools like pahole can add this
information.  So for each kind we record

- a kind name string
- kind-related flags
- length of singular element following struct btf_type
- length of each of the btf_vlen() elements following

In addition we make space in the metadata for
CRC32s computed over the BTF along with a CRC for
the base BTF; this allows split BTF to identify
a mismatch explicitly.

The ideas here were discussed at [1]. One additional
change is here that I believe will help BTF debugging;
support for adding a description string to BTF
metadata.  It can be used by pahole to describe
the version used, options etc.

Future work can take more advantage of these features
such as

- using base CRC to identify base/module BTF mismatch
  explicitly
- using absence of a base BTF CRC as evidence that
  BTF is standalone

...and new BTF kind addition should present less
trouble, provided the kinds are optional.  BTF
parsing _will_ still fail if a non-optional
kind is encountered, as the assumption is that
such kinds are needed.  To take a few examples,
the tag kinds are optional, however enum64
is required, so BTF containing an enum64
(that did not know about enum64) would be
rejected.  This makes sense as if for example
a struct contained an enum64 we would not
be able to fully represent that struct unless
we knew about enum64s.

Patch 1 is the UAPI changes, patches 2-3 provide
libbpf support for handling and using metadata.
Patch 4 adds kernel support for handling and using
metadata.  Patch 5 adds libbpf support to add
metadata.  Patch 6 adds BTF encoding flag
--btf_gen_meta for kernel/module BTF encoding.
Patch 7 adds bpftool support to dump header
and metadata info.  Patch 8 is a selftest
that validates metadata encoding and tests
that an unknown (optional) kind can be skipped
over without BTF failure.

Finally patch 9 is a patch for dwarves
to use the libbpf APIs to encode metadata.
dwarves has to be built with the updated
libbpf for this to work.

Changes tested on x86_64, aarch64.  BTF metadata
validation likely needs some tidying up but
wanted to get it out ASAP so probably still
rough around the edges.

[1] https://lore.kernel.org/bpf/CAEf4BzYjWHRdNNw4B=eOXOs_ONrDwrgX4bn=[e-mail]/

Alan Maguire (8):
  btf: add kind metadata encoding to UAPI
  libbpf: support handling of metadata section in BTF
  libbpf: use metadata to compute an unknown kind size
  btf: support kernel parsing of BTF with metadata, use it to parse BTF
    with unknown kinds
  libbpf: add metadata encoding support
  btf: generate metadata for vmlinux/module BTF
  bpftool: add BTF dump "format meta" to dump header/metadata
  selftests/bpf: test kind encoding/decoding

 include/uapi/linux/btf.h                      |  29 ++
 kernel/bpf/btf.c                              | 102 +++++-
 scripts/pahole-flags.sh                       |   2 +-
 tools/bpf/bpftool/btf.c                       |  46 +++
 tools/include/uapi/linux/btf.h                |  29 ++
 tools/lib/bpf/btf.c                           | 297 +++++++++++++++---
 tools/lib/bpf/btf.h                           |  11 +
 tools/lib/bpf/libbpf.map                      |   1 +
 .../selftests/bpf/prog_tests/btf_kind.c       | 138 ++++++++
 9 files changed, 595 insertions(+), 60 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/prog_tests/btf_kind.c