mbox series

[bpf-next,00/11] libbpf: split BTF support

Message ID 20201029005902.1706310-1-andrii@kernel.org (mailing list archive)
Headers show
Series libbpf: split BTF support | expand

Message

Andrii Nakryiko Oct. 29, 2020, 12:58 a.m. UTC
This patch set adds support for generating and deduplicating split BTF. This
is an enhancement to the BTF, which allows to designate one BTF as the "base
BTF" (e.g., vmlinux BTF), and one or more other BTFs as "split BTF" (e.g.,
kernel module BTF), which are building upon and extending base BTF with extra
types and strings.

Once loaded, split BTF appears as a single unified BTF superset of base BTF,
with continuous and transparent numbering scheme. This allows all the existing
users of BTF to work correctly and stay agnostic to the base/split BTFs
composition.  The only difference is in how to instantiate split BTF: it
requires base BTF to be alread instantiated and passed to btf__new_xxx_split()
or btf__parse_xxx_split() "constructors" explicitly.

This split approach is necessary if we are to have a reasonably-sized kernel
module BTFs. By deduping each kernel module's BTF individually, resulting
module BTFs contain copies of a lot of kernel types that are already present
in vmlinux BTF. Even those single copies result in a big BTF size bloat. On my
kernel configuration with 700 modules built, non-split BTF approach results in
115MBs of BTFs across all modules. With split BTF deduplication approach,
total size is down to 5.2MBs total, which is on part with vmlinux BTF (at
around 4MBs). This seems reasonable and practical. As to why we'd need kernel
module BTFs, that should be pretty obvious to anyone using BPF at this point,
as it allows all the BTF-powered features to be used with kernel modules:
tp_btf, fentry/fexit/fmod_ret, lsm, bpf_iter, etc.

This patch set is a pre-requisite to adding split BTF support to pahole, which
is a prerequisite to integrating split BTF into the Linux kernel build setup
to generate BTF for kernel modules. The latter will come as a follow-up patch
series once this series makes it to the libbpf and pahole makes use of it.

Patch #4 introduces necessary basic support for split BTF into libbpf APIs.
Patch #8 implements minimal changes to BTF dedup algorithm to allow
deduplicating split BTFs. Patch #11 adds extra -B flag to bpftool to allow to
specify the path to base BTF for cases when one wants to dump or inspect split
BTF. All the rest are refactorings, clean ups, bug fixes and selftests.

Andrii Nakryiko (11):
  libbpf: factor out common operations in BTF writing APIs
  selftest/bpf: relax btf_dedup test checks
  libbpf: unify and speed up BTF string deduplication
  libbpf: implement basic split BTF support
  selftests/bpf: add split BTF basic test
  selftests/bpf: add checking of raw type dump in BTF writer APIs
    selftests
  libbpf: fix BTF data layout checks and allow empty BTF
  libbpf: support BTF dedup of split BTFs
  libbpf: accomodate DWARF/compiler bug with duplicated identical arrays
  selftests/bpf: add split BTF dedup selftests
  tools/bpftool: add bpftool support for split BTF

 tools/bpf/bpftool/btf.c                       |   9 +-
 tools/bpf/bpftool/main.c                      |  15 +-
 tools/bpf/bpftool/main.h                      |   1 +
 tools/lib/bpf/btf.c                           | 814 ++++++++++--------
 tools/lib/bpf/btf.h                           |   8 +
 tools/lib/bpf/libbpf.map                      |   9 +
 tools/testing/selftests/bpf/Makefile          |   2 +-
 tools/testing/selftests/bpf/btf_helpers.c     | 259 ++++++
 tools/testing/selftests/bpf/btf_helpers.h     |  19 +
 tools/testing/selftests/bpf/prog_tests/btf.c  |  34 +-
 .../bpf/prog_tests/btf_dedup_split.c          | 326 +++++++
 .../selftests/bpf/prog_tests/btf_split.c      |  99 +++
 .../selftests/bpf/prog_tests/btf_write.c      |  48 +-
 tools/testing/selftests/bpf/test_progs.h      |  11 +
 14 files changed, 1294 insertions(+), 360 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/btf_helpers.c
 create mode 100644 tools/testing/selftests/bpf/btf_helpers.h
 create mode 100644 tools/testing/selftests/bpf/prog_tests/btf_dedup_split.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/btf_split.c

Comments

Song Liu Oct. 30, 2020, 12:33 a.m. UTC | #1
> On Oct 28, 2020, at 5:58 PM, Andrii Nakryiko <andrii@kernel.org> wrote:
> 
> This patch set adds support for generating and deduplicating split BTF. This
> is an enhancement to the BTF, which allows to designate one BTF as the "base
> BTF" (e.g., vmlinux BTF), and one or more other BTFs as "split BTF" (e.g.,
> kernel module BTF), which are building upon and extending base BTF with extra
> types and strings.
> 
> Once loaded, split BTF appears as a single unified BTF superset of base BTF,
> with continuous and transparent numbering scheme. This allows all the existing
> users of BTF to work correctly and stay agnostic to the base/split BTFs
> composition.  The only difference is in how to instantiate split BTF: it
> requires base BTF to be alread instantiated and passed to btf__new_xxx_split()
> or btf__parse_xxx_split() "constructors" explicitly.
> 
> This split approach is necessary if we are to have a reasonably-sized kernel
> module BTFs. By deduping each kernel module's BTF individually, resulting
> module BTFs contain copies of a lot of kernel types that are already present
> in vmlinux BTF. Even those single copies result in a big BTF size bloat. On my
> kernel configuration with 700 modules built, non-split BTF approach results in
> 115MBs of BTFs across all modules. With split BTF deduplication approach,
> total size is down to 5.2MBs total, which is on part with vmlinux BTF (at
> around 4MBs). This seems reasonable and practical. As to why we'd need kernel
> module BTFs, that should be pretty obvious to anyone using BPF at this point,
> as it allows all the BTF-powered features to be used with kernel modules:
> tp_btf, fentry/fexit/fmod_ret, lsm, bpf_iter, etc.

Some high level questions. Do we plan to use split BTF for in-tree modules
(those built together with the kernel) or out-of-tree modules (those built 
separately)? If it is for in-tree modules, is it possible to build split BTF
into vmlinux BTF? 

Thanks,
Song

[...]
Andrii Nakryiko Oct. 30, 2020, 2:33 a.m. UTC | #2
On Thu, Oct 29, 2020 at 5:33 PM Song Liu <songliubraving@fb.com> wrote:
>
>
>
> > On Oct 28, 2020, at 5:58 PM, Andrii Nakryiko <andrii@kernel.org> wrote:
> >
> > This patch set adds support for generating and deduplicating split BTF. This
> > is an enhancement to the BTF, which allows to designate one BTF as the "base
> > BTF" (e.g., vmlinux BTF), and one or more other BTFs as "split BTF" (e.g.,
> > kernel module BTF), which are building upon and extending base BTF with extra
> > types and strings.
> >
> > Once loaded, split BTF appears as a single unified BTF superset of base BTF,
> > with continuous and transparent numbering scheme. This allows all the existing
> > users of BTF to work correctly and stay agnostic to the base/split BTFs
> > composition.  The only difference is in how to instantiate split BTF: it
> > requires base BTF to be alread instantiated and passed to btf__new_xxx_split()
> > or btf__parse_xxx_split() "constructors" explicitly.
> >
> > This split approach is necessary if we are to have a reasonably-sized kernel
> > module BTFs. By deduping each kernel module's BTF individually, resulting
> > module BTFs contain copies of a lot of kernel types that are already present
> > in vmlinux BTF. Even those single copies result in a big BTF size bloat. On my
> > kernel configuration with 700 modules built, non-split BTF approach results in
> > 115MBs of BTFs across all modules. With split BTF deduplication approach,
> > total size is down to 5.2MBs total, which is on part with vmlinux BTF (at
> > around 4MBs). This seems reasonable and practical. As to why we'd need kernel
> > module BTFs, that should be pretty obvious to anyone using BPF at this point,
> > as it allows all the BTF-powered features to be used with kernel modules:
> > tp_btf, fentry/fexit/fmod_ret, lsm, bpf_iter, etc.
>
> Some high level questions. Do we plan to use split BTF for in-tree modules
> (those built together with the kernel) or out-of-tree modules (those built
> separately)? If it is for in-tree modules, is it possible to build split BTF
> into vmlinux BTF?

It will be possible to use for both in-tree and out-of-tree. For
in-tree, this will be integrated into the kernel build process. For
out-of-tree, whoever builds their kernel module will need to invoke
pahole -J with an extra flag pointing to the right vmlinux image (I
haven't looked into the exact details of this integration, maybe there
are already scripts in Linux repo that out-of-tree modules have to
use, in such case we can add this integration there).

Merging all in-tree modules' BTFs into vmlinux's BTF defeats the
purpose of the split BTF and will just increase the size of vmlinux
BTF unnecessarily.

>
> Thanks,
> Song
>
> [...]
Song Liu Oct. 30, 2020, 6:45 a.m. UTC | #3
> On Oct 29, 2020, at 7:33 PM, Andrii Nakryiko <andrii.nakryiko@gmail.com> wrote:
> 
> On Thu, Oct 29, 2020 at 5:33 PM Song Liu <songliubraving@fb.com> wrote:
>> 
>> 
>> 
>>> On Oct 28, 2020, at 5:58 PM, Andrii Nakryiko <andrii@kernel.org> wrote:
>>> 
>>> This patch set adds support for generating and deduplicating split BTF. This
>>> is an enhancement to the BTF, which allows to designate one BTF as the "base
>>> BTF" (e.g., vmlinux BTF), and one or more other BTFs as "split BTF" (e.g.,
>>> kernel module BTF), which are building upon and extending base BTF with extra
>>> types and strings.
>>> 
>>> Once loaded, split BTF appears as a single unified BTF superset of base BTF,
>>> with continuous and transparent numbering scheme. This allows all the existing
>>> users of BTF to work correctly and stay agnostic to the base/split BTFs
>>> composition.  The only difference is in how to instantiate split BTF: it
>>> requires base BTF to be alread instantiated and passed to btf__new_xxx_split()
>>> or btf__parse_xxx_split() "constructors" explicitly.
>>> 
>>> This split approach is necessary if we are to have a reasonably-sized kernel
>>> module BTFs. By deduping each kernel module's BTF individually, resulting
>>> module BTFs contain copies of a lot of kernel types that are already present
>>> in vmlinux BTF. Even those single copies result in a big BTF size bloat. On my
>>> kernel configuration with 700 modules built, non-split BTF approach results in
>>> 115MBs of BTFs across all modules. With split BTF deduplication approach,
>>> total size is down to 5.2MBs total, which is on part with vmlinux BTF (at
>>> around 4MBs). This seems reasonable and practical. As to why we'd need kernel
>>> module BTFs, that should be pretty obvious to anyone using BPF at this point,
>>> as it allows all the BTF-powered features to be used with kernel modules:
>>> tp_btf, fentry/fexit/fmod_ret, lsm, bpf_iter, etc.
>> 
>> Some high level questions. Do we plan to use split BTF for in-tree modules
>> (those built together with the kernel) or out-of-tree modules (those built
>> separately)? If it is for in-tree modules, is it possible to build split BTF
>> into vmlinux BTF?
> 
> It will be possible to use for both in-tree and out-of-tree. For
> in-tree, this will be integrated into the kernel build process. For
> out-of-tree, whoever builds their kernel module will need to invoke
> pahole -J with an extra flag pointing to the right vmlinux image (I
> haven't looked into the exact details of this integration, maybe there
> are already scripts in Linux repo that out-of-tree modules have to
> use, in such case we can add this integration there).

Thanks for the explanation. 

> 
> Merging all in-tree modules' BTFs into vmlinux's BTF defeats the
> purpose of the split BTF and will just increase the size of vmlinux
> BTF unnecessarily.

Is the purpose of split BTF to save memory used by module BTF? In the 
example above, I guess part of those 5.2MB will be loaded at run time, 
so the actual saving is less than 5.2MB. 5.2MB is really small for a 
decent system, e.g. ~0.03% of my laptop's main memory. 

Did I miss anything here? 

Song
Alan Maguire Oct. 30, 2020, 12:04 p.m. UTC | #4
On Thu, 29 Oct 2020, Andrii Nakryiko wrote:

> On Thu, Oct 29, 2020 at 5:33 PM Song Liu <songliubraving@fb.com> wrote:
> >
> >
> >
> > > On Oct 28, 2020, at 5:58 PM, Andrii Nakryiko <andrii@kernel.org> wrote:
> > >
> > > This patch set adds support for generating and deduplicating split BTF. This
> > > is an enhancement to the BTF, which allows to designate one BTF as the "base
> > > BTF" (e.g., vmlinux BTF), and one or more other BTFs as "split BTF" (e.g.,
> > > kernel module BTF), which are building upon and extending base BTF with extra
> > > types and strings.
> > >
> > > Once loaded, split BTF appears as a single unified BTF superset of base BTF,
> > > with continuous and transparent numbering scheme. This allows all the existing
> > > users of BTF to work correctly and stay agnostic to the base/split BTFs
> > > composition.  The only difference is in how to instantiate split BTF: it
> > > requires base BTF to be alread instantiated and passed to btf__new_xxx_split()
> > > or btf__parse_xxx_split() "constructors" explicitly.
> > >
> > > This split approach is necessary if we are to have a reasonably-sized kernel
> > > module BTFs. By deduping each kernel module's BTF individually, resulting
> > > module BTFs contain copies of a lot of kernel types that are already present
> > > in vmlinux BTF. Even those single copies result in a big BTF size bloat. On my
> > > kernel configuration with 700 modules built, non-split BTF approach results in
> > > 115MBs of BTFs across all modules. With split BTF deduplication approach,
> > > total size is down to 5.2MBs total, which is on part with vmlinux BTF (at
> > > around 4MBs). This seems reasonable and practical. As to why we'd need kernel
> > > module BTFs, that should be pretty obvious to anyone using BPF at this point,
> > > as it allows all the BTF-powered features to be used with kernel modules:
> > > tp_btf, fentry/fexit/fmod_ret, lsm, bpf_iter, etc.
> >
> > Some high level questions. Do we plan to use split BTF for in-tree modules
> > (those built together with the kernel) or out-of-tree modules (those built
> > separately)? If it is for in-tree modules, is it possible to build split BTF
> > into vmlinux BTF?
> 
> It will be possible to use for both in-tree and out-of-tree. For
> in-tree, this will be integrated into the kernel build process. For
> out-of-tree, whoever builds their kernel module will need to invoke
> pahole -J with an extra flag pointing to the right vmlinux image (I
> haven't looked into the exact details of this integration, maybe there
> are already scripts in Linux repo that out-of-tree modules have to
> use, in such case we can add this integration there).
> 
> Merging all in-tree modules' BTFs into vmlinux's BTF defeats the
> purpose of the split BTF and will just increase the size of vmlinux
> BTF unnecessarily.
>

Again more of a question about how module BTF will be exposed, but
I'm wondering if there will be a way for a consumer to ask for
type info across kernel and module BTF, i.e. something like
libbpf_find_kernel_btf_id() ? Similarly will __builtin_btf_type_id()
work across both vmlinux and modules? I'm thinking of the case where we 
potentially don't know which module a type is defined in.

I realize in some cases type names may refer to different types in 
different modules (not sure how frequent this is in practice?) but
I'm curious how the split model for modules will interact with existing 
APIs and helpers.

In some cases it's likely that modules may share types with
each other that they do not share with vmlinux; in such cases 
will those types get deduplicated also, or is deduplication just
between kernel/module, and not module/module? 

Sorry I know these questions aren't about this patchset in
particular, but I'm just trying to get a sense of the bigger
picture. Thanks!

Alan
Andrii Nakryiko Oct. 30, 2020, 6:30 p.m. UTC | #5
On Fri, Oct 30, 2020 at 5:06 AM Alan Maguire <alan.maguire@oracle.com> wrote:
>
> On Thu, 29 Oct 2020, Andrii Nakryiko wrote:
>
> > On Thu, Oct 29, 2020 at 5:33 PM Song Liu <songliubraving@fb.com> wrote:
> > >
> > >
> > >
> > > > On Oct 28, 2020, at 5:58 PM, Andrii Nakryiko <andrii@kernel.org> wrote:
> > > >
> > > > This patch set adds support for generating and deduplicating split BTF. This
> > > > is an enhancement to the BTF, which allows to designate one BTF as the "base
> > > > BTF" (e.g., vmlinux BTF), and one or more other BTFs as "split BTF" (e.g.,
> > > > kernel module BTF), which are building upon and extending base BTF with extra
> > > > types and strings.
> > > >
> > > > Once loaded, split BTF appears as a single unified BTF superset of base BTF,
> > > > with continuous and transparent numbering scheme. This allows all the existing
> > > > users of BTF to work correctly and stay agnostic to the base/split BTFs
> > > > composition.  The only difference is in how to instantiate split BTF: it
> > > > requires base BTF to be alread instantiated and passed to btf__new_xxx_split()
> > > > or btf__parse_xxx_split() "constructors" explicitly.
> > > >
> > > > This split approach is necessary if we are to have a reasonably-sized kernel
> > > > module BTFs. By deduping each kernel module's BTF individually, resulting
> > > > module BTFs contain copies of a lot of kernel types that are already present
> > > > in vmlinux BTF. Even those single copies result in a big BTF size bloat. On my
> > > > kernel configuration with 700 modules built, non-split BTF approach results in
> > > > 115MBs of BTFs across all modules. With split BTF deduplication approach,
> > > > total size is down to 5.2MBs total, which is on part with vmlinux BTF (at
> > > > around 4MBs). This seems reasonable and practical. As to why we'd need kernel
> > > > module BTFs, that should be pretty obvious to anyone using BPF at this point,
> > > > as it allows all the BTF-powered features to be used with kernel modules:
> > > > tp_btf, fentry/fexit/fmod_ret, lsm, bpf_iter, etc.
> > >
> > > Some high level questions. Do we plan to use split BTF for in-tree modules
> > > (those built together with the kernel) or out-of-tree modules (those built
> > > separately)? If it is for in-tree modules, is it possible to build split BTF
> > > into vmlinux BTF?
> >
> > It will be possible to use for both in-tree and out-of-tree. For
> > in-tree, this will be integrated into the kernel build process. For
> > out-of-tree, whoever builds their kernel module will need to invoke
> > pahole -J with an extra flag pointing to the right vmlinux image (I
> > haven't looked into the exact details of this integration, maybe there
> > are already scripts in Linux repo that out-of-tree modules have to
> > use, in such case we can add this integration there).
> >
> > Merging all in-tree modules' BTFs into vmlinux's BTF defeats the
> > purpose of the split BTF and will just increase the size of vmlinux
> > BTF unnecessarily.
> >
>
> Again more of a question about how module BTF will be exposed, but
> I'm wondering if there will be a way for a consumer to ask for
> type info across kernel and module BTF, i.e. something like
> libbpf_find_kernel_btf_id() ?

I'm still playing with the options, but I think libbpf will do all the
search across vmlinux and modules. I'm considering allowing users to
specify module name as an optional hint. Just in case if there are
conflicting types/functions in two different modules with the same
name.

> Similarly will __builtin_btf_type_id()
> work across both vmlinux and modules? I'm thinking of the case where we
> potentially don't know which module a type is defined in.

I think we'll need another built-in/relocation to specify
module/vmlinux ID. Type ID itself is not unique enough to identify the
module.

Alternatively, we can extend its return type to u64 and have BTF
object ID in upper 4 bytes, and BTF type ID in lower 4 bytes. Need to
think about this and discuss it with Yonghong.

>
> I realize in some cases type names may refer to different types in
> different modules (not sure how frequent this is in practice?) but
> I'm curious how the split model for modules will interact with existing
> APIs and helpers.
>
> In some cases it's likely that modules may share types with
> each other that they do not share with vmlinux; in such cases
> will those types get deduplicated also, or is deduplication just
> between kernel/module, and not module/module?

Yes, they will be duplicated in two modules. It's a start schema,
where vmlinux BTF is the base for all kernel modules. It's technically
possible to have a longer chain of BTFs, but we'd need to deal with
dependencies between modules, making sure that dependent BTF is loaded
and available first, etc. That can be added later without breaking
anything, if there is a need.

>
> Sorry I know these questions aren't about this patchset in
> particular, but I'm just trying to get a sense of the bigger
> picture. Thanks!

These are fair questions, I just didn't want to go into too many
details in this particular patch set, because it's pretty agnostic to
all of those concerns. The next patch set will be dealing with all the
details of kernel/user space interface.

>
> Alan