mbox series

[bpf-next,0/3] bpf: add signature

Message ID 20211203191844.69709-1-mcroce@linux.microsoft.com (mailing list archive)
Headers show
Series bpf: add signature | expand

Message

Matteo Croce Dec. 3, 2021, 7:18 p.m. UTC
From: Matteo Croce <mcroce@microsoft.com>

This series add signature verification for BPF files.
The first patch implements the signature validation in the kernel,
the second patch optionally makes the signature mandatory,
the third adds signature generation to bpftool.

This only works with CO-RE programs.

Matteo Croce (3):
  bpf: add signature to eBPF instructions
  bpf: add option to require BPF signature
  bpftool: add signature in skeleton

 crypto/asymmetric_keys/asymmetric_type.c |   1 +
 crypto/asymmetric_keys/pkcs7_verify.c    |   7 +-
 include/linux/verification.h             |   1 +
 include/uapi/linux/bpf.h                 |   2 +
 kernel/bpf/Kconfig                       |  14 ++
 kernel/bpf/syscall.c                     |  51 +++++-
 tools/bpf/bpftool/Makefile               |  14 +-
 tools/bpf/bpftool/gen.c                  |  33 ++++
 tools/bpf/bpftool/main.c                 |  28 +++
 tools/bpf/bpftool/main.h                 |   7 +
 tools/bpf/bpftool/sign.c                 | 218 +++++++++++++++++++++++
 tools/include/uapi/linux/bpf.h           |   2 +
 tools/lib/bpf/skel_internal.h            |   4 +
 13 files changed, 372 insertions(+), 10 deletions(-)
 create mode 100644 tools/bpf/bpftool/sign.c

Comments

Alexei Starovoitov Dec. 3, 2021, 7:22 p.m. UTC | #1
On Fri, Dec 3, 2021 at 11:18 AM Matteo Croce <mcroce@linux.microsoft.com> wrote:
>
> From: Matteo Croce <mcroce@microsoft.com>
>
> This series add signature verification for BPF files.
> The first patch implements the signature validation in the kernel,
> the second patch optionally makes the signature mandatory,
> the third adds signature generation to bpftool.

Matteo,

I think I already mentioned that it's no-go as-is.
We've agreed to go with John's suggestion.
Matteo Croce Dec. 3, 2021, 7:35 p.m. UTC | #2
On Fri, Dec 3, 2021 at 8:22 PM Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
>
> On Fri, Dec 3, 2021 at 11:18 AM Matteo Croce <mcroce@linux.microsoft.com> wrote:
> >
> > From: Matteo Croce <mcroce@microsoft.com>
> >
> > This series add signature verification for BPF files.
> > The first patch implements the signature validation in the kernel,
> > the second patch optionally makes the signature mandatory,
> > the third adds signature generation to bpftool.
>
> Matteo,
>
> I think I already mentioned that it's no-go as-is.
> We've agreed to go with John's suggestion.

Hi,

my previous attempt was loading a whole ELF file and parsing it in kernel.
In this series I just validate the instructions against a signature,
as with kernel CO-RE libbpf doesn't need to mangle it.

Which suggestion? I think I missed this one..
Alexei Starovoitov Dec. 3, 2021, 7:37 p.m. UTC | #3
On Fri, Dec 3, 2021 at 11:36 AM Matteo Croce <mcroce@linux.microsoft.com> wrote:
>
> On Fri, Dec 3, 2021 at 8:22 PM Alexei Starovoitov
> <alexei.starovoitov@gmail.com> wrote:
> >
> > On Fri, Dec 3, 2021 at 11:18 AM Matteo Croce <mcroce@linux.microsoft.com> wrote:
> > >
> > > From: Matteo Croce <mcroce@microsoft.com>
> > >
> > > This series add signature verification for BPF files.
> > > The first patch implements the signature validation in the kernel,
> > > the second patch optionally makes the signature mandatory,
> > > the third adds signature generation to bpftool.
> >
> > Matteo,
> >
> > I think I already mentioned that it's no-go as-is.
> > We've agreed to go with John's suggestion.
>
> Hi,
>
> my previous attempt was loading a whole ELF file and parsing it in kernel.
> In this series I just validate the instructions against a signature,
> as with kernel CO-RE libbpf doesn't need to mangle it.
>
> Which suggestion? I think I missed this one..

This talk and discussion:
https://linuxplumbersconf.org/event/11/contributions/947/
Luca Boccassi Dec. 3, 2021, 10:06 p.m. UTC | #4
On Fri, 2021-12-03 at 11:37 -0800, Alexei Starovoitov wrote:
> On Fri, Dec 3, 2021 at 11:36 AM Matteo Croce
> <mcroce@linux.microsoft.com> wrote:
> > 
> > On Fri, Dec 3, 2021 at 8:22 PM Alexei Starovoitov
> > <alexei.starovoitov@gmail.com> wrote:
> > > 
> > > On Fri, Dec 3, 2021 at 11:18 AM Matteo Croce
> > > <mcroce@linux.microsoft.com> wrote:
> > > > 
> > > > From: Matteo Croce <mcroce@microsoft.com>
> > > > 
> > > > This series add signature verification for BPF files.
> > > > The first patch implements the signature validation in the
> > > > kernel,
> > > > the second patch optionally makes the signature mandatory,
> > > > the third adds signature generation to bpftool.
> > > 
> > > Matteo,
> > > 
> > > I think I already mentioned that it's no-go as-is.
> > > We've agreed to go with John's suggestion.
> > 
> > Hi,
> > 
> > my previous attempt was loading a whole ELF file and parsing it in
> > kernel.
> > In this series I just validate the instructions against a
> > signature,
> > as with kernel CO-RE libbpf doesn't need to mangle it.
> > 
> > Which suggestion? I think I missed this one..
> 
> This talk and discussion:
> https://linuxplumbersconf.org/event/11/contributions/947/

Thanks for the link - but for those of us who don't have ~5 hours to
watch a video recording, would you mind sharing a one line summary,
please? Is there an alternative patch series implementing BPF signing
that you can link us so that we can look at it? Just a link or
googlable reference would be more than enough.

Thank you!
Alexei Starovoitov Dec. 3, 2021, 10:20 p.m. UTC | #5
On Fri, Dec 3, 2021 at 2:06 PM Luca Boccassi <bluca@debian.org> wrote:
>
> On Fri, 2021-12-03 at 11:37 -0800, Alexei Starovoitov wrote:
> > On Fri, Dec 3, 2021 at 11:36 AM Matteo Croce
> > <mcroce@linux.microsoft.com> wrote:
> > >
> > > On Fri, Dec 3, 2021 at 8:22 PM Alexei Starovoitov
> > > <alexei.starovoitov@gmail.com> wrote:
> > > >
> > > > On Fri, Dec 3, 2021 at 11:18 AM Matteo Croce
> > > > <mcroce@linux.microsoft.com> wrote:
> > > > >
> > > > > From: Matteo Croce <mcroce@microsoft.com>
> > > > >
> > > > > This series add signature verification for BPF files.
> > > > > The first patch implements the signature validation in the
> > > > > kernel,
> > > > > the second patch optionally makes the signature mandatory,
> > > > > the third adds signature generation to bpftool.
> > > >
> > > > Matteo,
> > > >
> > > > I think I already mentioned that it's no-go as-is.
> > > > We've agreed to go with John's suggestion.
> > >
> > > Hi,
> > >
> > > my previous attempt was loading a whole ELF file and parsing it in
> > > kernel.
> > > In this series I just validate the instructions against a
> > > signature,
> > > as with kernel CO-RE libbpf doesn't need to mangle it.
> > >
> > > Which suggestion? I think I missed this one..
> >
> > This talk and discussion:
> > https://linuxplumbersconf.org/event/11/contributions/947/
>
> Thanks for the link - but for those of us who don't have ~5 hours to
> watch a video recording, would you mind sharing a one line summary,
> please? Is there an alternative patch series implementing BPF signing
> that you can link us so that we can look at it? Just a link or
> googlable reference would be more than enough.

It's not 5 hours and you have to read slides and watch
John's presentation to follow the conversation.
Matteo Croce Dec. 4, 2021, 12:42 a.m. UTC | #6
On Fri, Dec 3, 2021 at 11:20 PM Alexei Starovoitov
<alexei.starovoitov@gmail.com> wrote:
>
> On Fri, Dec 3, 2021 at 2:06 PM Luca Boccassi <bluca@debian.org> wrote:
> >
> > On Fri, 2021-12-03 at 11:37 -0800, Alexei Starovoitov wrote:
> > > On Fri, Dec 3, 2021 at 11:36 AM Matteo Croce
> > > <mcroce@linux.microsoft.com> wrote:
> > > >
> > > > On Fri, Dec 3, 2021 at 8:22 PM Alexei Starovoitov
> > > > <alexei.starovoitov@gmail.com> wrote:
> > > > >
> > > > > On Fri, Dec 3, 2021 at 11:18 AM Matteo Croce
> > > > > <mcroce@linux.microsoft.com> wrote:
> > > > > >
> > > > > > From: Matteo Croce <mcroce@microsoft.com>
> > > > > >
> > > > > > This series add signature verification for BPF files.
> > > > > > The first patch implements the signature validation in the
> > > > > > kernel,
> > > > > > the second patch optionally makes the signature mandatory,
> > > > > > the third adds signature generation to bpftool.
> > > > >
> > > > > Matteo,
> > > > >
> > > > > I think I already mentioned that it's no-go as-is.
> > > > > We've agreed to go with John's suggestion.
> > > >
> > > > Hi,
> > > >
> > > > my previous attempt was loading a whole ELF file and parsing it in
> > > > kernel.
> > > > In this series I just validate the instructions against a
> > > > signature,
> > > > as with kernel CO-RE libbpf doesn't need to mangle it.
> > > >
> > > > Which suggestion? I think I missed this one..
> > >
> > > This talk and discussion:
> > > https://linuxplumbersconf.org/event/11/contributions/947/
> >
> > Thanks for the link - but for those of us who don't have ~5 hours to
> > watch a video recording, would you mind sharing a one line summary,
> > please? Is there an alternative patch series implementing BPF signing
> > that you can link us so that we can look at it? Just a link or
> > googlable reference would be more than enough.
>
> It's not 5 hours and you have to read slides and watch
> John's presentation to follow the conversation.

So, If I have understood correctly, the proposal is to validate the
tools which loads the BPF (e.g. perf, ip) with fs-verity, and only
allow BPF loading from those validated binaries?
That's nice, but I think that this could be complementary to the
instructions signature.
Imagine a validated binary being exploited somehow at runtime, that
could be vector of malicious BPF program load.
Can't we have both available, and use one or other, or even both
together depending on the use case?

Regards,
Alexei Starovoitov Dec. 4, 2021, 2:02 a.m. UTC | #7
On Fri, Dec 3, 2021 at 4:42 PM Matteo Croce <mcroce@linux.microsoft.com> wrote:
>
> On Fri, Dec 3, 2021 at 11:20 PM Alexei Starovoitov
> <alexei.starovoitov@gmail.com> wrote:
> >
> > On Fri, Dec 3, 2021 at 2:06 PM Luca Boccassi <bluca@debian.org> wrote:
> > >
> > > On Fri, 2021-12-03 at 11:37 -0800, Alexei Starovoitov wrote:
> > > > On Fri, Dec 3, 2021 at 11:36 AM Matteo Croce
> > > > <mcroce@linux.microsoft.com> wrote:
> > > > >
> > > > > On Fri, Dec 3, 2021 at 8:22 PM Alexei Starovoitov
> > > > > <alexei.starovoitov@gmail.com> wrote:
> > > > > >
> > > > > > On Fri, Dec 3, 2021 at 11:18 AM Matteo Croce
> > > > > > <mcroce@linux.microsoft.com> wrote:
> > > > > > >
> > > > > > > From: Matteo Croce <mcroce@microsoft.com>
> > > > > > >
> > > > > > > This series add signature verification for BPF files.
> > > > > > > The first patch implements the signature validation in the
> > > > > > > kernel,
> > > > > > > the second patch optionally makes the signature mandatory,
> > > > > > > the third adds signature generation to bpftool.
> > > > > >
> > > > > > Matteo,
> > > > > >
> > > > > > I think I already mentioned that it's no-go as-is.
> > > > > > We've agreed to go with John's suggestion.
> > > > >
> > > > > Hi,
> > > > >
> > > > > my previous attempt was loading a whole ELF file and parsing it in
> > > > > kernel.
> > > > > In this series I just validate the instructions against a
> > > > > signature,
> > > > > as with kernel CO-RE libbpf doesn't need to mangle it.
> > > > >
> > > > > Which suggestion? I think I missed this one..
> > > >
> > > > This talk and discussion:
> > > > https://linuxplumbersconf.org/event/11/contributions/947/
> > >
> > > Thanks for the link - but for those of us who don't have ~5 hours to
> > > watch a video recording, would you mind sharing a one line summary,
> > > please? Is there an alternative patch series implementing BPF signing
> > > that you can link us so that we can look at it? Just a link or
> > > googlable reference would be more than enough.
> >
> > It's not 5 hours and you have to read slides and watch
> > John's presentation to follow the conversation.
>
> So, If I have understood correctly, the proposal is to validate the
> tools which loads the BPF (e.g. perf, ip) with fs-verity, and only
> allow BPF loading from those validated binaries?
> That's nice, but I think that this could be complementary to the
> instructions signature.
> Imagine a validated binary being exploited somehow at runtime, that
> could be vector of malicious BPF program load.
> Can't we have both available, and use one or other, or even both
> together depending on the use case?

I'll let John comment.
John Fastabend Dec. 4, 2021, 3:39 a.m. UTC | #8
Alexei Starovoitov wrote:
> On Fri, Dec 3, 2021 at 4:42 PM Matteo Croce <mcroce@linux.microsoft.com> wrote:
> >
> > On Fri, Dec 3, 2021 at 11:20 PM Alexei Starovoitov
> > <alexei.starovoitov@gmail.com> wrote:
> > >
> > > On Fri, Dec 3, 2021 at 2:06 PM Luca Boccassi <bluca@debian.org> wrote:
> > > >
> > > > On Fri, 2021-12-03 at 11:37 -0800, Alexei Starovoitov wrote:
> > > > > On Fri, Dec 3, 2021 at 11:36 AM Matteo Croce
> > > > > <mcroce@linux.microsoft.com> wrote:
> > > > > >
> > > > > > On Fri, Dec 3, 2021 at 8:22 PM Alexei Starovoitov
> > > > > > <alexei.starovoitov@gmail.com> wrote:
> > > > > > >
> > > > > > > On Fri, Dec 3, 2021 at 11:18 AM Matteo Croce
> > > > > > > <mcroce@linux.microsoft.com> wrote:
> > > > > > > >
> > > > > > > > From: Matteo Croce <mcroce@microsoft.com>
> > > > > > > >
> > > > > > > > This series add signature verification for BPF files.
> > > > > > > > The first patch implements the signature validation in the
> > > > > > > > kernel,
> > > > > > > > the second patch optionally makes the signature mandatory,
> > > > > > > > the third adds signature generation to bpftool.
> > > > > > >
> > > > > > > Matteo,
> > > > > > >
> > > > > > > I think I already mentioned that it's no-go as-is.
> > > > > > > We've agreed to go with John's suggestion.
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > my previous attempt was loading a whole ELF file and parsing it in
> > > > > > kernel.
> > > > > > In this series I just validate the instructions against a
> > > > > > signature,
> > > > > > as with kernel CO-RE libbpf doesn't need to mangle it.
> > > > > >
> > > > > > Which suggestion? I think I missed this one..
> > > > >
> > > > > This talk and discussion:
> > > > > https://linuxplumbersconf.org/event/11/contributions/947/
> > > >
> > > > Thanks for the link - but for those of us who don't have ~5 hours to
> > > > watch a video recording, would you mind sharing a one line summary,
> > > > please? Is there an alternative patch series implementing BPF signing
> > > > that you can link us so that we can look at it? Just a link or
> > > > googlable reference would be more than enough.
> > >
> > > It's not 5 hours and you have to read slides and watch
> > > John's presentation to follow the conversation.
> >
> > So, If I have understood correctly, the proposal is to validate the
> > tools which loads the BPF (e.g. perf, ip) with fs-verity, and only
> > allow BPF loading from those validated binaries?
> > That's nice, but I think that this could be complementary to the
> > instructions signature.
> > Imagine a validated binary being exploited somehow at runtime, that
> > could be vector of malicious BPF program load.
> > Can't we have both available, and use one or other, or even both
> > together depending on the use case?
> 
> I'll let John comment.

I'll give the outline of the argument here.

I do not believe signing BPF instructions for real programs provides
much additional security. Given most real programs if the application
or loader is exploited at runtime we have all sorts of trouble. First
simply verifying the program doesn't prevent malicious use of the
program. If its in the network program this means DDOS, data exfiltration,
mitm attacks, many other possibilities. If its enforcement program
most enforcement actions are programmed from this application so system
security is lost already.  If its observability application simply
drops/manipulates observations that it wants. I don't know of any
useful programs that exist in isolation without user space input
and output as a critical component. If its not a privileged user,
well it better not be doing anything critical anyways or disabled
outright for the security focused.

Many critical programs can't be signed by the nature of the program.
Optimizing network app generates optimized code at runtime. Observability
tools JIT the code on the fly, similarly enforcement tools will do the
same. I think the power of being able to optimize JIT the code in
application and give to the kernel is something we will see more and
more of. Saying I'm only going to accept signed programs, for a
distribution or something other than niche use case, is non starter
IMO because it breaks so many real use cases. We should encourage
these optimizing use cases as I see it as critical to performance
and keeping overhead low.

From a purely security standpoint I believe you are better off
defining characteristics an application is allowed to have. For
example allowed to probe kernel memory, make these helpers calls,
have this many instructions, use this much memory, this much cpu,
etc. This lets you sandbox a BPF application (both user space and
kernel side) much nicer than any signing will allow.

If we want to 'sign' programs we should do that from a BPF program
directly where other metadata can be included in the policy. For
example having a hash of the program loaded along with the calls
made and process allows for rich policy decisions. I have other
use cases that need a hash/signature for data blobs, so its on
my todo list but not at the top yet.  But, being able to verify
arbitrary blob of data from BPF feels like a useful operation to me
in general. The fact in your case its a set of eBPF insns and in
my case its some key in a network header shouldn't matter.

The series as is, scanned commit descriptions, is going to break
lots of in-use-today programs if it was ever enabled. And
is not as flexible (can't support bpftrace, etc.) or powerful
(can't consider fine grained policy decisions) as above.

Add a function we can hook after verify (or before up for
debate) and helpers to verify signatures and/or generate
hashes and we get a better more general solution. And it can
also solve your use case even if I believe its not useful and
may break many BPF users running bpftrace, libbpf, etc.

Thanks,
John
Luca Boccassi Dec. 4, 2021, 12:37 p.m. UTC | #9
On Fri, 2021-12-03 at 19:39 -0800, John Fastabend wrote:
> Alexei Starovoitov wrote:
> > On Fri, Dec 3, 2021 at 4:42 PM Matteo Croce
> > <mcroce@linux.microsoft.com> wrote:
> > > 
> > > On Fri, Dec 3, 2021 at 11:20 PM Alexei Starovoitov
> > > <alexei.starovoitov@gmail.com> wrote:
> > > > 
> > > > On Fri, Dec 3, 2021 at 2:06 PM Luca Boccassi <bluca@debian.org>
> > > > wrote:
> > > > > 
> > > > > On Fri, 2021-12-03 at 11:37 -0800, Alexei Starovoitov wrote:
> > > > > > On Fri, Dec 3, 2021 at 11:36 AM Matteo Croce
> > > > > > <mcroce@linux.microsoft.com> wrote:
> > > > > > > 
> > > > > > > On Fri, Dec 3, 2021 at 8:22 PM Alexei Starovoitov
> > > > > > > <alexei.starovoitov@gmail.com> wrote:
> > > > > > > > 
> > > > > > > > On Fri, Dec 3, 2021 at 11:18 AM Matteo Croce
> > > > > > > > <mcroce@linux.microsoft.com> wrote:
> > > > > > > > > 
> > > > > > > > > From: Matteo Croce <mcroce@microsoft.com>
> > > > > > > > > 
> > > > > > > > > This series add signature verification for BPF files.
> > > > > > > > > The first patch implements the signature validation
> > > > > > > > > in the
> > > > > > > > > kernel,
> > > > > > > > > the second patch optionally makes the signature
> > > > > > > > > mandatory,
> > > > > > > > > the third adds signature generation to bpftool.
> > > > > > > > 
> > > > > > > > Matteo,
> > > > > > > > 
> > > > > > > > I think I already mentioned that it's no-go as-is.
> > > > > > > > We've agreed to go with John's suggestion.
> > > > > > > 
> > > > > > > Hi,
> > > > > > > 
> > > > > > > my previous attempt was loading a whole ELF file and
> > > > > > > parsing it in
> > > > > > > kernel.
> > > > > > > In this series I just validate the instructions against a
> > > > > > > signature,
> > > > > > > as with kernel CO-RE libbpf doesn't need to mangle it.
> > > > > > > 
> > > > > > > Which suggestion? I think I missed this one..
> > > > > > 
> > > > > > This talk and discussion:
> > > > > > https://linuxplumbersconf.org/event/11/contributions/947/
> > > > > 
> > > > > Thanks for the link - but for those of us who don't have ~5
> > > > > hours to
> > > > > watch a video recording, would you mind sharing a one line
> > > > > summary,
> > > > > please? Is there an alternative patch series implementing BPF
> > > > > signing
> > > > > that you can link us so that we can look at it? Just a link
> > > > > or
> > > > > googlable reference would be more than enough.
> > > > 
> > > > It's not 5 hours and you have to read slides and watch
> > > > John's presentation to follow the conversation.
> > > 
> > > So, If I have understood correctly, the proposal is to validate
> > > the
> > > tools which loads the BPF (e.g. perf, ip) with fs-verity, and
> > > only
> > > allow BPF loading from those validated binaries?
> > > That's nice, but I think that this could be complementary to the
> > > instructions signature.
> > > Imagine a validated binary being exploited somehow at runtime,
> > > that
> > > could be vector of malicious BPF program load.
> > > Can't we have both available, and use one or other, or even both
> > > together depending on the use case?
> > 
> > I'll let John comment.
> 
> I'll give the outline of the argument here.
> 
> I do not believe signing BPF instructions for real programs provides
> much additional security. Given most real programs if the application
> or loader is exploited at runtime we have all sorts of trouble. First
> simply verifying the program doesn't prevent malicious use of the
> program. If its in the network program this means DDOS, data
> exfiltration,
> mitm attacks, many other possibilities. If its enforcement program
> most enforcement actions are programmed from this application so
> system
> security is lost already.  If its observability application simply
> drops/manipulates observations that it wants. I don't know of any
> useful programs that exist in isolation without user space input
> and output as a critical component. If its not a privileged user,
> well it better not be doing anything critical anyways or disabled
> outright for the security focused.
> 
> Many critical programs can't be signed by the nature of the program.
> Optimizing network app generates optimized code at runtime.
> Observability
> tools JIT the code on the fly, similarly enforcement tools will do
> the
> same. I think the power of being able to optimize JIT the code in
> application and give to the kernel is something we will see more and
> more of. Saying I'm only going to accept signed programs, for a
> distribution or something other than niche use case, is non starter
> IMO because it breaks so many real use cases. We should encourage
> these optimizing use cases as I see it as critical to performance
> and keeping overhead low.
> 
> From a purely security standpoint I believe you are better off
> defining characteristics an application is allowed to have. For
> example allowed to probe kernel memory, make these helpers calls,
> have this many instructions, use this much memory, this much cpu,
> etc. This lets you sandbox a BPF application (both user space and
> kernel side) much nicer than any signing will allow.
> 
> If we want to 'sign' programs we should do that from a BPF program
> directly where other metadata can be included in the policy. For
> example having a hash of the program loaded along with the calls
> made and process allows for rich policy decisions. I have other
> use cases that need a hash/signature for data blobs, so its on
> my todo list but not at the top yet.  But, being able to verify
> arbitrary blob of data from BPF feels like a useful operation to me
> in general. The fact in your case its a set of eBPF insns and in
> my case its some key in a network header shouldn't matter.
> 
> The series as is, scanned commit descriptions, is going to break
> lots of in-use-today programs if it was ever enabled. And
> is not as flexible (can't support bpftrace, etc.) or powerful
> (can't consider fine grained policy decisions) as above.
> 
> Add a function we can hook after verify (or before up for
> debate) and helpers to verify signatures and/or generate
> hashes and we get a better more general solution. And it can
> also solve your use case even if I believe its not useful and
> may break many BPF users running bpftrace, libbpf, etc.
> 
> Thanks,
> John

Hello John,

Thank you for the summary, this is much clearer.

First of all, I think there's some misunderstanding: this series does
not enable optional signatures by default, and does not enable
mandatory signatures by default either. So I don't see how it would
break existing use cases as you are saying? Unless I'm missing
something?

There's a kconfig to enable optional signatures - if they are there,
they are verified, if they are not present then nothing different
happens. Unless I am missing something, this should be backward
compatible. This kconfig would likely be enabled in most use cases,
just like optionally signed kernel modules are.

Then there's a kconfig on top of that which makes signatures mandatory.
I would not imagine this to be enabled in may cases, just in custom
builds that have more stringent requirements. It certainly would not be
enabled in generalist distros. Perhaps a more flexible way would be to
introduce a sysctl, like fsverity has with
'fs.verity.require_signatures'? That would be just fine for our use
case. Matteo can we do that instead in the next revision?

Secondly, I understand that for your use case signing programs would
not be the best approach. That's fine, and I'm glad you are working on
an alternative that better fits your model, it will be very interesting
to see how it looks like once implemented. But that model doesn't fit
all cases. In our case at Microsoft, we absolutely want to be able to
pre-define at build time a list of BPF programs that are allowed to be
loaded, and reject anything else. Userspace processes in our case are
mostly old and crufty c++ programs that can most likely be pwned by
looking at them sideways, so they get locked down hard with multiple
redundant layers and so on and so forth. But right now for BPF you only
have a "can load BPF" or "cannot load BPF" knob, and that's it. This is
not good enough: we need to be able to define a list of allowed
payloads, and be able to enforce it, so when (not if) said processes do
get tricked into loading something else, it will fail, despite having
the capability of calling bpf(). Trying to define heuristics is also
not good enough for us - creative malicious actors have a tendency to
come up with ways to chain things that individually are allowed and
benign, but combined in a way that you just couldn't foresee. It would
certainly cover a lot of cases, but not all. A strictly pre-defined
list of what is allowed to run and what is not is what we need for our
case, so that we always know exactly what is going to run and what is
not, and can deal with the consequences accordingly, without nasty
surprises waiting around the corner. Now in my naive view the best way
to achieve this is via signatures and certs, as it's a well-understood
system, with processes already in place to revoke/rotate/etc, and it's
already used for kmods. An alternative would be hard-coding hashes I
guess, but that would be terribly inflexible.

Now in terms of _how_ the signatures are done and validated, I'm sure
there are multiple ways, and if some are better than what this series
implements, then that's not an issue, it can be reworked. But the core
requirement for us is: offline pre-defined list of what is allowed to
run and what is not, with ability for hard enforcement that cannot be
bypassed. Yes, you lose some features like JIT and so on: we don't
care, we don't need those for our use cases. If others have different
needs that's fine, this is all intended to be optional, not mandatory.
There are obviously trade-offs, as always when security is involved,
and each user can decide what's best for them.

Hope this makes sense. Thanks!
John Fastabend Dec. 6, 2021, 8:40 p.m. UTC | #10
Luca Boccassi wrote:

cutting to just the relevant pieces here.

[...]

> 
> > I'll give the outline of the argument here.
> > 
> > I do not believe signing BPF instructions for real programs provides
> > much additional security. Given most real programs if the application
> > or loader is exploited at runtime we have all sorts of trouble. First
> > simply verifying the program doesn't prevent malicious use of the
> > program. If its in the network program this means DDOS, data
> > exfiltration,
> > mitm attacks, many other possibilities. If its enforcement program
> > most enforcement actions are programmed from this application so
> > system
> > security is lost already.  If its observability application simply
> > drops/manipulates observations that it wants. I don't know of any
> > useful programs that exist in isolation without user space input
> > and output as a critical component. If its not a privileged user,
> > well it better not be doing anything critical anyways or disabled
> > outright for the security focused.
> > 
> > Many critical programs can't be signed by the nature of the program.
> > Optimizing network app generates optimized code at runtime.
> > Observability
> > tools JIT the code on the fly, similarly enforcement tools will do
> > the
> > same. I think the power of being able to optimize JIT the code in
> > application and give to the kernel is something we will see more and
> > more of. Saying I'm only going to accept signed programs, for a
> > distribution or something other than niche use case, is non starter
> > IMO because it breaks so many real use cases. We should encourage
> > these optimizing use cases as I see it as critical to performance
> > and keeping overhead low.
> > 
> > From a purely security standpoint I believe you are better off
> > defining characteristics an application is allowed to have. For
> > example allowed to probe kernel memory, make these helpers calls,
> > have this many instructions, use this much memory, this much cpu,
> > etc. This lets you sandbox a BPF application (both user space and
> > kernel side) much nicer than any signing will allow.
> > 
> > If we want to 'sign' programs we should do that from a BPF program
> > directly where other metadata can be included in the policy. For
> > example having a hash of the program loaded along with the calls
> > made and process allows for rich policy decisions. I have other
> > use cases that need a hash/signature for data blobs, so its on
> > my todo list but not at the top yet.  But, being able to verify
> > arbitrary blob of data from BPF feels like a useful operation to me
> > in general. The fact in your case its a set of eBPF insns and in
> > my case its some key in a network header shouldn't matter.
> > 
> > The series as is, scanned commit descriptions, is going to break
> > lots of in-use-today programs if it was ever enabled. And
> > is not as flexible (can't support bpftrace, etc.) or powerful
> > (can't consider fine grained policy decisions) as above.
> > 
> > Add a function we can hook after verify (or before up for
> > debate) and helpers to verify signatures and/or generate
> > hashes and we get a better more general solution. And it can
> > also solve your use case even if I believe its not useful and
> > may break many BPF users running bpftrace, libbpf, etc.
> > 
> > Thanks,
> > John
> 
> Hello John,
> 
> Thank you for the summary, this is much clearer.
> 
> First of all, I think there's some misunderstanding: this series does
> not enable optional signatures by default, and does not enable
> mandatory signatures by default either. So I don't see how it would
> break existing use cases as you are saying? Unless I'm missing
> something?
> 
> There's a kconfig to enable optional signatures - if they are there,
> they are verified, if they are not present then nothing different
> happens. Unless I am missing something, this should be backward
> compatible. This kconfig would likely be enabled in most use cases,
> just like optionally signed kernel modules are.

Agree, without enforcement things should continue to work.

> 
> Then there's a kconfig on top of that which makes signatures mandatory.
> I would not imagine this to be enabled in may cases, just in custom
> builds that have more stringent requirements. It certainly would not be
> enabled in generalist distros. Perhaps a more flexible way would be to
> introduce a sysctl, like fsverity has with
> 'fs.verity.require_signatures'? That would be just fine for our use
> case. Matteo can we do that instead in the next revision?

We want to manage this from BPF side directly. It looks
like policy decision and we have use cases that are not as
simple as yes/no with global switch. For example, in k8s world
this might be enabled via labels which are user specific per container
policy. e.g. lockdown some containers more strictly than others.

> 
> Secondly, I understand that for your use case signing programs would
> not be the best approach. That's fine, and I'm glad you are working on
> an alternative that better fits your model, it will be very interesting
> to see how it looks like once implemented. But that model doesn't fit
> all cases. In our case at Microsoft, we absolutely want to be able to
> pre-define at build time a list of BPF programs that are allowed to be
> loaded, and reject anything else. Userspace processes in our case are

By building this into BPF you can get the 'reject anything else' policy
and I get the metadata + reject/accept from the same hook. Its
just your program can be very simple.

> mostly old and crufty c++ programs that can most likely be pwned by
> looking at them sideways, so they get locked down hard with multiple
> redundant layers and so on and so forth. But right now for BPF you only
> have a "can load BPF" or "cannot load BPF" knob, and that's it. This is
> not good enough: we need to be able to define a list of allowed
> payloads, and be able to enforce it, so when (not if) said processes do
> get tricked into loading something else, it will fail, despite having

Yikes, this is a bit scary from a sec point of view right? Are those
programs read-only maps or can the C++ program also write into the
maps and control plane. Assuming they do some critical functions then
you really shouldn't be trusting them to not do all sorts of other
horrible things. Anyways not too important to this discussion.

I'll just reiterate (I think you get it though) that simply signing
enforcement doesn't mean now BPF is safe. Further these programs
have very high privileges and can do all sorts of things to the
system. But, sure sig enforcement locks down one avenue of loading
bogus program.

> the capability of calling bpf(). Trying to define heuristics is also
> not good enough for us - creative malicious actors have a tendency to
> come up with ways to chain things that individually are allowed and
> benign, but combined in a way that you just couldn't foresee. It would

Sure, but I would argue some things can be very restrictive and
generally useful. For example, never allow kernel memory read could be
enforced from BPF side directly. Never allow pkt redirect, etc.

> certainly cover a lot of cases, but not all. A strictly pre-defined
> list of what is allowed to run and what is not is what we need for our
> case, so that we always know exactly what is going to run and what is
> not, and can deal with the consequences accordingly, without nasty
> surprises waiting around the corner. Now in my naive view the best way
> to achieve this is via signatures and certs, as it's a well-understood
> system, with processes already in place to revoke/rotate/etc, and it's
> already used for kmods. An alternative would be hard-coding hashes I
> guess, but that would be terribly inflexible.

Another option would be to load your programs at boot time, presumably
with trusted boot enabled and then lock down BPF completely. Then
ensure all your BPF 'programs' are read-only from user<->kernel
interface and this should start looking fairly close to what you
want and all programs are correct by root of trust back to
trusted boot. Would assume you know what programs to load at boot
though. May or may not be a big assumption depending on your env.

> 
> Now in terms of _how_ the signatures are done and validated, I'm sure
> there are multiple ways, and if some are better than what this series
> implements, then that's not an issue, it can be reworked. But the core
> requirement for us is: offline pre-defined list of what is allowed to
> run and what is not, with ability for hard enforcement that cannot be
> bypassed. Yes, you lose some features like JIT and so on: we don't
> care, we don't need those for our use cases. If others have different
> needs that's fine, this is all intended to be optional, not mandatory.
> There are obviously trade-offs, as always when security is involved,
> and each user can decide what's best for them.
> 
> Hope this makes sense. Thanks!

I think I understand your use case. When done as BPF helper you
can get the behavior you want with a one line BPF program
loaded at boot.

int verify_all(struct bpf_prog **prog) {
	return verify_signature(prog->insn,
				prog->len * sizeof(struct bpf_insn),
			        signature, KEYRING, BPF_SIGTYPE);
}

And I can write some more specific things as,

int verify_blobs(void data) {
  int reject = verify_signature(data, data_len, sig, KEYRING, TYPE);
  struct policy_key *key = map_get_key();

  return policy(key, reject);  
}

map_get_key() looks into some datastor with the policy likely using
'current' to dig something up. It doesn't just apply to BPF progs
we can use it on other executables more generally. And I get more
interesting use cases like, allowing 'tc' programs unsigned, but
requiring kernel memory reads to require signatures or any N
other policies that may have value. Or only allowing my dbg user
to run read-only programs, because the dbg maybe shouldn't ever
be writing into packets, etc. Driving least privilege use cases
in fine detail.

By making it a BPF program we side step the debate where the kernel
tries to get the 'right' policy for you, me, everyone now and in
the future. The only way I can see to do this without getting N
policies baked into the kernel and at M different hook points is via
a BPF helper.

Thanks,
John
Arnaldo Carvalho de Melo Dec. 6, 2021, 9:11 p.m. UTC | #11
Em Mon, Dec 06, 2021 at 12:40:40PM -0800, John Fastabend escreveu:
> I'll just reiterate (I think you get it though) that simply signing
> enforcement doesn't mean now BPF is safe. Further these programs

I think this was clear from the get go, at most this would help with
fingerpointing :-) I.e. BPF signing is not about making things safer,
its just an attempt to know who messed up.

> have very high privileges and can do all sorts of things to the
> system. But, sure sig enforcement locks down one avenue of loading
> bogus program.
 
> > the capability of calling bpf(). Trying to define heuristics is also
> > not good enough for us - creative malicious actors have a tendency to
> > come up with ways to chain things that individually are allowed and
> > benign, but combined in a way that you just couldn't foresee. It would
 
> Sure, but I would argue some things can be very restrictive and
> generally useful. For example, never allow kernel memory read could be
> enforced from BPF side directly. Never allow pkt redirect, etc.

But this is something unrelated to BPF signing, right? Its something
desirable, I'd say this will be at some point required, i.e. one more
step in having BPF programs to be more like userspace apps, where you
can limit all sorts of things it can do, programmatically, a BPF ulimit,
hey, blimit?
 
- Arnaldo
Luca Boccassi Dec. 6, 2021, 10:59 p.m. UTC | #12
On Mon, 2021-12-06 at 12:40 -0800, John Fastabend wrote:
> Luca Boccassi wrote:
> 
> cutting to just the relevant pieces here.
> 
> [...]
> 
> > 
> > > I'll give the outline of the argument here.
> > > 
> > > I do not believe signing BPF instructions for real programs
> > > provides
> > > much additional security. Given most real programs if the
> > > application
> > > or loader is exploited at runtime we have all sorts of trouble.
> > > First
> > > simply verifying the program doesn't prevent malicious use of the
> > > program. If its in the network program this means DDOS, data
> > > exfiltration,
> > > mitm attacks, many other possibilities. If its enforcement
> > > program
> > > most enforcement actions are programmed from this application so
> > > system
> > > security is lost already.  If its observability application
> > > simply
> > > drops/manipulates observations that it wants. I don't know of any
> > > useful programs that exist in isolation without user space input
> > > and output as a critical component. If its not a privileged user,
> > > well it better not be doing anything critical anyways or disabled
> > > outright for the security focused.
> > > 
> > > Many critical programs can't be signed by the nature of the
> > > program.
> > > Optimizing network app generates optimized code at runtime.
> > > Observability
> > > tools JIT the code on the fly, similarly enforcement tools will
> > > do
> > > the
> > > same. I think the power of being able to optimize JIT the code in
> > > application and give to the kernel is something we will see more
> > > and
> > > more of. Saying I'm only going to accept signed programs, for a
> > > distribution or something other than niche use case, is non
> > > starter
> > > IMO because it breaks so many real use cases. We should encourage
> > > these optimizing use cases as I see it as critical to performance
> > > and keeping overhead low.
> > > 
> > > From a purely security standpoint I believe you are better off
> > > defining characteristics an application is allowed to have. For
> > > example allowed to probe kernel memory, make these helpers calls,
> > > have this many instructions, use this much memory, this much cpu,
> > > etc. This lets you sandbox a BPF application (both user space and
> > > kernel side) much nicer than any signing will allow.
> > > 
> > > If we want to 'sign' programs we should do that from a BPF
> > > program
> > > directly where other metadata can be included in the policy. For
> > > example having a hash of the program loaded along with the calls
> > > made and process allows for rich policy decisions. I have other
> > > use cases that need a hash/signature for data blobs, so its on
> > > my todo list but not at the top yet.  But, being able to verify
> > > arbitrary blob of data from BPF feels like a useful operation to
> > > me
> > > in general. The fact in your case its a set of eBPF insns and in
> > > my case its some key in a network header shouldn't matter.
> > > 
> > > The series as is, scanned commit descriptions, is going to break
> > > lots of in-use-today programs if it was ever enabled. And
> > > is not as flexible (can't support bpftrace, etc.) or powerful
> > > (can't consider fine grained policy decisions) as above.
> > > 
> > > Add a function we can hook after verify (or before up for
> > > debate) and helpers to verify signatures and/or generate
> > > hashes and we get a better more general solution. And it can
> > > also solve your use case even if I believe its not useful and
> > > may break many BPF users running bpftrace, libbpf, etc.
> > > 
> > > Thanks,
> > > John
> > 
> > Hello John,
> > 
> > Thank you for the summary, this is much clearer.
> > 
> > First of all, I think there's some misunderstanding: this series
> > does
> > not enable optional signatures by default, and does not enable
> > mandatory signatures by default either. So I don't see how it would
> > break existing use cases as you are saying? Unless I'm missing
> > something?
> > 
> > There's a kconfig to enable optional signatures - if they are
> > there,
> > they are verified, if they are not present then nothing different
> > happens. Unless I am missing something, this should be backward
> > compatible. This kconfig would likely be enabled in most use cases,
> > just like optionally signed kernel modules are.
> 
> Agree, without enforcement things should continue to work.
> 
> > 
> > Then there's a kconfig on top of that which makes signatures
> > mandatory.
> > I would not imagine this to be enabled in may cases, just in custom
> > builds that have more stringent requirements. It certainly would
> > not be
> > enabled in generalist distros. Perhaps a more flexible way would be
> > to
> > introduce a sysctl, like fsverity has with
> > 'fs.verity.require_signatures'? That would be just fine for our use
> > case. Matteo can we do that instead in the next revision?
> 
> We want to manage this from BPF side directly. It looks
> like policy decision and we have use cases that are not as
> simple as yes/no with global switch. For example, in k8s world
> this might be enabled via labels which are user specific per
> container
> policy. e.g. lockdown some containers more strictly than others.
> 
> > 
> > Secondly, I understand that for your use case signing programs
> > would
> > not be the best approach. That's fine, and I'm glad you are working
> > on
> > an alternative that better fits your model, it will be very
> > interesting
> > to see how it looks like once implemented. But that model doesn't
> > fit
> > all cases. In our case at Microsoft, we absolutely want to be able
> > to
> > pre-define at build time a list of BPF programs that are allowed to
> > be
> > loaded, and reject anything else. Userspace processes in our case
> > are
> 
> By building this into BPF you can get the 'reject anything else'
> policy
> and I get the metadata + reject/accept from the same hook. Its
> just your program can be very simple.
> 
> > mostly old and crufty c++ programs that can most likely be pwned by
> > looking at them sideways, so they get locked down hard with
> > multiple
> > redundant layers and so on and so forth. But right now for BPF you
> > only
> > have a "can load BPF" or "cannot load BPF" knob, and that's it.
> > This is
> > not good enough: we need to be able to define a list of allowed
> > payloads, and be able to enforce it, so when (not if) said
> > processes do
> > get tricked into loading something else, it will fail, despite
> > having
> 
> Yikes, this is a bit scary from a sec point of view right? Are those
> programs read-only maps or can the C++ program also write into the
> maps and control plane. Assuming they do some critical functions then
> you really shouldn't be trusting them to not do all sorts of other
> horrible things. Anyways not too important to this discussion.
> 
> I'll just reiterate (I think you get it though) that simply signing
> enforcement doesn't mean now BPF is safe. Further these programs
> have very high privileges and can do all sorts of things to the
> system. But, sure sig enforcement locks down one avenue of loading
> bogus program.

Oh it's terrifying - but business needs and all that.
But Arnaldo is spot on - it's not strictly about what is more secure,
but more about making it a known quantity. If we can prove what is
allowed to run and what not before any machine has even booted (barring
bugs in sig verification, of course) then the $org_security_team is
satisfied and can sign off on enabling bpf. Otherwise we can keep
dreaming.

> > the capability of calling bpf(). Trying to define heuristics is
> > also
> > not good enough for us - creative malicious actors have a tendency
> > to
> > come up with ways to chain things that individually are allowed and
> > benign, but combined in a way that you just couldn't foresee. It
> > would
> 
> Sure, but I would argue some things can be very restrictive and
> generally useful. For example, never allow kernel memory read could
> be
> enforced from BPF side directly. Never allow pkt redirect, etc.
> 
> > certainly cover a lot of cases, but not all. A strictly pre-defined
> > list of what is allowed to run and what is not is what we need for
> > our
> > case, so that we always know exactly what is going to run and what
> > is
> > not, and can deal with the consequences accordingly, without nasty
> > surprises waiting around the corner. Now in my naive view the best
> > way
> > to achieve this is via signatures and certs, as it's a well-
> > understood
> > system, with processes already in place to revoke/rotate/etc, and
> > it's
> > already used for kmods. An alternative would be hard-coding hashes
> > I
> > guess, but that would be terribly inflexible.
> 
> Another option would be to load your programs at boot time,
> presumably
> with trusted boot enabled and then lock down BPF completely. Then
> ensure all your BPF 'programs' are read-only from user<->kernel
> interface and this should start looking fairly close to what you
> want and all programs are correct by root of trust back to
> trusted boot. Would assume you know what programs to load at boot
> though. May or may not be a big assumption depending on your env.

One of the use cases we have for BPF is on-demand diagnostics, so
loading at boot and blocking afterwards would not work, I think.
Environment is constrained in terms of resources, so don't want to load
anything that is not needed.

> > 
> > Now in terms of _how_ the signatures are done and validated, I'm
> > sure
> > there are multiple ways, and if some are better than what this
> > series
> > implements, then that's not an issue, it can be reworked. But the
> > core
> > requirement for us is: offline pre-defined list of what is allowed
> > to
> > run and what is not, with ability for hard enforcement that cannot
> > be
> > bypassed. Yes, you lose some features like JIT and so on: we don't
> > care, we don't need those for our use cases. If others have
> > different
> > needs that's fine, this is all intended to be optional, not
> > mandatory.
> > There are obviously trade-offs, as always when security is
> > involved,
> > and each user can decide what's best for them.
> > 
> > Hope this makes sense. Thanks!
> 
> I think I understand your use case. When done as BPF helper you
> can get the behavior you want with a one line BPF program
> loaded at boot.
> 
> int verify_all(struct bpf_prog **prog) {
>         return verify_signature(prog->insn,
>                                 prog->len * sizeof(struct bpf_insn),
>                                 signature, KEYRING, BPF_SIGTYPE);
> }
> 
> And I can write some more specific things as,
> 
> int verify_blobs(void data) {
>   int reject = verify_signature(data, data_len, sig, KEYRING, TYPE);
>   struct policy_key *key = map_get_key();
> 
>   return policy(key, reject);  
> }
> 
> map_get_key() looks into some datastor with the policy likely using
> 'current' to dig something up. It doesn't just apply to BPF progs
> we can use it on other executables more generally. And I get more
> interesting use cases like, allowing 'tc' programs unsigned, but
> requiring kernel memory reads to require signatures or any N
> other policies that may have value. Or only allowing my dbg user
> to run read-only programs, because the dbg maybe shouldn't ever
> be writing into packets, etc. Driving least privilege use cases
> in fine detail.
> 
> By making it a BPF program we side step the debate where the kernel
> tries to get the 'right' policy for you, me, everyone now and in
> the future. The only way I can see to do this without getting N
> policies baked into the kernel and at M different hook points is via
> a BPF helper.
> 
> Thanks,
> John

Now this sounds like something that could work - we can prove that this
could be loaded before any writable fs comes up anywhere, so in
principle I think it would be acceptable and free of races. Matteo, we
should talk about this tomorrow.
And this requires some infrastructure work right? Is there a WIP git
tree somewhere that we can test out?

Thank you!
Luca Boccassi Dec. 8, 2021, 4:25 p.m. UTC | #13
On Mon, 2021-12-06 at 22:59 +0000, Luca Boccassi wrote:
> On Mon, 2021-12-06 at 12:40 -0800, John Fastabend wrote:
> > Luca Boccassi wrote:
> > 
> > cutting to just the relevant pieces here.
> > 
> > [...]
> > 
> > > 
> > > > I'll give the outline of the argument here.
> > > > 
> > > > I do not believe signing BPF instructions for real programs
> > > > provides
> > > > much additional security. Given most real programs if the
> > > > application
> > > > or loader is exploited at runtime we have all sorts of trouble.
> > > > First
> > > > simply verifying the program doesn't prevent malicious use of the
> > > > program. If its in the network program this means DDOS, data
> > > > exfiltration,
> > > > mitm attacks, many other possibilities. If its enforcement
> > > > program
> > > > most enforcement actions are programmed from this application so
> > > > system
> > > > security is lost already.  If its observability application
> > > > simply
> > > > drops/manipulates observations that it wants. I don't know of any
> > > > useful programs that exist in isolation without user space input
> > > > and output as a critical component. If its not a privileged user,
> > > > well it better not be doing anything critical anyways or disabled
> > > > outright for the security focused.
> > > > 
> > > > Many critical programs can't be signed by the nature of the
> > > > program.
> > > > Optimizing network app generates optimized code at runtime.
> > > > Observability
> > > > tools JIT the code on the fly, similarly enforcement tools will
> > > > do
> > > > the
> > > > same. I think the power of being able to optimize JIT the code in
> > > > application and give to the kernel is something we will see more
> > > > and
> > > > more of. Saying I'm only going to accept signed programs, for a
> > > > distribution or something other than niche use case, is non
> > > > starter
> > > > IMO because it breaks so many real use cases. We should encourage
> > > > these optimizing use cases as I see it as critical to performance
> > > > and keeping overhead low.
> > > > 
> > > > From a purely security standpoint I believe you are better off
> > > > defining characteristics an application is allowed to have. For
> > > > example allowed to probe kernel memory, make these helpers calls,
> > > > have this many instructions, use this much memory, this much cpu,
> > > > etc. This lets you sandbox a BPF application (both user space and
> > > > kernel side) much nicer than any signing will allow.
> > > > 
> > > > If we want to 'sign' programs we should do that from a BPF
> > > > program
> > > > directly where other metadata can be included in the policy. For
> > > > example having a hash of the program loaded along with the calls
> > > > made and process allows for rich policy decisions. I have other
> > > > use cases that need a hash/signature for data blobs, so its on
> > > > my todo list but not at the top yet.  But, being able to verify
> > > > arbitrary blob of data from BPF feels like a useful operation to
> > > > me
> > > > in general. The fact in your case its a set of eBPF insns and in
> > > > my case its some key in a network header shouldn't matter.
> > > > 
> > > > The series as is, scanned commit descriptions, is going to break
> > > > lots of in-use-today programs if it was ever enabled. And
> > > > is not as flexible (can't support bpftrace, etc.) or powerful
> > > > (can't consider fine grained policy decisions) as above.
> > > > 
> > > > Add a function we can hook after verify (or before up for
> > > > debate) and helpers to verify signatures and/or generate
> > > > hashes and we get a better more general solution. And it can
> > > > also solve your use case even if I believe its not useful and
> > > > may break many BPF users running bpftrace, libbpf, etc.
> > > > 
> > > > Thanks,
> > > > John
> > > 
> > > Hello John,
> > > 
> > > Thank you for the summary, this is much clearer.
> > > 
> > > First of all, I think there's some misunderstanding: this series
> > > does
> > > not enable optional signatures by default, and does not enable
> > > mandatory signatures by default either. So I don't see how it would
> > > break existing use cases as you are saying? Unless I'm missing
> > > something?
> > > 
> > > There's a kconfig to enable optional signatures - if they are
> > > there,
> > > they are verified, if they are not present then nothing different
> > > happens. Unless I am missing something, this should be backward
> > > compatible. This kconfig would likely be enabled in most use cases,
> > > just like optionally signed kernel modules are.
> > 
> > Agree, without enforcement things should continue to work.
> > 
> > > 
> > > Then there's a kconfig on top of that which makes signatures
> > > mandatory.
> > > I would not imagine this to be enabled in may cases, just in custom
> > > builds that have more stringent requirements. It certainly would
> > > not be
> > > enabled in generalist distros. Perhaps a more flexible way would be
> > > to
> > > introduce a sysctl, like fsverity has with
> > > 'fs.verity.require_signatures'? That would be just fine for our use
> > > case. Matteo can we do that instead in the next revision?
> > 
> > We want to manage this from BPF side directly. It looks
> > like policy decision and we have use cases that are not as
> > simple as yes/no with global switch. For example, in k8s world
> > this might be enabled via labels which are user specific per
> > container
> > policy. e.g. lockdown some containers more strictly than others.
> > 
> > > 
> > > Secondly, I understand that for your use case signing programs
> > > would
> > > not be the best approach. That's fine, and I'm glad you are working
> > > on
> > > an alternative that better fits your model, it will be very
> > > interesting
> > > to see how it looks like once implemented. But that model doesn't
> > > fit
> > > all cases. In our case at Microsoft, we absolutely want to be able
> > > to
> > > pre-define at build time a list of BPF programs that are allowed to
> > > be
> > > loaded, and reject anything else. Userspace processes in our case
> > > are
> > 
> > By building this into BPF you can get the 'reject anything else'
> > policy
> > and I get the metadata + reject/accept from the same hook. Its
> > just your program can be very simple.
> > 
> > > mostly old and crufty c++ programs that can most likely be pwned by
> > > looking at them sideways, so they get locked down hard with
> > > multiple
> > > redundant layers and so on and so forth. But right now for BPF you
> > > only
> > > have a "can load BPF" or "cannot load BPF" knob, and that's it.
> > > This is
> > > not good enough: we need to be able to define a list of allowed
> > > payloads, and be able to enforce it, so when (not if) said
> > > processes do
> > > get tricked into loading something else, it will fail, despite
> > > having
> > 
> > Yikes, this is a bit scary from a sec point of view right? Are those
> > programs read-only maps or can the C++ program also write into the
> > maps and control plane. Assuming they do some critical functions then
> > you really shouldn't be trusting them to not do all sorts of other
> > horrible things. Anyways not too important to this discussion.
> > 
> > I'll just reiterate (I think you get it though) that simply signing
> > enforcement doesn't mean now BPF is safe. Further these programs
> > have very high privileges and can do all sorts of things to the
> > system. But, sure sig enforcement locks down one avenue of loading
> > bogus program.
> 
> Oh it's terrifying - but business needs and all that.
> But Arnaldo is spot on - it's not strictly about what is more secure,
> but more about making it a known quantity. If we can prove what is
> allowed to run and what not before any machine has even booted (barring
> bugs in sig verification, of course) then the $org_security_team is
> satisfied and can sign off on enabling bpf. Otherwise we can keep
> dreaming.
> 
> > > the capability of calling bpf(). Trying to define heuristics is
> > > also
> > > not good enough for us - creative malicious actors have a tendency
> > > to
> > > come up with ways to chain things that individually are allowed and
> > > benign, but combined in a way that you just couldn't foresee. It
> > > would
> > 
> > Sure, but I would argue some things can be very restrictive and
> > generally useful. For example, never allow kernel memory read could
> > be
> > enforced from BPF side directly. Never allow pkt redirect, etc.
> > 
> > > certainly cover a lot of cases, but not all. A strictly pre-defined
> > > list of what is allowed to run and what is not is what we need for
> > > our
> > > case, so that we always know exactly what is going to run and what
> > > is
> > > not, and can deal with the consequences accordingly, without nasty
> > > surprises waiting around the corner. Now in my naive view the best
> > > way
> > > to achieve this is via signatures and certs, as it's a well-
> > > understood
> > > system, with processes already in place to revoke/rotate/etc, and
> > > it's
> > > already used for kmods. An alternative would be hard-coding hashes
> > > I
> > > guess, but that would be terribly inflexible.
> > 
> > Another option would be to load your programs at boot time,
> > presumably
> > with trusted boot enabled and then lock down BPF completely. Then
> > ensure all your BPF 'programs' are read-only from user<->kernel
> > interface and this should start looking fairly close to what you
> > want and all programs are correct by root of trust back to
> > trusted boot. Would assume you know what programs to load at boot
> > though. May or may not be a big assumption depending on your env.
> 
> One of the use cases we have for BPF is on-demand diagnostics, so
> loading at boot and blocking afterwards would not work, I think.
> Environment is constrained in terms of resources, so don't want to load
> anything that is not needed.
> 
> > > 
> > > Now in terms of _how_ the signatures are done and validated, I'm
> > > sure
> > > there are multiple ways, and if some are better than what this
> > > series
> > > implements, then that's not an issue, it can be reworked. But the
> > > core
> > > requirement for us is: offline pre-defined list of what is allowed
> > > to
> > > run and what is not, with ability for hard enforcement that cannot
> > > be
> > > bypassed. Yes, you lose some features like JIT and so on: we don't
> > > care, we don't need those for our use cases. If others have
> > > different
> > > needs that's fine, this is all intended to be optional, not
> > > mandatory.
> > > There are obviously trade-offs, as always when security is
> > > involved,
> > > and each user can decide what's best for them.
> > > 
> > > Hope this makes sense. Thanks!
> > 
> > I think I understand your use case. When done as BPF helper you
> > can get the behavior you want with a one line BPF program
> > loaded at boot.
> > 
> > int verify_all(struct bpf_prog **prog) {
> >         return verify_signature(prog->insn,
> >                                 prog->len * sizeof(struct bpf_insn),
> >                                 signature, KEYRING, BPF_SIGTYPE);
> > }
> > 
> > And I can write some more specific things as,
> > 
> > int verify_blobs(void data) {
> >   int reject = verify_signature(data, data_len, sig, KEYRING, TYPE);
> >   struct policy_key *key = map_get_key();
> > 
> >   return policy(key, reject);  
> > }
> > 
> > map_get_key() looks into some datastor with the policy likely using
> > 'current' to dig something up. It doesn't just apply to BPF progs
> > we can use it on other executables more generally. And I get more
> > interesting use cases like, allowing 'tc' programs unsigned, but
> > requiring kernel memory reads to require signatures or any N
> > other policies that may have value. Or only allowing my dbg user
> > to run read-only programs, because the dbg maybe shouldn't ever
> > be writing into packets, etc. Driving least privilege use cases
> > in fine detail.
> > 
> > By making it a BPF program we side step the debate where the kernel
> > tries to get the 'right' policy for you, me, everyone now and in
> > the future. The only way I can see to do this without getting N
> > policies baked into the kernel and at M different hook points is via
> > a BPF helper.
> > 
> > Thanks,
> > John
> 
> Now this sounds like something that could work - we can prove that this
> could be loaded before any writable fs comes up anywhere, so in
> principle I think it would be acceptable and free of races. Matteo, we
> should talk about this tomorrow.
> And this requires some infrastructure work right? Is there a WIP git
> tree somewhere that we can test out?
> 
> Thank you!

One question more question: with the signature + kconfig approach,
nothing can disable the signature check. But if the signature checker
is itself a bpf program, is there/can there be anything stopping root
from unloading it?
John Fastabend Dec. 8, 2021, 8:17 p.m. UTC | #14
[...]

> > > > Hope this makes sense. Thanks!
> > > 
> > > I think I understand your use case. When done as BPF helper you
> > > can get the behavior you want with a one line BPF program
> > > loaded at boot.
> > > 
> > > int verify_all(struct bpf_prog **prog) {
> > >         return verify_signature(prog->insn,
> > >                                 prog->len * sizeof(struct bpf_insn),
> > >                                 signature, KEYRING, BPF_SIGTYPE);
> > > }
> > > 
> > > And I can write some more specific things as,
> > > 
> > > int verify_blobs(void data) {
> > >   int reject = verify_signature(data, data_len, sig, KEYRING, TYPE);
> > >   struct policy_key *key = map_get_key();
> > > 
> > >   return policy(key, reject);  
> > > }
> > > 
> > > map_get_key() looks into some datastor with the policy likely using
> > > 'current' to dig something up. It doesn't just apply to BPF progs
> > > we can use it on other executables more generally. And I get more
> > > interesting use cases like, allowing 'tc' programs unsigned, but
> > > requiring kernel memory reads to require signatures or any N
> > > other policies that may have value. Or only allowing my dbg user
> > > to run read-only programs, because the dbg maybe shouldn't ever
> > > be writing into packets, etc. Driving least privilege use cases
> > > in fine detail.
> > > 
> > > By making it a BPF program we side step the debate where the kernel
> > > tries to get the 'right' policy for you, me, everyone now and in
> > > the future. The only way I can see to do this without getting N
> > > policies baked into the kernel and at M different hook points is via
> > > a BPF helper.
> > > 
> > > Thanks,
> > > John
> > 
> > Now this sounds like something that could work - we can prove that this
> > could be loaded before any writable fs comes up anywhere, so in
> > principle I think it would be acceptable and free of races. Matteo, we
> > should talk about this tomorrow.
> > And this requires some infrastructure work right? Is there a WIP git
> > tree somewhere that we can test out?
> > 
> > Thank you!
> 

I don't have a WIP tree, but I believe it should be fairly easy.
First I would add a wrapper BPF helper for verify_signature() so
we can call it from fentry/freturn context. That can be done on
its own IMO as its a generally useful operation.

Then I would stub a hook point into the BPF load path. The exact
place to put this is going to have some debate I think, but I
would place it immediately after the check_bpf call.

With above two you have enough to do sig verification iiuc.

Early boot loading I would have to check its current status. But I know
folks have been working on it. Maybe its done?

> One question more question: with the signature + kconfig approach,
> nothing can disable the signature check. But if the signature checker
> is itself a bpf program, is there/can there be anything stopping root
> from unloading it?

Interesting. Not that I'm aware of. Currently something with sufficient
privileges could unload the program. Maybe we should have a flag so
early boot programs can signal they shouldn't be unloaded ever. I would
be OK with this and also seems generally useful. I have a case where
I want to always set the socket cookie and we leave it running all the
time. It would be nice if it came up and was pinned at boot.

Maybe slightly better than a flag would be to have a new CAP support
that only early boot has like CAP_BPF_EARLY. From my point of view
this both seems doable with just some smallish changes on BPF side.

Thanks,
John
Luca Boccassi Dec. 9, 2021, 1:40 p.m. UTC | #15
On Wed, 2021-12-08 at 12:17 -0800, John Fastabend wrote:
> [...]
> 
> > > > > Hope this makes sense. Thanks!
> > > > 
> > > > I think I understand your use case. When done as BPF helper you
> > > > can get the behavior you want with a one line BPF program
> > > > loaded at boot.
> > > > 
> > > > int verify_all(struct bpf_prog **prog) {
> > > >         return verify_signature(prog->insn,
> > > >                                 prog->len * sizeof(struct bpf_insn),
> > > >                                 signature, KEYRING, BPF_SIGTYPE);
> > > > }
> > > > 
> > > > And I can write some more specific things as,
> > > > 
> > > > int verify_blobs(void data) {
> > > >   int reject = verify_signature(data, data_len, sig, KEYRING, TYPE);
> > > >   struct policy_key *key = map_get_key();
> > > > 
> > > >   return policy(key, reject);  
> > > > }
> > > > 
> > > > map_get_key() looks into some datastor with the policy likely using
> > > > 'current' to dig something up. It doesn't just apply to BPF progs
> > > > we can use it on other executables more generally. And I get more
> > > > interesting use cases like, allowing 'tc' programs unsigned, but
> > > > requiring kernel memory reads to require signatures or any N
> > > > other policies that may have value. Or only allowing my dbg user
> > > > to run read-only programs, because the dbg maybe shouldn't ever
> > > > be writing into packets, etc. Driving least privilege use cases
> > > > in fine detail.
> > > > 
> > > > By making it a BPF program we side step the debate where the kernel
> > > > tries to get the 'right' policy for you, me, everyone now and in
> > > > the future. The only way I can see to do this without getting N
> > > > policies baked into the kernel and at M different hook points is via
> > > > a BPF helper.
> > > > 
> > > > Thanks,
> > > > John
> > > 
> > > Now this sounds like something that could work - we can prove that this
> > > could be loaded before any writable fs comes up anywhere, so in
> > > principle I think it would be acceptable and free of races. Matteo, we
> > > should talk about this tomorrow.
> > > And this requires some infrastructure work right? Is there a WIP git
> > > tree somewhere that we can test out?
> > > 
> > > Thank you!
> > 
> 
> I don't have a WIP tree, but I believe it should be fairly easy.
> First I would add a wrapper BPF helper for verify_signature() so
> we can call it from fentry/freturn context. That can be done on
> its own IMO as its a generally useful operation.
> 
> Then I would stub a hook point into the BPF load path. The exact
> place to put this is going to have some debate I think, but I
> would place it immediately after the check_bpf call.
> 
> With above two you have enough to do sig verification iiuc.
> 
> Early boot loading I would have to check its current status. But I know
> folks have been working on it. Maybe its done?
> 
> > One question more question: with the signature + kconfig approach,
> > nothing can disable the signature check. But if the signature checker
> > is itself a bpf program, is there/can there be anything stopping root
> > from unloading it?
> 
> Interesting. Not that I'm aware of. Currently something with sufficient
> privileges could unload the program. Maybe we should have a flag so
> early boot programs can signal they shouldn't be unloaded ever. I would
> be OK with this and also seems generally useful. I have a case where
> I want to always set the socket cookie and we leave it running all the
> time. It would be nice if it came up and was pinned at boot.
> 
> Maybe slightly better than a flag would be to have a new CAP support
> that only early boot has like CAP_BPF_EARLY. From my point of view
> this both seems doable with just some smallish changes on BPF side.
> 
> Thanks,
> John

Thanks - again the means of enforcing this are not too important for
our use case, as long as there is something that works reliably and can
be attested.