| Message ID | 20211203191844.69709-1-mcroce@linux.microsoft.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | bpf: add signature | expand |
On Fri, Dec 3, 2021 at 11:18 AM Matteo Croce <mcroce@linux.microsoft.com> wrote: > > From: Matteo Croce <mcroce@microsoft.com> > > This series add signature verification for BPF files. > The first patch implements the signature validation in the kernel, > the second patch optionally makes the signature mandatory, > the third adds signature generation to bpftool. Matteo, I think I already mentioned that it's no-go as-is. We've agreed to go with John's suggestion.
On Fri, Dec 3, 2021 at 8:22 PM Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote: > > On Fri, Dec 3, 2021 at 11:18 AM Matteo Croce <mcroce@linux.microsoft.com> wrote: > > > > From: Matteo Croce <mcroce@microsoft.com> > > > > This series add signature verification for BPF files. > > The first patch implements the signature validation in the kernel, > > the second patch optionally makes the signature mandatory, > > the third adds signature generation to bpftool. > > Matteo, > > I think I already mentioned that it's no-go as-is. > We've agreed to go with John's suggestion. Hi, my previous attempt was loading a whole ELF file and parsing it in kernel. In this series I just validate the instructions against a signature, as with kernel CO-RE libbpf doesn't need to mangle it. Which suggestion? I think I missed this one..
On Fri, Dec 3, 2021 at 11:36 AM Matteo Croce <mcroce@linux.microsoft.com> wrote: > > On Fri, Dec 3, 2021 at 8:22 PM Alexei Starovoitov > <alexei.starovoitov@gmail.com> wrote: > > > > On Fri, Dec 3, 2021 at 11:18 AM Matteo Croce <mcroce@linux.microsoft.com> wrote: > > > > > > From: Matteo Croce <mcroce@microsoft.com> > > > > > > This series add signature verification for BPF files. > > > The first patch implements the signature validation in the kernel, > > > the second patch optionally makes the signature mandatory, > > > the third adds signature generation to bpftool. > > > > Matteo, > > > > I think I already mentioned that it's no-go as-is. > > We've agreed to go with John's suggestion. > > Hi, > > my previous attempt was loading a whole ELF file and parsing it in kernel. > In this series I just validate the instructions against a signature, > as with kernel CO-RE libbpf doesn't need to mangle it. > > Which suggestion? I think I missed this one.. This talk and discussion: https://linuxplumbersconf.org/event/11/contributions/947/
On Fri, 2021-12-03 at 11:37 -0800, Alexei Starovoitov wrote: > On Fri, Dec 3, 2021 at 11:36 AM Matteo Croce > <mcroce@linux.microsoft.com> wrote: > > > > On Fri, Dec 3, 2021 at 8:22 PM Alexei Starovoitov > > <alexei.starovoitov@gmail.com> wrote: > > > > > > On Fri, Dec 3, 2021 at 11:18 AM Matteo Croce > > > <mcroce@linux.microsoft.com> wrote: > > > > > > > > From: Matteo Croce <mcroce@microsoft.com> > > > > > > > > This series add signature verification for BPF files. > > > > The first patch implements the signature validation in the > > > > kernel, > > > > the second patch optionally makes the signature mandatory, > > > > the third adds signature generation to bpftool. > > > > > > Matteo, > > > > > > I think I already mentioned that it's no-go as-is. > > > We've agreed to go with John's suggestion. > > > > Hi, > > > > my previous attempt was loading a whole ELF file and parsing it in > > kernel. > > In this series I just validate the instructions against a > > signature, > > as with kernel CO-RE libbpf doesn't need to mangle it. > > > > Which suggestion? I think I missed this one.. > > This talk and discussion: > https://linuxplumbersconf.org/event/11/contributions/947/ Thanks for the link - but for those of us who don't have ~5 hours to watch a video recording, would you mind sharing a one line summary, please? Is there an alternative patch series implementing BPF signing that you can link us so that we can look at it? Just a link or googlable reference would be more than enough. Thank you!
On Fri, Dec 3, 2021 at 2:06 PM Luca Boccassi <bluca@debian.org> wrote: > > On Fri, 2021-12-03 at 11:37 -0800, Alexei Starovoitov wrote: > > On Fri, Dec 3, 2021 at 11:36 AM Matteo Croce > > <mcroce@linux.microsoft.com> wrote: > > > > > > On Fri, Dec 3, 2021 at 8:22 PM Alexei Starovoitov > > > <alexei.starovoitov@gmail.com> wrote: > > > > > > > > On Fri, Dec 3, 2021 at 11:18 AM Matteo Croce > > > > <mcroce@linux.microsoft.com> wrote: > > > > > > > > > > From: Matteo Croce <mcroce@microsoft.com> > > > > > > > > > > This series add signature verification for BPF files. > > > > > The first patch implements the signature validation in the > > > > > kernel, > > > > > the second patch optionally makes the signature mandatory, > > > > > the third adds signature generation to bpftool. > > > > > > > > Matteo, > > > > > > > > I think I already mentioned that it's no-go as-is. > > > > We've agreed to go with John's suggestion. > > > > > > Hi, > > > > > > my previous attempt was loading a whole ELF file and parsing it in > > > kernel. > > > In this series I just validate the instructions against a > > > signature, > > > as with kernel CO-RE libbpf doesn't need to mangle it. > > > > > > Which suggestion? I think I missed this one.. > > > > This talk and discussion: > > https://linuxplumbersconf.org/event/11/contributions/947/ > > Thanks for the link - but for those of us who don't have ~5 hours to > watch a video recording, would you mind sharing a one line summary, > please? Is there an alternative patch series implementing BPF signing > that you can link us so that we can look at it? Just a link or > googlable reference would be more than enough. It's not 5 hours and you have to read slides and watch John's presentation to follow the conversation.
On Fri, Dec 3, 2021 at 11:20 PM Alexei Starovoitov <alexei.starovoitov@gmail.com> wrote: > > On Fri, Dec 3, 2021 at 2:06 PM Luca Boccassi <bluca@debian.org> wrote: > > > > On Fri, 2021-12-03 at 11:37 -0800, Alexei Starovoitov wrote: > > > On Fri, Dec 3, 2021 at 11:36 AM Matteo Croce > > > <mcroce@linux.microsoft.com> wrote: > > > > > > > > On Fri, Dec 3, 2021 at 8:22 PM Alexei Starovoitov > > > > <alexei.starovoitov@gmail.com> wrote: > > > > > > > > > > On Fri, Dec 3, 2021 at 11:18 AM Matteo Croce > > > > > <mcroce@linux.microsoft.com> wrote: > > > > > > > > > > > > From: Matteo Croce <mcroce@microsoft.com> > > > > > > > > > > > > This series add signature verification for BPF files. > > > > > > The first patch implements the signature validation in the > > > > > > kernel, > > > > > > the second patch optionally makes the signature mandatory, > > > > > > the third adds signature generation to bpftool. > > > > > > > > > > Matteo, > > > > > > > > > > I think I already mentioned that it's no-go as-is. > > > > > We've agreed to go with John's suggestion. > > > > > > > > Hi, > > > > > > > > my previous attempt was loading a whole ELF file and parsing it in > > > > kernel. > > > > In this series I just validate the instructions against a > > > > signature, > > > > as with kernel CO-RE libbpf doesn't need to mangle it. > > > > > > > > Which suggestion? I think I missed this one.. > > > > > > This talk and discussion: > > > https://linuxplumbersconf.org/event/11/contributions/947/ > > > > Thanks for the link - but for those of us who don't have ~5 hours to > > watch a video recording, would you mind sharing a one line summary, > > please? Is there an alternative patch series implementing BPF signing > > that you can link us so that we can look at it? Just a link or > > googlable reference would be more than enough. > > It's not 5 hours and you have to read slides and watch > John's presentation to follow the conversation. So, If I have understood correctly, the proposal is to validate the tools which loads the BPF (e.g. perf, ip) with fs-verity, and only allow BPF loading from those validated binaries? That's nice, but I think that this could be complementary to the instructions signature. Imagine a validated binary being exploited somehow at runtime, that could be vector of malicious BPF program load. Can't we have both available, and use one or other, or even both together depending on the use case? Regards,
On Fri, Dec 3, 2021 at 4:42 PM Matteo Croce <mcroce@linux.microsoft.com> wrote: > > On Fri, Dec 3, 2021 at 11:20 PM Alexei Starovoitov > <alexei.starovoitov@gmail.com> wrote: > > > > On Fri, Dec 3, 2021 at 2:06 PM Luca Boccassi <bluca@debian.org> wrote: > > > > > > On Fri, 2021-12-03 at 11:37 -0800, Alexei Starovoitov wrote: > > > > On Fri, Dec 3, 2021 at 11:36 AM Matteo Croce > > > > <mcroce@linux.microsoft.com> wrote: > > > > > > > > > > On Fri, Dec 3, 2021 at 8:22 PM Alexei Starovoitov > > > > > <alexei.starovoitov@gmail.com> wrote: > > > > > > > > > > > > On Fri, Dec 3, 2021 at 11:18 AM Matteo Croce > > > > > > <mcroce@linux.microsoft.com> wrote: > > > > > > > > > > > > > > From: Matteo Croce <mcroce@microsoft.com> > > > > > > > > > > > > > > This series add signature verification for BPF files. > > > > > > > The first patch implements the signature validation in the > > > > > > > kernel, > > > > > > > the second patch optionally makes the signature mandatory, > > > > > > > the third adds signature generation to bpftool. > > > > > > > > > > > > Matteo, > > > > > > > > > > > > I think I already mentioned that it's no-go as-is. > > > > > > We've agreed to go with John's suggestion. > > > > > > > > > > Hi, > > > > > > > > > > my previous attempt was loading a whole ELF file and parsing it in > > > > > kernel. > > > > > In this series I just validate the instructions against a > > > > > signature, > > > > > as with kernel CO-RE libbpf doesn't need to mangle it. > > > > > > > > > > Which suggestion? I think I missed this one.. > > > > > > > > This talk and discussion: > > > > https://linuxplumbersconf.org/event/11/contributions/947/ > > > > > > Thanks for the link - but for those of us who don't have ~5 hours to > > > watch a video recording, would you mind sharing a one line summary, > > > please? Is there an alternative patch series implementing BPF signing > > > that you can link us so that we can look at it? Just a link or > > > googlable reference would be more than enough. > > > > It's not 5 hours and you have to read slides and watch > > John's presentation to follow the conversation. > > So, If I have understood correctly, the proposal is to validate the > tools which loads the BPF (e.g. perf, ip) with fs-verity, and only > allow BPF loading from those validated binaries? > That's nice, but I think that this could be complementary to the > instructions signature. > Imagine a validated binary being exploited somehow at runtime, that > could be vector of malicious BPF program load. > Can't we have both available, and use one or other, or even both > together depending on the use case? I'll let John comment.
Alexei Starovoitov wrote: > On Fri, Dec 3, 2021 at 4:42 PM Matteo Croce <mcroce@linux.microsoft.com> wrote: > > > > On Fri, Dec 3, 2021 at 11:20 PM Alexei Starovoitov > > <alexei.starovoitov@gmail.com> wrote: > > > > > > On Fri, Dec 3, 2021 at 2:06 PM Luca Boccassi <bluca@debian.org> wrote: > > > > > > > > On Fri, 2021-12-03 at 11:37 -0800, Alexei Starovoitov wrote: > > > > > On Fri, Dec 3, 2021 at 11:36 AM Matteo Croce > > > > > <mcroce@linux.microsoft.com> wrote: > > > > > > > > > > > > On Fri, Dec 3, 2021 at 8:22 PM Alexei Starovoitov > > > > > > <alexei.starovoitov@gmail.com> wrote: > > > > > > > > > > > > > > On Fri, Dec 3, 2021 at 11:18 AM Matteo Croce > > > > > > > <mcroce@linux.microsoft.com> wrote: > > > > > > > > > > > > > > > > From: Matteo Croce <mcroce@microsoft.com> > > > > > > > > > > > > > > > > This series add signature verification for BPF files. > > > > > > > > The first patch implements the signature validation in the > > > > > > > > kernel, > > > > > > > > the second patch optionally makes the signature mandatory, > > > > > > > > the third adds signature generation to bpftool. > > > > > > > > > > > > > > Matteo, > > > > > > > > > > > > > > I think I already mentioned that it's no-go as-is. > > > > > > > We've agreed to go with John's suggestion. > > > > > > > > > > > > Hi, > > > > > > > > > > > > my previous attempt was loading a whole ELF file and parsing it in > > > > > > kernel. > > > > > > In this series I just validate the instructions against a > > > > > > signature, > > > > > > as with kernel CO-RE libbpf doesn't need to mangle it. > > > > > > > > > > > > Which suggestion? I think I missed this one.. > > > > > > > > > > This talk and discussion: > > > > > https://linuxplumbersconf.org/event/11/contributions/947/ > > > > > > > > Thanks for the link - but for those of us who don't have ~5 hours to > > > > watch a video recording, would you mind sharing a one line summary, > > > > please? Is there an alternative patch series implementing BPF signing > > > > that you can link us so that we can look at it? Just a link or > > > > googlable reference would be more than enough. > > > > > > It's not 5 hours and you have to read slides and watch > > > John's presentation to follow the conversation. > > > > So, If I have understood correctly, the proposal is to validate the > > tools which loads the BPF (e.g. perf, ip) with fs-verity, and only > > allow BPF loading from those validated binaries? > > That's nice, but I think that this could be complementary to the > > instructions signature. > > Imagine a validated binary being exploited somehow at runtime, that > > could be vector of malicious BPF program load. > > Can't we have both available, and use one or other, or even both > > together depending on the use case? > > I'll let John comment. I'll give the outline of the argument here. I do not believe signing BPF instructions for real programs provides much additional security. Given most real programs if the application or loader is exploited at runtime we have all sorts of trouble. First simply verifying the program doesn't prevent malicious use of the program. If its in the network program this means DDOS, data exfiltration, mitm attacks, many other possibilities. If its enforcement program most enforcement actions are programmed from this application so system security is lost already. If its observability application simply drops/manipulates observations that it wants. I don't know of any useful programs that exist in isolation without user space input and output as a critical component. If its not a privileged user, well it better not be doing anything critical anyways or disabled outright for the security focused. Many critical programs can't be signed by the nature of the program. Optimizing network app generates optimized code at runtime. Observability tools JIT the code on the fly, similarly enforcement tools will do the same. I think the power of being able to optimize JIT the code in application and give to the kernel is something we will see more and more of. Saying I'm only going to accept signed programs, for a distribution or something other than niche use case, is non starter IMO because it breaks so many real use cases. We should encourage these optimizing use cases as I see it as critical to performance and keeping overhead low. From a purely security standpoint I believe you are better off defining characteristics an application is allowed to have. For example allowed to probe kernel memory, make these helpers calls, have this many instructions, use this much memory, this much cpu, etc. This lets you sandbox a BPF application (both user space and kernel side) much nicer than any signing will allow. If we want to 'sign' programs we should do that from a BPF program directly where other metadata can be included in the policy. For example having a hash of the program loaded along with the calls made and process allows for rich policy decisions. I have other use cases that need a hash/signature for data blobs, so its on my todo list but not at the top yet. But, being able to verify arbitrary blob of data from BPF feels like a useful operation to me in general. The fact in your case its a set of eBPF insns and in my case its some key in a network header shouldn't matter. The series as is, scanned commit descriptions, is going to break lots of in-use-today programs if it was ever enabled. And is not as flexible (can't support bpftrace, etc.) or powerful (can't consider fine grained policy decisions) as above. Add a function we can hook after verify (or before up for debate) and helpers to verify signatures and/or generate hashes and we get a better more general solution. And it can also solve your use case even if I believe its not useful and may break many BPF users running bpftrace, libbpf, etc. Thanks, John
On Fri, 2021-12-03 at 19:39 -0800, John Fastabend wrote: > Alexei Starovoitov wrote: > > On Fri, Dec 3, 2021 at 4:42 PM Matteo Croce > > <mcroce@linux.microsoft.com> wrote: > > > > > > On Fri, Dec 3, 2021 at 11:20 PM Alexei Starovoitov > > > <alexei.starovoitov@gmail.com> wrote: > > > > > > > > On Fri, Dec 3, 2021 at 2:06 PM Luca Boccassi <bluca@debian.org> > > > > wrote: > > > > > > > > > > On Fri, 2021-12-03 at 11:37 -0800, Alexei Starovoitov wrote: > > > > > > On Fri, Dec 3, 2021 at 11:36 AM Matteo Croce > > > > > > <mcroce@linux.microsoft.com> wrote: > > > > > > > > > > > > > > On Fri, Dec 3, 2021 at 8:22 PM Alexei Starovoitov > > > > > > > <alexei.starovoitov@gmail.com> wrote: > > > > > > > > > > > > > > > > On Fri, Dec 3, 2021 at 11:18 AM Matteo Croce > > > > > > > > <mcroce@linux.microsoft.com> wrote: > > > > > > > > > > > > > > > > > > From: Matteo Croce <mcroce@microsoft.com> > > > > > > > > > > > > > > > > > > This series add signature verification for BPF files. > > > > > > > > > The first patch implements the signature validation > > > > > > > > > in the > > > > > > > > > kernel, > > > > > > > > > the second patch optionally makes the signature > > > > > > > > > mandatory, > > > > > > > > > the third adds signature generation to bpftool. > > > > > > > > > > > > > > > > Matteo, > > > > > > > > > > > > > > > > I think I already mentioned that it's no-go as-is. > > > > > > > > We've agreed to go with John's suggestion. > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > my previous attempt was loading a whole ELF file and > > > > > > > parsing it in > > > > > > > kernel. > > > > > > > In this series I just validate the instructions against a > > > > > > > signature, > > > > > > > as with kernel CO-RE libbpf doesn't need to mangle it. > > > > > > > > > > > > > > Which suggestion? I think I missed this one.. > > > > > > > > > > > > This talk and discussion: > > > > > > https://linuxplumbersconf.org/event/11/contributions/947/ > > > > > > > > > > Thanks for the link - but for those of us who don't have ~5 > > > > > hours to > > > > > watch a video recording, would you mind sharing a one line > > > > > summary, > > > > > please? Is there an alternative patch series implementing BPF > > > > > signing > > > > > that you can link us so that we can look at it? Just a link > > > > > or > > > > > googlable reference would be more than enough. > > > > > > > > It's not 5 hours and you have to read slides and watch > > > > John's presentation to follow the conversation. > > > > > > So, If I have understood correctly, the proposal is to validate > > > the > > > tools which loads the BPF (e.g. perf, ip) with fs-verity, and > > > only > > > allow BPF loading from those validated binaries? > > > That's nice, but I think that this could be complementary to the > > > instructions signature. > > > Imagine a validated binary being exploited somehow at runtime, > > > that > > > could be vector of malicious BPF program load. > > > Can't we have both available, and use one or other, or even both > > > together depending on the use case? > > > > I'll let John comment. > > I'll give the outline of the argument here. > > I do not believe signing BPF instructions for real programs provides > much additional security. Given most real programs if the application > or loader is exploited at runtime we have all sorts of trouble. First > simply verifying the program doesn't prevent malicious use of the > program. If its in the network program this means DDOS, data > exfiltration, > mitm attacks, many other possibilities. If its enforcement program > most enforcement actions are programmed from this application so > system > security is lost already. If its observability application simply > drops/manipulates observations that it wants. I don't know of any > useful programs that exist in isolation without user space input > and output as a critical component. If its not a privileged user, > well it better not be doing anything critical anyways or disabled > outright for the security focused. > > Many critical programs can't be signed by the nature of the program. > Optimizing network app generates optimized code at runtime. > Observability > tools JIT the code on the fly, similarly enforcement tools will do > the > same. I think the power of being able to optimize JIT the code in > application and give to the kernel is something we will see more and > more of. Saying I'm only going to accept signed programs, for a > distribution or something other than niche use case, is non starter > IMO because it breaks so many real use cases. We should encourage > these optimizing use cases as I see it as critical to performance > and keeping overhead low. > > From a purely security standpoint I believe you are better off > defining characteristics an application is allowed to have. For > example allowed to probe kernel memory, make these helpers calls, > have this many instructions, use this much memory, this much cpu, > etc. This lets you sandbox a BPF application (both user space and > kernel side) much nicer than any signing will allow. > > If we want to 'sign' programs we should do that from a BPF program > directly where other metadata can be included in the policy. For > example having a hash of the program loaded along with the calls > made and process allows for rich policy decisions. I have other > use cases that need a hash/signature for data blobs, so its on > my todo list but not at the top yet. But, being able to verify > arbitrary blob of data from BPF feels like a useful operation to me > in general. The fact in your case its a set of eBPF insns and in > my case its some key in a network header shouldn't matter. > > The series as is, scanned commit descriptions, is going to break > lots of in-use-today programs if it was ever enabled. And > is not as flexible (can't support bpftrace, etc.) or powerful > (can't consider fine grained policy decisions) as above. > > Add a function we can hook after verify (or before up for > debate) and helpers to verify signatures and/or generate > hashes and we get a better more general solution. And it can > also solve your use case even if I believe its not useful and > may break many BPF users running bpftrace, libbpf, etc. > > Thanks, > John Hello John, Thank you for the summary, this is much clearer. First of all, I think there's some misunderstanding: this series does not enable optional signatures by default, and does not enable mandatory signatures by default either. So I don't see how it would break existing use cases as you are saying? Unless I'm missing something? There's a kconfig to enable optional signatures - if they are there, they are verified, if they are not present then nothing different happens. Unless I am missing something, this should be backward compatible. This kconfig would likely be enabled in most use cases, just like optionally signed kernel modules are. Then there's a kconfig on top of that which makes signatures mandatory. I would not imagine this to be enabled in may cases, just in custom builds that have more stringent requirements. It certainly would not be enabled in generalist distros. Perhaps a more flexible way would be to introduce a sysctl, like fsverity has with 'fs.verity.require_signatures'? That would be just fine for our use case. Matteo can we do that instead in the next revision? Secondly, I understand that for your use case signing programs would not be the best approach. That's fine, and I'm glad you are working on an alternative that better fits your model, it will be very interesting to see how it looks like once implemented. But that model doesn't fit all cases. In our case at Microsoft, we absolutely want to be able to pre-define at build time a list of BPF programs that are allowed to be loaded, and reject anything else. Userspace processes in our case are mostly old and crufty c++ programs that can most likely be pwned by looking at them sideways, so they get locked down hard with multiple redundant layers and so on and so forth. But right now for BPF you only have a "can load BPF" or "cannot load BPF" knob, and that's it. This is not good enough: we need to be able to define a list of allowed payloads, and be able to enforce it, so when (not if) said processes do get tricked into loading something else, it will fail, despite having the capability of calling bpf(). Trying to define heuristics is also not good enough for us - creative malicious actors have a tendency to come up with ways to chain things that individually are allowed and benign, but combined in a way that you just couldn't foresee. It would certainly cover a lot of cases, but not all. A strictly pre-defined list of what is allowed to run and what is not is what we need for our case, so that we always know exactly what is going to run and what is not, and can deal with the consequences accordingly, without nasty surprises waiting around the corner. Now in my naive view the best way to achieve this is via signatures and certs, as it's a well-understood system, with processes already in place to revoke/rotate/etc, and it's already used for kmods. An alternative would be hard-coding hashes I guess, but that would be terribly inflexible. Now in terms of _how_ the signatures are done and validated, I'm sure there are multiple ways, and if some are better than what this series implements, then that's not an issue, it can be reworked. But the core requirement for us is: offline pre-defined list of what is allowed to run and what is not, with ability for hard enforcement that cannot be bypassed. Yes, you lose some features like JIT and so on: we don't care, we don't need those for our use cases. If others have different needs that's fine, this is all intended to be optional, not mandatory. There are obviously trade-offs, as always when security is involved, and each user can decide what's best for them. Hope this makes sense. Thanks!
Luca Boccassi wrote: cutting to just the relevant pieces here. [...] > > > I'll give the outline of the argument here. > > > > I do not believe signing BPF instructions for real programs provides > > much additional security. Given most real programs if the application > > or loader is exploited at runtime we have all sorts of trouble. First > > simply verifying the program doesn't prevent malicious use of the > > program. If its in the network program this means DDOS, data > > exfiltration, > > mitm attacks, many other possibilities. If its enforcement program > > most enforcement actions are programmed from this application so > > system > > security is lost already. If its observability application simply > > drops/manipulates observations that it wants. I don't know of any > > useful programs that exist in isolation without user space input > > and output as a critical component. If its not a privileged user, > > well it better not be doing anything critical anyways or disabled > > outright for the security focused. > > > > Many critical programs can't be signed by the nature of the program. > > Optimizing network app generates optimized code at runtime. > > Observability > > tools JIT the code on the fly, similarly enforcement tools will do > > the > > same. I think the power of being able to optimize JIT the code in > > application and give to the kernel is something we will see more and > > more of. Saying I'm only going to accept signed programs, for a > > distribution or something other than niche use case, is non starter > > IMO because it breaks so many real use cases. We should encourage > > these optimizing use cases as I see it as critical to performance > > and keeping overhead low. > > > > From a purely security standpoint I believe you are better off > > defining characteristics an application is allowed to have. For > > example allowed to probe kernel memory, make these helpers calls, > > have this many instructions, use this much memory, this much cpu, > > etc. This lets you sandbox a BPF application (both user space and > > kernel side) much nicer than any signing will allow. > > > > If we want to 'sign' programs we should do that from a BPF program > > directly where other metadata can be included in the policy. For > > example having a hash of the program loaded along with the calls > > made and process allows for rich policy decisions. I have other > > use cases that need a hash/signature for data blobs, so its on > > my todo list but not at the top yet. But, being able to verify > > arbitrary blob of data from BPF feels like a useful operation to me > > in general. The fact in your case its a set of eBPF insns and in > > my case its some key in a network header shouldn't matter. > > > > The series as is, scanned commit descriptions, is going to break > > lots of in-use-today programs if it was ever enabled. And > > is not as flexible (can't support bpftrace, etc.) or powerful > > (can't consider fine grained policy decisions) as above. > > > > Add a function we can hook after verify (or before up for > > debate) and helpers to verify signatures and/or generate > > hashes and we get a better more general solution. And it can > > also solve your use case even if I believe its not useful and > > may break many BPF users running bpftrace, libbpf, etc. > > > > Thanks, > > John > > Hello John, > > Thank you for the summary, this is much clearer. > > First of all, I think there's some misunderstanding: this series does > not enable optional signatures by default, and does not enable > mandatory signatures by default either. So I don't see how it would > break existing use cases as you are saying? Unless I'm missing > something? > > There's a kconfig to enable optional signatures - if they are there, > they are verified, if they are not present then nothing different > happens. Unless I am missing something, this should be backward > compatible. This kconfig would likely be enabled in most use cases, > just like optionally signed kernel modules are. Agree, without enforcement things should continue to work. > > Then there's a kconfig on top of that which makes signatures mandatory. > I would not imagine this to be enabled in may cases, just in custom > builds that have more stringent requirements. It certainly would not be > enabled in generalist distros. Perhaps a more flexible way would be to > introduce a sysctl, like fsverity has with > 'fs.verity.require_signatures'? That would be just fine for our use > case. Matteo can we do that instead in the next revision? We want to manage this from BPF side directly. It looks like policy decision and we have use cases that are not as simple as yes/no with global switch. For example, in k8s world this might be enabled via labels which are user specific per container policy. e.g. lockdown some containers more strictly than others. > > Secondly, I understand that for your use case signing programs would > not be the best approach. That's fine, and I'm glad you are working on > an alternative that better fits your model, it will be very interesting > to see how it looks like once implemented. But that model doesn't fit > all cases. In our case at Microsoft, we absolutely want to be able to > pre-define at build time a list of BPF programs that are allowed to be > loaded, and reject anything else. Userspace processes in our case are By building this into BPF you can get the 'reject anything else' policy and I get the metadata + reject/accept from the same hook. Its just your program can be very simple. > mostly old and crufty c++ programs that can most likely be pwned by > looking at them sideways, so they get locked down hard with multiple > redundant layers and so on and so forth. But right now for BPF you only > have a "can load BPF" or "cannot load BPF" knob, and that's it. This is > not good enough: we need to be able to define a list of allowed > payloads, and be able to enforce it, so when (not if) said processes do > get tricked into loading something else, it will fail, despite having Yikes, this is a bit scary from a sec point of view right? Are those programs read-only maps or can the C++ program also write into the maps and control plane. Assuming they do some critical functions then you really shouldn't be trusting them to not do all sorts of other horrible things. Anyways not too important to this discussion. I'll just reiterate (I think you get it though) that simply signing enforcement doesn't mean now BPF is safe. Further these programs have very high privileges and can do all sorts of things to the system. But, sure sig enforcement locks down one avenue of loading bogus program. > the capability of calling bpf(). Trying to define heuristics is also > not good enough for us - creative malicious actors have a tendency to > come up with ways to chain things that individually are allowed and > benign, but combined in a way that you just couldn't foresee. It would Sure, but I would argue some things can be very restrictive and generally useful. For example, never allow kernel memory read could be enforced from BPF side directly. Never allow pkt redirect, etc. > certainly cover a lot of cases, but not all. A strictly pre-defined > list of what is allowed to run and what is not is what we need for our > case, so that we always know exactly what is going to run and what is > not, and can deal with the consequences accordingly, without nasty > surprises waiting around the corner. Now in my naive view the best way > to achieve this is via signatures and certs, as it's a well-understood > system, with processes already in place to revoke/rotate/etc, and it's > already used for kmods. An alternative would be hard-coding hashes I > guess, but that would be terribly inflexible. Another option would be to load your programs at boot time, presumably with trusted boot enabled and then lock down BPF completely. Then ensure all your BPF 'programs' are read-only from user<->kernel interface and this should start looking fairly close to what you want and all programs are correct by root of trust back to trusted boot. Would assume you know what programs to load at boot though. May or may not be a big assumption depending on your env. > > Now in terms of _how_ the signatures are done and validated, I'm sure > there are multiple ways, and if some are better than what this series > implements, then that's not an issue, it can be reworked. But the core > requirement for us is: offline pre-defined list of what is allowed to > run and what is not, with ability for hard enforcement that cannot be > bypassed. Yes, you lose some features like JIT and so on: we don't > care, we don't need those for our use cases. If others have different > needs that's fine, this is all intended to be optional, not mandatory. > There are obviously trade-offs, as always when security is involved, > and each user can decide what's best for them. > > Hope this makes sense. Thanks! I think I understand your use case. When done as BPF helper you can get the behavior you want with a one line BPF program loaded at boot. int verify_all(struct bpf_prog **prog) { return verify_signature(prog->insn, prog->len * sizeof(struct bpf_insn), signature, KEYRING, BPF_SIGTYPE); } And I can write some more specific things as, int verify_blobs(void data) { int reject = verify_signature(data, data_len, sig, KEYRING, TYPE); struct policy_key *key = map_get_key(); return policy(key, reject); } map_get_key() looks into some datastor with the policy likely using 'current' to dig something up. It doesn't just apply to BPF progs we can use it on other executables more generally. And I get more interesting use cases like, allowing 'tc' programs unsigned, but requiring kernel memory reads to require signatures or any N other policies that may have value. Or only allowing my dbg user to run read-only programs, because the dbg maybe shouldn't ever be writing into packets, etc. Driving least privilege use cases in fine detail. By making it a BPF program we side step the debate where the kernel tries to get the 'right' policy for you, me, everyone now and in the future. The only way I can see to do this without getting N policies baked into the kernel and at M different hook points is via a BPF helper. Thanks, John
Em Mon, Dec 06, 2021 at 12:40:40PM -0800, John Fastabend escreveu: > I'll just reiterate (I think you get it though) that simply signing > enforcement doesn't mean now BPF is safe. Further these programs I think this was clear from the get go, at most this would help with fingerpointing :-) I.e. BPF signing is not about making things safer, its just an attempt to know who messed up. > have very high privileges and can do all sorts of things to the > system. But, sure sig enforcement locks down one avenue of loading > bogus program. > > the capability of calling bpf(). Trying to define heuristics is also > > not good enough for us - creative malicious actors have a tendency to > > come up with ways to chain things that individually are allowed and > > benign, but combined in a way that you just couldn't foresee. It would > Sure, but I would argue some things can be very restrictive and > generally useful. For example, never allow kernel memory read could be > enforced from BPF side directly. Never allow pkt redirect, etc. But this is something unrelated to BPF signing, right? Its something desirable, I'd say this will be at some point required, i.e. one more step in having BPF programs to be more like userspace apps, where you can limit all sorts of things it can do, programmatically, a BPF ulimit, hey, blimit? - Arnaldo
On Mon, 2021-12-06 at 12:40 -0800, John Fastabend wrote: > Luca Boccassi wrote: > > cutting to just the relevant pieces here. > > [...] > > > > > > I'll give the outline of the argument here. > > > > > > I do not believe signing BPF instructions for real programs > > > provides > > > much additional security. Given most real programs if the > > > application > > > or loader is exploited at runtime we have all sorts of trouble. > > > First > > > simply verifying the program doesn't prevent malicious use of the > > > program. If its in the network program this means DDOS, data > > > exfiltration, > > > mitm attacks, many other possibilities. If its enforcement > > > program > > > most enforcement actions are programmed from this application so > > > system > > > security is lost already. If its observability application > > > simply > > > drops/manipulates observations that it wants. I don't know of any > > > useful programs that exist in isolation without user space input > > > and output as a critical component. If its not a privileged user, > > > well it better not be doing anything critical anyways or disabled > > > outright for the security focused. > > > > > > Many critical programs can't be signed by the nature of the > > > program. > > > Optimizing network app generates optimized code at runtime. > > > Observability > > > tools JIT the code on the fly, similarly enforcement tools will > > > do > > > the > > > same. I think the power of being able to optimize JIT the code in > > > application and give to the kernel is something we will see more > > > and > > > more of. Saying I'm only going to accept signed programs, for a > > > distribution or something other than niche use case, is non > > > starter > > > IMO because it breaks so many real use cases. We should encourage > > > these optimizing use cases as I see it as critical to performance > > > and keeping overhead low. > > > > > > From a purely security standpoint I believe you are better off > > > defining characteristics an application is allowed to have. For > > > example allowed to probe kernel memory, make these helpers calls, > > > have this many instructions, use this much memory, this much cpu, > > > etc. This lets you sandbox a BPF application (both user space and > > > kernel side) much nicer than any signing will allow. > > > > > > If we want to 'sign' programs we should do that from a BPF > > > program > > > directly where other metadata can be included in the policy. For > > > example having a hash of the program loaded along with the calls > > > made and process allows for rich policy decisions. I have other > > > use cases that need a hash/signature for data blobs, so its on > > > my todo list but not at the top yet. But, being able to verify > > > arbitrary blob of data from BPF feels like a useful operation to > > > me > > > in general. The fact in your case its a set of eBPF insns and in > > > my case its some key in a network header shouldn't matter. > > > > > > The series as is, scanned commit descriptions, is going to break > > > lots of in-use-today programs if it was ever enabled. And > > > is not as flexible (can't support bpftrace, etc.) or powerful > > > (can't consider fine grained policy decisions) as above. > > > > > > Add a function we can hook after verify (or before up for > > > debate) and helpers to verify signatures and/or generate > > > hashes and we get a better more general solution. And it can > > > also solve your use case even if I believe its not useful and > > > may break many BPF users running bpftrace, libbpf, etc. > > > > > > Thanks, > > > John > > > > Hello John, > > > > Thank you for the summary, this is much clearer. > > > > First of all, I think there's some misunderstanding: this series > > does > > not enable optional signatures by default, and does not enable > > mandatory signatures by default either. So I don't see how it would > > break existing use cases as you are saying? Unless I'm missing > > something? > > > > There's a kconfig to enable optional signatures - if they are > > there, > > they are verified, if they are not present then nothing different > > happens. Unless I am missing something, this should be backward > > compatible. This kconfig would likely be enabled in most use cases, > > just like optionally signed kernel modules are. > > Agree, without enforcement things should continue to work. > > > > > Then there's a kconfig on top of that which makes signatures > > mandatory. > > I would not imagine this to be enabled in may cases, just in custom > > builds that have more stringent requirements. It certainly would > > not be > > enabled in generalist distros. Perhaps a more flexible way would be > > to > > introduce a sysctl, like fsverity has with > > 'fs.verity.require_signatures'? That would be just fine for our use > > case. Matteo can we do that instead in the next revision? > > We want to manage this from BPF side directly. It looks > like policy decision and we have use cases that are not as > simple as yes/no with global switch. For example, in k8s world > this might be enabled via labels which are user specific per > container > policy. e.g. lockdown some containers more strictly than others. > > > > > Secondly, I understand that for your use case signing programs > > would > > not be the best approach. That's fine, and I'm glad you are working > > on > > an alternative that better fits your model, it will be very > > interesting > > to see how it looks like once implemented. But that model doesn't > > fit > > all cases. In our case at Microsoft, we absolutely want to be able > > to > > pre-define at build time a list of BPF programs that are allowed to > > be > > loaded, and reject anything else. Userspace processes in our case > > are > > By building this into BPF you can get the 'reject anything else' > policy > and I get the metadata + reject/accept from the same hook. Its > just your program can be very simple. > > > mostly old and crufty c++ programs that can most likely be pwned by > > looking at them sideways, so they get locked down hard with > > multiple > > redundant layers and so on and so forth. But right now for BPF you > > only > > have a "can load BPF" or "cannot load BPF" knob, and that's it. > > This is > > not good enough: we need to be able to define a list of allowed > > payloads, and be able to enforce it, so when (not if) said > > processes do > > get tricked into loading something else, it will fail, despite > > having > > Yikes, this is a bit scary from a sec point of view right? Are those > programs read-only maps or can the C++ program also write into the > maps and control plane. Assuming they do some critical functions then > you really shouldn't be trusting them to not do all sorts of other > horrible things. Anyways not too important to this discussion. > > I'll just reiterate (I think you get it though) that simply signing > enforcement doesn't mean now BPF is safe. Further these programs > have very high privileges and can do all sorts of things to the > system. But, sure sig enforcement locks down one avenue of loading > bogus program. Oh it's terrifying - but business needs and all that. But Arnaldo is spot on - it's not strictly about what is more secure, but more about making it a known quantity. If we can prove what is allowed to run and what not before any machine has even booted (barring bugs in sig verification, of course) then the $org_security_team is satisfied and can sign off on enabling bpf. Otherwise we can keep dreaming. > > the capability of calling bpf(). Trying to define heuristics is > > also > > not good enough for us - creative malicious actors have a tendency > > to > > come up with ways to chain things that individually are allowed and > > benign, but combined in a way that you just couldn't foresee. It > > would > > Sure, but I would argue some things can be very restrictive and > generally useful. For example, never allow kernel memory read could > be > enforced from BPF side directly. Never allow pkt redirect, etc. > > > certainly cover a lot of cases, but not all. A strictly pre-defined > > list of what is allowed to run and what is not is what we need for > > our > > case, so that we always know exactly what is going to run and what > > is > > not, and can deal with the consequences accordingly, without nasty > > surprises waiting around the corner. Now in my naive view the best > > way > > to achieve this is via signatures and certs, as it's a well- > > understood > > system, with processes already in place to revoke/rotate/etc, and > > it's > > already used for kmods. An alternative would be hard-coding hashes > > I > > guess, but that would be terribly inflexible. > > Another option would be to load your programs at boot time, > presumably > with trusted boot enabled and then lock down BPF completely. Then > ensure all your BPF 'programs' are read-only from user<->kernel > interface and this should start looking fairly close to what you > want and all programs are correct by root of trust back to > trusted boot. Would assume you know what programs to load at boot > though. May or may not be a big assumption depending on your env. One of the use cases we have for BPF is on-demand diagnostics, so loading at boot and blocking afterwards would not work, I think. Environment is constrained in terms of resources, so don't want to load anything that is not needed. > > > > Now in terms of _how_ the signatures are done and validated, I'm > > sure > > there are multiple ways, and if some are better than what this > > series > > implements, then that's not an issue, it can be reworked. But the > > core > > requirement for us is: offline pre-defined list of what is allowed > > to > > run and what is not, with ability for hard enforcement that cannot > > be > > bypassed. Yes, you lose some features like JIT and so on: we don't > > care, we don't need those for our use cases. If others have > > different > > needs that's fine, this is all intended to be optional, not > > mandatory. > > There are obviously trade-offs, as always when security is > > involved, > > and each user can decide what's best for them. > > > > Hope this makes sense. Thanks! > > I think I understand your use case. When done as BPF helper you > can get the behavior you want with a one line BPF program > loaded at boot. > > int verify_all(struct bpf_prog **prog) { > return verify_signature(prog->insn, > prog->len * sizeof(struct bpf_insn), > signature, KEYRING, BPF_SIGTYPE); > } > > And I can write some more specific things as, > > int verify_blobs(void data) { > int reject = verify_signature(data, data_len, sig, KEYRING, TYPE); > struct policy_key *key = map_get_key(); > > return policy(key, reject); > } > > map_get_key() looks into some datastor with the policy likely using > 'current' to dig something up. It doesn't just apply to BPF progs > we can use it on other executables more generally. And I get more > interesting use cases like, allowing 'tc' programs unsigned, but > requiring kernel memory reads to require signatures or any N > other policies that may have value. Or only allowing my dbg user > to run read-only programs, because the dbg maybe shouldn't ever > be writing into packets, etc. Driving least privilege use cases > in fine detail. > > By making it a BPF program we side step the debate where the kernel > tries to get the 'right' policy for you, me, everyone now and in > the future. The only way I can see to do this without getting N > policies baked into the kernel and at M different hook points is via > a BPF helper. > > Thanks, > John Now this sounds like something that could work - we can prove that this could be loaded before any writable fs comes up anywhere, so in principle I think it would be acceptable and free of races. Matteo, we should talk about this tomorrow. And this requires some infrastructure work right? Is there a WIP git tree somewhere that we can test out? Thank you!
On Mon, 2021-12-06 at 22:59 +0000, Luca Boccassi wrote: > On Mon, 2021-12-06 at 12:40 -0800, John Fastabend wrote: > > Luca Boccassi wrote: > > > > cutting to just the relevant pieces here. > > > > [...] > > > > > > > > > I'll give the outline of the argument here. > > > > > > > > I do not believe signing BPF instructions for real programs > > > > provides > > > > much additional security. Given most real programs if the > > > > application > > > > or loader is exploited at runtime we have all sorts of trouble. > > > > First > > > > simply verifying the program doesn't prevent malicious use of the > > > > program. If its in the network program this means DDOS, data > > > > exfiltration, > > > > mitm attacks, many other possibilities. If its enforcement > > > > program > > > > most enforcement actions are programmed from this application so > > > > system > > > > security is lost already. If its observability application > > > > simply > > > > drops/manipulates observations that it wants. I don't know of any > > > > useful programs that exist in isolation without user space input > > > > and output as a critical component. If its not a privileged user, > > > > well it better not be doing anything critical anyways or disabled > > > > outright for the security focused. > > > > > > > > Many critical programs can't be signed by the nature of the > > > > program. > > > > Optimizing network app generates optimized code at runtime. > > > > Observability > > > > tools JIT the code on the fly, similarly enforcement tools will > > > > do > > > > the > > > > same. I think the power of being able to optimize JIT the code in > > > > application and give to the kernel is something we will see more > > > > and > > > > more of. Saying I'm only going to accept signed programs, for a > > > > distribution or something other than niche use case, is non > > > > starter > > > > IMO because it breaks so many real use cases. We should encourage > > > > these optimizing use cases as I see it as critical to performance > > > > and keeping overhead low. > > > > > > > > From a purely security standpoint I believe you are better off > > > > defining characteristics an application is allowed to have. For > > > > example allowed to probe kernel memory, make these helpers calls, > > > > have this many instructions, use this much memory, this much cpu, > > > > etc. This lets you sandbox a BPF application (both user space and > > > > kernel side) much nicer than any signing will allow. > > > > > > > > If we want to 'sign' programs we should do that from a BPF > > > > program > > > > directly where other metadata can be included in the policy. For > > > > example having a hash of the program loaded along with the calls > > > > made and process allows for rich policy decisions. I have other > > > > use cases that need a hash/signature for data blobs, so its on > > > > my todo list but not at the top yet. But, being able to verify > > > > arbitrary blob of data from BPF feels like a useful operation to > > > > me > > > > in general. The fact in your case its a set of eBPF insns and in > > > > my case its some key in a network header shouldn't matter. > > > > > > > > The series as is, scanned commit descriptions, is going to break > > > > lots of in-use-today programs if it was ever enabled. And > > > > is not as flexible (can't support bpftrace, etc.) or powerful > > > > (can't consider fine grained policy decisions) as above. > > > > > > > > Add a function we can hook after verify (or before up for > > > > debate) and helpers to verify signatures and/or generate > > > > hashes and we get a better more general solution. And it can > > > > also solve your use case even if I believe its not useful and > > > > may break many BPF users running bpftrace, libbpf, etc. > > > > > > > > Thanks, > > > > John > > > > > > Hello John, > > > > > > Thank you for the summary, this is much clearer. > > > > > > First of all, I think there's some misunderstanding: this series > > > does > > > not enable optional signatures by default, and does not enable > > > mandatory signatures by default either. So I don't see how it would > > > break existing use cases as you are saying? Unless I'm missing > > > something? > > > > > > There's a kconfig to enable optional signatures - if they are > > > there, > > > they are verified, if they are not present then nothing different > > > happens. Unless I am missing something, this should be backward > > > compatible. This kconfig would likely be enabled in most use cases, > > > just like optionally signed kernel modules are. > > > > Agree, without enforcement things should continue to work. > > > > > > > > Then there's a kconfig on top of that which makes signatures > > > mandatory. > > > I would not imagine this to be enabled in may cases, just in custom > > > builds that have more stringent requirements. It certainly would > > > not be > > > enabled in generalist distros. Perhaps a more flexible way would be > > > to > > > introduce a sysctl, like fsverity has with > > > 'fs.verity.require_signatures'? That would be just fine for our use > > > case. Matteo can we do that instead in the next revision? > > > > We want to manage this from BPF side directly. It looks > > like policy decision and we have use cases that are not as > > simple as yes/no with global switch. For example, in k8s world > > this might be enabled via labels which are user specific per > > container > > policy. e.g. lockdown some containers more strictly than others. > > > > > > > > Secondly, I understand that for your use case signing programs > > > would > > > not be the best approach. That's fine, and I'm glad you are working > > > on > > > an alternative that better fits your model, it will be very > > > interesting > > > to see how it looks like once implemented. But that model doesn't > > > fit > > > all cases. In our case at Microsoft, we absolutely want to be able > > > to > > > pre-define at build time a list of BPF programs that are allowed to > > > be > > > loaded, and reject anything else. Userspace processes in our case > > > are > > > > By building this into BPF you can get the 'reject anything else' > > policy > > and I get the metadata + reject/accept from the same hook. Its > > just your program can be very simple. > > > > > mostly old and crufty c++ programs that can most likely be pwned by > > > looking at them sideways, so they get locked down hard with > > > multiple > > > redundant layers and so on and so forth. But right now for BPF you > > > only > > > have a "can load BPF" or "cannot load BPF" knob, and that's it. > > > This is > > > not good enough: we need to be able to define a list of allowed > > > payloads, and be able to enforce it, so when (not if) said > > > processes do > > > get tricked into loading something else, it will fail, despite > > > having > > > > Yikes, this is a bit scary from a sec point of view right? Are those > > programs read-only maps or can the C++ program also write into the > > maps and control plane. Assuming they do some critical functions then > > you really shouldn't be trusting them to not do all sorts of other > > horrible things. Anyways not too important to this discussion. > > > > I'll just reiterate (I think you get it though) that simply signing > > enforcement doesn't mean now BPF is safe. Further these programs > > have very high privileges and can do all sorts of things to the > > system. But, sure sig enforcement locks down one avenue of loading > > bogus program. > > Oh it's terrifying - but business needs and all that. > But Arnaldo is spot on - it's not strictly about what is more secure, > but more about making it a known quantity. If we can prove what is > allowed to run and what not before any machine has even booted (barring > bugs in sig verification, of course) then the $org_security_team is > satisfied and can sign off on enabling bpf. Otherwise we can keep > dreaming. > > > > the capability of calling bpf(). Trying to define heuristics is > > > also > > > not good enough for us - creative malicious actors have a tendency > > > to > > > come up with ways to chain things that individually are allowed and > > > benign, but combined in a way that you just couldn't foresee. It > > > would > > > > Sure, but I would argue some things can be very restrictive and > > generally useful. For example, never allow kernel memory read could > > be > > enforced from BPF side directly. Never allow pkt redirect, etc. > > > > > certainly cover a lot of cases, but not all. A strictly pre-defined > > > list of what is allowed to run and what is not is what we need for > > > our > > > case, so that we always know exactly what is going to run and what > > > is > > > not, and can deal with the consequences accordingly, without nasty > > > surprises waiting around the corner. Now in my naive view the best > > > way > > > to achieve this is via signatures and certs, as it's a well- > > > understood > > > system, with processes already in place to revoke/rotate/etc, and > > > it's > > > already used for kmods. An alternative would be hard-coding hashes > > > I > > > guess, but that would be terribly inflexible. > > > > Another option would be to load your programs at boot time, > > presumably > > with trusted boot enabled and then lock down BPF completely. Then > > ensure all your BPF 'programs' are read-only from user<->kernel > > interface and this should start looking fairly close to what you > > want and all programs are correct by root of trust back to > > trusted boot. Would assume you know what programs to load at boot > > though. May or may not be a big assumption depending on your env. > > One of the use cases we have for BPF is on-demand diagnostics, so > loading at boot and blocking afterwards would not work, I think. > Environment is constrained in terms of resources, so don't want to load > anything that is not needed. > > > > > > > Now in terms of _how_ the signatures are done and validated, I'm > > > sure > > > there are multiple ways, and if some are better than what this > > > series > > > implements, then that's not an issue, it can be reworked. But the > > > core > > > requirement for us is: offline pre-defined list of what is allowed > > > to > > > run and what is not, with ability for hard enforcement that cannot > > > be > > > bypassed. Yes, you lose some features like JIT and so on: we don't > > > care, we don't need those for our use cases. If others have > > > different > > > needs that's fine, this is all intended to be optional, not > > > mandatory. > > > There are obviously trade-offs, as always when security is > > > involved, > > > and each user can decide what's best for them. > > > > > > Hope this makes sense. Thanks! > > > > I think I understand your use case. When done as BPF helper you > > can get the behavior you want with a one line BPF program > > loaded at boot. > > > > int verify_all(struct bpf_prog **prog) { > > return verify_signature(prog->insn, > > prog->len * sizeof(struct bpf_insn), > > signature, KEYRING, BPF_SIGTYPE); > > } > > > > And I can write some more specific things as, > > > > int verify_blobs(void data) { > > int reject = verify_signature(data, data_len, sig, KEYRING, TYPE); > > struct policy_key *key = map_get_key(); > > > > return policy(key, reject); > > } > > > > map_get_key() looks into some datastor with the policy likely using > > 'current' to dig something up. It doesn't just apply to BPF progs > > we can use it on other executables more generally. And I get more > > interesting use cases like, allowing 'tc' programs unsigned, but > > requiring kernel memory reads to require signatures or any N > > other policies that may have value. Or only allowing my dbg user > > to run read-only programs, because the dbg maybe shouldn't ever > > be writing into packets, etc. Driving least privilege use cases > > in fine detail. > > > > By making it a BPF program we side step the debate where the kernel > > tries to get the 'right' policy for you, me, everyone now and in > > the future. The only way I can see to do this without getting N > > policies baked into the kernel and at M different hook points is via > > a BPF helper. > > > > Thanks, > > John > > Now this sounds like something that could work - we can prove that this > could be loaded before any writable fs comes up anywhere, so in > principle I think it would be acceptable and free of races. Matteo, we > should talk about this tomorrow. > And this requires some infrastructure work right? Is there a WIP git > tree somewhere that we can test out? > > Thank you! One question more question: with the signature + kconfig approach, nothing can disable the signature check. But if the signature checker is itself a bpf program, is there/can there be anything stopping root from unloading it?
[...] > > > > Hope this makes sense. Thanks! > > > > > > I think I understand your use case. When done as BPF helper you > > > can get the behavior you want with a one line BPF program > > > loaded at boot. > > > > > > int verify_all(struct bpf_prog **prog) { > > > return verify_signature(prog->insn, > > > prog->len * sizeof(struct bpf_insn), > > > signature, KEYRING, BPF_SIGTYPE); > > > } > > > > > > And I can write some more specific things as, > > > > > > int verify_blobs(void data) { > > > int reject = verify_signature(data, data_len, sig, KEYRING, TYPE); > > > struct policy_key *key = map_get_key(); > > > > > > return policy(key, reject); > > > } > > > > > > map_get_key() looks into some datastor with the policy likely using > > > 'current' to dig something up. It doesn't just apply to BPF progs > > > we can use it on other executables more generally. And I get more > > > interesting use cases like, allowing 'tc' programs unsigned, but > > > requiring kernel memory reads to require signatures or any N > > > other policies that may have value. Or only allowing my dbg user > > > to run read-only programs, because the dbg maybe shouldn't ever > > > be writing into packets, etc. Driving least privilege use cases > > > in fine detail. > > > > > > By making it a BPF program we side step the debate where the kernel > > > tries to get the 'right' policy for you, me, everyone now and in > > > the future. The only way I can see to do this without getting N > > > policies baked into the kernel and at M different hook points is via > > > a BPF helper. > > > > > > Thanks, > > > John > > > > Now this sounds like something that could work - we can prove that this > > could be loaded before any writable fs comes up anywhere, so in > > principle I think it would be acceptable and free of races. Matteo, we > > should talk about this tomorrow. > > And this requires some infrastructure work right? Is there a WIP git > > tree somewhere that we can test out? > > > > Thank you! > I don't have a WIP tree, but I believe it should be fairly easy. First I would add a wrapper BPF helper for verify_signature() so we can call it from fentry/freturn context. That can be done on its own IMO as its a generally useful operation. Then I would stub a hook point into the BPF load path. The exact place to put this is going to have some debate I think, but I would place it immediately after the check_bpf call. With above two you have enough to do sig verification iiuc. Early boot loading I would have to check its current status. But I know folks have been working on it. Maybe its done? > One question more question: with the signature + kconfig approach, > nothing can disable the signature check. But if the signature checker > is itself a bpf program, is there/can there be anything stopping root > from unloading it? Interesting. Not that I'm aware of. Currently something with sufficient privileges could unload the program. Maybe we should have a flag so early boot programs can signal they shouldn't be unloaded ever. I would be OK with this and also seems generally useful. I have a case where I want to always set the socket cookie and we leave it running all the time. It would be nice if it came up and was pinned at boot. Maybe slightly better than a flag would be to have a new CAP support that only early boot has like CAP_BPF_EARLY. From my point of view this both seems doable with just some smallish changes on BPF side. Thanks, John
On Wed, 2021-12-08 at 12:17 -0800, John Fastabend wrote: > [...] > > > > > > Hope this makes sense. Thanks! > > > > > > > > I think I understand your use case. When done as BPF helper you > > > > can get the behavior you want with a one line BPF program > > > > loaded at boot. > > > > > > > > int verify_all(struct bpf_prog **prog) { > > > > return verify_signature(prog->insn, > > > > prog->len * sizeof(struct bpf_insn), > > > > signature, KEYRING, BPF_SIGTYPE); > > > > } > > > > > > > > And I can write some more specific things as, > > > > > > > > int verify_blobs(void data) { > > > > int reject = verify_signature(data, data_len, sig, KEYRING, TYPE); > > > > struct policy_key *key = map_get_key(); > > > > > > > > return policy(key, reject); > > > > } > > > > > > > > map_get_key() looks into some datastor with the policy likely using > > > > 'current' to dig something up. It doesn't just apply to BPF progs > > > > we can use it on other executables more generally. And I get more > > > > interesting use cases like, allowing 'tc' programs unsigned, but > > > > requiring kernel memory reads to require signatures or any N > > > > other policies that may have value. Or only allowing my dbg user > > > > to run read-only programs, because the dbg maybe shouldn't ever > > > > be writing into packets, etc. Driving least privilege use cases > > > > in fine detail. > > > > > > > > By making it a BPF program we side step the debate where the kernel > > > > tries to get the 'right' policy for you, me, everyone now and in > > > > the future. The only way I can see to do this without getting N > > > > policies baked into the kernel and at M different hook points is via > > > > a BPF helper. > > > > > > > > Thanks, > > > > John > > > > > > Now this sounds like something that could work - we can prove that this > > > could be loaded before any writable fs comes up anywhere, so in > > > principle I think it would be acceptable and free of races. Matteo, we > > > should talk about this tomorrow. > > > And this requires some infrastructure work right? Is there a WIP git > > > tree somewhere that we can test out? > > > > > > Thank you! > > > > I don't have a WIP tree, but I believe it should be fairly easy. > First I would add a wrapper BPF helper for verify_signature() so > we can call it from fentry/freturn context. That can be done on > its own IMO as its a generally useful operation. > > Then I would stub a hook point into the BPF load path. The exact > place to put this is going to have some debate I think, but I > would place it immediately after the check_bpf call. > > With above two you have enough to do sig verification iiuc. > > Early boot loading I would have to check its current status. But I know > folks have been working on it. Maybe its done? > > > One question more question: with the signature + kconfig approach, > > nothing can disable the signature check. But if the signature checker > > is itself a bpf program, is there/can there be anything stopping root > > from unloading it? > > Interesting. Not that I'm aware of. Currently something with sufficient > privileges could unload the program. Maybe we should have a flag so > early boot programs can signal they shouldn't be unloaded ever. I would > be OK with this and also seems generally useful. I have a case where > I want to always set the socket cookie and we leave it running all the > time. It would be nice if it came up and was pinned at boot. > > Maybe slightly better than a flag would be to have a new CAP support > that only early boot has like CAP_BPF_EARLY. From my point of view > this both seems doable with just some smallish changes on BPF side. > > Thanks, > John Thanks - again the means of enforcing this are not too important for our use case, as long as there is something that works reliably and can be attested.
From: Matteo Croce <mcroce@microsoft.com> This series add signature verification for BPF files. The first patch implements the signature validation in the kernel, the second patch optionally makes the signature mandatory, the third adds signature generation to bpftool. This only works with CO-RE programs. Matteo Croce (3): bpf: add signature to eBPF instructions bpf: add option to require BPF signature bpftool: add signature in skeleton crypto/asymmetric_keys/asymmetric_type.c | 1 + crypto/asymmetric_keys/pkcs7_verify.c | 7 +- include/linux/verification.h | 1 + include/uapi/linux/bpf.h | 2 + kernel/bpf/Kconfig | 14 ++ kernel/bpf/syscall.c | 51 +++++- tools/bpf/bpftool/Makefile | 14 +- tools/bpf/bpftool/gen.c | 33 ++++ tools/bpf/bpftool/main.c | 28 +++ tools/bpf/bpftool/main.h | 7 + tools/bpf/bpftool/sign.c | 218 +++++++++++++++++++++++ tools/include/uapi/linux/bpf.h | 2 + tools/lib/bpf/skel_internal.h | 4 + 13 files changed, 372 insertions(+), 10 deletions(-) create mode 100644 tools/bpf/bpftool/sign.c