mbox series

[bpf-next,v8,0/8] MAC and Audit policy using eBPF (KRSI)

Message ID 20200327192854.31150-1-kpsingh@chromium.org (mailing list archive)
Headers show
Series MAC and Audit policy using eBPF (KRSI) | expand

Message

KP Singh March 27, 2020, 7:28 p.m. UTC 
From: KP Singh <kpsingh@google.com>

# v7 -> v8

  https://lore.kernel.org/bpf/20200326142823.26277-1-kpsingh@chromium.org/

* Removed CAP_MAC_ADMIN check from bpf_lsm_verify_prog. LSMs can add it
  in their own bpf_prog hook. This can be revisited as a separate patch.
* Added Andrii and James' Ack/Review tags.
* Fixed an indentation issue and missing newlines in selftest error
  a cases.
* Updated a comment as suggested by Alexei.
* Updated the documentation to use the newer libbpf API and some other
  fixes.
* Rebase

# v6 -> v7

  https://lore.kernel.org/bpf/20200325152629.6904-1-kpsingh@chromium.org/

* Removed __weak from the LSM attachment nops per Kees' suggestion.
  Will send a separate patch (if needed) to update the noinline
  definition in include/linux/compiler_attributes.h.
* waitpid to wait specifically for the forked child in selftests.
* Comment format fixes in security/... as suggested by Casey.
* Added Acks from Kees and Andrii and Casey's Reviewed-by: tags to
  the respective patches.
* Rebase

# v5 -> v6

  https://lore.kernel.org/bpf/20200323164415.12943-1-kpsingh@chromium.org/

* Updated LSM_HOOK macro to define a default value and cleaned up the
  BPF LSM hook declarations.
* Added Yonghong's Acks and Kees' Reviewed-by tags.
* Simplification of the selftest code.
* Rebase and fixes suggested by Andrii and Yonghong and some other minor
  fixes noticed in internal review.

# v4 -> v5

  https://lore.kernel.org/bpf/20200220175250.10795-1-kpsingh@chromium.org/

* Removed static keys and special casing of BPF calls from the LSM
  framework.
* Initialized the BPF callbacks (nops) as proper LSM hooks.
* Updated to using the newly introduced BPF_TRAMP_MODIFY_RETURN
  trampolines in https://lkml.org/lkml/2020/3/4/877
* Addressed Andrii's feedback and rebased.

# v3 -> v4

* Moved away from allocating a separate security_hook_heads and adding a
  new special case for arch_prepare_bpf_trampoline to using BPF fexit
  trampolines called from the right place in the LSM hook and toggled by
  static keys based on the discussion in:

  https://lore.kernel.org/bpf/CAG48ez25mW+_oCxgCtbiGMX07g_ph79UOJa07h=o_6B6+Q-u5g@mail.gmail.com/

* Since the code does not deal with security_hook_heads anymore, it goes
  from "being a BPF LSM" to "BPF program attachment to LSM hooks".
* Added a new test case which ensures that the BPF programs' return value
  is reflected by the LSM hook.

# v2 -> v3 does not change the overall design and has some minor fixes:

* LSM_ORDER_LAST is introduced to represent the behaviour of the BPF LSM
* Fixed the inadvertent clobbering of the LSM Hook error codes
* Added GPL license requirement to the commit log
* The lsm_hook_idx is now the more conventional 0-based index
* Some changes were split into a separate patch ("Load btf_vmlinux only
  once per object")

  https://lore.kernel.org/bpf/20200117212825.11755-1-kpsingh@chromium.org/

* Addressed Andrii's feedback on the BTF implementation
* Documentation update for using generated vmlinux.h to simplify
  programs
* Rebase

# Changes since v1

  https://lore.kernel.org/bpf/20191220154208.15895-1-kpsingh@chromium.org

* Eliminate the requirement to maintain LSM hooks separately in
  security/bpf/hooks.h Use BPF trampolines to dynamically allocate
  security hooks
* Drop the use of securityfs as bpftool provides the required
  introspection capabilities.  Update the tests to use the bpf_skeleton
  and global variables
* Use O_CLOEXEC anonymous fds to represent BPF attachment in line with
  the other BPF programs with the possibility to use bpf program pinning
  in the future to provide "permanent attachment".
* Drop the logic based on prog names for handling re-attachment.
* Drop bpf_lsm_event_output from this series and send it as a separate
  patch.

# Motivation

Google does analysis of rich runtime security data to detect and thwart
threats in real-time. Currently, this is done in custom kernel modules
but we would like to replace this with something that's upstream and
useful to others.

The current kernel infrastructure for providing telemetry (Audit, Perf
etc.) is disjoint from access enforcement (i.e. LSMs).  Augmenting the
information provided by audit requires kernel changes to audit, its
policy language and user-space components. Furthermore, building a MAC
policy based on the newly added telemetry data requires changes to
various LSMs and their respective policy languages.

This patchset allows BPF programs to be attached to LSM hooks This
facilitates a unified and dynamic (not requiring re-compilation of the
kernel) audit and MAC policy.

# Why an LSM?

Linux Security Modules target security behaviours rather than the
kernel's API. For example, it's easy to miss out a newly added system
call for executing processes (eg. execve, execveat etc.) but the LSM
framework ensures that all process executions trigger the relevant hooks
irrespective of how the process was executed.

Allowing users to implement LSM hooks at runtime also benefits the LSM
eco-system by enabling a quick feedback loop from the security community
about the kind of behaviours that the LSM Framework should be targeting.

# How does it work?

The patchset introduces a new eBPF (https://docs.cilium.io/en/v1.6/bpf/)
program type BPF_PROG_TYPE_LSM which can only be attached to LSM hooks.
Loading and attachment of BPF programs requires CAP_SYS_ADMIN.

The new LSM registers nop functions (bpf_lsm_<hook_name>) as LSM hook
callbacks. Their purpose is to provide a definite point where BPF
programs can be attached as BPF_TRAMP_MODIFY_RETURN trampoline programs
for hooks that return an int, and BPF_TRAMP_FEXIT trampoline programs
for void LSM hooks.

Audit logs can be written using a format chosen by the eBPF program to
the perf events buffer or to global eBPF variables or maps and can be
further processed in user-space.

# BTF Based Design

The current design uses BTF:

  * https://facebookmicrosites.github.io/bpf/blog/2018/11/14/btf-enhancement.html
  * https://lwn.net/Articles/803258

which allows verifiable read-only structure accesses by field names
rather than fixed offsets. This allows accessing the hook parameters
using a dynamically created context which provides a certain degree of
ABI stability:


// Only declare the structure and fields intended to be used
// in the program
struct vm_area_struct {
  unsigned long vm_start;
} __attribute__((preserve_access_index));

// Declare the eBPF program mprotect_audit which attaches to
// to the file_mprotect LSM hook and accepts three arguments.
SEC("lsm/file_mprotect")
int BPF_PROG(mprotect_audit, struct vm_area_struct *vma,
       unsigned long reqprot, unsigned long prot, int ret)
{
  unsigned long vm_start = vma->vm_start;

  return 0;
}

By relocating field offsets, BTF makes a large portion of kernel data
structures readily accessible across kernel versions without requiring a
large corpus of BPF helper functions and requiring recompilation with
every kernel version. The BTF type information is also used by the BPF
verifier to validate memory accesses within the BPF program and also
prevents arbitrary writes to the kernel memory.

The limitations of BTF compatibility are described in BPF Co-Re
(http://vger.kernel.org/bpfconf2019_talks/bpf-core.pdf, i.e. field
renames, #defines and changes to the signature of LSM hooks).  This
design imposes that the MAC policy (eBPF programs) be updated when the
inspected kernel structures change outside of BTF compatibility
guarantees. In practice, this is only required when a structure field
used by a current policy is removed (or renamed) or when the used LSM
hooks change. We expect the maintenance cost of these changes to be
acceptable as compared to the design presented in the RFC.

(https://lore.kernel.org/bpf/20190910115527.5235-1-kpsingh@chromium.org/).

# Usage Examples

A simple example and some documentation is included in the patchset.
In order to better illustrate the capabilities of the framework some
more advanced prototype (not-ready for review) code has also been
published separately:

* Logging execution events (including environment variables and
  arguments)
  https://github.com/sinkap/linux-krsi/blob/patch/v1/examples/samples/bpf/lsm_audit_env.c

* Detecting deletion of running executables:
  https://github.com/sinkap/linux-krsi/blob/patch/v1/examples/samples/bpf/lsm_detect_exec_unlink.c

* Detection of writes to /proc/<pid>/mem:
  https://github.com/sinkap/linux-krsi/blob/patch/v1/examples/samples/bpf/lsm_audit_env.c

We have updated Google's internal telemetry infrastructure and have
started deploying this LSM on our Linux Workstations. This gives us more
confidence in the real-world applications of such a system.

KP Singh (8):
  bpf: Introduce BPF_PROG_TYPE_LSM
  security: Refactor declaration of LSM hooks
  bpf: lsm: provide attachment points for BPF LSM programs
  bpf: lsm: Implement attach, detach and execution
  bpf: lsm: Initialize the BPF LSM hooks
  tools/libbpf: Add support for BPF_PROG_TYPE_LSM
  bpf: lsm: Add selftests for BPF_PROG_TYPE_LSM
  bpf: lsm: Add Documentation

 Documentation/bpf/bpf_lsm.rst                 | 146 ++++
 Documentation/bpf/index.rst                   |   1 +
 MAINTAINERS                                   |   1 +
 include/linux/bpf.h                           |   3 +
 include/linux/bpf_lsm.h                       |  33 +
 include/linux/bpf_types.h                     |   4 +
 include/linux/lsm_hook_defs.h                 | 381 +++++++++++
 include/linux/lsm_hooks.h                     | 628 +-----------------
 include/uapi/linux/bpf.h                      |   2 +
 init/Kconfig                                  |  12 +
 kernel/bpf/Makefile                           |   1 +
 kernel/bpf/bpf_lsm.c                          |  59 ++
 kernel/bpf/btf.c                              |   9 +-
 kernel/bpf/syscall.c                          |  57 +-
 kernel/bpf/trampoline.c                       |  17 +-
 kernel/bpf/verifier.c                         |  19 +-
 kernel/trace/bpf_trace.c                      |  12 +-
 security/Kconfig                              |  10 +-
 security/Makefile                             |   2 +
 security/bpf/Makefile                         |   5 +
 security/bpf/hooks.c                          |  26 +
 security/security.c                           |  41 +-
 tools/include/uapi/linux/bpf.h                |   2 +
 tools/lib/bpf/bpf.c                           |   3 +-
 tools/lib/bpf/libbpf.c                        |  39 +-
 tools/lib/bpf/libbpf.h                        |   4 +
 tools/lib/bpf/libbpf.map                      |   3 +
 tools/lib/bpf/libbpf_probes.c                 |   1 +
 tools/testing/selftests/bpf/config            |   2 +
 .../selftests/bpf/prog_tests/test_lsm.c       |  86 +++
 tools/testing/selftests/bpf/progs/lsm.c       |  48 ++
 31 files changed, 987 insertions(+), 670 deletions(-)
 create mode 100644 Documentation/bpf/bpf_lsm.rst
 create mode 100644 include/linux/bpf_lsm.h
 create mode 100644 include/linux/lsm_hook_defs.h
 create mode 100644 kernel/bpf/bpf_lsm.c
 create mode 100644 security/bpf/Makefile
 create mode 100644 security/bpf/hooks.c
 create mode 100644 tools/testing/selftests/bpf/prog_tests/test_lsm.c
 create mode 100644 tools/testing/selftests/bpf/progs/lsm.c

Comments

Daniel Borkmann March 28, 2020, 5:18 p.m. UTC | #1 
Hey KP,

On 3/27/20 8:28 PM, KP Singh wrote:
> From: KP Singh <kpsingh@google.com>
> 
> # v7 -> v8
> 
>    https://lore.kernel.org/bpf/20200326142823.26277-1-kpsingh@chromium.org/
> 
> * Removed CAP_MAC_ADMIN check from bpf_lsm_verify_prog. LSMs can add it
>    in their own bpf_prog hook. This can be revisited as a separate patch.
> * Added Andrii and James' Ack/Review tags.
> * Fixed an indentation issue and missing newlines in selftest error
>    a cases.
> * Updated a comment as suggested by Alexei.
> * Updated the documentation to use the newer libbpf API and some other
>    fixes.
> * Rebase
> 
> # v6 -> v7
> 
>    https://lore.kernel.org/bpf/20200325152629.6904-1-kpsingh@chromium.org/
> 
[...]
> KP Singh (8):
>    bpf: Introduce BPF_PROG_TYPE_LSM
>    security: Refactor declaration of LSM hooks
>    bpf: lsm: provide attachment points for BPF LSM programs
>    bpf: lsm: Implement attach, detach and execution
>    bpf: lsm: Initialize the BPF LSM hooks
>    tools/libbpf: Add support for BPF_PROG_TYPE_LSM
>    bpf: lsm: Add selftests for BPF_PROG_TYPE_LSM
>    bpf: lsm: Add Documentation

I was about to apply, but then I'm getting the following selftest issue on
the added LSM one, ptal:

# ./test_progs
[...]
#65/1 test_global_func1.o:OK
#65/2 test_global_func2.o:OK
#65/3 test_global_func3.o:OK
#65/4 test_global_func4.o:OK
#65/5 test_global_func5.o:OK
#65/6 test_global_func6.o:OK
#65/7 test_global_func7.o:OK
#65 test_global_funcs:OK
test_test_lsm:PASS:skel_load 0 nsec
test_test_lsm:PASS:attach 0 nsec
test_test_lsm:PASS:exec_cmd 0 nsec
test_test_lsm:FAIL:bprm_count bprm_count = 0
test_test_lsm:FAIL:heap_mprotect want errno=EPERM, got 22
#66 test_lsm:FAIL
test_test_overhead:PASS:obj_open_file 0 nsec
test_test_overhead:PASS:find_probe 0 nsec
test_test_overhead:PASS:find_probe 0 nsec
test_test_overhead:PASS:find_probe 0 nsec
test_test_overhead:PASS:find_probe 0 nsec
test_test_overhead:PASS:find_probe 0 nsec
Caught signal #11!
Stack trace:
./test_progs(crash_handler+0x31)[0x56100f25eb51]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f9d8d225890]
/lib/x86_64-linux-gnu/libc.so.6(+0x18ef2d)[0x7f9d8cfb0f2d]
/lib/x86_64-linux-gnu/libc.so.6(__libc_calloc+0x372)[0x7f9d8cebc3a2]
/usr/local/lib/libelf.so.1(+0x33ce)[0x7f9d8d85a3ce]
/usr/local/lib/libelf.so.1(+0x3fb2)[0x7f9d8d85afb2]
./test_progs(btf__parse_elf+0x15d)[0x56100f27a141]
./test_progs(libbpf_find_kernel_btf+0x169)[0x56100f27ee83]
./test_progs(+0x43906)[0x56100f266906]
./test_progs(bpf_object__load_xattr+0xe5)[0x56100f26e93c]
./test_progs(bpf_object__load+0x47)[0x56100f26eafd]
./test_progs(test_test_overhead+0x252)[0x56100f24a922]
./test_progs(main+0x212)[0x56100f22f772]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x7f9d8ce43b97]
./test_progs(_start+0x2a)[0x56100f22f8fa]
Segmentation fault (core dumped)
#

(Before the series, it runs through fine on my side.)

Thanks,
Daniel
KP Singh March 28, 2020, 7:56 p.m. UTC | #2 
On 28-Mar 18:18, Daniel Borkmann wrote:
> Hey KP,
> 
> On 3/27/20 8:28 PM, KP Singh wrote:
> > From: KP Singh <kpsingh@google.com>
> > 
> > # v7 -> v8
> > 
> >    https://lore.kernel.org/bpf/20200326142823.26277-1-kpsingh@chromium.org/
> > 
> > * Removed CAP_MAC_ADMIN check from bpf_lsm_verify_prog. LSMs can add it
> >    in their own bpf_prog hook. This can be revisited as a separate patch.
> > * Added Andrii and James' Ack/Review tags.
> > * Fixed an indentation issue and missing newlines in selftest error
> >    a cases.
> > * Updated a comment as suggested by Alexei.
> > * Updated the documentation to use the newer libbpf API and some other
> >    fixes.
> > * Rebase
> > 
> > # v6 -> v7
> > 
> >    https://lore.kernel.org/bpf/20200325152629.6904-1-kpsingh@chromium.org/
> > 
> [...]
> > KP Singh (8):
> >    bpf: Introduce BPF_PROG_TYPE_LSM
> >    security: Refactor declaration of LSM hooks
> >    bpf: lsm: provide attachment points for BPF LSM programs
> >    bpf: lsm: Implement attach, detach and execution
> >    bpf: lsm: Initialize the BPF LSM hooks
> >    tools/libbpf: Add support for BPF_PROG_TYPE_LSM
> >    bpf: lsm: Add selftests for BPF_PROG_TYPE_LSM
> >    bpf: lsm: Add Documentation
> 
> I was about to apply, but then I'm getting the following selftest issue on
> the added LSM one, ptal:
> 
> # ./test_progs
> [...]
> #65/1 test_global_func1.o:OK
> #65/2 test_global_func2.o:OK
> #65/3 test_global_func3.o:OK
> #65/4 test_global_func4.o:OK
> #65/5 test_global_func5.o:OK
> #65/6 test_global_func6.o:OK
> #65/7 test_global_func7.o:OK
> #65 test_global_funcs:OK
> test_test_lsm:PASS:skel_load 0 nsec
> test_test_lsm:PASS:attach 0 nsec
> test_test_lsm:PASS:exec_cmd 0 nsec
> test_test_lsm:FAIL:bprm_count bprm_count = 0
> test_test_lsm:FAIL:heap_mprotect want errno=EPERM, got 22

The test seems to pass for me [classic, "works on my machine" ;)]

  ./test_progs -t test_lsm
  #66 test_lsm:OK
  Summary: 1/0 PASSED, 0 SKIPPED, 0 FAILED

and also in the complete run of test_progs.

Since the attachment succeeds and the hook does not get called, it
seems like "bpf" LSM is not being initialized and the hook, although
present, does not get called.

This indicates that "bpf" is not in CONFIG_LSM. It should, however, be
there by default as we added it to default value of CONFIG_LSM and
also for other DEFAULT_SECURITY_* options.

Let me know if that's the case and it fixes it.

- KP

> #66 test_lsm:FAIL
> test_test_overhead:PASS:obj_open_file 0 nsec
> test_test_overhead:PASS:find_probe 0 nsec
> test_test_overhead:PASS:find_probe 0 nsec
> test_test_overhead:PASS:find_probe 0 nsec
> test_test_overhead:PASS:find_probe 0 nsec
> test_test_overhead:PASS:find_probe 0 nsec
> Caught signal #11!
> Stack trace:
> ./test_progs(crash_handler+0x31)[0x56100f25eb51]
> /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f9d8d225890]
> /lib/x86_64-linux-gnu/libc.so.6(+0x18ef2d)[0x7f9d8cfb0f2d]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_calloc+0x372)[0x7f9d8cebc3a2]
> /usr/local/lib/libelf.so.1(+0x33ce)[0x7f9d8d85a3ce]
> /usr/local/lib/libelf.so.1(+0x3fb2)[0x7f9d8d85afb2]
> ./test_progs(btf__parse_elf+0x15d)[0x56100f27a141]
> ./test_progs(libbpf_find_kernel_btf+0x169)[0x56100f27ee83]
> ./test_progs(+0x43906)[0x56100f266906]
> ./test_progs(bpf_object__load_xattr+0xe5)[0x56100f26e93c]
> ./test_progs(bpf_object__load+0x47)[0x56100f26eafd]
> ./test_progs(test_test_overhead+0x252)[0x56100f24a922]
> ./test_progs(main+0x212)[0x56100f22f772]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x7f9d8ce43b97]
> ./test_progs(_start+0x2a)[0x56100f22f8fa]
> Segmentation fault (core dumped)
> #
> 
> (Before the series, it runs through fine on my side.)
> 
> Thanks,
> Daniel
Kees Cook March 28, 2020, 9:50 p.m. UTC | #3 
On Sat, Mar 28, 2020 at 08:56:36PM +0100, KP Singh wrote:
> Since the attachment succeeds and the hook does not get called, it
> seems like "bpf" LSM is not being initialized and the hook, although
> present, does not get called.
> 
> This indicates that "bpf" is not in CONFIG_LSM. It should, however, be
> there by default as we added it to default value of CONFIG_LSM and
> also for other DEFAULT_SECURITY_* options.
> 
> Let me know if that's the case and it fixes it.

Is the selftest expected to at least fail cleanly (i.e. not segfault)
when the BPF LSF is not built into the kernel?
KP Singh March 28, 2020, 10:30 p.m. UTC | #4 
On Sat, Mar 28, 2020 at 10:50 PM Kees Cook <keescook@chromium.org> wrote:
>
> On Sat, Mar 28, 2020 at 08:56:36PM +0100, KP Singh wrote:
> > Since the attachment succeeds and the hook does not get called, it
> > seems like "bpf" LSM is not being initialized and the hook, although
> > present, does not get called.
> >
> > This indicates that "bpf" is not in CONFIG_LSM. It should, however, be
> > there by default as we added it to default value of CONFIG_LSM and
> > also for other DEFAULT_SECURITY_* options.
> >
> > Let me know if that's the case and it fixes it.
>
> Is the selftest expected to at least fail cleanly (i.e. not segfault)

I am not sure where the crash comes from, it does not look like it's test_lsm,
it seems to happen in test_overhead. Both seem to run fine for me.

- KP

> when the BPF LSF is not built into the kernel?
>
> --
> Kees Cook
KP Singh March 29, 2020, 12:07 a.m. UTC | #5 
On 28-Mar 23:30, KP Singh wrote:
> On Sat, Mar 28, 2020 at 10:50 PM Kees Cook <keescook@chromium.org> wrote:
> >
> > On Sat, Mar 28, 2020 at 08:56:36PM +0100, KP Singh wrote:
> > > Since the attachment succeeds and the hook does not get called, it
> > > seems like "bpf" LSM is not being initialized and the hook, although
> > > present, does not get called.
> > >
> > > This indicates that "bpf" is not in CONFIG_LSM. It should, however, be
> > > there by default as we added it to default value of CONFIG_LSM and
> > > also for other DEFAULT_SECURITY_* options.
> > >
> > > Let me know if that's the case and it fixes it.
> >
> > Is the selftest expected to at least fail cleanly (i.e. not segfault)
> 
> I am not sure where the crash comes from, it does not look like it's test_lsm,
> it seems to happen in test_overhead. Both seem to run fine for me.

So I was able to reproduce the crash:

* Remove "bpf" from CONFIG_LSM

./test_progs -n 66,67
test_test_lsm:PASS:skel_load 0 nsec
test_test_lsm:PASS:attach 0 nsec
test_test_lsm:PASS:exec_cmd 0 nsec
test_test_lsm:FAIL:bprm_count bprm_count = 0
test_test_lsm:FAIL:heap_mprotect want errno=EPERM, got 0
#66 test_lsm:FAIL
Caught signal #11!
Stack trace:
./test_progs(crash_handler+0x1f)[0x55b7f9867acf]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x13520)[0x7fcf1467e520]
/lib/x86_64-linux-gnu/libc.so.6(+0x15f73d)[0x7fcf1460a73d]
/lib/x86_64-linux-gnu/libc.so.6(__libc_calloc+0x2ca)[0x7fcf1453286a]
/usr/lib/x86_64-linux-gnu/libelf.so.1(+0x37

[snip]

* The crash went away when I removed the heap_mprotect call, now the BPF
  hook attached did not allow this operation, so it had no side-effects.
  Which lead me to believe the crash could be a side-effect of this
  operation. So I did:

--- a/tools/testing/selftests/bpf/prog_tests/test_lsm.c
+++ b/tools/testing/selftests/bpf/prog_tests/test_lsm.c
@@ -29,7 +29,7 @@ int heap_mprotect(void)
        if (buf == NULL)
                return -ENOMEM;

-       ret = mprotect(buf, sz, PROT_READ | PROT_EXEC);
+       ret = mprotect(buf, sz, PROT_READ | PROT_WRITE | PROT_EXEC);
        free(buf);
        return ret;
 }

and the crash went away. Which made me realize that the free
operation does not like memory without PROT_WRITE, So I did this:

diff --git a/tools/testing/selftests/bpf/prog_tests/test_lsm.c b/tools/testing/selftests/bpf/prog_tests/test_lsm.c
index fcd839e88540..78f125cc09b3 100644
--- a/tools/testing/selftests/bpf/prog_tests/test_lsm.c
+++ b/tools/testing/selftests/bpf/prog_tests/test_lsm.c
@@ -30,7 +30,7 @@ int heap_mprotect(void)
                return -ENOMEM;

        ret = mprotect(buf, sz, PROT_READ | PROT_EXEC);
-       free(buf);
+       // free(buf);
        return ret;
 }

and the crash went away as well. So it indeed was a combination of:

* CONFIG_LSM not enabling the hook
* mprotect marking the memory as non-writeable
* free being called on the memory.

I will send a v9 which has the PROT_WRITE on the mprotect. Thanks
for noticing this!

- KP

> 
> - KP
> 
> > when the BPF LSF is not built into the kernel?
> >
> > --
> > Kees Cook
Daniel Borkmann March 29, 2020, 12:15 a.m. UTC | #6 
On 3/29/20 1:07 AM, KP Singh wrote:
> On 28-Mar 23:30, KP Singh wrote:
>> On Sat, Mar 28, 2020 at 10:50 PM Kees Cook <keescook@chromium.org> wrote:
>>>
>>> On Sat, Mar 28, 2020 at 08:56:36PM +0100, KP Singh wrote:
>>>> Since the attachment succeeds and the hook does not get called, it
>>>> seems like "bpf" LSM is not being initialized and the hook, although
>>>> present, does not get called.
>>>>
>>>> This indicates that "bpf" is not in CONFIG_LSM. It should, however, be
>>>> there by default as we added it to default value of CONFIG_LSM and
>>>> also for other DEFAULT_SECURITY_* options.
>>>>
>>>> Let me know if that's the case and it fixes it.
>>>
>>> Is the selftest expected to at least fail cleanly (i.e. not segfault)
>>
>> I am not sure where the crash comes from, it does not look like it's test_lsm,
>> it seems to happen in test_overhead. Both seem to run fine for me.
> 
> So I was able to reproduce the crash:
> 
> * Remove "bpf" from CONFIG_LSM
> 
> ./test_progs -n 66,67
> test_test_lsm:PASS:skel_load 0 nsec
> test_test_lsm:PASS:attach 0 nsec
> test_test_lsm:PASS:exec_cmd 0 nsec
> test_test_lsm:FAIL:bprm_count bprm_count = 0
> test_test_lsm:FAIL:heap_mprotect want errno=EPERM, got 0
> #66 test_lsm:FAIL
> Caught signal #11!
> Stack trace:
> ./test_progs(crash_handler+0x1f)[0x55b7f9867acf]
> /lib/x86_64-linux-gnu/libpthread.so.0(+0x13520)[0x7fcf1467e520]
> /lib/x86_64-linux-gnu/libc.so.6(+0x15f73d)[0x7fcf1460a73d]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_calloc+0x2ca)[0x7fcf1453286a]
> /usr/lib/x86_64-linux-gnu/libelf.so.1(+0x37
> 
> [snip]
> 
> * The crash went away when I removed the heap_mprotect call, now the BPF
>    hook attached did not allow this operation, so it had no side-effects.
>    Which lead me to believe the crash could be a side-effect of this
>    operation. So I did:
> 
> --- a/tools/testing/selftests/bpf/prog_tests/test_lsm.c
> +++ b/tools/testing/selftests/bpf/prog_tests/test_lsm.c
> @@ -29,7 +29,7 @@ int heap_mprotect(void)
>          if (buf == NULL)
>                  return -ENOMEM;
> 
> -       ret = mprotect(buf, sz, PROT_READ | PROT_EXEC);
> +       ret = mprotect(buf, sz, PROT_READ | PROT_WRITE | PROT_EXEC);
>          free(buf);
>          return ret;
>   }
> 
> and the crash went away. Which made me realize that the free
> operation does not like memory without PROT_WRITE, So I did this:
> 
> diff --git a/tools/testing/selftests/bpf/prog_tests/test_lsm.c b/tools/testing/selftests/bpf/prog_tests/test_lsm.c
> index fcd839e88540..78f125cc09b3 100644
> --- a/tools/testing/selftests/bpf/prog_tests/test_lsm.c
> +++ b/tools/testing/selftests/bpf/prog_tests/test_lsm.c
> @@ -30,7 +30,7 @@ int heap_mprotect(void)
>                  return -ENOMEM;
> 
>          ret = mprotect(buf, sz, PROT_READ | PROT_EXEC);
> -       free(buf);
> +       // free(buf);
>          return ret;
>   }
> 
> and the crash went away as well. So it indeed was a combination of:
> 
> * CONFIG_LSM not enabling the hook
> * mprotect marking the memory as non-writeable
> * free being called on the memory.
> 
> I will send a v9 which has the PROT_WRITE on the mprotect. Thanks
> for noticing this!

And also explains the stack trace for __libc_calloc() where it's trying to zero the
area later on.

Thanks for the quick debugging,
Daniel