| Message ID | 20200519151457.31618-2-william.c.roberts@intel.com (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
| Series | ci: run SE Linux kernel test suite | expand |
On Tue, May 19, 2020 at 11:15 AM <bill.c.roberts@gmail.com> wrote: > From: William Roberts <william.c.roberts@intel.com> > > The current CI runs the userspace tooling and librariers against > policy files, but cannot test against an SE Linux enabled kernel. Thus, > some tests are not being done in the CI. Travis, unfortunately only > provides Ubuntu images, so in order to run against a modern distro with > SE Linux in enforcing mode, we need to launch a KVM with something like > Fedora. > > This patch enables this support by launching a Fedora32 Cloud Image with > the selinux userspace library passed on from the travis clone, it then > builds and replaces the current selinux bits on the Fedora image and > runs the SE Linux testsuite. > > Signed-off-by: William Roberts <william.c.roberts@intel.com> > --- > .travis.yml | 8 +++ > scripts/ci/README.md | 8 +++ > scripts/ci/fedora-test-runner.sh | 79 +++++++++++++++++++++ > scripts/ci/travis-kvm-setup.sh | 113 +++++++++++++++++++++++++++++++ > 4 files changed, 208 insertions(+) > create mode 100644 scripts/ci/README.md > create mode 100755 scripts/ci/fedora-test-runner.sh > create mode 100755 scripts/ci/travis-kvm-setup.sh ... > diff --git a/scripts/ci/fedora-test-runner.sh b/scripts/ci/fedora-test-runner.sh > new file mode 100755 > index 000000000000..8d4b1bf7b8f5 > --- /dev/null > +++ b/scripts/ci/fedora-test-runner.sh > @@ -0,0 +1,79 @@ > +#!/usr/bin/env bash > + > +set -ev > + > +# CI Debug output if things go squirrely. > +getenforce > +id -Z > +nproc > +pwd Granted my automated testing experience has been with Fedora Rawhide and not the stable Fedora releases, but there have been occasions where the system is broken in some way which prevents the necessary test setup. My current approach is to put the system in permissive mode and leave it there until just before I run my tests. > +dnf install -y \ Another speaking from experience comment: you probably want to add "--allowerasing" and "--skip-broken" to the dnf command line. If you can cope with the extra overhead, I would even suggest a "dnf clean all -y" at the start. > + git \ > + audit-libs-devel \ > + bison \ > + bzip2-devel \ > + CUnit-devel \ > + diffutils \ > + flex \ > + gcc \ > + gettext \ > + glib2-devel \ > + make \ > + libcap-devel \ > + libcap-ng-devel \ > + pam-devel \ > + pcre-devel \ > + xmlto \ > + python3-devel \ > + ruby-devel \ > + swig \ > + perl-Test \ > + perl-Test-Harness \ > + perl-Test-Simple \ > + selinux-policy-devel \ > + gcc \ > + libselinux-devel \ > + net-tools \ > + netlabel_tools \ > + iptables \ > + lksctp-tools-devel \ > + attr \ > + libbpf-devel \ > + keyutils-libs-devel \ > + kernel-devel \ > + quota \ > + xfsprogs-devel \ > + libuuid-devel \ > + kernel-devel-$(uname -r) \ > + kernel-modules-$(uname -r) > + > +# > +# Move to selinux code and build > +# > +cd ~/selinux > + > +# Show HEAD commit for sanity checking > +git log -1 > + > +# > +# Build and replace userspace components > +# > +# Note: You can't use parallel builds here (make -jX), you'll end up > +# with race conditions that manifest like: > +# semanage_store.lo: file not recognized: file format not recognized > +# > +make LIBDIR=/usr/lib64 SHLIBDIR=/lib64 install install-pywrap relabel > + > +# > +# Get the selinux testsuite, but don't clone it in ~/selinux, move to ~ > +# first. > +# > +cd ~ > +git clone --depth=1 https://github.com/SELinuxProject/selinux-testsuite.git > +cd selinux-testsuite > + > +# > +# Run the test suite > +# > +make test > diff --git a/scripts/ci/travis-kvm-setup.sh b/scripts/ci/travis-kvm-setup.sh > new file mode 100755 > index 000000000000..19287fd21642 > --- /dev/null > +++ b/scripts/ci/travis-kvm-setup.sh > @@ -0,0 +1,113 @@ > +#!/usr/bin/env bash ... > +# > +# Great we have a host running, ssh into it. We specify -o so > +# we don't get blocked on asking to add the servers key to > +# our known_hosts. > +# > +ssh -o StrictHostKeyChecking=no "root@$ipaddy" "/root/selinux/$TEST_RUNNER" Depending on the tests, you'll get better output in the logs if you add "-tt" to the SSH command line. You may also want to add "-o LogLevel=QUIET" too. > + > +exit 0 Did you want to return the return value from SSH/$TEST_RUNNER?
On Tue, May 19, 2020 at 5:00 PM Paul Moore <paul@paul-moore.com> wrote: > > On Tue, May 19, 2020 at 11:15 AM <bill.c.roberts@gmail.com> wrote: > > From: William Roberts <william.c.roberts@intel.com> > > > > The current CI runs the userspace tooling and librariers against > > policy files, but cannot test against an SE Linux enabled kernel. Thus, > > some tests are not being done in the CI. Travis, unfortunately only > > provides Ubuntu images, so in order to run against a modern distro with > > SE Linux in enforcing mode, we need to launch a KVM with something like > > Fedora. > > > > This patch enables this support by launching a Fedora32 Cloud Image with > > the selinux userspace library passed on from the travis clone, it then > > builds and replaces the current selinux bits on the Fedora image and > > runs the SE Linux testsuite. > > > > Signed-off-by: William Roberts <william.c.roberts@intel.com> > > --- > > .travis.yml | 8 +++ > > scripts/ci/README.md | 8 +++ > > scripts/ci/fedora-test-runner.sh | 79 +++++++++++++++++++++ > > scripts/ci/travis-kvm-setup.sh | 113 +++++++++++++++++++++++++++++++ > > 4 files changed, 208 insertions(+) > > create mode 100644 scripts/ci/README.md > > create mode 100755 scripts/ci/fedora-test-runner.sh > > create mode 100755 scripts/ci/travis-kvm-setup.sh > > ... > > > diff --git a/scripts/ci/fedora-test-runner.sh b/scripts/ci/fedora-test-runner.sh > > new file mode 100755 > > index 000000000000..8d4b1bf7b8f5 > > --- /dev/null > > +++ b/scripts/ci/fedora-test-runner.sh > > @@ -0,0 +1,79 @@ > > +#!/usr/bin/env bash > > + > > +set -ev > > + > > +# CI Debug output if things go squirrely. > > +getenforce > > +id -Z > > +nproc > > +pwd > > Granted my automated testing experience has been with Fedora Rawhide > and not the stable Fedora releases, but there have been occasions > where the system is broken in some way which prevents the necessary > test setup. My current approach is to put the system in permissive > mode and leave it there until just before I run my tests. We can do that. > > > +dnf install -y \ > > Another speaking from experience comment: you probably want to add > "--allowerasing" and "--skip-broken" to the dnf command line. If you > can cope with the extra overhead, I would even suggest a "dnf clean > all -y" at the start. I don't see why we couldn't. Not really sure if it matters for those first two things, considering that were starting in the same state each time. Its not like we need to work on a system that's already been played with. I can add them and see what kind of increase it adds to the CI time. I'm sure it's tolerable (obviously set permissive is fast, I'm talking about the dnf stuff adding time). > > > + git \ > > + audit-libs-devel \ > > + bison \ > > + bzip2-devel \ > > + CUnit-devel \ > > + diffutils \ > > + flex \ > > + gcc \ > > + gettext \ > > + glib2-devel \ > > + make \ > > + libcap-devel \ > > + libcap-ng-devel \ > > + pam-devel \ > > + pcre-devel \ > > + xmlto \ > > + python3-devel \ > > + ruby-devel \ > > + swig \ > > + perl-Test \ > > + perl-Test-Harness \ > > + perl-Test-Simple \ > > + selinux-policy-devel \ > > + gcc \ > > + libselinux-devel \ > > + net-tools \ > > + netlabel_tools \ > > + iptables \ > > + lksctp-tools-devel \ > > + attr \ > > + libbpf-devel \ > > + keyutils-libs-devel \ > > + kernel-devel \ > > + quota \ > > + xfsprogs-devel \ > > + libuuid-devel \ > > + kernel-devel-$(uname -r) \ > > + kernel-modules-$(uname -r) > > + > > +# > > +# Move to selinux code and build > > +# > > +cd ~/selinux > > + > > +# Show HEAD commit for sanity checking > > +git log -1 > > + > > +# > > +# Build and replace userspace components > > +# > > +# Note: You can't use parallel builds here (make -jX), you'll end up > > +# with race conditions that manifest like: > > +# semanage_store.lo: file not recognized: file format not recognized > > +# > > +make LIBDIR=/usr/lib64 SHLIBDIR=/lib64 install install-pywrap relabel > > + > > +# > > +# Get the selinux testsuite, but don't clone it in ~/selinux, move to ~ > > +# first. > > +# > > +cd ~ > > +git clone --depth=1 https://github.com/SELinuxProject/selinux-testsuite.git > > +cd selinux-testsuite > > + > > +# > > +# Run the test suite > > +# > > +make test > > diff --git a/scripts/ci/travis-kvm-setup.sh b/scripts/ci/travis-kvm-setup.sh > > new file mode 100755 > > index 000000000000..19287fd21642 > > --- /dev/null > > +++ b/scripts/ci/travis-kvm-setup.sh > > @@ -0,0 +1,113 @@ > > +#!/usr/bin/env bash > > ... > > > +# > > +# Great we have a host running, ssh into it. We specify -o so > > +# we don't get blocked on asking to add the servers key to > > +# our known_hosts. > > +# > > +ssh -o StrictHostKeyChecking=no "root@$ipaddy" "/root/selinux/$TEST_RUNNER" > > Depending on the tests, you'll get better output in the logs if you > add "-tt" to the SSH command line. You may also want to add "-o > LogLevel=QUIET" too. +1 > > > + > > +exit 0 > > Did you want to return the return value from SSH/$TEST_RUNNER? set -e, ssh will cause the shell to exit with whatever code it returns that's not 0. > > -- > paul moore > www.paul-moore.com
On Tue, May 19, 2020 at 6:16 PM William Roberts <bill.c.roberts@gmail.com> wrote: > On Tue, May 19, 2020 at 5:00 PM Paul Moore <paul@paul-moore.com> wrote: > > On Tue, May 19, 2020 at 11:15 AM <bill.c.roberts@gmail.com> wrote: ... > > > +dnf install -y \ > > > > Another speaking from experience comment: you probably want to add > > "--allowerasing" and "--skip-broken" to the dnf command line. If you > > can cope with the extra overhead, I would even suggest a "dnf clean > > all -y" at the start. > > I don't see why we couldn't. > > Not really sure if it matters for those first two things, considering > that were starting > in the same state each time. Its not like we need to work on a system > that's already > been played with. FWIW, the VM I use for my automated kernel testing doesn't get played with (other than to fix dnf/Rawhide problems) and I can tell you from experience that Rawhide finds itself broken in new and exciting ways :) You are using a stable Fedora release so it probably won't be too bad, I'm just trying to share some of the lessons I've learned.
On Tue, May 19, 2020 at 5:23 PM Paul Moore <paul@paul-moore.com> wrote: > > On Tue, May 19, 2020 at 6:16 PM William Roberts > <bill.c.roberts@gmail.com> wrote: > > On Tue, May 19, 2020 at 5:00 PM Paul Moore <paul@paul-moore.com> wrote: > > > On Tue, May 19, 2020 at 11:15 AM <bill.c.roberts@gmail.com> wrote: > > ... > > > > > +dnf install -y \ > > > > > > Another speaking from experience comment: you probably want to add > > > "--allowerasing" and "--skip-broken" to the dnf command line. If you > > > can cope with the extra overhead, I would even suggest a "dnf clean > > > all -y" at the start. > > > > I don't see why we couldn't. > > > > Not really sure if it matters for those first two things, considering > > that were starting > > in the same state each time. Its not like we need to work on a system > > that's already > > been played with. > > FWIW, the VM I use for my automated kernel testing doesn't get played > with (other than to fix dnf/Rawhide problems) and I can tell you from > experience that Rawhide finds itself broken in new and exciting ways > :) > > You are using a stable Fedora release so it probably won't be too bad, > I'm just trying to share some of the lessons I've learned. > Sure, and I appreciate that. It actually doesn't add any time, to give you an idea the CI build time went from 14 to 11 mins. So that'll show you how much jitter there is. Ill send a V2 all fixed up shortly
On Wed, May 20, 2020 at 10:13 AM William Roberts <bill.c.roberts@gmail.com> wrote: > > On Tue, May 19, 2020 at 5:23 PM Paul Moore <paul@paul-moore.com> wrote: > > > > On Tue, May 19, 2020 at 6:16 PM William Roberts > > <bill.c.roberts@gmail.com> wrote: > > > On Tue, May 19, 2020 at 5:00 PM Paul Moore <paul@paul-moore.com> wrote: > > > > On Tue, May 19, 2020 at 11:15 AM <bill.c.roberts@gmail.com> wrote: > > > > ... > > > > > > > +dnf install -y \ > > > > > > > > Another speaking from experience comment: you probably want to add > > > > "--allowerasing" and "--skip-broken" to the dnf command line. If you > > > > can cope with the extra overhead, I would even suggest a "dnf clean > > > > all -y" at the start. > > > > > > I don't see why we couldn't. > > > > > > Not really sure if it matters for those first two things, considering > > > that were starting > > > in the same state each time. Its not like we need to work on a system > > > that's already > > > been played with. > > > > FWIW, the VM I use for my automated kernel testing doesn't get played > > with (other than to fix dnf/Rawhide problems) and I can tell you from > > experience that Rawhide finds itself broken in new and exciting ways > > :) > > > > You are using a stable Fedora release so it probably won't be too bad, > > I'm just trying to share some of the lessons I've learned. > > > > Sure, and I appreciate that. It actually doesn't add any time, to give > you an idea > the CI build time went from 14 to 11 mins. So that'll show you how much > jitter there is. Ill send a V2 all fixed up shortly Not true, I lied. I didn't have it back in enforcing mode so the test suite was skipped. I still don't see it adding much in the way of time.
diff --git a/.travis.yml b/.travis.yml index c36e721a5e1d..63a856672f9b 100644 --- a/.travis.yml +++ b/.travis.yml @@ -34,6 +34,14 @@ matrix: env: PYVER=python3.8 RUBYLIBVER=2.7 LINKER=gold - compiler: clang env: PYVER=python3.8 RUBYLIBVER=2.7 LINKER=bfd + include: + - compiler: gcc + env: TRAVIS_RUN_KVM=true + install: + - skip + before_script: + - skip + script: scripts/ci/travis-kvm-setup.sh # Use Travis-CI Ubuntu 18.04 Bionic Beaver, "full image" variant sudo: required diff --git a/scripts/ci/README.md b/scripts/ci/README.md new file mode 100644 index 000000000000..04a134a438c2 --- /dev/null +++ b/scripts/ci/README.md @@ -0,0 +1,8 @@ +# Continuous Integration Scripts + +The scripts under `scripts/ci` are designed specifically +for the Travis CI system. While nothing prevents you +from mimicking that environment and using them locally, +they are not applicable for general consumption. Any +thing in this directory should never be considered as +a stable API. diff --git a/scripts/ci/fedora-test-runner.sh b/scripts/ci/fedora-test-runner.sh new file mode 100755 index 000000000000..8d4b1bf7b8f5 --- /dev/null +++ b/scripts/ci/fedora-test-runner.sh @@ -0,0 +1,79 @@ +#!/usr/bin/env bash + +set -ev + +# CI Debug output if things go squirrely. +getenforce +id -Z +nproc +pwd + +dnf install -y \ + git \ + audit-libs-devel \ + bison \ + bzip2-devel \ + CUnit-devel \ + diffutils \ + flex \ + gcc \ + gettext \ + glib2-devel \ + make \ + libcap-devel \ + libcap-ng-devel \ + pam-devel \ + pcre-devel \ + xmlto \ + python3-devel \ + ruby-devel \ + swig \ + perl-Test \ + perl-Test-Harness \ + perl-Test-Simple \ + selinux-policy-devel \ + gcc \ + libselinux-devel \ + net-tools \ + netlabel_tools \ + iptables \ + lksctp-tools-devel \ + attr \ + libbpf-devel \ + keyutils-libs-devel \ + kernel-devel \ + quota \ + xfsprogs-devel \ + libuuid-devel \ + kernel-devel-$(uname -r) \ + kernel-modules-$(uname -r) + +# +# Move to selinux code and build +# +cd ~/selinux + +# Show HEAD commit for sanity checking +git log -1 + +# +# Build and replace userspace components +# +# Note: You can't use parallel builds here (make -jX), you'll end up +# with race conditions that manifest like: +# semanage_store.lo: file not recognized: file format not recognized +# +make LIBDIR=/usr/lib64 SHLIBDIR=/lib64 install install-pywrap relabel + +# +# Get the selinux testsuite, but don't clone it in ~/selinux, move to ~ +# first. +# +cd ~ +git clone --depth=1 https://github.com/SELinuxProject/selinux-testsuite.git +cd selinux-testsuite + +# +# Run the test suite +# +make test diff --git a/scripts/ci/travis-kvm-setup.sh b/scripts/ci/travis-kvm-setup.sh new file mode 100755 index 000000000000..19287fd21642 --- /dev/null +++ b/scripts/ci/travis-kvm-setup.sh @@ -0,0 +1,113 @@ +#!/usr/bin/env bash + +set -ev + +TEST_RUNNER="scripts/ci/fedora-test-runner.sh" + +# +# Travis gives us 7.5GB of RAM and two cores: +# https://docs.travis-ci.com/user/reference/overview/ +# +MEMORY=4096 +VCPUS=2 + +# Install these here so other builds don't have to wait on these deps to download and install +sudo apt-get install qemu-kvm libvirt-bin virtinst bridge-utils cpu-checker libguestfs-tools + +sudo usermod -a -G kvm,libvirt,libvirt-qemu $USER + +# Verify that KVM is working, useful if Travis every changes anything. +kvm-ok + +sudo systemctl enable libvirtd +sudo systemctl start libvirtd + +# Set up a key so we can ssh into the VM +ssh-keygen -N "" -f "$HOME/.ssh/id_rsa" + +# +# Get the Fedora Cloud Image, It is a base image that small and ready to go, extract it and modify it with virt-sysprep +# - https://alt.fedoraproject.org/en/verify.html +cd $HOME +wget https://download.fedoraproject.org/pub/fedora/linux/releases/32/Cloud/x86_64/images/Fedora-Cloud-Base-32-1.6.x86_64.raw.xz + +# Verify the image +curl https://getfedora.org/static/fedora.gpg | gpg --import +wget https://getfedora.org/static/checksums/Fedora-Cloud-32-1.6-x86_64-CHECKSUM +gpg --verify-files *-CHECKSUM +sha256sum --ignore-missing -c *-CHECKSUM + +# Extract the image +unxz -T0 Fedora-Cloud-Base-32-1.6.x86_64.raw.xz + +# Search is needed for $HOME so virt service can access the image file. +chmod a+x $HOME + +# +# Modify the virtual image to: +# - Enable a login, we just use root +# - Enable passwordless login +# - Force a relabel to fix labels on ssh keys +# +sudo virt-sysprep -a "$HOME/Fedora-Cloud-Base-32-1.6.x86_64.raw" \ + --root-password password:123456 \ + --hostname fedoravm \ + --append-line '/etc/ssh/sshd_config:PermitRootLogin yes' \ + --append-line '/etc/ssh/sshd_config:PubkeyAuthentication yes' \ + --mkdir /root/.ssh \ + --upload "$HOME/.ssh/id_rsa.pub:/root/.ssh/authorized_keys" \ + --chmod '0600:/root/.ssh/authorized_keys' \ + --run-command 'chown root:root /root/.ssh/authorized_keys' \ + --copy-in "$TRAVIS_BUILD_DIR:/root" \ + --network \ + --selinux-relabel + +# +# Now we create a domain by using virt-install. This not only creates the domain, but runs the VM as well +# It should be ready to go for ssh, once ssh starts. +# +sudo virt-install \ + --name fedoravm \ + --memory $MEMORY \ + --vcpus $VCPUS \ + --disk "$HOME/Fedora-Cloud-Base-32-1.6.x86_64.raw" \ + --import --noautoconsole + +# +# Here comes the tricky part, we have to figure out when the VM comes up AND we need the ip address for ssh. So we +# can check the net-dhcp leases, for our host. We have to poll, and we will poll for up 3 minutes in 6 second +# intervals, so 30 poll attempts (0-29 inclusive). I don't know of a better way to do this. +# +# We have a full reboot + relabel, so first sleep gets us close +# +sleep 30 +for i in $(seq 0 29); do + echo "loop $i" + sleep 6s + # Get the leases, but tee it so it's easier to debug + sudo virsh net-dhcp-leases default | tee dhcp-leases.txt + + # get our ipaddress + ipaddy=$(grep fedoravm dhcp-leases.txt | awk {'print $5'} | cut -d'/' -f 1-1) + if [ -n "$ipaddy" ]; then + # found it, we're done looking, print it for debug logs + echo "ipaddy: $ipaddy" + break + fi + # it's empty/not found, loop back and try again. +done + +# Did we find it? If not die. +if [ -z "$ipaddy" ]; then + echo "ipaddy zero length, exiting with error 1" + exit 1 +fi + +# +# Great we have a host running, ssh into it. We specify -o so +# we don't get blocked on asking to add the servers key to +# our known_hosts. +# +ssh -o StrictHostKeyChecking=no "root@$ipaddy" "/root/selinux/$TEST_RUNNER" + +exit 0