[v2] ci: run SELinux kernel test suite
diff mbox series

Message ID 20200520163421.27965-2-william.c.roberts@intel.com
State Changes Requested
Headers show
Series
  • [v2] ci: run SELinux kernel test suite
Related show

Commit Message

William Roberts May 20, 2020, 4:34 p.m. UTC
From: William Roberts <william.c.roberts@intel.com>

The current Travis CI runs the userspace tooling and libraries against
policy files, but cannot test against an SELinux enabled kernel. Thus,
some tests are not being done in the CI. Travis, unfortunately only
provides Ubuntu images, so in order to run against a modern distro with
SELinux in enforcing mode, we need to launch a KVM with something like
Fedora.

This patch enables this support by launching a Fedora32 Cloud Image with
the SELinux userspace library passed on from the Travis clone, it then
builds and replaces the current SELinux bits on the Fedora32 image and
runs the SELinux testsuite.

Signed-off-by: William Roberts <william.c.roberts@intel.com>
---
 .travis.yml                      |   8 +++
 scripts/ci/README.md             |   8 +++
 scripts/ci/fedora-test-runner.sh |  89 ++++++++++++++++++++++++
 scripts/ci/travis-kvm-setup.sh   | 113 +++++++++++++++++++++++++++++++
 4 files changed, 218 insertions(+)
 create mode 100644 scripts/ci/README.md
 create mode 100755 scripts/ci/fedora-test-runner.sh
 create mode 100755 scripts/ci/travis-kvm-setup.sh

Comments

Ondrej Mosnacek May 21, 2020, 8:50 a.m. UTC | #1
On Wed, May 20, 2020 at 6:34 PM <bill.c.roberts@gmail.com> wrote:
> From: William Roberts <william.c.roberts@intel.com>
>
> The current Travis CI runs the userspace tooling and libraries against
> policy files, but cannot test against an SELinux enabled kernel. Thus,
> some tests are not being done in the CI. Travis, unfortunately only
> provides Ubuntu images, so in order to run against a modern distro with
> SELinux in enforcing mode, we need to launch a KVM with something like
> Fedora.
>
> This patch enables this support by launching a Fedora32 Cloud Image with
> the SELinux userspace library passed on from the Travis clone, it then
> builds and replaces the current SELinux bits on the Fedora32 image and
> runs the SELinux testsuite.
>
> Signed-off-by: William Roberts <william.c.roberts@intel.com>

From the text above I infer that this patch is intended against the
userspace repo, right?

If so, I don't quite see the usefulness of running the
selinux-testsuite on every userspace change... It is mainly intended
for testing the kernel and only a small part of its running time is
spent on running (i.e. testing in a sense) the SELinux userspace
programs. Not to mention that in your patch it runs with the userspace
shipped in Fedora and not the version from the given commit...

However, it could be very useful if this was added to the testsuite's
CI instead so that it can verify that the testsuite patches don't
break something. But note that you'd need to modify the script a bit
to copy over the testsuite snapshot being tested to the VM and run
that, instead of the current master.

Anyway, thank you for working on this! I never realized that it could
be so easy to run a Fedora VM from Travis. If I find some time maybe I
can find some more ways to use this... For example we could run the
Fedora/RHEL SELinux userspace tests from [1] after installing (not yet
sure how) the userspace built from the currently tested userspace repo
commit.

[1] https://src.fedoraproject.org/tests/selinux/tree/master
Stephen Smalley May 21, 2020, 12:52 p.m. UTC | #2
On Thu, May 21, 2020 at 4:51 AM Ondrej Mosnacek <omosnace@redhat.com> wrote:
>
> On Wed, May 20, 2020 at 6:34 PM <bill.c.roberts@gmail.com> wrote:
> > From: William Roberts <william.c.roberts@intel.com>
> >
> > The current Travis CI runs the userspace tooling and libraries against
> > policy files, but cannot test against an SELinux enabled kernel. Thus,
> > some tests are not being done in the CI. Travis, unfortunately only
> > provides Ubuntu images, so in order to run against a modern distro with
> > SELinux in enforcing mode, we need to launch a KVM with something like
> > Fedora.
> >
> > This patch enables this support by launching a Fedora32 Cloud Image with
> > the SELinux userspace library passed on from the Travis clone, it then
> > builds and replaces the current SELinux bits on the Fedora32 image and
> > runs the SELinux testsuite.
> >
> > Signed-off-by: William Roberts <william.c.roberts@intel.com>
>
> From the text above I infer that this patch is intended against the
> userspace repo, right?
>
> If so, I don't quite see the usefulness of running the
> selinux-testsuite on every userspace change... It is mainly intended
> for testing the kernel and only a small part of its running time is
> spent on running (i.e. testing in a sense) the SELinux userspace
> programs. Not to mention that in your patch it runs with the userspace
> shipped in Fedora and not the version from the given commit...

Last I looked, his script builds and installs the userspace code on
top of the Fedora libraries and programs (make LIBDIR=... install...)
and then runs the testsuite.  That was my suggestion.  While it is the
kernel testsuite, it exercises a lot of SELinux userspace
functionality that isn't tested by the userspace tests.

>
> However, it could be very useful if this was added to the testsuite's
> CI instead so that it can verify that the testsuite patches don't
> break something. But note that you'd need to modify the script a bit
> to copy over the testsuite snapshot being tested to the VM and run
> that, instead of the current master.
>
> Anyway, thank you for working on this! I never realized that it could
> be so easy to run a Fedora VM from Travis. If I find some time maybe I
> can find some more ways to use this... For example we could run the
> Fedora/RHEL SELinux userspace tests from [1] after installing (not yet
> sure how) the userspace built from the currently tested userspace repo
> commit.
>
> [1] https://src.fedoraproject.org/tests/selinux/tree/master
Ondrej Mosnacek May 21, 2020, 12:58 p.m. UTC | #3
On Thu, May 21, 2020 at 2:52 PM Stephen Smalley
<stephen.smalley.work@gmail.com> wrote:
> On Thu, May 21, 2020 at 4:51 AM Ondrej Mosnacek <omosnace@redhat.com> wrote:
> >
> > On Wed, May 20, 2020 at 6:34 PM <bill.c.roberts@gmail.com> wrote:
> > > From: William Roberts <william.c.roberts@intel.com>
> > >
> > > The current Travis CI runs the userspace tooling and libraries against
> > > policy files, but cannot test against an SELinux enabled kernel. Thus,
> > > some tests are not being done in the CI. Travis, unfortunately only
> > > provides Ubuntu images, so in order to run against a modern distro with
> > > SELinux in enforcing mode, we need to launch a KVM with something like
> > > Fedora.
> > >
> > > This patch enables this support by launching a Fedora32 Cloud Image with
> > > the SELinux userspace library passed on from the Travis clone, it then
> > > builds and replaces the current SELinux bits on the Fedora32 image and
> > > runs the SELinux testsuite.
> > >
> > > Signed-off-by: William Roberts <william.c.roberts@intel.com>
> >
> > From the text above I infer that this patch is intended against the
> > userspace repo, right?
> >
> > If so, I don't quite see the usefulness of running the
> > selinux-testsuite on every userspace change... It is mainly intended
> > for testing the kernel and only a small part of its running time is
> > spent on running (i.e. testing in a sense) the SELinux userspace
> > programs. Not to mention that in your patch it runs with the userspace
> > shipped in Fedora and not the version from the given commit...
>
> Last I looked, his script builds and installs the userspace code on
> top of the Fedora libraries and programs (make LIBDIR=... install...)
> and then runs the testsuite.  That was my suggestion.

Ah, yes, I can see that line now. Sorry, somehow I missed it before.

> While it is the
> kernel testsuite, it exercises a lot of SELinux userspace
> functionality that isn't tested by the userspace tests.

OK, I suppose it's better than nothing...
William Roberts May 21, 2020, 2:11 p.m. UTC | #4
On Thu, May 21, 2020 at 7:58 AM Ondrej Mosnacek <omosnace@redhat.com> wrote:
>
> On Thu, May 21, 2020 at 2:52 PM Stephen Smalley
> <stephen.smalley.work@gmail.com> wrote:
> > On Thu, May 21, 2020 at 4:51 AM Ondrej Mosnacek <omosnace@redhat.com> wrote:
> > >
> > > On Wed, May 20, 2020 at 6:34 PM <bill.c.roberts@gmail.com> wrote:
> > > > From: William Roberts <william.c.roberts@intel.com>
> > > >
> > > > The current Travis CI runs the userspace tooling and libraries against
> > > > policy files, but cannot test against an SELinux enabled kernel. Thus,
> > > > some tests are not being done in the CI. Travis, unfortunately only
> > > > provides Ubuntu images, so in order to run against a modern distro with
> > > > SELinux in enforcing mode, we need to launch a KVM with something like
> > > > Fedora.
> > > >
> > > > This patch enables this support by launching a Fedora32 Cloud Image with
> > > > the SELinux userspace library passed on from the Travis clone, it then
> > > > builds and replaces the current SELinux bits on the Fedora32 image and
> > > > runs the SELinux testsuite.
> > > >
> > > > Signed-off-by: William Roberts <william.c.roberts@intel.com>
> > >
> > > From the text above I infer that this patch is intended against the
> > > userspace repo, right?
> > >
> > > If so, I don't quite see the usefulness of running the
> > > selinux-testsuite on every userspace change... It is mainly intended
> > > for testing the kernel and only a small part of its running time is
> > > spent on running (i.e. testing in a sense) the SELinux userspace
> > > programs. Not to mention that in your patch it runs with the userspace
> > > shipped in Fedora and not the version from the given commit...

Totally doable, and you just copy + edit those scripts to achieve
that. Currently we inject the travis repo
with patches, which is the selinux userspace, into the VM image with
virt-sysprep.
But if this was selinux-testsuite ci you would just virt-sysprep
inject it's repo.

> >
> > Last I looked, his script builds and installs the userspace code on
> > top of the Fedora libraries and programs (make LIBDIR=... install...)
> > and then runs the testsuite.  That was my suggestion.
>
> Ah, yes, I can see that line now. Sorry, somehow I missed it before.
>
> > While it is the
> > kernel testsuite, it exercises a lot of SELinux userspace
> > functionality that isn't tested by the userspace tests.
>
> OK, I suppose it's better than nothing...
>

Stephen pointed out the additional ways userspace gets tested, and
perhaps my title and description
of the patch could be better. But the main point is to increase the
test coverage
and perform the testing steps we expect are done before a release in
the CI. We should have
the testing coverage and the confidence to release userspace from master at any
point. We also have forward facing proof that tests are being executed
and we can make sure
nothing regresses.

My ultimate goal here, is to help make sure that if Petr gets hit by a
bus, releases will
move forward without worry and without any change in quality among the various
maintainers.

Additionally, we pick up some cross project testing and can find other
surprises.
Nicolas Iooss May 21, 2020, 7:54 p.m. UTC | #5
On Wed, May 20, 2020 at 6:34 PM <bill.c.roberts@gmail.com> wrote:
>
> From: William Roberts <william.c.roberts@intel.com>
>
> The current Travis CI runs the userspace tooling and libraries against
> policy files, but cannot test against an SELinux enabled kernel. Thus,
> some tests are not being done in the CI. Travis, unfortunately only
> provides Ubuntu images, so in order to run against a modern distro with
> SELinux in enforcing mode, we need to launch a KVM with something like
> Fedora.
>
> This patch enables this support by launching a Fedora32 Cloud Image with
> the SELinux userspace library passed on from the Travis clone, it then
> builds and replaces the current SELinux bits on the Fedora32 image and
> runs the SELinux testsuite.
>
> Signed-off-by: William Roberts <william.c.roberts@intel.com>

Hi,
Thanks for working on this. The CI scripts are quite easy to follow,
thanks to all the comments :)

Anyway, here are some suggestions/comments in order to make this patch
even better (if you want to do a v3):

* ShellCheck detects that some strings are not quoted, that using
*-CHECKSUM is dangerous (using ./*-CHECKSUM prevents ill things from
happening if a file in the directory happens to match the pattern and
start with a dash), etc. Most issues reported by ShellCheck are not
relevant here, but one:

    ipaddy=$(grep fedoravm dhcp-leases.txt | awk {'print $5'} | cut
-d'/' -f 1-1)
                                                 ^-- SC1083: This { is
literal. Check expression (missing ;/\n?) or quote it.

Using awk '{print $5}' (single quotes outside of the braces) seems to
better match the intent. Moreover the whole $(...) expression could be
double-quoted.

* fedora-test-runner.sh uses ~, which is expanded to /root as it does
not make sense to run this script as non-root user. In my humble
opinion, using /root instead of ~ makes the script even easier to
read.

* There is a comment about issues with -jX with "make
LIBDIR=/usr/lib64 SHLIBDIR=/lib64 install install-pywrap relabel". I
usually split the targets when using "make -jX" (ie. make -j$(nproc)
install && make -j$(nproc) install-pywrap && make relabel), so if not
using -jX is really an issue, this could be considered.

* Does the script runs "make test" in selinux/? I see the "make test"
for selinux-testsuite, but if the main point of fedora-test-runner.sh
is to run selinux's tests (in a VM), I missed the command that does
so.

Thanks,
Nicolas

> ---
>  .travis.yml                      |   8 +++
>  scripts/ci/README.md             |   8 +++
>  scripts/ci/fedora-test-runner.sh |  89 ++++++++++++++++++++++++
>  scripts/ci/travis-kvm-setup.sh   | 113 +++++++++++++++++++++++++++++++
>  4 files changed, 218 insertions(+)
>  create mode 100644 scripts/ci/README.md
>  create mode 100755 scripts/ci/fedora-test-runner.sh
>  create mode 100755 scripts/ci/travis-kvm-setup.sh
>
> diff --git a/.travis.yml b/.travis.yml
> index c36e721a5e1d..63a856672f9b 100644
> --- a/.travis.yml
> +++ b/.travis.yml
> @@ -34,6 +34,14 @@ matrix:
>        env: PYVER=python3.8 RUBYLIBVER=2.7 LINKER=gold
>      - compiler: clang
>        env: PYVER=python3.8 RUBYLIBVER=2.7 LINKER=bfd
> +  include:
> +    - compiler: gcc
> +      env: TRAVIS_RUN_KVM=true
> +      install:
> +        - skip
> +      before_script:
> +        - skip
> +      script: scripts/ci/travis-kvm-setup.sh
>
>  # Use Travis-CI Ubuntu 18.04 Bionic Beaver, "full image" variant
>  sudo: required
> diff --git a/scripts/ci/README.md b/scripts/ci/README.md
> new file mode 100644
> index 000000000000..04a134a438c2
> --- /dev/null
> +++ b/scripts/ci/README.md
> @@ -0,0 +1,8 @@
> +# Continuous Integration Scripts
> +
> +The scripts under `scripts/ci` are designed specifically
> +for the Travis CI system. While nothing prevents you
> +from mimicking that environment and using them locally,
> +they are not applicable for general consumption. Any
> +thing in this directory should never be considered as
> +a stable API.
> diff --git a/scripts/ci/fedora-test-runner.sh b/scripts/ci/fedora-test-runner.sh
> new file mode 100755
> index 000000000000..14bcf5fc469d
> --- /dev/null
> +++ b/scripts/ci/fedora-test-runner.sh
> @@ -0,0 +1,89 @@
> +#!/usr/bin/env bash
> +
> +set -ev
> +
> +# CI Debug output if things go squirrely.
> +getenforce
> +id -Z
> +nproc
> +pwd
> +
> +# Turn off enforcing for the setup to prevent any weirdness from breaking
> +# the CI.
> +setenforce 0
> +
> +dnf clean all -y
> +dnf install -y \
> +    --allowerasing \
> +    --skip-broken \
> +    git \
> +    audit-libs-devel \
> +    bison \
> +    bzip2-devel \
> +    CUnit-devel \
> +    diffutils \
> +    flex \
> +    gcc \
> +    gettext \
> +    glib2-devel \
> +    make \
> +    libcap-devel \
> +    libcap-ng-devel \
> +    pam-devel \
> +    pcre-devel \
> +    xmlto \
> +    python3-devel \
> +    ruby-devel \
> +    swig \
> +    perl-Test \
> +    perl-Test-Harness \
> +    perl-Test-Simple \
> +    selinux-policy-devel \
> +    gcc \
> +    libselinux-devel \
> +    net-tools \
> +    netlabel_tools \
> +    iptables \
> +    lksctp-tools-devel \
> +    attr \
> +    libbpf-devel \
> +    keyutils-libs-devel \
> +    kernel-devel \
> +    quota \
> +    xfsprogs-devel \
> +    libuuid-devel \
> +    kernel-devel-$(uname -r) \
> +    kernel-modules-$(uname -r)
> +
> +#
> +# Move to selinux code and build
> +#
> +cd ~/selinux
> +
> +# Show HEAD commit for sanity checking
> +git log -1
> +
> +#
> +# Build and replace userspace components
> +#
> +# Note: You can't use parallel builds here (make -jX), you'll end up
> +# with race conditions that manifest like:
> +# semanage_store.lo: file not recognized: file format not recognized
> +#
> +make LIBDIR=/usr/lib64 SHLIBDIR=/lib64 install install-pywrap relabel
> +
> +#
> +# Get the selinux testsuite, but don't clone it in ~/selinux, move to ~
> +# first.
> +#
> +cd ~
> +git clone --depth=1 https://github.com/SELinuxProject/selinux-testsuite.git
> +cd selinux-testsuite
> +
> +# The testsuite must be run in enforcing mode
> +setenforce 1
> +
> +#
> +# Run the test suite
> +#
> +make test
> diff --git a/scripts/ci/travis-kvm-setup.sh b/scripts/ci/travis-kvm-setup.sh
> new file mode 100755
> index 000000000000..66606e9d4a5b
> --- /dev/null
> +++ b/scripts/ci/travis-kvm-setup.sh
> @@ -0,0 +1,113 @@
> +#!/usr/bin/env bash
> +
> +set -ev
> +
> +TEST_RUNNER="scripts/ci/fedora-test-runner.sh"
> +
> +#
> +# Travis gives us 7.5GB of RAM and two cores:
> +# https://docs.travis-ci.com/user/reference/overview/
> +#
> +MEMORY=4096
> +VCPUS=2
> +
> +# Install these here so other builds don't have to wait on these deps to download and install
> +sudo apt-get install qemu-kvm libvirt-bin virtinst bridge-utils cpu-checker libguestfs-tools
> +
> +sudo usermod -a -G kvm,libvirt,libvirt-qemu $USER
> +
> +# Verify that KVM is working, useful if Travis every changes anything.
> +kvm-ok
> +
> +sudo systemctl enable libvirtd
> +sudo systemctl start libvirtd
> +
> +# Set up a key so we can ssh into the VM
> +ssh-keygen -N "" -f "$HOME/.ssh/id_rsa"
> +
> +#
> +# Get the Fedora Cloud Image, It is a base image that small and ready to go, extract it and modify it with virt-sysprep
> +#  - https://alt.fedoraproject.org/en/verify.html
> +cd $HOME
> +wget https://download.fedoraproject.org/pub/fedora/linux/releases/32/Cloud/x86_64/images/Fedora-Cloud-Base-32-1.6.x86_64.raw.xz
> +
> +# Verify the image
> +curl https://getfedora.org/static/fedora.gpg | gpg --import
> +wget https://getfedora.org/static/checksums/Fedora-Cloud-32-1.6-x86_64-CHECKSUM
> +gpg --verify-files *-CHECKSUM
> +sha256sum --ignore-missing -c *-CHECKSUM
> +
> +# Extract the image
> +unxz -T0 Fedora-Cloud-Base-32-1.6.x86_64.raw.xz
> +
> +# Search is needed for $HOME so virt service can access the image file.
> +chmod a+x $HOME
> +
> +#
> +# Modify the virtual image to:
> +#   - Enable a login, we just use root
> +#   - Enable passwordless login
> +#     - Force a relabel to fix labels on ssh keys
> +#
> +sudo virt-sysprep -a "$HOME/Fedora-Cloud-Base-32-1.6.x86_64.raw" \
> +  --root-password password:123456 \
> +  --hostname fedoravm \
> +  --append-line '/etc/ssh/sshd_config:PermitRootLogin yes' \
> +  --append-line '/etc/ssh/sshd_config:PubkeyAuthentication yes' \
> +  --mkdir /root/.ssh \
> +  --upload "$HOME/.ssh/id_rsa.pub:/root/.ssh/authorized_keys" \
> +  --chmod '0600:/root/.ssh/authorized_keys' \
> +  --run-command 'chown root:root /root/.ssh/authorized_keys' \
> +  --copy-in "$TRAVIS_BUILD_DIR:/root" \
> +  --network \
> +  --selinux-relabel
> +
> +#
> +# Now we create a domain by using virt-install. This not only creates the domain, but runs the VM as well
> +# It should be ready to go for ssh, once ssh starts.
> +#
> +sudo virt-install \
> +  --name fedoravm \
> +  --memory $MEMORY \
> +  --vcpus $VCPUS \
> +  --disk "$HOME/Fedora-Cloud-Base-32-1.6.x86_64.raw" \
> +  --import --noautoconsole
> +
> +#
> +# Here comes the tricky part, we have to figure out when the VM comes up AND we need the ip address for ssh. So we
> +# can check the net-dhcp leases, for our host. We have to poll, and we will poll for up 3 minutes in 6 second
> +# intervals, so 30 poll attempts (0-29 inclusive). I don't know of a better way to do this.
> +#
> +# We have a full reboot + relabel, so first sleep gets us close
> +#
> +sleep 30
> +for i in $(seq 0 29); do
> +    echo "loop $i"
> +    sleep 6s
> +    # Get the leases, but tee it so it's easier to debug
> +    sudo virsh net-dhcp-leases default | tee dhcp-leases.txt
> +
> +    # get our ipaddress
> +    ipaddy=$(grep fedoravm dhcp-leases.txt | awk {'print $5'} | cut -d'/' -f 1-1)
> +    if [ -n "$ipaddy" ]; then
> +        # found it, we're done looking, print it for debug logs
> +        echo "ipaddy: $ipaddy"
> +        break
> +    fi
> +    # it's empty/not found, loop back and try again.
> +done
> +
> +# Did we find it? If not die.
> +if [ -z "$ipaddy" ]; then
> +    echo "ipaddy zero length, exiting with error 1"
> +    exit 1
> +fi
> +
> +#
> +# Great we have a host running, ssh into it. We specify -o so
> +# we don't get blocked on asking to add the servers key to
> +# our known_hosts.
> +#
> +ssh -tt -o StrictHostKeyChecking=no -o LogLevel=QUIET "root@$ipaddy" "/root/selinux/$TEST_RUNNER"
> +
> +exit 0
> --
> 2.17.1
>
William Roberts May 21, 2020, 8:52 p.m. UTC | #6
On Thu, May 21, 2020 at 2:54 PM Nicolas Iooss <nicolas.iooss@m4x.org> wrote:
>
> On Wed, May 20, 2020 at 6:34 PM <bill.c.roberts@gmail.com> wrote:
> >
> > From: William Roberts <william.c.roberts@intel.com>
> >
> > The current Travis CI runs the userspace tooling and libraries against
> > policy files, but cannot test against an SELinux enabled kernel. Thus,
> > some tests are not being done in the CI. Travis, unfortunately only
> > provides Ubuntu images, so in order to run against a modern distro with
> > SELinux in enforcing mode, we need to launch a KVM with something like
> > Fedora.
> >
> > This patch enables this support by launching a Fedora32 Cloud Image with
> > the SELinux userspace library passed on from the Travis clone, it then
> > builds and replaces the current SELinux bits on the Fedora32 image and
> > runs the SELinux testsuite.
> >
> > Signed-off-by: William Roberts <william.c.roberts@intel.com>
>
> Hi,
> Thanks for working on this. The CI scripts are quite easy to follow,
> thanks to all the comments :)
>
> Anyway, here are some suggestions/comments in order to make this patch
> even better (if you want to do a v3):
>
> * ShellCheck detects that some strings are not quoted, that using
> *-CHECKSUM is dangerous (using ./*-CHECKSUM prevents ill things from
> happening if a file in the directory happens to match the pattern and
> start with a dash), etc. Most issues reported by ShellCheck are not
> relevant here, but one:
>
>     ipaddy=$(grep fedoravm dhcp-leases.txt | awk {'print $5'} | cut
> -d'/' -f 1-1)
>                                                  ^-- SC1083: This { is
> literal. Check expression (missing ;/\n?) or quote it.
>
> Using awk '{print $5}' (single quotes outside of the braces) seems to
> better match the intent. Moreover the whole $(...) expression could be
> double-quoted.
>
> * fedora-test-runner.sh uses ~, which is expanded to /root as it does
> not make sense to run this script as non-root user. In my humble
> opinion, using /root instead of ~ makes the script even easier to
> read.

Fixing all the shell check issues for v3 seems prudent, so will
respin with those changes.

>
> * There is a comment about issues with -jX with "make
> LIBDIR=/usr/lib64 SHLIBDIR=/lib64 install install-pywrap relabel". I
> usually split the targets when using "make -jX" (ie. make -j$(nproc)
> install && make -j$(nproc) install-pywrap && make relabel), so if not
> using -jX is really an issue, this could be considered.

Oh thanks for the suggestion, I didn't think of that,
I can do that to reduce whatever time we can.

>
> * Does the script runs "make test" in selinux/? I see the "make test"
> for selinux-testsuite, but if the main point of fedora-test-runner.sh
> is to run selinux's tests (in a VM), I missed the command that does
> so.

I thought about that, but libselinux and things that use libselinux:
- libselinux
- libsemanage
- mcstrans
- policycoreutils
- restorecond
- sandbox
All have *empty* Makefile test targets *with the exception of sandbox*.

We already run make check on the Ubuntu VM on Travis, which tests all
the projects
that have defined test targets and since they don't use libselinux likely
don't need a libselinux enabled host.  I'm not opposed to it, just trying to
trim down to things that aren't already being done.

I guess at a minimum it ensures that tests didn't break between Ubuntu and
Fedora, perhaps over things like dependency versions (python version
for instance) and
perhaps we pick up the sandbox tests. But the Ubuntu VM's already test
with multiple (22
different build flavors) of different python, ruby, compilers and
linkers and those only
take a few minutes to complete.

I'm definitely not opposed to adding it, perhaps i'm ignorant to the
additional things
that might get covered and thus someone can make stronger arguments to
adding it than I can.

>
> Thanks,
> Nicolas
>
> > ---
> >  .travis.yml                      |   8 +++
> >  scripts/ci/README.md             |   8 +++
> >  scripts/ci/fedora-test-runner.sh |  89 ++++++++++++++++++++++++
> >  scripts/ci/travis-kvm-setup.sh   | 113 +++++++++++++++++++++++++++++++
> >  4 files changed, 218 insertions(+)
> >  create mode 100644 scripts/ci/README.md
> >  create mode 100755 scripts/ci/fedora-test-runner.sh
> >  create mode 100755 scripts/ci/travis-kvm-setup.sh
> >
> > diff --git a/.travis.yml b/.travis.yml
> > index c36e721a5e1d..63a856672f9b 100644
> > --- a/.travis.yml
> > +++ b/.travis.yml
> > @@ -34,6 +34,14 @@ matrix:
> >        env: PYVER=python3.8 RUBYLIBVER=2.7 LINKER=gold
> >      - compiler: clang
> >        env: PYVER=python3.8 RUBYLIBVER=2.7 LINKER=bfd
> > +  include:
> > +    - compiler: gcc
> > +      env: TRAVIS_RUN_KVM=true
> > +      install:
> > +        - skip
> > +      before_script:
> > +        - skip
> > +      script: scripts/ci/travis-kvm-setup.sh
> >
> >  # Use Travis-CI Ubuntu 18.04 Bionic Beaver, "full image" variant
> >  sudo: required
> > diff --git a/scripts/ci/README.md b/scripts/ci/README.md
> > new file mode 100644
> > index 000000000000..04a134a438c2
> > --- /dev/null
> > +++ b/scripts/ci/README.md
> > @@ -0,0 +1,8 @@
> > +# Continuous Integration Scripts
> > +
> > +The scripts under `scripts/ci` are designed specifically
> > +for the Travis CI system. While nothing prevents you
> > +from mimicking that environment and using them locally,
> > +they are not applicable for general consumption. Any
> > +thing in this directory should never be considered as
> > +a stable API.
> > diff --git a/scripts/ci/fedora-test-runner.sh b/scripts/ci/fedora-test-runner.sh
> > new file mode 100755
> > index 000000000000..14bcf5fc469d
> > --- /dev/null
> > +++ b/scripts/ci/fedora-test-runner.sh
> > @@ -0,0 +1,89 @@
> > +#!/usr/bin/env bash
> > +
> > +set -ev
> > +
> > +# CI Debug output if things go squirrely.
> > +getenforce
> > +id -Z
> > +nproc
> > +pwd
> > +
> > +# Turn off enforcing for the setup to prevent any weirdness from breaking
> > +# the CI.
> > +setenforce 0
> > +
> > +dnf clean all -y
> > +dnf install -y \
> > +    --allowerasing \
> > +    --skip-broken \
> > +    git \
> > +    audit-libs-devel \
> > +    bison \
> > +    bzip2-devel \
> > +    CUnit-devel \
> > +    diffutils \
> > +    flex \
> > +    gcc \
> > +    gettext \
> > +    glib2-devel \
> > +    make \
> > +    libcap-devel \
> > +    libcap-ng-devel \
> > +    pam-devel \
> > +    pcre-devel \
> > +    xmlto \
> > +    python3-devel \
> > +    ruby-devel \
> > +    swig \
> > +    perl-Test \
> > +    perl-Test-Harness \
> > +    perl-Test-Simple \
> > +    selinux-policy-devel \
> > +    gcc \
> > +    libselinux-devel \
> > +    net-tools \
> > +    netlabel_tools \
> > +    iptables \
> > +    lksctp-tools-devel \
> > +    attr \
> > +    libbpf-devel \
> > +    keyutils-libs-devel \
> > +    kernel-devel \
> > +    quota \
> > +    xfsprogs-devel \
> > +    libuuid-devel \
> > +    kernel-devel-$(uname -r) \
> > +    kernel-modules-$(uname -r)
> > +
> > +#
> > +# Move to selinux code and build
> > +#
> > +cd ~/selinux
> > +
> > +# Show HEAD commit for sanity checking
> > +git log -1
> > +
> > +#
> > +# Build and replace userspace components
> > +#
> > +# Note: You can't use parallel builds here (make -jX), you'll end up
> > +# with race conditions that manifest like:
> > +# semanage_store.lo: file not recognized: file format not recognized
> > +#
> > +make LIBDIR=/usr/lib64 SHLIBDIR=/lib64 install install-pywrap relabel
> > +
> > +#
> > +# Get the selinux testsuite, but don't clone it in ~/selinux, move to ~
> > +# first.
> > +#
> > +cd ~
> > +git clone --depth=1 https://github.com/SELinuxProject/selinux-testsuite.git
> > +cd selinux-testsuite
> > +
> > +# The testsuite must be run in enforcing mode
> > +setenforce 1
> > +
> > +#
> > +# Run the test suite
> > +#
> > +make test
> > diff --git a/scripts/ci/travis-kvm-setup.sh b/scripts/ci/travis-kvm-setup.sh
> > new file mode 100755
> > index 000000000000..66606e9d4a5b
> > --- /dev/null
> > +++ b/scripts/ci/travis-kvm-setup.sh
> > @@ -0,0 +1,113 @@
> > +#!/usr/bin/env bash
> > +
> > +set -ev
> > +
> > +TEST_RUNNER="scripts/ci/fedora-test-runner.sh"
> > +
> > +#
> > +# Travis gives us 7.5GB of RAM and two cores:
> > +# https://docs.travis-ci.com/user/reference/overview/
> > +#
> > +MEMORY=4096
> > +VCPUS=2
> > +
> > +# Install these here so other builds don't have to wait on these deps to download and install
> > +sudo apt-get install qemu-kvm libvirt-bin virtinst bridge-utils cpu-checker libguestfs-tools
> > +
> > +sudo usermod -a -G kvm,libvirt,libvirt-qemu $USER
> > +
> > +# Verify that KVM is working, useful if Travis every changes anything.
> > +kvm-ok
> > +
> > +sudo systemctl enable libvirtd
> > +sudo systemctl start libvirtd
> > +
> > +# Set up a key so we can ssh into the VM
> > +ssh-keygen -N "" -f "$HOME/.ssh/id_rsa"
> > +
> > +#
> > +# Get the Fedora Cloud Image, It is a base image that small and ready to go, extract it and modify it with virt-sysprep
> > +#  - https://alt.fedoraproject.org/en/verify.html
> > +cd $HOME
> > +wget https://download.fedoraproject.org/pub/fedora/linux/releases/32/Cloud/x86_64/images/Fedora-Cloud-Base-32-1.6.x86_64.raw.xz
> > +
> > +# Verify the image
> > +curl https://getfedora.org/static/fedora.gpg | gpg --import
> > +wget https://getfedora.org/static/checksums/Fedora-Cloud-32-1.6-x86_64-CHECKSUM
> > +gpg --verify-files *-CHECKSUM
> > +sha256sum --ignore-missing -c *-CHECKSUM
> > +
> > +# Extract the image
> > +unxz -T0 Fedora-Cloud-Base-32-1.6.x86_64.raw.xz
> > +
> > +# Search is needed for $HOME so virt service can access the image file.
> > +chmod a+x $HOME
> > +
> > +#
> > +# Modify the virtual image to:
> > +#   - Enable a login, we just use root
> > +#   - Enable passwordless login
> > +#     - Force a relabel to fix labels on ssh keys
> > +#
> > +sudo virt-sysprep -a "$HOME/Fedora-Cloud-Base-32-1.6.x86_64.raw" \
> > +  --root-password password:123456 \
> > +  --hostname fedoravm \
> > +  --append-line '/etc/ssh/sshd_config:PermitRootLogin yes' \
> > +  --append-line '/etc/ssh/sshd_config:PubkeyAuthentication yes' \
> > +  --mkdir /root/.ssh \
> > +  --upload "$HOME/.ssh/id_rsa.pub:/root/.ssh/authorized_keys" \
> > +  --chmod '0600:/root/.ssh/authorized_keys' \
> > +  --run-command 'chown root:root /root/.ssh/authorized_keys' \
> > +  --copy-in "$TRAVIS_BUILD_DIR:/root" \
> > +  --network \
> > +  --selinux-relabel
> > +
> > +#
> > +# Now we create a domain by using virt-install. This not only creates the domain, but runs the VM as well
> > +# It should be ready to go for ssh, once ssh starts.
> > +#
> > +sudo virt-install \
> > +  --name fedoravm \
> > +  --memory $MEMORY \
> > +  --vcpus $VCPUS \
> > +  --disk "$HOME/Fedora-Cloud-Base-32-1.6.x86_64.raw" \
> > +  --import --noautoconsole
> > +
> > +#
> > +# Here comes the tricky part, we have to figure out when the VM comes up AND we need the ip address for ssh. So we
> > +# can check the net-dhcp leases, for our host. We have to poll, and we will poll for up 3 minutes in 6 second
> > +# intervals, so 30 poll attempts (0-29 inclusive). I don't know of a better way to do this.
> > +#
> > +# We have a full reboot + relabel, so first sleep gets us close
> > +#
> > +sleep 30
> > +for i in $(seq 0 29); do
> > +    echo "loop $i"
> > +    sleep 6s
> > +    # Get the leases, but tee it so it's easier to debug
> > +    sudo virsh net-dhcp-leases default | tee dhcp-leases.txt
> > +
> > +    # get our ipaddress
> > +    ipaddy=$(grep fedoravm dhcp-leases.txt | awk {'print $5'} | cut -d'/' -f 1-1)
> > +    if [ -n "$ipaddy" ]; then
> > +        # found it, we're done looking, print it for debug logs
> > +        echo "ipaddy: $ipaddy"
> > +        break
> > +    fi
> > +    # it's empty/not found, loop back and try again.
> > +done
> > +
> > +# Did we find it? If not die.
> > +if [ -z "$ipaddy" ]; then
> > +    echo "ipaddy zero length, exiting with error 1"
> > +    exit 1
> > +fi
> > +
> > +#
> > +# Great we have a host running, ssh into it. We specify -o so
> > +# we don't get blocked on asking to add the servers key to
> > +# our known_hosts.
> > +#
> > +ssh -tt -o StrictHostKeyChecking=no -o LogLevel=QUIET "root@$ipaddy" "/root/selinux/$TEST_RUNNER"
> > +
> > +exit 0
> > --
> > 2.17.1
> >
>
William Roberts May 21, 2020, 10:39 p.m. UTC | #7
<snip>
> * fedora-test-runner.sh uses ~, which is expanded to /root as it does
> not make sense to run this script as non-root user. In my humble
> opinion, using /root instead of ~ makes the script even easier to
> read.

I forgot to mention this in my previous email, this is the only thing
I really don't
want to change. You could, in theory, configure any user to run this.

<snip>
Ondrej Mosnacek May 22, 2020, 7:40 a.m. UTC | #8
On Thu, May 21, 2020 at 4:12 PM William Roberts
<bill.c.roberts@gmail.com> wrote:
> On Thu, May 21, 2020 at 7:58 AM Ondrej Mosnacek <omosnace@redhat.com> wrote:
> >
> > On Thu, May 21, 2020 at 2:52 PM Stephen Smalley
> > <stephen.smalley.work@gmail.com> wrote:
[...]
> > > Last I looked, his script builds and installs the userspace code on
> > > top of the Fedora libraries and programs (make LIBDIR=... install...)
> > > and then runs the testsuite.  That was my suggestion.
> >
> > Ah, yes, I can see that line now. Sorry, somehow I missed it before.
> >
> > > While it is the
> > > kernel testsuite, it exercises a lot of SELinux userspace
> > > functionality that isn't tested by the userspace tests.
> >
> > OK, I suppose it's better than nothing...
> >
>
> Stephen pointed out the additional ways userspace gets tested, and
> perhaps my title and description
> of the patch could be better. But the main point is to increase the
> test coverage
> and perform the testing steps we expect are done before a release in
> the CI. We should have
> the testing coverage and the confidence to release userspace from master at any
> point. We also have forward facing proof that tests are being executed
> and we can make sure
> nothing regresses.
>
> My ultimate goal here, is to help make sure that if Petr gets hit by a
> bus, releases will
> move forward without worry and without any change in quality among the various
> maintainers.
>
> Additionally, we pick up some cross project testing and can find other
> surprises.

Ah, so you want to move an integration test, which we would normally
run only before release, down to per-commit testing, which is fine
because it is quite fast... OK, it started to make sense to me now :)

If I find time I'll have a closer look at the scripts. I already see
some tiny possible improvements...

--
Ondrej Mosnacek <omosnace at redhat dot com>
Software Engineer, Security Technologies
Red Hat, Inc.
Nicolas Iooss May 22, 2020, 7:07 p.m. UTC | #9
On Fri, May 22, 2020 at 12:39 AM William Roberts
<bill.c.roberts@gmail.com> wrote:
>
> <snip>
> > * fedora-test-runner.sh uses ~, which is expanded to /root as it does
> > not make sense to run this script as non-root user. In my humble
> > opinion, using /root instead of ~ makes the script even easier to
> > read.
>
> I forgot to mention this in my previous email, this is the only thing
> I really don't
> want to change. You could, in theory, configure any user to run this.
>
> <snip>

The script executes "setenforce 0", runs dnf to install packages and
overwrites binaries and libraries in system directories (/usr/bin,
/usr/sbin, /usr/lib64, etc.). How do you allow any user to perform
these actions, without being root?

Anyway, if you do not want to hardcode /root, to could use "$HOME"
instead of ~. It makes things appear less magical, in my humble
opinion.

Nicolas
William Roberts May 23, 2020, 12:21 a.m. UTC | #10
On Fri, May 22, 2020 at 2:07 PM Nicolas Iooss <nicolas.iooss@m4x.org> wrote:
>
> On Fri, May 22, 2020 at 12:39 AM William Roberts
> <bill.c.roberts@gmail.com> wrote:
> >
> > <snip>
> > > * fedora-test-runner.sh uses ~, which is expanded to /root as it does
> > > not make sense to run this script as non-root user. In my humble
> > > opinion, using /root instead of ~ makes the script even easier to
> > > read.
> >
> > I forgot to mention this in my previous email, this is the only thing
> > I really don't
> > want to change. You could, in theory, configure any user to run this.
> >
> > <snip>
>
> The script executes "setenforce 0", runs dnf to install packages and
> overwrites binaries and libraries in system directories (/usr/bin,
> /usr/sbin, /usr/lib64, etc.). How do you allow any user to perform
> these actions, without being root?

Its called capabilities. I could build you a box where root is neutered and
cannot do anything.

>
> Anyway, if you do not want to hardcode /root, to could use "$HOME"
> instead of ~. It makes things appear less magical, in my humble
> opinion.

Sure, it actually makes it consistent with the other script.

>
> Nicolas
>
William Roberts May 24, 2020, 4:18 p.m. UTC | #11
On Fri, May 22, 2020 at 2:40 AM Ondrej Mosnacek <omosnace@redhat.com> wrote:
>
> On Thu, May 21, 2020 at 4:12 PM William Roberts
> <bill.c.roberts@gmail.com> wrote:
> > On Thu, May 21, 2020 at 7:58 AM Ondrej Mosnacek <omosnace@redhat.com> wrote:
> > >
> > > On Thu, May 21, 2020 at 2:52 PM Stephen Smalley
> > > <stephen.smalley.work@gmail.com> wrote:
> [...]
> > > > Last I looked, his script builds and installs the userspace code on
> > > > top of the Fedora libraries and programs (make LIBDIR=... install...)
> > > > and then runs the testsuite.  That was my suggestion.
> > >
> > > Ah, yes, I can see that line now. Sorry, somehow I missed it before.
> > >
> > > > While it is the
> > > > kernel testsuite, it exercises a lot of SELinux userspace
> > > > functionality that isn't tested by the userspace tests.
> > >
> > > OK, I suppose it's better than nothing...
> > >
> >
> > Stephen pointed out the additional ways userspace gets tested, and
> > perhaps my title and description
> > of the patch could be better. But the main point is to increase the
> > test coverage
> > and perform the testing steps we expect are done before a release in
> > the CI. We should have
> > the testing coverage and the confidence to release userspace from master at any
> > point. We also have forward facing proof that tests are being executed
> > and we can make sure
> > nothing regresses.
> >
> > My ultimate goal here, is to help make sure that if Petr gets hit by a
> > bus, releases will
> > move forward without worry and without any change in quality among the various
> > maintainers.
> >
> > Additionally, we pick up some cross project testing and can find other
> > surprises.
>
> Ah, so you want to move an integration test, which we would normally
> run only before release, down to per-commit testing, which is fine
> because it is quite fast... OK, it started to make sense to me now :)

Exactly, plus we pick up a little more test coverage on the userspace bits
by swapping out what's installed in the VM with the current build and running
the tests. The speed is less important, it's just fast enough where our CI isn't
going to take years to complete. CI doesn't need to be super snappy per se,
but it also cannot take a fortnight.

>
> If I find time I'll have a closer look at the scripts. I already see
> some tiny possible improvements... I have Nicolas's last comments
addressed and staged, so ill wait a few days and see what you come
back with and re-send a V3.
Stephen Smalley May 29, 2020, 1:24 p.m. UTC | #12
On Sun, May 24, 2020 at 12:18 PM William Roberts
<bill.c.roberts@gmail.com> wrote:
>
> On Fri, May 22, 2020 at 2:40 AM Ondrej Mosnacek <omosnace@redhat.com> wrote:
> >
> > On Thu, May 21, 2020 at 4:12 PM William Roberts
> > <bill.c.roberts@gmail.com> wrote:
> > > On Thu, May 21, 2020 at 7:58 AM Ondrej Mosnacek <omosnace@redhat.com> wrote:
> > > >
> > > > On Thu, May 21, 2020 at 2:52 PM Stephen Smalley
> > > > <stephen.smalley.work@gmail.com> wrote:
> > [...]
> > > > > Last I looked, his script builds and installs the userspace code on
> > > > > top of the Fedora libraries and programs (make LIBDIR=... install...)
> > > > > and then runs the testsuite.  That was my suggestion.
> > > >
> > > > Ah, yes, I can see that line now. Sorry, somehow I missed it before.
> > > >
> > > > > While it is the
> > > > > kernel testsuite, it exercises a lot of SELinux userspace
> > > > > functionality that isn't tested by the userspace tests.
> > > >
> > > > OK, I suppose it's better than nothing...
> > > >
> > >
> > > Stephen pointed out the additional ways userspace gets tested, and
> > > perhaps my title and description
> > > of the patch could be better. But the main point is to increase the
> > > test coverage
> > > and perform the testing steps we expect are done before a release in
> > > the CI. We should have
> > > the testing coverage and the confidence to release userspace from master at any
> > > point. We also have forward facing proof that tests are being executed
> > > and we can make sure
> > > nothing regresses.
> > >
> > > My ultimate goal here, is to help make sure that if Petr gets hit by a
> > > bus, releases will
> > > move forward without worry and without any change in quality among the various
> > > maintainers.
> > >
> > > Additionally, we pick up some cross project testing and can find other
> > > surprises.
> >
> > Ah, so you want to move an integration test, which we would normally
> > run only before release, down to per-commit testing, which is fine
> > because it is quite fast... OK, it started to make sense to me now :)
>
> Exactly, plus we pick up a little more test coverage on the userspace bits
> by swapping out what's installed in the VM with the current build and running
> the tests. The speed is less important, it's just fast enough where our CI isn't
> going to take years to complete. CI doesn't need to be super snappy per se,
> but it also cannot take a fortnight.
>
> >
> > If I find time I'll have a closer look at the scripts. I already see
> > some tiny possible improvements... I have Nicolas's last comments
> addressed and staged, so ill wait a few days and see what you come
> back with and re-send a V3.

This is looking good to me.  Only questions I have are:
1) Have you confirmed that a testsuite failure within the VM gets
correctly propagated up and treated as a failure by travis-ci itself?
2) Have you seen any problems with instability in running of the tests
due to the additional complexity and time?  I've certainly already
seen instances where travis-ci of selinux or selinux-testsuite fails
randomly due to timeouts or something when downloading external
components.
William Roberts May 29, 2020, 3:33 p.m. UTC | #13
On Fri, May 29, 2020 at 8:24 AM Stephen Smalley
<stephen.smalley.work@gmail.com> wrote:
>
> On Sun, May 24, 2020 at 12:18 PM William Roberts
> <bill.c.roberts@gmail.com> wrote:
> >
> > On Fri, May 22, 2020 at 2:40 AM Ondrej Mosnacek <omosnace@redhat.com> wrote:
> > >
> > > On Thu, May 21, 2020 at 4:12 PM William Roberts
> > > <bill.c.roberts@gmail.com> wrote:
> > > > On Thu, May 21, 2020 at 7:58 AM Ondrej Mosnacek <omosnace@redhat.com> wrote:
> > > > >
> > > > > On Thu, May 21, 2020 at 2:52 PM Stephen Smalley
> > > > > <stephen.smalley.work@gmail.com> wrote:
> > > [...]
> > > > > > Last I looked, his script builds and installs the userspace code on
> > > > > > top of the Fedora libraries and programs (make LIBDIR=... install...)
> > > > > > and then runs the testsuite.  That was my suggestion.
> > > > >
> > > > > Ah, yes, I can see that line now. Sorry, somehow I missed it before.
> > > > >
> > > > > > While it is the
> > > > > > kernel testsuite, it exercises a lot of SELinux userspace
> > > > > > functionality that isn't tested by the userspace tests.
> > > > >
> > > > > OK, I suppose it's better than nothing...
> > > > >
> > > >
> > > > Stephen pointed out the additional ways userspace gets tested, and
> > > > perhaps my title and description
> > > > of the patch could be better. But the main point is to increase the
> > > > test coverage
> > > > and perform the testing steps we expect are done before a release in
> > > > the CI. We should have
> > > > the testing coverage and the confidence to release userspace from master at any
> > > > point. We also have forward facing proof that tests are being executed
> > > > and we can make sure
> > > > nothing regresses.
> > > >
> > > > My ultimate goal here, is to help make sure that if Petr gets hit by a
> > > > bus, releases will
> > > > move forward without worry and without any change in quality among the various
> > > > maintainers.
> > > >
> > > > Additionally, we pick up some cross project testing and can find other
> > > > surprises.
> > >
> > > Ah, so you want to move an integration test, which we would normally
> > > run only before release, down to per-commit testing, which is fine
> > > because it is quite fast... OK, it started to make sense to me now :)
> >
> > Exactly, plus we pick up a little more test coverage on the userspace bits
> > by swapping out what's installed in the VM with the current build and running
> > the tests. The speed is less important, it's just fast enough where our CI isn't
> > going to take years to complete. CI doesn't need to be super snappy per se,
> > but it also cannot take a fortnight.
> >
> > >
> > > If I find time I'll have a closer look at the scripts. I already see
> > > some tiny possible improvements... I have Nicolas's last comments
> > addressed and staged, so ill wait a few days and see what you come
> > back with and re-send a V3.
>
> This is looking good to me.  Only questions I have are:
> 1) Have you confirmed that a testsuite failure within the VM gets
> correctly propagated up and treated as a failure by travis-ci itself?

Yes, when I was testing and didn't have SCTP support, the test suite
would fail and propagate back.

> 2) Have you seen any problems with instability in running of the tests
> due to the additional complexity and time?  I've certainly already
> seen instances where travis-ci of selinux or selinux-testsuite fails
> randomly due to timeouts or something when downloading external
> components.

No, and I did a bunch of additional runs, like 10-12 and never saw additional
failures. I see some of those intermittent travis CI failures with all
my projects
on Travis, but I don't see this as adding any more issues. Timeouts are caused
by nothing sent to stdout for 10 mins (IIRC), so we should be good there.

Im just waiting on Ondrej, he said he had some feedback and I was gonna send
round 3. If I don't here anything back on Monday ill send round 3.
Ondrej Mosnacek May 29, 2020, 6:42 p.m. UTC | #14
Apologies for getting back to this so late... Just some small nitpicks.

On Wed, May 20, 2020 at 6:34 PM <bill.c.roberts@gmail.com> wrote:
> From: William Roberts <william.c.roberts@intel.com>
>
> The current Travis CI runs the userspace tooling and libraries against
> policy files, but cannot test against an SELinux enabled kernel. Thus,
> some tests are not being done in the CI. Travis, unfortunately only
> provides Ubuntu images, so in order to run against a modern distro with
> SELinux in enforcing mode, we need to launch a KVM with something like
> Fedora.
>
> This patch enables this support by launching a Fedora32 Cloud Image with
> the SELinux userspace library passed on from the Travis clone, it then
> builds and replaces the current SELinux bits on the Fedora32 image and
> runs the SELinux testsuite.
>
> Signed-off-by: William Roberts <william.c.roberts@intel.com>
> ---
>  .travis.yml                      |   8 +++
>  scripts/ci/README.md             |   8 +++
>  scripts/ci/fedora-test-runner.sh |  89 ++++++++++++++++++++++++
>  scripts/ci/travis-kvm-setup.sh   | 113 +++++++++++++++++++++++++++++++
>  4 files changed, 218 insertions(+)
>  create mode 100644 scripts/ci/README.md
>  create mode 100755 scripts/ci/fedora-test-runner.sh
>  create mode 100755 scripts/ci/travis-kvm-setup.sh
>
[...]
> diff --git a/scripts/ci/fedora-test-runner.sh b/scripts/ci/fedora-test-runner.sh
> new file mode 100755
> index 000000000000..14bcf5fc469d
> --- /dev/null
> +++ b/scripts/ci/fedora-test-runner.sh
> @@ -0,0 +1,89 @@
> +#!/usr/bin/env bash
> +
> +set -ev
> +
> +# CI Debug output if things go squirrely.
> +getenforce
> +id -Z
> +nproc
> +pwd

I'd add also "uname -r" here to dump the running kernel version (will
probably be also printed later somewhere, but better to have it also
in one place with the other debug info).

> +
> +# Turn off enforcing for the setup to prevent any weirdness from breaking
> +# the CI.
> +setenforce 0
> +
> +dnf clean all -y
> +dnf install -y \
[...]
> diff --git a/scripts/ci/travis-kvm-setup.sh b/scripts/ci/travis-kvm-setup.sh
> new file mode 100755
> index 000000000000..66606e9d4a5b
> --- /dev/null
> +++ b/scripts/ci/travis-kvm-setup.sh
> @@ -0,0 +1,113 @@
> +#!/usr/bin/env bash
> +
> +set -ev
> +
> +TEST_RUNNER="scripts/ci/fedora-test-runner.sh"
> +
> +#
> +# Travis gives us 7.5GB of RAM and two cores:
> +# https://docs.travis-ci.com/user/reference/overview/
> +#
> +MEMORY=4096
> +VCPUS=2

Why not "VCPUS=$(nproc)"?

> +
> +# Install these here so other builds don't have to wait on these deps to download and install
> +sudo apt-get install qemu-kvm libvirt-bin virtinst bridge-utils cpu-checker libguestfs-tools
> +
> +sudo usermod -a -G kvm,libvirt,libvirt-qemu $USER
> +
> +# Verify that KVM is working, useful if Travis every changes anything.

s/every/ever/

> +kvm-ok
> +
> +sudo systemctl enable libvirtd
> +sudo systemctl start libvirtd
> +
> +# Set up a key so we can ssh into the VM
> +ssh-keygen -N "" -f "$HOME/.ssh/id_rsa"
> +
> +#
> +# Get the Fedora Cloud Image, It is a base image that small and ready to go, extract it and modify it with virt-sysprep
> +#  - https://alt.fedoraproject.org/en/verify.html
> +cd $HOME
> +wget https://download.fedoraproject.org/pub/fedora/linux/releases/32/Cloud/x86_64/images/Fedora-Cloud-Base-32-1.6.x86_64.raw.xz

I'd suggest extracting the Fedora release version (32) + the image
version (1.6) into variables, so they can be easily bumped later.

> +
> +# Verify the image
> +curl https://getfedora.org/static/fedora.gpg | gpg --import
> +wget https://getfedora.org/static/checksums/Fedora-Cloud-32-1.6-x86_64-CHECKSUM
> +gpg --verify-files *-CHECKSUM
> +sha256sum --ignore-missing -c *-CHECKSUM
> +
> +# Extract the image
> +unxz -T0 Fedora-Cloud-Base-32-1.6.x86_64.raw.xz
> +
> +# Search is needed for $HOME so virt service can access the image file.
> +chmod a+x $HOME
> +
> +#
> +# Modify the virtual image to:
> +#   - Enable a login, we just use root
> +#   - Enable passwordless login
> +#     - Force a relabel to fix labels on ssh keys
> +#
> +sudo virt-sysprep -a "$HOME/Fedora-Cloud-Base-32-1.6.x86_64.raw" \
> +  --root-password password:123456 \

Do you need to set the password when you use an SSH key to login?

> +  --hostname fedoravm \
> +  --append-line '/etc/ssh/sshd_config:PermitRootLogin yes' \
> +  --append-line '/etc/ssh/sshd_config:PubkeyAuthentication yes' \
> +  --mkdir /root/.ssh \
> +  --upload "$HOME/.ssh/id_rsa.pub:/root/.ssh/authorized_keys" \
> +  --chmod '0600:/root/.ssh/authorized_keys' \
> +  --run-command 'chown root:root /root/.ssh/authorized_keys' \

Could these be replaced with just "--ssh-inject root"?

> +  --copy-in "$TRAVIS_BUILD_DIR:/root" \
> +  --network \
> +  --selinux-relabel
> +
> +#
> +# Now we create a domain by using virt-install. This not only creates the domain, but runs the VM as well
> +# It should be ready to go for ssh, once ssh starts.
> +#
> +sudo virt-install \
> +  --name fedoravm \
> +  --memory $MEMORY \
> +  --vcpus $VCPUS \
> +  --disk "$HOME/Fedora-Cloud-Base-32-1.6.x86_64.raw" \
> +  --import --noautoconsole
> +
> +#
> +# Here comes the tricky part, we have to figure out when the VM comes up AND we need the ip address for ssh. So we
> +# can check the net-dhcp leases, for our host. We have to poll, and we will poll for up 3 minutes in 6 second
> +# intervals, so 30 poll attempts (0-29 inclusive). I don't know of a better way to do this.
> +#
> +# We have a full reboot + relabel, so first sleep gets us close
> +#
> +sleep 30
> +for i in $(seq 0 29); do
> +    echo "loop $i"
> +    sleep 6s
> +    # Get the leases, but tee it so it's easier to debug
> +    sudo virsh net-dhcp-leases default | tee dhcp-leases.txt
> +
> +    # get our ipaddress
> +    ipaddy=$(grep fedoravm dhcp-leases.txt | awk {'print $5'} | cut -d'/' -f 1-1)

Looks cleaner this way:
[...] | awk '{ print $5 }' | cut -d / -f 1)

> +    if [ -n "$ipaddy" ]; then
> +        # found it, we're done looking, print it for debug logs
> +        echo "ipaddy: $ipaddy"
> +        break
> +    fi
> +    # it's empty/not found, loop back and try again.
> +done
> +
> +# Did we find it? If not die.
> +if [ -z "$ipaddy" ]; then
> +    echo "ipaddy zero length, exiting with error 1"
> +    exit 1
> +fi
> +
> +#
> +# Great we have a host running, ssh into it. We specify -o so
> +# we don't get blocked on asking to add the servers key to
> +# our known_hosts.
> +#
> +ssh -tt -o StrictHostKeyChecking=no -o LogLevel=QUIET "root@$ipaddy" "/root/selinux/$TEST_RUNNER"
> +
> +exit 0
> --
> 2.17.1
William Roberts May 29, 2020, 7:17 p.m. UTC | #15
On Fri, May 29, 2020 at 1:43 PM Ondrej Mosnacek <omosnace@redhat.com> wrote:
>
> Apologies for getting back to this so late... Just some small nitpicks.
>
> On Wed, May 20, 2020 at 6:34 PM <bill.c.roberts@gmail.com> wrote:
> > From: William Roberts <william.c.roberts@intel.com>
> >
> > The current Travis CI runs the userspace tooling and libraries against
> > policy files, but cannot test against an SELinux enabled kernel. Thus,
> > some tests are not being done in the CI. Travis, unfortunately only
> > provides Ubuntu images, so in order to run against a modern distro with
> > SELinux in enforcing mode, we need to launch a KVM with something like
> > Fedora.
> >
> > This patch enables this support by launching a Fedora32 Cloud Image with
> > the SELinux userspace library passed on from the Travis clone, it then
> > builds and replaces the current SELinux bits on the Fedora32 image and
> > runs the SELinux testsuite.
> >
> > Signed-off-by: William Roberts <william.c.roberts@intel.com>
> > ---
> >  .travis.yml                      |   8 +++
> >  scripts/ci/README.md             |   8 +++
> >  scripts/ci/fedora-test-runner.sh |  89 ++++++++++++++++++++++++
> >  scripts/ci/travis-kvm-setup.sh   | 113 +++++++++++++++++++++++++++++++
> >  4 files changed, 218 insertions(+)
> >  create mode 100644 scripts/ci/README.md
> >  create mode 100755 scripts/ci/fedora-test-runner.sh
> >  create mode 100755 scripts/ci/travis-kvm-setup.sh
> >
> [...]
> > diff --git a/scripts/ci/fedora-test-runner.sh b/scripts/ci/fedora-test-runner.sh
> > new file mode 100755
> > index 000000000000..14bcf5fc469d
> > --- /dev/null
> > +++ b/scripts/ci/fedora-test-runner.sh
> > @@ -0,0 +1,89 @@
> > +#!/usr/bin/env bash
> > +
> > +set -ev
> > +
> > +# CI Debug output if things go squirrely.
> > +getenforce
> > +id -Z
> > +nproc
> > +pwd
>
> I'd add also "uname -r" here to dump the running kernel version (will
> probably be also printed later somewhere, but better to have it also
> in one place with the other debug info).
>
> > +
> > +# Turn off enforcing for the setup to prevent any weirdness from breaking
> > +# the CI.
> > +setenforce 0
> > +
> > +dnf clean all -y
> > +dnf install -y \
> [...]
> > diff --git a/scripts/ci/travis-kvm-setup.sh b/scripts/ci/travis-kvm-setup.sh
> > new file mode 100755
> > index 000000000000..66606e9d4a5b
> > --- /dev/null
> > +++ b/scripts/ci/travis-kvm-setup.sh
> > @@ -0,0 +1,113 @@
> > +#!/usr/bin/env bash
> > +
> > +set -ev
> > +
> > +TEST_RUNNER="scripts/ci/fedora-test-runner.sh"
> > +
> > +#
> > +# Travis gives us 7.5GB of RAM and two cores:
> > +# https://docs.travis-ci.com/user/reference/overview/
> > +#
> > +MEMORY=4096
> > +VCPUS=2
>
> Why not "VCPUS=$(nproc)"?

+1: Initially I just had this set to what travis provides. I don't
know why I didn't do that.

>
> > +
> > +# Install these here so other builds don't have to wait on these deps to download and install
> > +sudo apt-get install qemu-kvm libvirt-bin virtinst bridge-utils cpu-checker libguestfs-tools
> > +
> > +sudo usermod -a -G kvm,libvirt,libvirt-qemu $USER
> > +
> > +# Verify that KVM is working, useful if Travis every changes anything.
>
> s/every/ever/

+1

>
> > +kvm-ok
> > +
> > +sudo systemctl enable libvirtd
> > +sudo systemctl start libvirtd
> > +
> > +# Set up a key so we can ssh into the VM
> > +ssh-keygen -N "" -f "$HOME/.ssh/id_rsa"
> > +
> > +#
> > +# Get the Fedora Cloud Image, It is a base image that small and ready to go, extract it and modify it with virt-sysprep
> > +#  - https://alt.fedoraproject.org/en/verify.html
> > +cd $HOME
> > +wget https://download.fedoraproject.org/pub/fedora/linux/releases/32/Cloud/x86_64/images/Fedora-Cloud-Base-32-1.6.x86_64.raw.xz
>
> I'd suggest extracting the Fedora release version (32) + the image
> version (1.6) into variables, so they can be easily bumped later.

Sure, I forget why I ended up not doing it this way. I remember it
being late at night, and cursing for some reason.

>
> > +
> > +# Verify the image
> > +curl https://getfedora.org/static/fedora.gpg | gpg --import
> > +wget https://getfedora.org/static/checksums/Fedora-Cloud-32-1.6-x86_64-CHECKSUM
> > +gpg --verify-files *-CHECKSUM
> > +sha256sum --ignore-missing -c *-CHECKSUM
> > +
> > +# Extract the image
> > +unxz -T0 Fedora-Cloud-Base-32-1.6.x86_64.raw.xz
> > +
> > +# Search is needed for $HOME so virt service can access the image file.
> > +chmod a+x $HOME
> > +
> > +#
> > +# Modify the virtual image to:
> > +#   - Enable a login, we just use root
> > +#   - Enable passwordless login
> > +#     - Force a relabel to fix labels on ssh keys
> > +#
> > +sudo virt-sysprep -a "$HOME/Fedora-Cloud-Base-32-1.6.x86_64.raw" \
> > +  --root-password password:123456 \
>
> Do you need to set the password when you use an SSH key to login?

Yeah the account is disabled unless you do this. Plus it was helpful when
using the scripts locally and using virsh console

>
> > +  --hostname fedoravm \
> > +  --append-line '/etc/ssh/sshd_config:PermitRootLogin yes' \
> > +  --append-line '/etc/ssh/sshd_config:PubkeyAuthentication yes' \
> > +  --mkdir /root/.ssh \
> > +  --upload "$HOME/.ssh/id_rsa.pub:/root/.ssh/authorized_keys" \
> > +  --chmod '0600:/root/.ssh/authorized_keys' \
> > +  --run-command 'chown root:root /root/.ssh/authorized_keys' \
>
> Could these be replaced with just "--ssh-inject root"?

No, and I went through immense pain trying to get it to work. The
reason is that the tool
will dump it under /home/root instead of /root. So it won't get picked
up without some
other magic anyways.

>
> > +  --copy-in "$TRAVIS_BUILD_DIR:/root" \
> > +  --network \
> > +  --selinux-relabel
> > +
> > +#
> > +# Now we create a domain by using virt-install. This not only creates the domain, but runs the VM as well
> > +# It should be ready to go for ssh, once ssh starts.
> > +#
> > +sudo virt-install \
> > +  --name fedoravm \
> > +  --memory $MEMORY \
> > +  --vcpus $VCPUS \
> > +  --disk "$HOME/Fedora-Cloud-Base-32-1.6.x86_64.raw" \
> > +  --import --noautoconsole
> > +
> > +#
> > +# Here comes the tricky part, we have to figure out when the VM comes up AND we need the ip address for ssh. So we
> > +# can check the net-dhcp leases, for our host. We have to poll, and we will poll for up 3 minutes in 6 second
> > +# intervals, so 30 poll attempts (0-29 inclusive). I don't know of a better way to do this.
> > +#
> > +# We have a full reboot + relabel, so first sleep gets us close
> > +#
> > +sleep 30
> > +for i in $(seq 0 29); do
> > +    echo "loop $i"
> > +    sleep 6s
> > +    # Get the leases, but tee it so it's easier to debug
> > +    sudo virsh net-dhcp-leases default | tee dhcp-leases.txt
> > +
> > +    # get our ipaddress
> > +    ipaddy=$(grep fedoravm dhcp-leases.txt | awk {'print $5'} | cut -d'/' -f 1-1)
>
> Looks cleaner this way:
> [...] | awk '{ print $5 }' | cut -d / -f 1)
>
> > +    if [ -n "$ipaddy" ]; then
> > +        # found it, we're done looking, print it for debug logs
> > +        echo "ipaddy: $ipaddy"
> > +        break
> > +    fi
> > +    # it's empty/not found, loop back and try again.
> > +done
> > +
> > +# Did we find it? If not die.
> > +if [ -z "$ipaddy" ]; then
> > +    echo "ipaddy zero length, exiting with error 1"
> > +    exit 1
> > +fi
> > +
> > +#
> > +# Great we have a host running, ssh into it. We specify -o so
> > +# we don't get blocked on asking to add the servers key to
> > +# our known_hosts.
> > +#
> > +ssh -tt -o StrictHostKeyChecking=no -o LogLevel=QUIET "root@$ipaddy" "/root/selinux/$TEST_RUNNER"
> > +
> > +exit 0
> > --
> > 2.17.1
>

Thanks Ondrej.

Bill

Patch
diff mbox series

diff --git a/.travis.yml b/.travis.yml
index c36e721a5e1d..63a856672f9b 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -34,6 +34,14 @@  matrix:
       env: PYVER=python3.8 RUBYLIBVER=2.7 LINKER=gold
     - compiler: clang
       env: PYVER=python3.8 RUBYLIBVER=2.7 LINKER=bfd
+  include:
+    - compiler: gcc
+      env: TRAVIS_RUN_KVM=true
+      install:
+        - skip
+      before_script:
+        - skip
+      script: scripts/ci/travis-kvm-setup.sh
 
 # Use Travis-CI Ubuntu 18.04 Bionic Beaver, "full image" variant
 sudo: required
diff --git a/scripts/ci/README.md b/scripts/ci/README.md
new file mode 100644
index 000000000000..04a134a438c2
--- /dev/null
+++ b/scripts/ci/README.md
@@ -0,0 +1,8 @@ 
+# Continuous Integration Scripts
+
+The scripts under `scripts/ci` are designed specifically
+for the Travis CI system. While nothing prevents you
+from mimicking that environment and using them locally,
+they are not applicable for general consumption. Any
+thing in this directory should never be considered as
+a stable API.
diff --git a/scripts/ci/fedora-test-runner.sh b/scripts/ci/fedora-test-runner.sh
new file mode 100755
index 000000000000..14bcf5fc469d
--- /dev/null
+++ b/scripts/ci/fedora-test-runner.sh
@@ -0,0 +1,89 @@ 
+#!/usr/bin/env bash
+
+set -ev
+
+# CI Debug output if things go squirrely.
+getenforce
+id -Z
+nproc
+pwd
+
+# Turn off enforcing for the setup to prevent any weirdness from breaking
+# the CI.
+setenforce 0
+
+dnf clean all -y
+dnf install -y \
+    --allowerasing \
+    --skip-broken \
+    git \
+    audit-libs-devel \
+    bison \
+    bzip2-devel \
+    CUnit-devel \
+    diffutils \
+    flex \
+    gcc \
+    gettext \
+    glib2-devel \
+    make \
+    libcap-devel \
+    libcap-ng-devel \
+    pam-devel \
+    pcre-devel \
+    xmlto \
+    python3-devel \
+    ruby-devel \
+    swig \
+    perl-Test \
+    perl-Test-Harness \
+    perl-Test-Simple \
+    selinux-policy-devel \
+    gcc \
+    libselinux-devel \
+    net-tools \
+    netlabel_tools \
+    iptables \
+    lksctp-tools-devel \
+    attr \
+    libbpf-devel \
+    keyutils-libs-devel \
+    kernel-devel \
+    quota \
+    xfsprogs-devel \
+    libuuid-devel \
+    kernel-devel-$(uname -r) \
+    kernel-modules-$(uname -r)
+
+#
+# Move to selinux code and build
+#
+cd ~/selinux
+
+# Show HEAD commit for sanity checking
+git log -1
+
+#
+# Build and replace userspace components
+#
+# Note: You can't use parallel builds here (make -jX), you'll end up
+# with race conditions that manifest like:
+# semanage_store.lo: file not recognized: file format not recognized
+#
+make LIBDIR=/usr/lib64 SHLIBDIR=/lib64 install install-pywrap relabel
+
+#
+# Get the selinux testsuite, but don't clone it in ~/selinux, move to ~
+# first.
+#
+cd ~
+git clone --depth=1 https://github.com/SELinuxProject/selinux-testsuite.git
+cd selinux-testsuite
+
+# The testsuite must be run in enforcing mode
+setenforce 1
+
+#
+# Run the test suite
+#
+make test
diff --git a/scripts/ci/travis-kvm-setup.sh b/scripts/ci/travis-kvm-setup.sh
new file mode 100755
index 000000000000..66606e9d4a5b
--- /dev/null
+++ b/scripts/ci/travis-kvm-setup.sh
@@ -0,0 +1,113 @@ 
+#!/usr/bin/env bash
+
+set -ev
+
+TEST_RUNNER="scripts/ci/fedora-test-runner.sh"
+
+#
+# Travis gives us 7.5GB of RAM and two cores:
+# https://docs.travis-ci.com/user/reference/overview/
+#
+MEMORY=4096
+VCPUS=2
+
+# Install these here so other builds don't have to wait on these deps to download and install
+sudo apt-get install qemu-kvm libvirt-bin virtinst bridge-utils cpu-checker libguestfs-tools
+
+sudo usermod -a -G kvm,libvirt,libvirt-qemu $USER
+
+# Verify that KVM is working, useful if Travis every changes anything.
+kvm-ok
+
+sudo systemctl enable libvirtd
+sudo systemctl start libvirtd
+
+# Set up a key so we can ssh into the VM
+ssh-keygen -N "" -f "$HOME/.ssh/id_rsa"
+
+#
+# Get the Fedora Cloud Image, It is a base image that small and ready to go, extract it and modify it with virt-sysprep
+#  - https://alt.fedoraproject.org/en/verify.html
+cd $HOME
+wget https://download.fedoraproject.org/pub/fedora/linux/releases/32/Cloud/x86_64/images/Fedora-Cloud-Base-32-1.6.x86_64.raw.xz
+
+# Verify the image
+curl https://getfedora.org/static/fedora.gpg | gpg --import
+wget https://getfedora.org/static/checksums/Fedora-Cloud-32-1.6-x86_64-CHECKSUM
+gpg --verify-files *-CHECKSUM
+sha256sum --ignore-missing -c *-CHECKSUM
+
+# Extract the image
+unxz -T0 Fedora-Cloud-Base-32-1.6.x86_64.raw.xz
+
+# Search is needed for $HOME so virt service can access the image file.
+chmod a+x $HOME
+
+#
+# Modify the virtual image to:
+#   - Enable a login, we just use root
+#   - Enable passwordless login
+#     - Force a relabel to fix labels on ssh keys
+#
+sudo virt-sysprep -a "$HOME/Fedora-Cloud-Base-32-1.6.x86_64.raw" \
+  --root-password password:123456 \
+  --hostname fedoravm \
+  --append-line '/etc/ssh/sshd_config:PermitRootLogin yes' \
+  --append-line '/etc/ssh/sshd_config:PubkeyAuthentication yes' \
+  --mkdir /root/.ssh \
+  --upload "$HOME/.ssh/id_rsa.pub:/root/.ssh/authorized_keys" \
+  --chmod '0600:/root/.ssh/authorized_keys' \
+  --run-command 'chown root:root /root/.ssh/authorized_keys' \
+  --copy-in "$TRAVIS_BUILD_DIR:/root" \
+  --network \
+  --selinux-relabel
+
+#
+# Now we create a domain by using virt-install. This not only creates the domain, but runs the VM as well
+# It should be ready to go for ssh, once ssh starts.
+#
+sudo virt-install \
+  --name fedoravm \
+  --memory $MEMORY \
+  --vcpus $VCPUS \
+  --disk "$HOME/Fedora-Cloud-Base-32-1.6.x86_64.raw" \
+  --import --noautoconsole
+
+#
+# Here comes the tricky part, we have to figure out when the VM comes up AND we need the ip address for ssh. So we
+# can check the net-dhcp leases, for our host. We have to poll, and we will poll for up 3 minutes in 6 second
+# intervals, so 30 poll attempts (0-29 inclusive). I don't know of a better way to do this.
+#
+# We have a full reboot + relabel, so first sleep gets us close
+#
+sleep 30
+for i in $(seq 0 29); do
+    echo "loop $i"
+    sleep 6s
+    # Get the leases, but tee it so it's easier to debug
+    sudo virsh net-dhcp-leases default | tee dhcp-leases.txt
+
+    # get our ipaddress
+    ipaddy=$(grep fedoravm dhcp-leases.txt | awk {'print $5'} | cut -d'/' -f 1-1)
+    if [ -n "$ipaddy" ]; then
+        # found it, we're done looking, print it for debug logs
+        echo "ipaddy: $ipaddy"
+        break
+    fi
+    # it's empty/not found, loop back and try again.
+done
+
+# Did we find it? If not die.
+if [ -z "$ipaddy" ]; then
+    echo "ipaddy zero length, exiting with error 1"
+    exit 1
+fi
+
+#
+# Great we have a host running, ssh into it. We specify -o so
+# we don't get blocked on asking to add the servers key to
+# our known_hosts.
+#
+ssh -tt -o StrictHostKeyChecking=no -o LogLevel=QUIET "root@$ipaddy" "/root/selinux/$TEST_RUNNER"
+
+exit 0