mbox series

[RFC,v1,00/31] kunit: Introducing KUnit, the Linux kernel unit testing framework

Message ID 20181016235120.138227-1-brendanhiggins@google.com (mailing list archive)
Headers show
Series kunit: Introducing KUnit, the Linux kernel unit testing framework | expand

Message

Brendan Higgins Oct. 16, 2018, 11:50 p.m. UTC
This patch set proposes KUnit, a lightweight unit testing and mocking
framework for the Linux kernel.

Unlike Autotest and kselftest, KUnit is a true unit testing framework;
it does not require installing the kernel on a test machine or in a VM
and does not require tests to be written in userspace running on a host
kernel. Additionally, KUnit is fast: From invocation to completion KUnit
can run several dozen tests in under a second. Currently, the entire
KUnit test suite for KUnit runs in under a second from the initial
invocation (build time excluded).

KUnit is heavily inspired by JUnit, Python's unittest.mock, and
Googletest/Googlemock for C++. KUnit provides facilities for defining
unit test cases, grouping related test cases into test suites, providing
common infrastructure for running tests, mocking, spying, and much more.

## What's so special about unit testing?

A unit test is supposed to test a single unit of code in isolation,
hence the name. There should be no dependencies outside the control of
the test; this means no external dependencies, which makes tests orders
of magnitudes faster. Likewise, since there are no external dependencies,
there are no hoops to jump through to run the tests. Additionally, this
makes unit tests deterministic: a failing unit test always indicates a
problem. Finally, because unit tests necessarily have finer granularity,
they are able to test all code paths easily solving the classic problem
of difficulty in exercising error handling code.

## Is KUnit trying to replace other testing frameworks for the kernel?

No. Most existing tests for the Linux kernel are end-to-end tests, which
have their place. A well tested system has lots of unit tests, a
reasonable number of integration tests, and some end-to-end tests. KUnit
is just trying to address the unit test space which is currently not
being addressed.

## More information on KUnit

There is a bunch of documentation near the end of this patch set that
describes how to use KUnit and best practices for writing unit tests.
For convenience I am hosting the compiled docs here:
https://google.github.io/kunit-docs/third_party/kernel/docs/

Comments

Daniel Vetter Oct. 17, 2018, 9:08 a.m. UTC | #1
On Tue, Oct 16, 2018 at 04:50:49PM -0700, Brendan Higgins wrote:
> This patch set proposes KUnit, a lightweight unit testing and mocking
> framework for the Linux kernel.
> 
> Unlike Autotest and kselftest, KUnit is a true unit testing framework;
> it does not require installing the kernel on a test machine or in a VM
> and does not require tests to be written in userspace running on a host
> kernel. Additionally, KUnit is fast: From invocation to completion KUnit
> can run several dozen tests in under a second. Currently, the entire
> KUnit test suite for KUnit runs in under a second from the initial
> invocation (build time excluded).
> 
> KUnit is heavily inspired by JUnit, Python's unittest.mock, and
> Googletest/Googlemock for C++. KUnit provides facilities for defining
> unit test cases, grouping related test cases into test suites, providing
> common infrastructure for running tests, mocking, spying, and much more.
> 
> ## What's so special about unit testing?
> 
> A unit test is supposed to test a single unit of code in isolation,
> hence the name. There should be no dependencies outside the control of
> the test; this means no external dependencies, which makes tests orders
> of magnitudes faster. Likewise, since there are no external dependencies,
> there are no hoops to jump through to run the tests. Additionally, this
> makes unit tests deterministic: a failing unit test always indicates a
> problem. Finally, because unit tests necessarily have finer granularity,
> they are able to test all code paths easily solving the classic problem
> of difficulty in exercising error handling code.
> 
> ## Is KUnit trying to replace other testing frameworks for the kernel?
> 
> No. Most existing tests for the Linux kernel are end-to-end tests, which
> have their place. A well tested system has lots of unit tests, a
> reasonable number of integration tests, and some end-to-end tests. KUnit
> is just trying to address the unit test space which is currently not
> being addressed.
> 
> ## More information on KUnit
> 
> There is a bunch of documentation near the end of this patch set that
> describes how to use KUnit and best practices for writing unit tests.
> For convenience I am hosting the compiled docs here:
> https://google.github.io/kunit-docs/third_party/kernel/docs/

We've started our own unit test framework (very early, and not any real
infrastructure yet) under drivers/gpu/drm/selftests. They're all meant to
test functions and small pieces of functionality of our libraries in
isolation, without any need for a real (or virtual) gpu driver. It kinda
integrates both with kselftests and with our graphics test suite, but
directly running tests using UML as part of the build process sounds much,
much better. Having proper and standardized infrastructure for kernel unit
tests sounds terrific.

In other words: I want.

Please keep dri-devel@lists.freedesktop.org in the loop on future
versions.

Cheers, Daniel
Bird, Tim Oct. 17, 2018, 5:49 p.m. UTC | #2
> -----Original Message-----
> From: Brendan Higgins
> 
> This patch set proposes KUnit, a lightweight unit testing and mocking
> framework for the Linux kernel.

I'm interested in this, and think the kernel might benefit from this,
but I have lots of questions.

> Unlike Autotest and kselftest, KUnit is a true unit testing framework;
> it does not require installing the kernel on a test machine or in a VM
> and does not require tests to be written in userspace running on a host
> kernel.


This is stated here and a few places in the documentation.  Just to clarify,
KUnit works by compiling the unit under test, along with the test code
itself, and then runs it on the machine where the compilation took
place?  Is this right?  How does cross-compiling enter into the equation?
If not what I described, then what exactly is happening?

Sorry - I haven't had time to look through the patches in detail.

Another issue is, what requirements does this place on the tested
code?  Is extra instrumentation required?  I didn't see any, but I
didn't look exhaustively at the code.

Are all unit tests stored separately from the unit-under-test, or are
they expected to be in the same directory?  Who is expected to
maintain the unit tests?  How often are they expected to change?
(Would it be every time the unit-under-test changed?)

Does the test code require the same level of expertise to write
and maintain as the unit-under-test code?  That is, could this be
a new opportunity for additional developers (especially relative
newcomers) to add value to the kernel by writing and maintaining
test code, or does this add to the already large burden of code
maintenance for our existing maintainers.

Thanks,
 -- Tim

...
Rob Herring (Arm) Oct. 17, 2018, 8:43 p.m. UTC | #3
On Tue, Oct 16, 2018 at 6:53 PM Brendan Higgins
<brendanhiggins@google.com> wrote:
>
> This patch set proposes KUnit, a lightweight unit testing and mocking
> framework for the Linux kernel.
>
> Unlike Autotest and kselftest, KUnit is a true unit testing framework;
> it does not require installing the kernel on a test machine or in a VM
> and does not require tests to be written in userspace running on a host
> kernel. Additionally, KUnit is fast: From invocation to completion KUnit
> can run several dozen tests in under a second. Currently, the entire
> KUnit test suite for KUnit runs in under a second from the initial
> invocation (build time excluded).
>
> KUnit is heavily inspired by JUnit, Python's unittest.mock, and
> Googletest/Googlemock for C++. KUnit provides facilities for defining
> unit test cases, grouping related test cases into test suites, providing
> common infrastructure for running tests, mocking, spying, and much more.

I very much like this. The DT code too has unit tests with our own,
simple infrastructure. They too can run under UML (and every other
arch).

Rob
Brendan Higgins Oct. 17, 2018, 10:22 p.m. UTC | #4
On Wed, Oct 17, 2018 at 10:49 AM <Tim.Bird@sony.com> wrote:
>
> > -----Original Message-----
> > From: Brendan Higgins
> >
> > This patch set proposes KUnit, a lightweight unit testing and mocking
> > framework for the Linux kernel.
>
> I'm interested in this, and think the kernel might benefit from this,
> but I have lots of questions.
>
Awesome!

> > Unlike Autotest and kselftest, KUnit is a true unit testing framework;
> > it does not require installing the kernel on a test machine or in a VM
> > and does not require tests to be written in userspace running on a host
> > kernel.
>
>
> This is stated here and a few places in the documentation.  Just to clarify,
> KUnit works by compiling the unit under test, along with the test code
> itself, and then runs it on the machine where the compilation took
> place?  Is this right?  How does cross-compiling enter into the equation?
> If not what I described, then what exactly is happening?
>
Yep, that's exactly right!

The test and the code under test are linked together in the same
binary and are compiled under Kbuild. Right now I am linking
everything into a UML kernel, but I would ultimately like to make
tests compile into completely independent test binaries. So each test
file would get compiled into its own test binary and would link
against only the code needed to run the test, but we are a bit of a
ways off from that.

For now, tests compile as part of a UML kernel and a test script boots
the UML kernel, tests run as part of the boot process, and the script
extracts test results and reports them.

I intentionally made it so the KUnit test libraries could be
relatively easily ported to other architectures, but in the long term,
tests that depend on being built into a real kernel that boots on real
hardware would be a lot more difficult to maintain and we would never
be able to provide the kind of resources and infrastructure as we
could for tests that run as normal user space binaries.

Does that answer your question?

> Sorry - I haven't had time to look through the patches in detail.
>
> Another issue is, what requirements does this place on the tested
> code?  Is extra instrumentation required?  I didn't see any, but I
> didn't look exhaustively at the code.
>
Nope, no special instrumentation. As long as the code under tests can
be compiled under COMPILE_TEST for the host architecture, you should
be able to use KUnit.

> Are all unit tests stored separately from the unit-under-test, or are
> they expected to be in the same directory?  Who is expected to
> maintain the unit tests?  How often are they expected to change?
> (Would it be every time the unit-under-test changed?)
>
Tests are in the same directory as the code under test. For example,
if I have a driver drivers/i2c/busses/i2c-aspeed.c, I would write a
test drivers/i2c/busses/i2c-aspeed-test.c (that's my opinion anyway).

Unit tests should be the responsibility of the person who is
responsible for the code. So one way to do this would be that unit
tests should be the responsibility of the maintainer who would in turn
require that new tests are written for any new code added, and that
all tests should pass for every patch sent for review.

A well written unit test tests public interfaces (by public I just
mean functions exported outside of a .c file, so non-static functions
and functions which are shared as a member of a struct) so a unit test
should change at a slower rate than the code under test, but you would
likely have to change the test anytime the public interface changes
(intended behavior changes, function signature changes, new public
feature added, etc). More succinctly, if the contract that your code
provide changes your test should probably change, if the contract
doesn't change, your test probably shouldn't change. Does that make
sense?

> Does the test code require the same level of expertise to write
> and maintain as the unit-under-test code?  That is, could this be
> a new opportunity for additional developers (especially relative
> newcomers) to add value to the kernel by writing and maintaining
> test code, or does this add to the already large burden of code
> maintenance for our existing maintainers.

So a couple things, in order to write a unit test, the person who
writes the test must understand what the code they are testing is
supposed to do. To some extent that will probably require someone with
some expertise to ensure that the test makes sense, and indeed a
change that breaks a test should be accompanied by a update to the
test.

On the other hand, I think understanding what pre-existing code does
and is supposed to do is much easier than writing new code from
scratch, and probably doesn't require too much expertise. I actually
did a bit of an experiment internally on this: I had some people with
no prior knowledge of the kernel write some tests for existing kernel
code and they were able to do it with only minimal guidance. I was so
happy with the result that I was already thinking that it might have
some potential for onboarding newcomers.

Now, how much burden does this add to maintainers? As someone who
pretty regularly reviews code that come in with unit tests and code
that comes in without unit tests. I find it much easier to review code
that comes in with unit tests. I would actually say that from the
standpoint of being an owner of a code base, unit tests actually
reduce the amount of work I have to do overall. Code with unit tests
is usually cleaner, the tests tell me exactly what the code is
supposed to do, and I can run the tests (or ideally have an automated
service run the tests) that tell me that the code actually does what
the tests say it should. Even when it comes to writing code I find
that writing code with unit tests ends up saving me time overall.
Randy Dunlap Oct. 17, 2018, 11:12 p.m. UTC | #5
On 10/16/18 4:50 PM, Brendan Higgins wrote:
> This patch set proposes KUnit, a lightweight unit testing and mocking
> framework for the Linux kernel.

Hi,

Just a general comment:

Documentation/process/submitting-patches.rst says:
<<Describe your changes in imperative mood, e.g. "make xyzzy do frotz"
instead of "[This patch] makes xyzzy do frotz" or "[I] changed xyzzy
to do frotz", as if you are giving orders to the codebase to change
its behaviour.>>

That also means saying things like:
... test: add
instead of ... test: added
and "enable" instead of "enabled"
and "improve" instead of "improved"
and "implement" instead of "implemented".



thanks.
Brendan Higgins Oct. 18, 2018, 2:05 a.m. UTC | #6
On Wed, Oct 17, 2018 at 4:12 PM Randy Dunlap <rdunlap@infradead.org> wrote:
>
> On 10/16/18 4:50 PM, Brendan Higgins wrote:
> > This patch set proposes KUnit, a lightweight unit testing and mocking
> > framework for the Linux kernel.
>
> Hi,
>
> Just a general comment:
>
> Documentation/process/submitting-patches.rst says:
> <<Describe your changes in imperative mood, e.g. "make xyzzy do frotz"
> instead of "[This patch] makes xyzzy do frotz" or "[I] changed xyzzy
> to do frotz", as if you are giving orders to the codebase to change
> its behaviour.>>
>

Thanks! I will fix this in the next revision.
Dan Williams Oct. 18, 2018, 3:55 a.m. UTC | #7
On Tue, Oct 16, 2018 at 4:54 PM Brendan Higgins
<brendanhiggins@google.com> wrote:
>
> This patch set proposes KUnit, a lightweight unit testing and mocking
> framework for the Linux kernel.
>
> Unlike Autotest and kselftest, KUnit is a true unit testing framework;
> it does not require installing the kernel on a test machine or in a VM
> and does not require tests to be written in userspace running on a host
> kernel. Additionally, KUnit is fast: From invocation to completion KUnit
> can run several dozen tests in under a second. Currently, the entire
> KUnit test suite for KUnit runs in under a second from the initial
> invocation (build time excluded).
>
> KUnit is heavily inspired by JUnit, Python's unittest.mock, and
> Googletest/Googlemock for C++. KUnit provides facilities for defining
> unit test cases, grouping related test cases into test suites, providing
> common infrastructure for running tests, mocking, spying, and much more.
>
> ## What's so special about unit testing?
>
> A unit test is supposed to test a single unit of code in isolation,
> hence the name. There should be no dependencies outside the control of
> the test; this means no external dependencies, which makes tests orders
> of magnitudes faster. Likewise, since there are no external dependencies,
> there are no hoops to jump through to run the tests. Additionally, this
> makes unit tests deterministic: a failing unit test always indicates a
> problem. Finally, because unit tests necessarily have finer granularity,
> they are able to test all code paths easily solving the classic problem
> of difficulty in exercising error handling code.
>
> ## Is KUnit trying to replace other testing frameworks for the kernel?
>
> No. Most existing tests for the Linux kernel are end-to-end tests, which
> have their place. A well tested system has lots of unit tests, a
> reasonable number of integration tests, and some end-to-end tests. KUnit
> is just trying to address the unit test space which is currently not
> being addressed.
>
> ## More information on KUnit
>
> There is a bunch of documentation near the end of this patch set that
> describes how to use KUnit and best practices for writing unit tests.
> For convenience I am hosting the compiled docs here:
> https://google.github.io/kunit-docs/third_party/kernel/docs/

Nice! I've been using mocking techniques in kernel code for the
libnvdimm test infrastructure in tools/testing/nvdimm/. It's part unit
test infrastructure, part emulation, and I've always had the feeling
it's all a bit too adhoc. I'm going to take a look and see what can be
converted to kunit. Please include linux-nvdimm@lists.01.org on future
postings.

I'll shamelessly plug my lwn article about unit testing
https://lwn.net/Articles/654071/ because it's always good to find
fellow co-travelers to compare notes and advocate for more test
oriented kernel development.
Brendan Higgins Oct. 19, 2018, 6:27 a.m. UTC | #8
On Wed, Oct 17, 2018 at 8:55 PM Dan Williams <dan.j.williams@intel.com> wrote:
<snip>
> > ## More information on KUnit
> >
> > There is a bunch of documentation near the end of this patch set that
> > describes how to use KUnit and best practices for writing unit tests.
> > For convenience I am hosting the compiled docs here:
> > https://google.github.io/kunit-docs/third_party/kernel/docs/
>
> Nice! I've been using mocking techniques in kernel code for the
> libnvdimm test infrastructure in tools/testing/nvdimm/. It's part unit
> test infrastructure, part emulation, and I've always had the feeling
> it's all a bit too adhoc. I'm going to take a look and see what can be
> converted to kunit. Please include linux-nvdimm@lists.01.org on future
> postings.

Great to hear!

Interesting, is this kind of like the nfsim stuff?
>
> I'll shamelessly plug my lwn article about unit testing
> https://lwn.net/Articles/654071/ because it's always good to find
> fellow co-travelers to compare notes and advocate for more test
> oriented kernel development.

Most definitely! I will take a look, and be in touch.

Cheers!