| Message ID | 20181023235750.103146-1-brendanhiggins@google.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | kunit: introduce KUnit, the Linux kernel unit testing framework | expand |
On Tue, Oct 23, 2018 at 04:57:36PM -0700, Brendan Higgins wrote: > This patch set proposes KUnit, a lightweight unit testing and mocking > framework for the Linux kernel. > > Unlike Autotest and kselftest, KUnit is a true unit testing framework; > it does not require installing the kernel on a test machine or in a VM > and does not require tests to be written in userspace running on a host > kernel. Additionally, KUnit is fast: From invocation to completion KUnit > can run several dozen tests in under a second. Currently, the entire > KUnit test suite for KUnit runs in under a second from the initial > invocation (build time excluded). > > KUnit is heavily inspired by JUnit, Python's unittest.mock, and > Googletest/Googlemock for C++. KUnit provides facilities for defining > unit test cases, grouping related test cases into test suites, providing > common infrastructure for running tests, mocking, spying, and much more. > > ## What's so special about unit testing? > > A unit test is supposed to test a single unit of code in isolation, > hence the name. There should be no dependencies outside the control of > the test; this means no external dependencies, which makes tests orders > of magnitudes faster. Likewise, since there are no external dependencies, > there are no hoops to jump through to run the tests. Additionally, this > makes unit tests deterministic: a failing unit test always indicates a > problem. Finally, because unit tests necessarily have finer granularity, > they are able to test all code paths easily solving the classic problem > of difficulty in exercising error handling code. > > ## Is KUnit trying to replace other testing frameworks for the kernel? > > No. Most existing tests for the Linux kernel are end-to-end tests, which > have their place. A well tested system has lots of unit tests, a > reasonable number of integration tests, and some end-to-end tests. KUnit > is just trying to address the unit test space which is currently not > being addressed. > > ## More information on KUnit > > There is a bunch of documentation near the end of this patch set that > describes how to use KUnit and best practices for writing unit tests. > For convenience I am hosting the compiled docs here: > https://google.github.io/kunit-docs/third_party/kernel/docs/ > > ## Changes Since Last Version > > - Updated patchset to apply cleanly on 4.19. > - Stripped down patchset to focus on just the core features (I dropped > mocking, spying, and the MMIO stuff for now; you can find these > patches here: https://kunit-review.googlesource.com/c/linux/+/1132), > as suggested by Rob. > - Cleaned up some of the commit messages and tweaked commit order a > bit based on suggestions. Do you have some example unit tests somewhere? The docs are all neat, but real example helps a lot with the tried&true art of copypasting :-) I'd like to give this a test spin with some of the unit tests we already have in drm. And especially figuring out how we could integrate this with our existing infrastructure. -Daniel
On 10/23/2018 05:57 PM, Brendan Higgins wrote: > This patch set proposes KUnit, a lightweight unit testing and mocking > framework for the Linux kernel. > > Unlike Autotest and kselftest, KUnit is a true unit testing framework; > it does not require installing the kernel on a test machine or in a VM > and does not require tests to be written in userspace running on a host > kernel. Additionally, KUnit is fast: From invocation to completion KUnit > can run several dozen tests in under a second. Currently, the entire > KUnit test suite for KUnit runs in under a second from the initial > invocation (build time excluded). > > KUnit is heavily inspired by JUnit, Python's unittest.mock, and > Googletest/Googlemock for C++. KUnit provides facilities for defining > unit test cases, grouping related test cases into test suites, providing > common infrastructure for running tests, mocking, spying, and much more. > > ## What's so special about unit testing? > > A unit test is supposed to test a single unit of code in isolation, > hence the name. There should be no dependencies outside the control of > the test; this means no external dependencies, which makes tests orders > of magnitudes faster. Likewise, since there are no external dependencies, > there are no hoops to jump through to run the tests. Additionally, this > makes unit tests deterministic: a failing unit test always indicates a > problem. Finally, because unit tests necessarily have finer granularity, > they are able to test all code paths easily solving the classic problem > of difficulty in exercising error handling code. > > ## Is KUnit trying to replace other testing frameworks for the kernel? > > No. Most existing tests for the Linux kernel are end-to-end tests, which > have their place. A well tested system has lots of unit tests, a > reasonable number of integration tests, and some end-to-end tests. KUnit > is just trying to address the unit test space which is currently not > being addressed. > > ## More information on KUnit > > There is a bunch of documentation near the end of this patch set that > describes how to use KUnit and best practices for writing unit tests. > For convenience I am hosting the compiled docs here: > https://google.github.io/kunit-docs/third_party/kernel/docs/ > > ## Changes Since Last Version > > - Updated patchset to apply cleanly on 4.19. > - Stripped down patchset to focus on just the core features (I dropped > mocking, spying, and the MMIO stuff for now; you can find these > patches here: https://kunit-review.googlesource.com/c/linux/+/1132), > as suggested by Rob. > - Cleaned up some of the commit messages and tweaked commit order a > bit based on suggestions. > I am a bit behind on reviewing this patch series. Just a quick note that I will start looking at these early next week. thanks, -- Shuah
On Wed, Oct 24, 2018 at 2:14 AM Daniel Vetter <daniel@ffwll.ch> wrote: > > On Tue, Oct 23, 2018 at 04:57:36PM -0700, Brendan Higgins wrote: <snip> > > ## Changes Since Last Version > > > > - Updated patchset to apply cleanly on 4.19. > > - Stripped down patchset to focus on just the core features (I dropped > > mocking, spying, and the MMIO stuff for now; you can find these > > patches here: https://kunit-review.googlesource.com/c/linux/+/1132), > > as suggested by Rob. > > - Cleaned up some of the commit messages and tweaked commit order a > > bit based on suggestions. > > Do you have some example unit tests somewhere? The docs are all neat, but > real example helps a lot with the tried&true art of copypasting :-) Yes! So I have quite a few tests for KUnit itself, but they are mostly in the patches that I cut from the previous version of the RFC: https://kunit.googlesource.com/linux/+/c58019fb4fe15f820e51f857ae4ff14cd34075af/kunit/mock-macro-test.c https://kunit.googlesource.com/linux/+/c58019fb4fe15f820e51f857ae4ff14cd34075af/kunit/mock-test.c https://kunit.googlesource.com/linux/+/c58019fb4fe15f820e51f857ae4ff14cd34075af/kunit/string-stream-test.c https://kunit.googlesource.com/linux/+/c58019fb4fe15f820e51f857ae4ff14cd34075af/kunit/test-stream-test.c https://kunit.googlesource.com/linux/+/c58019fb4fe15f820e51f857ae4ff14cd34075af/kunit/test-test.c There are some in this patchset (like here: https://lore.kernel.org/patchwork/patch/1002904/), but I should probably include some more... In any case, if you are looking for a more "realistic" example, you could look at these sets of tests, which I think is pretty good, and actually found a couple bugs in existing code: https://kunit.googlesource.com/linux/+/e10484ad2f9fc7926412ec84739fe105981b4771/drivers/i2c/busses/i2c-aspeed-test.c https://kunit.googlesource.com/linux/+/4b41b1594ffeab70f1b4364721940cb909509d97/net/ipv6/route-test.c > > I'd like to give this a test spin with some of the unit tests we already > have in drm. And especially figuring out how we could integrate this with > our existing infrastructure. Awesome! I am looking forward to hearing how it goes! Cheers
Hi Brendan, On 10/23/2018 05:57 PM, Brendan Higgins wrote: > This patch set proposes KUnit, a lightweight unit testing and mocking > framework for the Linux kernel. > > Unlike Autotest and kselftest, KUnit is a true unit testing framework; > it does not require installing the kernel on a test machine or in a VM > and does not require tests to be written in userspace running on a host > kernel. Additionally, KUnit is fast: From invocation to completion KUnit > can run several dozen tests in under a second. Currently, the entire > KUnit test suite for KUnit runs in under a second from the initial > invocation (build time excluded). > > KUnit is heavily inspired by JUnit, Python's unittest.mock, and > Googletest/Googlemock for C++. KUnit provides facilities for defining > unit test cases, grouping related test cases into test suites, providing > common infrastructure for running tests, mocking, spying, and much more. > > ## What's so special about unit testing? > > A unit test is supposed to test a single unit of code in isolation, > hence the name. There should be no dependencies outside the control of > the test; this means no external dependencies, which makes tests orders > of magnitudes faster. Likewise, since there are no external dependencies, > there are no hoops to jump through to run the tests. Additionally, this > makes unit tests deterministic: a failing unit test always indicates a > problem. Finally, because unit tests necessarily have finer granularity, > they are able to test all code paths easily solving the classic problem > of difficulty in exercising error handling code. > > ## Is KUnit trying to replace other testing frameworks for the kernel? > > No. Most existing tests for the Linux kernel are end-to-end tests, which > have their place. A well tested system has lots of unit tests, a > reasonable number of integration tests, and some end-to-end tests. KUnit > is just trying to address the unit test space which is currently not > being addressed. > > ## More information on KUnit > > There is a bunch of documentation near the end of this patch set that > describes how to use KUnit and best practices for writing unit tests. > For convenience I am hosting the compiled docs here: > https://google.github.io/kunit-docs/third_party/kernel/docs/ > > ## Changes Since Last Version > > - Updated patchset to apply cleanly on 4.19. > - Stripped down patchset to focus on just the core features (I dropped > mocking, spying, and the MMIO stuff for now; you can find these > patches here: https://kunit-review.googlesource.com/c/linux/+/1132), > as suggested by Rob. > - Cleaned up some of the commit messages and tweaked commit order a > bit based on suggestions. > Framework looks good. I think it would be helpful to include a real test in the patch series to get a feel for how effective it is. On one hand, KUnit stands on its own as its own and maybe it should be placed in under tools/testing/KUnit, however I am wondering would it be beneficial for the framework to under selftests. I am a bit concerned about the number of test framework we have at the moment and are we running the risk of fragmenting the landscape. I am concerned if this would lead to developer confusion as to where to add tests. That being said, I don't have a strong opinion one way or the other. btw I started playing with kunit following the instructions and ran into problems: ./tools/testing/kunit/kunit.py usage: kunit.py [-h] {run,new} ... Helps writing and running KUnit tests. positional arguments: {run,new} run Runs KUnit tests. new Prints out boilerplate for writing new tests. optional arguments: -h, --help show this help message and exit ./tools/testing/kunit/kunit.py run Regenerating .config ... ERROR:root:Provided Kconfig is not contained in validated .config! thanks, -- Shuah
On Fri, Nov 2, 2018 at 11:23 AM Shuah Khan <shuah@kernel.org> wrote: > > Hi Brendan, <snip> > Framework looks good. I think it would be helpful to include a real test Great to hear! > in the patch series to get a feel for how effective it is. Alright, will do. Rob suggested converting https://elixir.bootlin.com/linux/v4.19/source/drivers/of/unittest.c to KUnit, so that might offer a good comparison. > > On one hand, KUnit stands on its own as its own and maybe it should be placed in > under tools/testing/KUnit, however I am wondering would it be beneficial for the > framework to under selftests. > > I am a bit concerned about the number of test framework we have at the moment and > are we running the risk of fragmenting the landscape. I am concerned if this would > lead to developer confusion as to where to add tests. On one hand separating them makes sense because they are different things. kselftest seems to do its job, writing end-to-end tests, pretty well. Trying to associate KUnit with it could also be confusing since they have different requirements and should be used for different things. But on the other hand, you are right, it would be nice to have a coherent experience for developers. In either case, we will still need developers to understand when they should be writing unit tests and when they should be writing end-to-end tests. So in the end, I don't think it will matter too much, we will still need to address this confusion either way. The main reason I made it separate was just because I thought people would find it strange to have essentially normal kernel code depend on something in `tools/testing/`. You, me, and Greg had talked about this elsewhere, in summary, I am trying to do in-tree unit testing, so tests should live side-by-side with the code it tests, and the tests run at the same level of abstraction as the code that is under test. The tests do not run as userspace programs on an installed kernel under test. The goal is to be able to run arbitrary kernel code in isolation. Although it would be *possible* to put KUnit under `tools/testing/`, my understanding is that `tools/testing/` is for programs that compile independently of the kernel, rather they do not link against anything built-in to the kernel. (I saw there are some test drivers, but they seem to be modeled as out of tree drivers that have no option to be built into the kernel.) KUnit does not fall into this category as KUnit links directly against arbitrary kernel code, and actually reuses kernel infrastructure to provide an environment to run kernel code in. I do not feel strongly about where KUnit lives, but by my understanding, putting KUnit in `tools/testing/` might break assumptions about the relationship between `tools/` and the rest of the kernel. I know you previously said that there are unit tests under kselftest (like maybe this one: https://elixir.bootlin.com/linux/v4.19/source/lib/locking-selftest.c ?). I agree that something like this example is trying to be a unit test, but the kselftest infrastructure is built around the idea of booting kernels and running tests against them; I know that some kselftests load test modules, but the point is the intention is to run the kernel somewhere. My end goal with KUnit is that we should not need to run a kernel anywhere (I know right now I am using UML, but the goal is to get away from that). So I think that decidedly makes them two different things. The number of test frameworks is a problem, I definitely agree with that. Most of the other tests I have seen for the kernel are indisputably end-to-end tests; it seems that kselftest is decidedly the favorite, maybe we should move those under kselftest? I know there are other things that don't fit. Maybe we could start off with some suggestions for best practices for what to use and when? > > That being said, I don't have a strong opinion one way or the other. I think it is pretty common on other projects to have unit tests separated from end-to-end tests in some way or another. You want to be able to run unit tests quickly and all the time, whereas you usually only want to run your end-to-end tests when you make a submission or a release. Regardless of where we put KUnit and what its relationship to kselftest becomes, we should definitely make it easy to run unit tests separate from all the end-to-end tests. > > btw I started playing with kunit following the instructions and ran into problems: > > ./tools/testing/kunit/kunit.py > usage: kunit.py [-h] {run,new} ... > > Helps writing and running KUnit tests. > > positional arguments: > {run,new} > run Runs KUnit tests. > new Prints out boilerplate for writing new tests. > > optional arguments: > -h, --help show this help message and exit > > ./tools/testing/kunit/kunit.py run > Regenerating .config ... > ERROR:root:Provided Kconfig is not contained in validated .config! Oh sorry, I need to write some documentation for that. I take it you checked out https://kunit.googlesource.com/linux/+/kunit/alpha/master ? If so, you need a "kunitconfig" in the root of your kernel source with the following Kconfig options: CONFIG_TEST=y CONFIG_TEST_TEST=y CONFIG_EXAMPLE_TEST=y I am guessing what happened is that you used the "stable" branch which we have been using internally, but follow the instructions I posted for this patchset. Some of the Kconfig options have deviated between them. Sorry about the confusion. > > thanks, > -- Shuah Thank you!
On 11/6/18 5:17 PM, Brendan Higgins wrote: > On Fri, Nov 2, 2018 at 11:23 AM Shuah Khan <shuah@kernel.org> wrote: >> >> Hi Brendan, > <snip> >> Framework looks good. I think it would be helpful to include a real test > > Great to hear! > >> in the patch series to get a feel for how effective it is. > > Alright, will do. Rob suggested converting > https://elixir.bootlin.com/linux/v4.19/source/drivers/of/unittest.c to > KUnit, so that might offer a good comparison. drivers/of/unittest.c might be a bit bigger and more complex test than you want to start with. -Frank < snip >
On Wed, Nov 7, 2018 at 9:46 AM Frank Rowand <frowand.list@gmail.com> wrote: > > On 11/6/18 5:17 PM, Brendan Higgins wrote: > > On Fri, Nov 2, 2018 at 11:23 AM Shuah Khan <shuah@kernel.org> wrote: > >> > >> Hi Brendan, > > <snip> > >> Framework looks good. I think it would be helpful to include a real test > > > > Great to hear! > > > >> in the patch series to get a feel for how effective it is. > > > > Alright, will do. Rob suggested converting > > https://elixir.bootlin.com/linux/v4.19/source/drivers/of/unittest.c to > > KUnit, so that might offer a good comparison. > > drivers/of/unittest.c might be a bit bigger and more complex test than > you want to start with. I already got it working under KUnit. Admittedly, structurally it is not so different, so it will require some work to make it pretty. I think now it is a question of whether you want me to start here or not. Probably best just to share what I have now along with Shuah's requested changes and we can go from there. Let me know what you think. Cheers
On Tue, 2018-10-23 at 16:57 -0700, Brendan Higgins wrote: > This patch set proposes KUnit, a lightweight unit testing and mocking > framework for the Linux kernel. > > Unlike Autotest and kselftest, KUnit is a true unit testing framework; First thanks to Hidenori Yamaji for making me aware of these threads! I'd like to kindly remind Brendan, and inform others who might have missed out on it, about our (somewhat different approach) to this space at Oracle: KTF (Kernel Test Framework). KTF is a product of our experience with driver testing within Oracle since 2011, developed as part of a project that was not made public until 2016. During the project, we experimented with multiple approaches to enable more test driven work with kernel code. KTF is the "testing within the kernel" part of this. While we realize there are quite a few testing frameworks out there, KTF makes it easy to run selected tests in kernel context directly, and as such provides a valuable approach to unit testing. Brendan, I regret you weren't at this year's testing and fuzzing workshop at LPC last week so we could have continued our discussions from last year there! I hope we can work on this for a while longer before anything gets merged. Maybe it can be topic for a longer session in a future test related workshop? Links to more info about KTF: ------ Git repo: https://github.com/oracle/ktf Formatted docs: http://heim.ifi.uio.no/~knuto/ktf/ LWN mention from my presentation at LPC'17: https://lwn.net/Articles/735034/ Oracle blog post: https://blogs.oracle.com/linux/oracles-new-kernel-test-framework-for-linux-v2 OSS'18 presentation slides: https://events.linuxfoundation.org/wp-content/uploads/2017/12/Test-Driven-Kernel-Development-Knut-Omang-Oracle.pdf In the documentation (see http://heim.ifi.uio.no/~knuto/ktf/introduction.html) we present some more motivation for choices made with KTF. As described in that introduction, we believe in a more pragmatic approach to unit testing for the kernel than the classical "mock everything" approach, except for typical heavily algorithmic components that has interfaces simple to mock, such as container implementations, or components like page table traversal algorithms or memory allocators, where the benefit of being able to "listen" on the mock interfaces needed pays handsomely off. We also used strategies to compile kernel code in user mode, for parts of the code which seemed easy enough to mock interfaces for. I also looked at UML back then, but dismissed it in favor of the more lightweight approach of just compiling the code under test directly in user mode, with a minimal partly hand crafted, flat mock layer. > KUnit is heavily inspired by JUnit, Python's unittest.mock, and > Googletest/Googlemock for C++. KUnit provides facilities for defining > unit test cases, grouping related test cases into test suites, providing > common infrastructure for running tests, mocking, spying, and much more. I am curious, with the intention of only running in user mode anyway, why not try to build upon Googletest/Googlemock (or a similar C unit test framework if C is desired), instead of "reinventing" specific kernel macros for the tests? > A unit test is supposed to test a single unit of code in isolation, > hence the name. There should be no dependencies outside the control of > the test; this means no external dependencies, which makes tests orders > of magnitudes faster. Likewise, since there are no external dependencies, > there are no hoops to jump through to run the tests. Additionally, this > makes unit tests deterministic: a failing unit test always indicates a > problem. Finally, because unit tests necessarily have finer granularity, > they are able to test all code paths easily solving the classic problem > of difficulty in exercising error handling code. I think it is clearly a trade-off here: Tests run in an isolated, mocked environment are subject to fewer external components. But the more complex the mock environment gets, the more likely it also is to be a source of errors, nondeterminism and capacity limits itself, also the mock code would typically be less well tested than the mocked parts of the kernel, so it is by no means any silver bullet, precise testing with a real kernel on real hardware is still often necessary and desired. If the code under test is fairly standalone and complex enough, building a mock environment for it and test it independently may be worth it, but pragmatically, if the same functionality can be relatively easily exercised within the kernel, that would be my first choice. I like to think about all sorts of testing and assertion making as adding more redundancy. When errors surface you can never be sure whether it is a problem with the test, the test framework, the environment, or an actual error, and all places have to be fixed before the test can pass. Thanks, Knut
On Fri, Nov 23, 2018 at 9:15 PM Knut Omang <knut.omang@oracle.com> wrote: > > On Tue, 2018-10-23 at 16:57 -0700, Brendan Higgins wrote: <snip> > > Brendan, I regret you weren't at this year's testing and fuzzing workshop at > LPC last week so we could have continued our discussions from last year there! Likewise! Unfortunately, I could not make it. So it goes. > > I hope we can work on this for a while longer before anything gets merged. > Maybe it can be topic for a longer session in a future test related workshop? I don't see why we cannot just discuss it here as we are already doing. Besides, you are mostly interested in out of tree testing, right? I don't see how this precludes anything that you are trying to do with KTF. I think the best way to develop something like what I am trying to do with KUnit is gradually, in tree, and with the active input and participation of the Linux kernel community. > > Links to more info about KTF: > ------ > Git repo: https://github.com/oracle/ktf > Formatted docs: http://heim.ifi.uio.no/~knuto/ktf/ > > LWN mention from my presentation at LPC'17: https://lwn.net/Articles/735034/ > Oracle blog post: https://blogs.oracle.com/linux/oracles-new-kernel-test-framework-for-linux-v2 > OSS'18 presentation slides: https://events.linuxfoundation.org/wp-content/uploads/2017/12/Test-Driven-Kernel-Development-Knut-Omang-Oracle.pdf > > In the documentation (see http://heim.ifi.uio.no/~knuto/ktf/introduction.html) > we present some more motivation for choices made with KTF. > As described in that introduction, we believe in a more pragmatic approach > to unit testing for the kernel than the classical "mock everything" approach, > except for typical heavily algorithmic components that has interfaces simple to mock, > such as container implementations, or components like page table traversal > algorithms or memory allocators, where the benefit of being able to "listen" > on the mock interfaces needed pays handsomely off. I am not advocating that we mock everything. Using as much real code dependencies as possible for code under test is a pretty common position, and one which I adhere to myself. > > We also used strategies to compile kernel code in user mode, > for parts of the code which seemed easy enough to mock interfaces for. > I also looked at UML back then, but dismissed it in favor of the > more lightweight approach of just compiling the code under test > directly in user mode, with a minimal partly hand crafted, flat mock layer. Is this new? When I tried your code out, I had to install the kernel objects into my host kernel. Indeed, your documentation references having to install kernel modules on the host: http://heim.ifi.uio.no/~knuto/ktf/installation.html > > > KUnit is heavily inspired by JUnit, Python's unittest.mock, and > > Googletest/Googlemock for C++. KUnit provides facilities for defining > > unit test cases, grouping related test cases into test suites, providing > > common infrastructure for running tests, mocking, spying, and much more. > > I am curious, with the intention of only running in user mode anyway, I made it possible to "port" KUnit to other architectures. Nevertheless, I believe all unit tests should be able to run without depending on hardware or some special test harness. If I see a unit test, I should not need to know anything about it just to run it. Since there is no way to have all possible hardware configurations a priori, all tests must be able to be run in a place that doesn't depend in hardware; hence they should all be runnable as just normal plane old user space programs with no dependency on a host kernel or host hardware. > why not try to build upon Googletest/Googlemock (or a similar C unit > test framework if C is desired), instead of "reinventing" > specific kernel macros for the tests? I would love to reuse Googletest/Googlemock if it were possible; I have used it a lot on other projects that I have worked on and think it is great, but I need something I can check into the Linux kernel; this requirement rules out Googletest/Googlemock since it depends on C++. There are existing frameworks for C, true, but we then need to check that into the Linux kernel or have the kernel depend on that; to me that seemed like a lot more work than just reimplementing what we need, which isn't much. Most of the hard parts are specific to the kernel anyway. > > > A unit test is supposed to test a single unit of code in isolation, > > hence the name. There should be no dependencies outside the control of > > the test; this means no external dependencies, which makes tests orders > > of magnitudes faster. Likewise, since there are no external dependencies, > > there are no hoops to jump through to run the tests. Additionally, this > > makes unit tests deterministic: a failing unit test always indicates a > > problem. Finally, because unit tests necessarily have finer granularity, > > they are able to test all code paths easily solving the classic problem > > of difficulty in exercising error handling code. > > I think it is clearly a trade-off here: Tests run in an isolated, mocked > environment are subject to fewer external components. But the more complex > the mock environment gets, the more likely it also is to be a source of errors, > nondeterminism and capacity limits itself, also the mock code would typically be > less well tested than the mocked parts of the kernel, so it is by no means any > silver bullet, precise testing with a real kernel on real hardware is still > often necessary and desired. I think you are misunderstand me. By isolation, I just mean no code under test should depend on anything outside of the control of the test environment. As I mention above, reusing real code for test dependencies is highly encouraged. As for running against hardware, yes, we need tests for that too, but that falls under integration testing; it is possible to use what I have here as a basis for that, but for right now, I just want to focus on the problem of unit testing: I think this patchset is large enough as it is. > > If the code under test is fairly standalone and complex enough, building a mock > environment for it and test it independently may be worth it, but pragmatically, > if the same functionality can be relatively easily exercised within the kernel, > that would be my first choice. > > I like to think about all sorts of testing and assertion making as adding more > redundancy. When errors surface you can never be sure whether it is a problem > with the test, the test framework, the environment, or an actual error, and > all places have to be fixed before the test can pass. Yep, I totally agree, but this is why I think test isolation is so important. If one test, or one piece of code is running that doesn't need to be, it makes debugging tests that much more complicated. Cheers!
On Mon, 2018-11-26 at 17:41 -0800, Brendan Higgins wrote: > On Fri, Nov 23, 2018 at 9:15 PM Knut Omang <knut.omang@oracle.com> wrote: > > > > On Tue, 2018-10-23 at 16:57 -0700, Brendan Higgins wrote: > <snip> > > > > Brendan, I regret you weren't at this year's testing and fuzzing workshop at > > LPC last week so we could have continued our discussions from last year there! > > Likewise! Unfortunately, I could not make it. So it goes. > > > > > I hope we can work on this for a while longer before anything gets merged. > > Maybe it can be topic for a longer session in a future test related workshop? > > I don't see why we cannot just discuss it here as we are already > doing. Yes, as long as we are not wasting all the Cc:'ed people's valuable time. > Besides, you are mostly interested in out of tree testing, > right? I don't see how this precludes anything that you are trying to > do with KTF. Both approaches provide assertion macros for running tests inside the kernel, I doubt the kernel community would like to see yet two such sets of macros, with differing syntax merged. One of the good reasons to have a *framework* is that it is widely used and understood, so that people coming from one part of the kernel can easily run, understand and relate to selected tests from another part. The goal with KTF is to allow this for any kernel, old or new, not just kernels built specifically for testing purposes. We felt that providing it as a separate git module (still open source, for anyone to contribute to) is a more efficient approach until we have more examples and experience with using it in different parts of the kernel. We can definitely post the kernel side of KTF as a patch series fairly soon if the community wants us to. Except for hybrid tests, the ktf.ko module works fine independently of any user side support, just using the debugfs interface to run and examine tests. I think there are good uses cases for having the ability to maintain a single source for tests that can be run against multiple kernels, also distro kernels that the test framework cannot expect to be able to modify, except from using the module interfaces. And there are good arguments for having (at least parts of) the test framework easily available within the kernel under test. > > Links to more info about KTF: > > ------ > > Git repo: https://github.com/oracle/ktf > > Formatted docs: http://heim.ifi.uio.no/~knuto/ktf/ > > > > LWN mention from my presentation at LPC'17: https://lwn.net/Articles/735034/ > > Oracle blog post: https://blogs.oracle.com/linux/oracles-new-kernel-test-framework-for > -linux-v2 > > OSS'18 presentation slides: https://events.linuxfoundation.org/wp-content/uploads/2017 > /12/Test-Driven-Kernel-Development-Knut-Omang-Oracle.pdf > > > > In the documentation (see http://heim.ifi.uio.no/~knuto/ktf/introduction.html) > > we present some more motivation for choices made with KTF. > > As described in that introduction, we believe in a more pragmatic approach > > to unit testing for the kernel than the classical "mock everything" approach, > > except for typical heavily algorithmic components that has interfaces simple to mock, > > such as container implementations, or components like page table traversal > > algorithms or memory allocators, where the benefit of being able to "listen" > > on the mock interfaces needed pays handsomely off. > > I am not advocating that we mock everything. Using as much real code > dependencies as possible for code under test is a pretty common > position, and one which I adhere to myself. > > > > > We also used strategies to compile kernel code in user mode, > > for parts of the code which seemed easy enough to mock interfaces for. > > I also looked at UML back then, but dismissed it in favor of the > > more lightweight approach of just compiling the code under test > > directly in user mode, with a minimal partly hand crafted, flat mock layer. > > Is this new? When I tried your code out, I had to install the kernel > objects into my host kernel. Indeed, your documentation references > having to install kernel modules on the host: > http://heim.ifi.uio.no/~knuto/ktf/installation.html That is correct. The user land testing is a separate code base that need some more care still to make it generic enough to serve as an RFC, so you haven't seen it (yet). > > > KUnit is heavily inspired by JUnit, Python's unittest.mock, and > > > Googletest/Googlemock for C++. KUnit provides facilities for defining > > > unit test cases, grouping related test cases into test suites, providing > > > common infrastructure for running tests, mocking, spying, and much more. > > > > I am curious, with the intention of only running in user mode anyway, > > I made it possible to "port" KUnit to other architectures. > Nevertheless, I believe all unit tests should be able to run without > depending on hardware or some special test harness. If I see a unit > test, I should not need to know anything about it just to run it. > Since there is no way to have all possible hardware configurations a > priori, all tests must be able to be run in a place that doesn't > depend in hardware; hence they should all be runnable as just normal > plane old user space programs with no dependency on a host kernel or > host hardware. > > > why not try to build upon Googletest/Googlemock (or a similar C unit > > test framework if C is desired), instead of "reinventing" > > specific kernel macros for the tests? > > I would love to reuse Googletest/Googlemock if it were possible; I have done it with googletest, so it *is* possible ;-) > I have used it a lot on other projects that I have worked on and think > it is great, but I need something I can check into the Linux kernel; > this requirement rules out Googletest/Googlemock since it depends on > C++. There are existing frameworks for C, true, but we then need to > check that into the Linux kernel or have the kernel depend on that; I think that is limiting the scope of it. Certainly developing the kernel already depends on a lot of user land tools, such as git for instance. Having another package to install for running tests might not be such a big deal, and saves this list from even more traffic. So figuring out what to put in the kernel repo and what to leave outside is IMHO part of the challenge. > to me that seemed like a lot more work than just reimplementing what we > need, which isn't much. Most of the hard parts are specific to the > kernel anyway. > > > A unit test is supposed to test a single unit of code in isolation, > > > hence the name. There should be no dependencies outside the control of > > > the test; this means no external dependencies, which makes tests orders > > > of magnitudes faster. Likewise, since there are no external dependencies, > > > there are no hoops to jump through to run the tests. Additionally, this > > > makes unit tests deterministic: a failing unit test always indicates a > > > problem. Finally, because unit tests necessarily have finer granularity, > > > they are able to test all code paths easily solving the classic problem > > > of difficulty in exercising error handling code. > > > > I think it is clearly a trade-off here: Tests run in an isolated, mocked > > environment are subject to fewer external components. But the more complex > > the mock environment gets, the more likely it also is to be a source of errors, > > nondeterminism and capacity limits itself, also the mock code would typically be > > less well tested than the mocked parts of the kernel, so it is by no means any > > silver bullet, precise testing with a real kernel on real hardware is still > > often necessary and desired. > > I think you are misunderstand me. By isolation, I just mean no code > under test should depend on anything outside of the control of the > test environment. And this approach is good, but it only covers the needs for a limited part of the kernel code. It can also be done entirely in user space, using user land test frameworks, the biggest challenge is in the mocking. Lots of the code in the kernel behave based on interaction with other subsystems, and with hardware. > As I mention above, reusing real code for test > dependencies is highly encouraged. > > As for running against hardware, yes, we need tests for that too, but > that falls under integration testing; it is possible to use what I > have here as a basis for that, but for right now, I just want to focus > on the problem of unit testing: I think this patchset is large enough > as it is. > > > If the code under test is fairly standalone and complex enough, building a mock > > environment for it and test it independently may be worth it, but pragmatically, > > if the same functionality can be relatively easily exercised within the kernel, > > that would be my first choice. > > > > I like to think about all sorts of testing and assertion making as adding more > > redundancy. When errors surface you can never be sure whether it is a problem > > with the test, the test framework, the environment, or an actual error, and > > all places have to be fixed before the test can pass. > > Yep, I totally agree, but this is why I think test isolation is so > important. If one test, or one piece of code is running that doesn't > need to be, it makes debugging tests that much more complicated. Yes, and another dimension to this that we have focused on with KTF, and where the Googletest frontend gives additional value, is that the tests should be both usable for smoke test and continuous integration needs, but at the same time be easy to execute standalone, test by test, with extra debugging, to allow them to be used by developers as part of a short cycle development process. I think the solution needs to allow a pragmatic approach, time an resources are limited. Sometimes an isolated test is possible, sometimes a test that executes inside a real environment is a better return on investment! Cheers, Knut
On 11/28/18 12:54 PM, Knut Omang wrote: > On Mon, 2018-11-26 at 17:41 -0800, Brendan Higgins wrote: >> On Fri, Nov 23, 2018 at 9:15 PM Knut Omang <knut.omang@oracle.com> wrote: >>> >>> On Tue, 2018-10-23 at 16:57 -0700, Brendan Higgins wrote: >> <snip> >>> >>> Brendan, I regret you weren't at this year's testing and fuzzing workshop at >>> LPC last week so we could have continued our discussions from last year there! >> >> Likewise! Unfortunately, I could not make it. So it goes. >> >>> >>> I hope we can work on this for a while longer before anything gets merged. >>> Maybe it can be topic for a longer session in a future test related workshop? >> >> I don't see why we cannot just discuss it here as we are already >> doing. > > Yes, as long as we are not wasting all the Cc:'ed people's valuable time. > >> Besides, you are mostly interested in out of tree testing, >> right? I don't see how this precludes anything that you are trying to >> do with KTF. > > Both approaches provide assertion macros for running tests inside the kernel, > I doubt the kernel community would like to see yet two such sets of macros, > with differing syntax merged. One of the good reasons to have a *framework* > is that it is widely used and understood, so that people coming from one part of the > kernel can easily run, understand and relate to selected tests from another part. > > The goal with KTF is to allow this for any kernel, old or new, not just kernels > built specifically for testing purposes. We felt that providing it as a separate git > module (still open source, for anyone to contribute to) is a more efficient approach > until we have more examples and experience with using it in different parts > of the kernel. We can definitely post the kernel side of KTF as a patch series fairly soon > if the community wants us to. Except for hybrid tests, the ktf.ko module works fine > independently of any user side support, just using the debugfs interface to run and > examine tests. > Having test framework in the kernel sources tree has benefits. It allows us (kernel developers) to do co-development of kernel features and tests for these features. It encourages developers to write regression tests. More importantly, kernel features and tests for these features are included in the same release in most cases and/or allows us freedom to do so if test framework and tests are part of the kernel sources. We have seen this with our experience with kselftest. It would not have see the same level of attention and growth if it stayed outside the kernel sources. Most kernel developers would not want to include a externally maintained module for testing. As a general rule, it has to be easy to run tests. > I think there are good uses cases for having the ability to maintain a > single source for tests that can be run against multiple kernels, > also distro kernels that the test framework cannot expect to be able to modify, > except from using the module interfaces. Same reasons as above. Having the tests included in the kernel sources makes it easier for distros to run those tests and include running them during their qualification. > > And there are good arguments for having (at least parts of) > the test framework easily available within the kernel under test. > When Kernel unit, functional, and regressions tests reside in the kernel sources, they evolve quicker as kernel developers contribute tests as part of their kernel work-flow. Maintaining tests and framework separately will make it harder to maintain them and keep them updated for us the kernel community. thanks, -- Shuah
On Wed, Nov 28, 2018 at 01:50:01PM -0700, shuah wrote: > On 11/28/18 12:54 PM, Knut Omang wrote: > > On Mon, 2018-11-26 at 17:41 -0800, Brendan Higgins wrote: > > Both approaches provide assertion macros for running tests inside the kernel, > > I doubt the kernel community would like to see yet two such sets of macros, > > with differing syntax merged. One of the good reasons to have a *framework* > > is that it is widely used and understood, so that people coming from one part of the > > kernel can easily run, understand and relate to selected tests from another part. > > > > The goal with KTF is to allow this for any kernel, old or new, not just kernels > > built specifically for testing purposes. We felt that providing it as a separate git > > module (still open source, for anyone to contribute to) is a more efficient approach > > until we have more examples and experience with using it in different parts > > of the kernel. We can definitely post the kernel side of KTF as a patch series fairly soon > > if the community wants us to. Except for hybrid tests, the ktf.ko module works fine > > independently of any user side support, just using the debugfs interface to run and > > examine tests. > > > > Having test framework in the kernel sources tree has benefits. It allows > us (kernel developers) to do co-development of kernel features and tests > for these features. Agreed! > It encourages developers to write regression tests. More importantly, > kernel features and tests for these features are included in the same > release in most cases and/or allows us freedom to do so if test framework > and tests are part of the kernel sources. > > We have seen this with our experience with kselftest. It would not have > see the same level of attention and growth if it stayed outside the > kernel sources. > > Most kernel developers would not want to include a externally maintained > module for testing. As a general rule, it has to be easy to run tests. > > > I think there are good uses cases for having the ability to maintain a > > single source for tests that can be run against multiple kernels, > > also distro kernels that the test framework cannot expect to be able to modify, > > except from using the module interfaces. > > Same reasons as above. Having the tests included in the kernel sources > makes it easier for distros to run those tests and include running them > during their qualification. Also... selftests are an example of tests which *are* upstream and yet there are teams out there using them to test these tests on older kernels. So the scripts for instance are supposed to work with older kernels. So if you expand on a feature your selftest script should detect if the new mechanism is present or not, and also be backward compatible with older kernels. > > And there are good arguments for having (at least parts of) > > the test framework easily available within the kernel under test. > > > > When Kernel unit, functional, and regressions tests reside in the kernel > sources, they evolve quicker as kernel developers contribute tests as > part of their kernel work-flow. Maintaining tests and framework > separately will make it harder to maintain them and keep them updated > for us the kernel community. Agreed! Also, I actually see no issue with having *both* kunit / ktest merged upstream. IMHO we should not be forcing people to pick one or the other but rather we should: let the best test framework win. Similar as we did with LSMs. Each test framework has its own gains / advantages. Luis