mbox series

[v18,00/19] kunit: introduce KUnit, the Linux kernel unit testing framework

Message ID 20190923090249.127984-1-brendanhiggins@google.com (mailing list archive)
Headers show
Series kunit: introduce KUnit, the Linux kernel unit testing framework | expand

Message

Brendan Higgins Sept. 23, 2019, 9:02 a.m. UTC
## TL;DR

This revision addresses comments from Linus[1] and Randy[2], by moving
top level `kunit/` directory to `lib/kunit/` and likewise moves top
level Kconfig entry under lib/Kconfig.debug, so the KUnit submenu now
shows up under the "Kernel Hacking" menu.

As a consequence of this, I rewrote patch 06/18 (kbuild: enable building
KUnit) - now 06/19 (lib: enable building KUnit in lib/), and now needs
to be re-acked/reviewed.

## Background

This patch set proposes KUnit, a lightweight unit testing and mocking
framework for the Linux kernel.

Unlike Autotest and kselftest, KUnit is a true unit testing framework;
it does not require installing the kernel on a test machine or in a VM
(however, KUnit still allows you to run tests on test machines or in VMs
if you want[3]) and does not require tests to be written in userspace
running on a host kernel. Additionally, KUnit is fast: From invocation
to completion KUnit can run several dozen tests in about a second.
Currently, the entire KUnit test suite for KUnit runs in under a second
from the initial invocation (build time excluded).

KUnit is heavily inspired by JUnit, Python's unittest.mock, and
Googletest/Googlemock for C++. KUnit provides facilities for defining
unit test cases, grouping related test cases into test suites, providing
common infrastructure for running tests, mocking, spying, and much more.

### What's so special about unit testing?

A unit test is supposed to test a single unit of code in isolation,
hence the name. There should be no dependencies outside the control of
the test; this means no external dependencies, which makes tests orders
of magnitudes faster. Likewise, since there are no external dependencies,
there are no hoops to jump through to run the tests. Additionally, this
makes unit tests deterministic: a failing unit test always indicates a
problem. Finally, because unit tests necessarily have finer granularity,
they are able to test all code paths easily solving the classic problem
of difficulty in exercising error handling code.

### Is KUnit trying to replace other testing frameworks for the kernel?

No. Most existing tests for the Linux kernel are end-to-end tests, which
have their place. A well tested system has lots of unit tests, a
reasonable number of integration tests, and some end-to-end tests. KUnit
is just trying to address the unit test space which is currently not
being addressed.

### More information on KUnit

There is a bunch of documentation near the end of this patch set that
describes how to use KUnit and best practices for writing unit tests.
For convenience I am hosting the compiled docs here[4].

Additionally for convenience, I have applied these patches to a
branch[5]. The repo may be cloned with:
git clone https://kunit.googlesource.com/linux
This patchset is on the kunit/initial/v5.3/v18 branch.

## History since v15

### v18

 - Addrssed comments on 07/19 (kunit: test: add initial tests) from
   Randy Dunlap by removing redundant dependencies from Kconfig entries.

### v17

 - Addressed comments on 06/19 (lib: enable building KUnit in lib/) from
   Stephen Boyd by moving KUnit submenu ahead of Runtime Testing
   submenu.

### v16

 - Addressed comments from Linus Torvalds by moving all kunit/ paths to
   lib/kunit/.
 - Addressed comments by Randy Dunlap by moving KUnit Kconfig under
   lib/Kconfig.debug so the KUnit submenu shows up under the "Kernel
   Hacking" menu.

[1] https://www.lkml.org/lkml/2019/9/20/696
[2] https://www.lkml.org/lkml/2019/9/20/738
[3] https://google.github.io/kunit-docs/third_party/kernel/docs/usage.html#kunit-on-non-uml-architectures
[4] https://google.github.io/kunit-docs/third_party/kernel/docs/
[5] https://kunit.googlesource.com/linux/+/kunit/initial/v5.3/v18

---
Avinash Kondareddy (1):
  kunit: test: add tests for KUnit managed resources

Brendan Higgins (16):
  kunit: test: add KUnit test runner core
  kunit: test: add test resource management API
  kunit: test: add string_stream a std::stream like string builder
  kunit: test: add assertion printing library
  kunit: test: add the concept of expectations
  lib: enable building KUnit in lib/
  kunit: test: add initial tests
  objtool: add kunit_try_catch_throw to the noreturn list
  kunit: test: add support for test abort
  kunit: test: add tests for kunit test abort
  kunit: test: add the concept of assertions
  kunit: defconfig: add defconfigs for building KUnit tests
  Documentation: kunit: add documentation for KUnit
  MAINTAINERS: add entry for KUnit the unit testing framework
  MAINTAINERS: add proc sysctl KUnit test to PROC SYSCTL section
  kunit: fix failure to build without printk

Felix Guo (1):
  kunit: tool: add Python wrappers for running KUnit tests

Iurii Zaikin (1):
  kernel/sysctl-test: Add null pointer test for sysctl.c:proc_dointvec()

 Documentation/dev-tools/index.rst             |    1 +
 Documentation/dev-tools/kunit/api/index.rst   |   16 +
 Documentation/dev-tools/kunit/api/test.rst    |   11 +
 Documentation/dev-tools/kunit/faq.rst         |   62 +
 Documentation/dev-tools/kunit/index.rst       |   79 +
 Documentation/dev-tools/kunit/start.rst       |  180 ++
 Documentation/dev-tools/kunit/usage.rst       |  576 +++++++
 MAINTAINERS                                   |   13 +
 arch/um/configs/kunit_defconfig               |    3 +
 include/kunit/assert.h                        |  356 ++++
 include/kunit/string-stream.h                 |   51 +
 include/kunit/test.h                          | 1490 +++++++++++++++++
 include/kunit/try-catch.h                     |   75 +
 kernel/Makefile                               |    2 +
 kernel/sysctl-test.c                          |  392 +++++
 lib/Kconfig.debug                             |   13 +
 lib/Makefile                                  |    2 +
 lib/kunit/Kconfig                             |   36 +
 lib/kunit/Makefile                            |    9 +
 lib/kunit/assert.c                            |  141 ++
 lib/kunit/example-test.c                      |   88 +
 lib/kunit/string-stream-test.c                |   52 +
 lib/kunit/string-stream.c                     |  217 +++
 lib/kunit/test-test.c                         |  331 ++++
 lib/kunit/test.c                              |  478 ++++++
 lib/kunit/try-catch.c                         |  118 ++
 tools/objtool/check.c                         |    1 +
 tools/testing/kunit/.gitignore                |    3 +
 tools/testing/kunit/configs/all_tests.config  |    3 +
 tools/testing/kunit/kunit.py                  |  136 ++
 tools/testing/kunit/kunit_config.py           |   66 +
 tools/testing/kunit/kunit_kernel.py           |  149 ++
 tools/testing/kunit/kunit_parser.py           |  310 ++++
 tools/testing/kunit/kunit_tool_test.py        |  206 +++
 .../test_is_test_passed-all_passed.log        |   32 +
 .../test_data/test_is_test_passed-crash.log   |   69 +
 .../test_data/test_is_test_passed-failure.log |   36 +
 .../test_is_test_passed-no_tests_run.log      |   75 +
 .../test_output_isolated_correctly.log        |  106 ++
 .../test_data/test_read_from_file.kconfig     |   17 +
 40 files changed, 6001 insertions(+)
 create mode 100644 Documentation/dev-tools/kunit/api/index.rst
 create mode 100644 Documentation/dev-tools/kunit/api/test.rst
 create mode 100644 Documentation/dev-tools/kunit/faq.rst
 create mode 100644 Documentation/dev-tools/kunit/index.rst
 create mode 100644 Documentation/dev-tools/kunit/start.rst
 create mode 100644 Documentation/dev-tools/kunit/usage.rst
 create mode 100644 arch/um/configs/kunit_defconfig
 create mode 100644 include/kunit/assert.h
 create mode 100644 include/kunit/string-stream.h
 create mode 100644 include/kunit/test.h
 create mode 100644 include/kunit/try-catch.h
 create mode 100644 kernel/sysctl-test.c
 create mode 100644 lib/kunit/Kconfig
 create mode 100644 lib/kunit/Makefile
 create mode 100644 lib/kunit/assert.c
 create mode 100644 lib/kunit/example-test.c
 create mode 100644 lib/kunit/string-stream-test.c
 create mode 100644 lib/kunit/string-stream.c
 create mode 100644 lib/kunit/test-test.c
 create mode 100644 lib/kunit/test.c
 create mode 100644 lib/kunit/try-catch.c
 create mode 100644 tools/testing/kunit/.gitignore
 create mode 100644 tools/testing/kunit/configs/all_tests.config
 create mode 100755 tools/testing/kunit/kunit.py
 create mode 100644 tools/testing/kunit/kunit_config.py
 create mode 100644 tools/testing/kunit/kunit_kernel.py
 create mode 100644 tools/testing/kunit/kunit_parser.py
 create mode 100755 tools/testing/kunit/kunit_tool_test.py
 create mode 100644 tools/testing/kunit/test_data/test_is_test_passed-all_passed.log
 create mode 100644 tools/testing/kunit/test_data/test_is_test_passed-crash.log
 create mode 100644 tools/testing/kunit/test_data/test_is_test_passed-failure.log
 create mode 100644 tools/testing/kunit/test_data/test_is_test_passed-no_tests_run.log
 create mode 100644 tools/testing/kunit/test_data/test_output_isolated_correctly.log
 create mode 100644 tools/testing/kunit/test_data/test_read_from_file.kconfig

Comments

Theodore Ts'o Oct. 4, 2019, 9:38 p.m. UTC | #1
On Mon, Sep 23, 2019 at 02:02:30AM -0700, Brendan Higgins wrote:
> ## TL;DR
> 
> This revision addresses comments from Linus[1] and Randy[2], by moving
> top level `kunit/` directory to `lib/kunit/` and likewise moves top
> level Kconfig entry under lib/Kconfig.debug, so the KUnit submenu now
> shows up under the "Kernel Hacking" menu.

This question is primarily directed at Shuah and Linus....

What's the current status of the kunit series now that Brendan has
moved it out of the top-level kunit directory as Linus has requested?

There doesn't appear to have been many comments or changes since since
September 23rd, and I was very much hoping they could land before
-rc2, since I've been hoping to add unit tests for ext4.

Is kunit likely to be able to be landed in Linus's tree during this
development cycle?

Many thanks!

					- Ted
Linus Torvalds Oct. 4, 2019, 9:42 p.m. UTC | #2
On Fri, Oct 4, 2019 at 2:39 PM Theodore Y. Ts'o <tytso@mit.edu> wrote:
>
> This question is primarily directed at Shuah and Linus....
>
> What's the current status of the kunit series now that Brendan has
> moved it out of the top-level kunit directory as Linus has requested?

We seemed to decide to just wait for 5.5, but there is nothing that
looks to block that. And I encouraged Shuah to find more kunit cases
for when it _does_ get merged.

So if the kunit branch is stable, and people want to start using it
for their unit tests, then I think that would be a good idea, and then
during the 5.5 merge window we'll not just get the infrastructure,
we'll get a few more users too and not just examples.

             Linus
Shuah Oct. 4, 2019, 9:59 p.m. UTC | #3
On 10/4/19 3:42 PM, Linus Torvalds wrote:
> On Fri, Oct 4, 2019 at 2:39 PM Theodore Y. Ts'o <tytso@mit.edu> wrote:
>>
>> This question is primarily directed at Shuah and Linus....
>>
>> What's the current status of the kunit series now that Brendan has
>> moved it out of the top-level kunit directory as Linus has requested?
> 

The move happened smack in the middle of merge window and landed in
linux-next towards the end of the merge window.

> We seemed to decide to just wait for 5.5, but there is nothing that
> looks to block that. And I encouraged Shuah to find more kunit cases
> for when it _does_ get merged.
> 

Right. I communicated that to Brendan that we could work on adding more
kunit based tests which would help get more mileage on the kunit.

> So if the kunit branch is stable, and people want to start using it
> for their unit tests, then I think that would be a good idea, and then
> during the 5.5 merge window we'll not just get the infrastructure,
> we'll get a few more users too and not just examples.
> 
thanks,
-- Shuah
Brendan Higgins Oct. 4, 2019, 10:27 p.m. UTC | #4
On Fri, Oct 04, 2019 at 03:59:10PM -0600, shuah wrote:
> On 10/4/19 3:42 PM, Linus Torvalds wrote:
> > On Fri, Oct 4, 2019 at 2:39 PM Theodore Y. Ts'o <tytso@mit.edu> wrote:
> > > 
> > > This question is primarily directed at Shuah and Linus....
> > > 
> > > What's the current status of the kunit series now that Brendan has
> > > moved it out of the top-level kunit directory as Linus has requested?
> > 
> 
> The move happened smack in the middle of merge window and landed in
> linux-next towards the end of the merge window.
> 
> > We seemed to decide to just wait for 5.5, but there is nothing that
> > looks to block that. And I encouraged Shuah to find more kunit cases
> > for when it _does_ get merged.
> > 
> 
> Right. I communicated that to Brendan that we could work on adding more
> kunit based tests which would help get more mileage on the kunit.
> 
> > So if the kunit branch is stable, and people want to start using it
> > for their unit tests, then I think that would be a good idea, and then
> > during the 5.5 merge window we'll not just get the infrastructure,
> > we'll get a few more users too and not just examples.

I was planning on holding off on accepting more tests/changes until
KUnit is in torvalds/master. As much as I would like to go around
promoting it, I don't really want to promote too much complexity in a
non-upstream branch before getting it upstream because I don't want to
risk adding something that might cause it to get rejected again.

To be clear, I can understand from your perspective why getting more
tests/usage before accepting it is a good thing. The more people that
play around with it, the more likely that someone will find an issue
with it, and more likely that what is accepted into torvalds/master is
of high quality.

However, if I encourage arbitrary tests/improvements into my KUnit
branch, it further diverges away from torvalds/master, and is more
likely that there will be a merge conflict or issue that is not related
to the core KUnit changes that will cause the whole thing to be
rejected again in v5.5.

I don't know. I guess we could maybe address that situation by splitting
up the pull request into features and tests when we go to send it in,
but that seems to invite a lot of unnecessary complexity. I actually
already had some other tests/changes ready to send for review, but was
holding off until the initial set of patches mad it in.

Looking forward to hearing other people's thoughts.
Shuah Oct. 4, 2019, 10:47 p.m. UTC | #5
On 10/4/19 4:27 PM, Brendan Higgins wrote:
> On Fri, Oct 04, 2019 at 03:59:10PM -0600, shuah wrote:
>> On 10/4/19 3:42 PM, Linus Torvalds wrote:
>>> On Fri, Oct 4, 2019 at 2:39 PM Theodore Y. Ts'o <tytso@mit.edu> wrote:
>>>>
>>>> This question is primarily directed at Shuah and Linus....
>>>>
>>>> What's the current status of the kunit series now that Brendan has
>>>> moved it out of the top-level kunit directory as Linus has requested?
>>>
>>
>> The move happened smack in the middle of merge window and landed in
>> linux-next towards the end of the merge window.
>>
>>> We seemed to decide to just wait for 5.5, but there is nothing that
>>> looks to block that. And I encouraged Shuah to find more kunit cases
>>> for when it _does_ get merged.
>>>
>>
>> Right. I communicated that to Brendan that we could work on adding more
>> kunit based tests which would help get more mileage on the kunit.
>>
>>> So if the kunit branch is stable, and people want to start using it
>>> for their unit tests, then I think that would be a good idea, and then
>>> during the 5.5 merge window we'll not just get the infrastructure,
>>> we'll get a few more users too and not just examples.
> 
> I was planning on holding off on accepting more tests/changes until
> KUnit is in torvalds/master. As much as I would like to go around
> promoting it, I don't really want to promote too much complexity in a
> non-upstream branch before getting it upstream because I don't want to
> risk adding something that might cause it to get rejected again.
> 
> To be clear, I can understand from your perspective why getting more
> tests/usage before accepting it is a good thing. The more people that
> play around with it, the more likely that someone will find an issue
> with it, and more likely that what is accepted into torvalds/master is
> of high quality.
> 
> However, if I encourage arbitrary tests/improvements into my KUnit
> branch, it further diverges away from torvalds/master, and is more
> likely that there will be a merge conflict or issue that is not related
> to the core KUnit changes that will cause the whole thing to be
> rejected again in v5.5.
> 

The idea is that the new development will happen based on kunit in
linux-kselftest next. It will work just fine. As we accepts patches,
they will go on top of kunit that is in linux-next now.

thanks,
-- Shuah
Brendan Higgins Oct. 4, 2019, 11:10 p.m. UTC | #6
On Fri, Oct 4, 2019 at 3:47 PM shuah <shuah@kernel.org> wrote:
>
> On 10/4/19 4:27 PM, Brendan Higgins wrote:
> > On Fri, Oct 04, 2019 at 03:59:10PM -0600, shuah wrote:
> >> On 10/4/19 3:42 PM, Linus Torvalds wrote:
> >>> On Fri, Oct 4, 2019 at 2:39 PM Theodore Y. Ts'o <tytso@mit.edu> wrote:
> >>>>
> >>>> This question is primarily directed at Shuah and Linus....
> >>>>
> >>>> What's the current status of the kunit series now that Brendan has
> >>>> moved it out of the top-level kunit directory as Linus has requested?
> >>>
> >>
> >> The move happened smack in the middle of merge window and landed in
> >> linux-next towards the end of the merge window.
> >>
> >>> We seemed to decide to just wait for 5.5, but there is nothing that
> >>> looks to block that. And I encouraged Shuah to find more kunit cases
> >>> for when it _does_ get merged.
> >>>
> >>
> >> Right. I communicated that to Brendan that we could work on adding more
> >> kunit based tests which would help get more mileage on the kunit.
> >>
> >>> So if the kunit branch is stable, and people want to start using it
> >>> for their unit tests, then I think that would be a good idea, and then
> >>> during the 5.5 merge window we'll not just get the infrastructure,
> >>> we'll get a few more users too and not just examples.
> >
> > I was planning on holding off on accepting more tests/changes until
> > KUnit is in torvalds/master. As much as I would like to go around
> > promoting it, I don't really want to promote too much complexity in a
> > non-upstream branch before getting it upstream because I don't want to
> > risk adding something that might cause it to get rejected again.
> >
> > To be clear, I can understand from your perspective why getting more
> > tests/usage before accepting it is a good thing. The more people that
> > play around with it, the more likely that someone will find an issue
> > with it, and more likely that what is accepted into torvalds/master is
> > of high quality.
> >
> > However, if I encourage arbitrary tests/improvements into my KUnit
> > branch, it further diverges away from torvalds/master, and is more
> > likely that there will be a merge conflict or issue that is not related
> > to the core KUnit changes that will cause the whole thing to be
> > rejected again in v5.5.
> >
>
> The idea is that the new development will happen based on kunit in
> linux-kselftest next. It will work just fine. As we accepts patches,
> they will go on top of kunit that is in linux-next now.

But then wouldn't we want to limit what KUnit changes are going into
linux-kselftest next for v5.5? For example, we probably don't want to
do anymore feature development on it until it is in v5.5, since the
goal is to make it more stable, right?

I am guessing that it will probably be fine, but it still sounds like
we need to establish some ground rules, and play it *very* safe.
Shuah Oct. 4, 2019, 11:15 p.m. UTC | #7
On 10/4/19 5:10 PM, Brendan Higgins wrote:
> On Fri, Oct 4, 2019 at 3:47 PM shuah <shuah@kernel.org> wrote:
>>
>> On 10/4/19 4:27 PM, Brendan Higgins wrote:
>>> On Fri, Oct 04, 2019 at 03:59:10PM -0600, shuah wrote:
>>>> On 10/4/19 3:42 PM, Linus Torvalds wrote:
>>>>> On Fri, Oct 4, 2019 at 2:39 PM Theodore Y. Ts'o <tytso@mit.edu> wrote:
>>>>>>
>>>>>> This question is primarily directed at Shuah and Linus....
>>>>>>
>>>>>> What's the current status of the kunit series now that Brendan has
>>>>>> moved it out of the top-level kunit directory as Linus has requested?
>>>>>
>>>>
>>>> The move happened smack in the middle of merge window and landed in
>>>> linux-next towards the end of the merge window.
>>>>
>>>>> We seemed to decide to just wait for 5.5, but there is nothing that
>>>>> looks to block that. And I encouraged Shuah to find more kunit cases
>>>>> for when it _does_ get merged.
>>>>>
>>>>
>>>> Right. I communicated that to Brendan that we could work on adding more
>>>> kunit based tests which would help get more mileage on the kunit.
>>>>
>>>>> So if the kunit branch is stable, and people want to start using it
>>>>> for their unit tests, then I think that would be a good idea, and then
>>>>> during the 5.5 merge window we'll not just get the infrastructure,
>>>>> we'll get a few more users too and not just examples.
>>>
>>> I was planning on holding off on accepting more tests/changes until
>>> KUnit is in torvalds/master. As much as I would like to go around
>>> promoting it, I don't really want to promote too much complexity in a
>>> non-upstream branch before getting it upstream because I don't want to
>>> risk adding something that might cause it to get rejected again.
>>>
>>> To be clear, I can understand from your perspective why getting more
>>> tests/usage before accepting it is a good thing. The more people that
>>> play around with it, the more likely that someone will find an issue
>>> with it, and more likely that what is accepted into torvalds/master is
>>> of high quality.
>>>
>>> However, if I encourage arbitrary tests/improvements into my KUnit
>>> branch, it further diverges away from torvalds/master, and is more
>>> likely that there will be a merge conflict or issue that is not related
>>> to the core KUnit changes that will cause the whole thing to be
>>> rejected again in v5.5.
>>>
>>
>> The idea is that the new development will happen based on kunit in
>> linux-kselftest next. It will work just fine. As we accepts patches,
>> they will go on top of kunit that is in linux-next now.
> 
> But then wouldn't we want to limit what KUnit changes are going into
> linux-kselftest next for v5.5? For example, we probably don't want to
> do anymore feature development on it until it is in v5.5, since the
> goal is to make it more stable, right?
> 
> I am guessing that it will probably be fine, but it still sounds like
> we need to establish some ground rules, and play it *very* safe.
> 

How about we identify a small number tests that can add value and focus
on them. I am thinking a number between 2 and 5. This way we get a feel
for the API, if it changes for the better great, if it doesn't have to,
then you know you already did a great job.

Does that sound reasonable to you?

thanks,
-- Shuah
Theodore Ts'o Oct. 4, 2019, 11:29 p.m. UTC | #8
On Fri, Oct 04, 2019 at 04:47:09PM -0600, shuah wrote:
> > However, if I encourage arbitrary tests/improvements into my KUnit
> > branch, it further diverges away from torvalds/master, and is more
> > likely that there will be a merge conflict or issue that is not related
> > to the core KUnit changes that will cause the whole thing to be
> > rejected again in v5.5.
> 
> The idea is that the new development will happen based on kunit in
> linux-kselftest next. It will work just fine. As we accepts patches,
> they will go on top of kunit that is in linux-next now.

I don't see how this would work.  If I add unit tests to ext4, they
would be in fs/ext4.  And to the extent that I need to add test mocks
to allow the unit tests to work, they will involve changes to existing
files in fs/ext4.  I can't put them in the ext4.git tree without
pulling in the kunit changes into the ext4 git tree.  And if they ext4
unit tests land in the kunit tree, it would be a receipe for large
numbers of merge conflicts.

						- Ted
Brendan Higgins Oct. 4, 2019, 11:52 p.m. UTC | #9
On Fri, Oct 4, 2019 at 4:30 PM Theodore Y. Ts'o <tytso@mit.edu> wrote:
>
> On Fri, Oct 04, 2019 at 04:47:09PM -0600, shuah wrote:
> > > However, if I encourage arbitrary tests/improvements into my KUnit
> > > branch, it further diverges away from torvalds/master, and is more
> > > likely that there will be a merge conflict or issue that is not related
> > > to the core KUnit changes that will cause the whole thing to be
> > > rejected again in v5.5.
> >
> > The idea is that the new development will happen based on kunit in
> > linux-kselftest next. It will work just fine. As we accepts patches,
> > they will go on top of kunit that is in linux-next now.
>
> I don't see how this would work.  If I add unit tests to ext4, they
> would be in fs/ext4.  And to the extent that I need to add test mocks
> to allow the unit tests to work, they will involve changes to existing
> files in fs/ext4.  I can't put them in the ext4.git tree without
> pulling in the kunit changes into the ext4 git tree.  And if they ext4
> unit tests land in the kunit tree, it would be a receipe for large
> numbers of merge conflicts.

That's where I was originally coming from.

So here's a dumb idea: what if we merged KUnit through the ext4 tree?
I imagine that could potentially get very confusing when we go back to
sending changes in through the kselftest tree, but at least it means
that ext4 can use it in the meantime, which means that it at least
gets to be useful to one group of people. Also, since Ted seems pretty
keen on using this, I imagine it is much more likely to produce real
world, immediately useful tests not written by me (I'm not being lazy,
I just think it is better to get other people's experiences other than
my own).

Thoughts?
Shuah Oct. 4, 2019, 11:57 p.m. UTC | #10
On 10/4/19 5:52 PM, Brendan Higgins wrote:
> On Fri, Oct 4, 2019 at 4:30 PM Theodore Y. Ts'o <tytso@mit.edu> wrote:
>>
>> On Fri, Oct 04, 2019 at 04:47:09PM -0600, shuah wrote:
>>>> However, if I encourage arbitrary tests/improvements into my KUnit
>>>> branch, it further diverges away from torvalds/master, and is more
>>>> likely that there will be a merge conflict or issue that is not related
>>>> to the core KUnit changes that will cause the whole thing to be
>>>> rejected again in v5.5.
>>>
>>> The idea is that the new development will happen based on kunit in
>>> linux-kselftest next. It will work just fine. As we accepts patches,
>>> they will go on top of kunit that is in linux-next now.
>>
>> I don't see how this would work.  If I add unit tests to ext4, they
>> would be in fs/ext4.  And to the extent that I need to add test mocks
>> to allow the unit tests to work, they will involve changes to existing
>> files in fs/ext4.  I can't put them in the ext4.git tree without
>> pulling in the kunit changes into the ext4 git tree.  And if they ext4
>> unit tests land in the kunit tree, it would be a receipe for large
>> numbers of merge conflicts.
> 
> That's where I was originally coming from.
> 
> So here's a dumb idea: what if we merged KUnit through the ext4 tree?
> I imagine that could potentially get very confusing when we go back to
> sending changes in through the kselftest tree, but at least it means
> that ext4 can use it in the meantime, which means that it at least
> gets to be useful to one group of people. Also, since Ted seems pretty
> keen on using this, I imagine it is much more likely to produce real
> world, immediately useful tests not written by me (I'm not being lazy,
> I just think it is better to get other people's experiences other than
> my own).
> 

That doesn't make sense does it? The tests might not be limited to
fs/ext4. We might have other sub-systems that add tests.

So, we will have to work to make linux-next as the base for new
development and limit the number of tests to where it will be
easier work in this mode for 5.5. We can stage the pull requests
so that kunit lands first followed by tests.

We have a similar situation with kselftest as well. Sub-systems
send tests that depend on their tress separately.

thanks,
-- Shuah
Brendan Higgins Oct. 5, 2019, 12:33 a.m. UTC | #11
On Fri, Oct 4, 2019 at 4:57 PM shuah <shuah@kernel.org> wrote:
>
> On 10/4/19 5:52 PM, Brendan Higgins wrote:
> > On Fri, Oct 4, 2019 at 4:30 PM Theodore Y. Ts'o <tytso@mit.edu> wrote:
> >>
> >> On Fri, Oct 04, 2019 at 04:47:09PM -0600, shuah wrote:
> >>>> However, if I encourage arbitrary tests/improvements into my KUnit
> >>>> branch, it further diverges away from torvalds/master, and is more
> >>>> likely that there will be a merge conflict or issue that is not related
> >>>> to the core KUnit changes that will cause the whole thing to be
> >>>> rejected again in v5.5.
> >>>
> >>> The idea is that the new development will happen based on kunit in
> >>> linux-kselftest next. It will work just fine. As we accepts patches,
> >>> they will go on top of kunit that is in linux-next now.
> >>
> >> I don't see how this would work.  If I add unit tests to ext4, they
> >> would be in fs/ext4.  And to the extent that I need to add test mocks
> >> to allow the unit tests to work, they will involve changes to existing
> >> files in fs/ext4.  I can't put them in the ext4.git tree without
> >> pulling in the kunit changes into the ext4 git tree.  And if they ext4
> >> unit tests land in the kunit tree, it would be a receipe for large
> >> numbers of merge conflicts.
> >
> > That's where I was originally coming from.
> >
> > So here's a dumb idea: what if we merged KUnit through the ext4 tree?
> > I imagine that could potentially get very confusing when we go back to
> > sending changes in through the kselftest tree, but at least it means
> > that ext4 can use it in the meantime, which means that it at least
> > gets to be useful to one group of people. Also, since Ted seems pretty
> > keen on using this, I imagine it is much more likely to produce real
> > world, immediately useful tests not written by me (I'm not being lazy,
> > I just think it is better to get other people's experiences other than
> > my own).
> >
>
> That doesn't make sense does it? The tests might not be limited to
> fs/ext4. We might have other sub-systems that add tests.

Well, I have some small isolated examples that I think would probably
work no matter what, so we can get some usage outside of ext4. Also,
if we want to limit the number of tests, then we don't expect to get
much usage outside of ext4 before v5.5 anyway. I just figure, it's
better to make it work for one person than no one, right?

In any case, I admit it is not a great idea. I just thought it had
some interesting advantages over going in through linux-kselftest that
were worth exploring.

> So, we will have to work to make linux-next as the base for new
> development and limit the number of tests to where it will be
> easier work in this mode for 5.5. We can stage the pull requests
> so that kunit lands first followed by tests.

So we are going to encourage maintainers to allow tests in their tree
based on KUnit on the assumption that KUnit will get merged before
there changes? That sounds like a huge burden, not just on us, but on
other maintainers and users.

I think if we are going to allow tests before KUnit is merged, we
should have the tests come in through the same tree as KUnit.

> We have a similar situation with kselftest as well. Sub-systems
> send tests that depend on their tress separately.

Well it is different if the maintainer wants to send the test in
through their tree, right? Otherwise, it won't matter what the state
of linux-next is and it won't matter when linux-kselftest gets merged.
Or am I not understanding you?
Shuah Oct. 5, 2019, 12:49 a.m. UTC | #12
On 10/4/19 6:33 PM, Brendan Higgins wrote:
> On Fri, Oct 4, 2019 at 4:57 PM shuah <shuah@kernel.org> wrote:
>>
>> On 10/4/19 5:52 PM, Brendan Higgins wrote:
>>> On Fri, Oct 4, 2019 at 4:30 PM Theodore Y. Ts'o <tytso@mit.edu> wrote:
>>>>
>>>> On Fri, Oct 04, 2019 at 04:47:09PM -0600, shuah wrote:
>>>>>> However, if I encourage arbitrary tests/improvements into my KUnit
>>>>>> branch, it further diverges away from torvalds/master, and is more
>>>>>> likely that there will be a merge conflict or issue that is not related
>>>>>> to the core KUnit changes that will cause the whole thing to be
>>>>>> rejected again in v5.5.
>>>>>
>>>>> The idea is that the new development will happen based on kunit in
>>>>> linux-kselftest next. It will work just fine. As we accepts patches,
>>>>> they will go on top of kunit that is in linux-next now.
>>>>
>>>> I don't see how this would work.  If I add unit tests to ext4, they
>>>> would be in fs/ext4.  And to the extent that I need to add test mocks
>>>> to allow the unit tests to work, they will involve changes to existing
>>>> files in fs/ext4.  I can't put them in the ext4.git tree without
>>>> pulling in the kunit changes into the ext4 git tree.  And if they ext4
>>>> unit tests land in the kunit tree, it would be a receipe for large
>>>> numbers of merge conflicts.
>>>
>>> That's where I was originally coming from.
>>>
>>> So here's a dumb idea: what if we merged KUnit through the ext4 tree?
>>> I imagine that could potentially get very confusing when we go back to
>>> sending changes in through the kselftest tree, but at least it means
>>> that ext4 can use it in the meantime, which means that it at least
>>> gets to be useful to one group of people. Also, since Ted seems pretty
>>> keen on using this, I imagine it is much more likely to produce real
>>> world, immediately useful tests not written by me (I'm not being lazy,
>>> I just think it is better to get other people's experiences other than
>>> my own).
>>>
>>
>> That doesn't make sense does it? The tests might not be limited to
>> fs/ext4. We might have other sub-systems that add tests.
> 
> Well, I have some small isolated examples that I think would probably
> work no matter what, so we can get some usage outside of ext4. Also,
> if we want to limit the number of tests, then we don't expect to get
> much usage outside of ext4 before v5.5 anyway. I just figure, it's
> better to make it work for one person than no one, right?
> 
> In any case, I admit it is not a great idea. I just thought it had
> some interesting advantages over going in through linux-kselftest that
> were worth exploring.
> 
>> So, we will have to work to make linux-next as the base for new
>> development and limit the number of tests to where it will be
>> easier work in this mode for 5.5. We can stage the pull requests
>> so that kunit lands first followed by tests.
> 
> So we are going to encourage maintainers to allow tests in their tree
> based on KUnit on the assumption that KUnit will get merged before
> there changes? That sounds like a huge burden, not just on us, but on
> other maintainers and users.
> 
> I think if we are going to allow tests before KUnit is merged, we
> should have the tests come in through the same tree as KUnit.
> 
>> We have a similar situation with kselftest as well. Sub-systems
>> send tests that depend on their tress separately.
> 
> Well it is different if the maintainer wants to send the test in
> through their tree, right? Otherwise, it won't matter what the state
> of linux-next is and it won't matter when linux-kselftest gets merged.
> Or am I not understanding you?
> 

Let's talk about current state. Right now kunit is in linux-next and
we want to add a few more tests. We will have to coordinate the effort.
Once kunit get into mainline, then the need for this coordination goes
down.

Let's focus on the next few weeks first so we can get this into mainline
in 5.5.

The two of us can chat next week and come up with a plan.

thanks,
-- Shuah
Brendan Higgins Oct. 5, 2019, 1:18 a.m. UTC | #13
On Fri, Oct 4, 2019 at 5:49 PM shuah <shuah@kernel.org> wrote:
>
> On 10/4/19 6:33 PM, Brendan Higgins wrote:
> > On Fri, Oct 4, 2019 at 4:57 PM shuah <shuah@kernel.org> wrote:
> >>
> >> On 10/4/19 5:52 PM, Brendan Higgins wrote:
> >>> On Fri, Oct 4, 2019 at 4:30 PM Theodore Y. Ts'o <tytso@mit.edu> wrote:
> >>>>
> >>>> On Fri, Oct 04, 2019 at 04:47:09PM -0600, shuah wrote:
> >>>>>> However, if I encourage arbitrary tests/improvements into my KUnit
> >>>>>> branch, it further diverges away from torvalds/master, and is more
> >>>>>> likely that there will be a merge conflict or issue that is not related
> >>>>>> to the core KUnit changes that will cause the whole thing to be
> >>>>>> rejected again in v5.5.
> >>>>>
> >>>>> The idea is that the new development will happen based on kunit in
> >>>>> linux-kselftest next. It will work just fine. As we accepts patches,
> >>>>> they will go on top of kunit that is in linux-next now.
> >>>>
> >>>> I don't see how this would work.  If I add unit tests to ext4, they
> >>>> would be in fs/ext4.  And to the extent that I need to add test mocks
> >>>> to allow the unit tests to work, they will involve changes to existing
> >>>> files in fs/ext4.  I can't put them in the ext4.git tree without
> >>>> pulling in the kunit changes into the ext4 git tree.  And if they ext4
> >>>> unit tests land in the kunit tree, it would be a receipe for large
> >>>> numbers of merge conflicts.
> >>>
> >>> That's where I was originally coming from.
> >>>
> >>> So here's a dumb idea: what if we merged KUnit through the ext4 tree?
> >>> I imagine that could potentially get very confusing when we go back to
> >>> sending changes in through the kselftest tree, but at least it means
> >>> that ext4 can use it in the meantime, which means that it at least
> >>> gets to be useful to one group of people. Also, since Ted seems pretty
> >>> keen on using this, I imagine it is much more likely to produce real
> >>> world, immediately useful tests not written by me (I'm not being lazy,
> >>> I just think it is better to get other people's experiences other than
> >>> my own).
> >>>
> >>
> >> That doesn't make sense does it? The tests might not be limited to
> >> fs/ext4. We might have other sub-systems that add tests.
> >
> > Well, I have some small isolated examples that I think would probably
> > work no matter what, so we can get some usage outside of ext4. Also,
> > if we want to limit the number of tests, then we don't expect to get
> > much usage outside of ext4 before v5.5 anyway. I just figure, it's
> > better to make it work for one person than no one, right?
> >
> > In any case, I admit it is not a great idea. I just thought it had
> > some interesting advantages over going in through linux-kselftest that
> > were worth exploring.
> >
> >> So, we will have to work to make linux-next as the base for new
> >> development and limit the number of tests to where it will be
> >> easier work in this mode for 5.5. We can stage the pull requests
> >> so that kunit lands first followed by tests.
> >
> > So we are going to encourage maintainers to allow tests in their tree
> > based on KUnit on the assumption that KUnit will get merged before
> > there changes? That sounds like a huge burden, not just on us, but on
> > other maintainers and users.
> >
> > I think if we are going to allow tests before KUnit is merged, we
> > should have the tests come in through the same tree as KUnit.
> >
> >> We have a similar situation with kselftest as well. Sub-systems
> >> send tests that depend on their tress separately.
> >
> > Well it is different if the maintainer wants to send the test in
> > through their tree, right? Otherwise, it won't matter what the state
> > of linux-next is and it won't matter when linux-kselftest gets merged.
> > Or am I not understanding you?
> >
>
> Let's talk about current state. Right now kunit is in linux-next and
> we want to add a few more tests. We will have to coordinate the effort.
> Once kunit get into mainline, then the need for this coordination goes
> down.

Sure, I was just thinking that getting other people to write the tests
would be better. Since not only is it then useful to someone else, it
provides the best possible exercise of KUnit.

Nevertheless, it would probably just be easier to get a handful of
example tests, and it is less likely to result in any issues for v5.5,
so that's probably better. (I think that is what you are hinting at
here. ;-) )

Hey Ted, do you know if that ext4 timestamp test can go in through
linux-kselftest? It seemed fairly self-contained. Or is that what you
were saying wouldn't work for you?

> Let's focus on the next few weeks first so we can get this into mainline
> in 5.5.

I agree. That is the most important thing.

> The two of us can chat next week and come up with a plan.

Sure.

Cheers!
Theodore Ts'o Oct. 6, 2019, 4:54 p.m. UTC | #14
On Fri, Oct 04, 2019 at 06:18:04PM -0700, Brendan Higgins wrote:
> > Let's talk about current state. Right now kunit is in linux-next and
> > we want to add a few more tests. We will have to coordinate the effort.
> > Once kunit get into mainline, then the need for this coordination goes
> > down.
> 
> Sure, I was just thinking that getting other people to write the tests
> would be better. Since not only is it then useful to someone else, it
> provides the best possible exercise of KUnit.

Well, one thing we *can* do is if (a) if we can create a kselftest
branch which we know is stable and won't change, and (b) we can get
assurances that Linus *will* accept that branch during the next merge
window, those subsystems which want to use kself test can simply pull
it into their tree.

We've done this before in the file system world, when there has been
some common set of changes needed to improve, say, Direct I/O, where
the changes are put into a standalone branch, say, in the xfs tree,
and those file systems which need it as a building block can pull it
into their tree, and then add the changes needed to use those changes
into their file system git tree.  These changes are generally not
terribly controversial, and we've not had to worry about people want
to bikeshed the changes.

There is a risk with doing this of course, which is that if the branch
*is* controversial, or gets bike-shedded for some reason, then Linus
gets upset and any branches which depended on said branch will get
rejected at the next merge window.  Which is the requirement for (a)
and (b) above.  Presumably, the fact that people were unwilling to let
Kunit land during this merge window might will *because* we think more
changes might be pending?

The other thing I suppose I can do is to let the ext4 kunit tests land
in ext4 tree, but with the necessary #ifdef's around things like
"#include <kunit/test.h>" so that the build won't blow up w/o kunit
changes being in the tree that I'm building.  It means I won't be able
to run the tests without creating a test integration branch and
merging in kunit by hand, which will super-annoying, of course.  And
if some of the bike-shedding is in Kunit's interfaces, then that
becomes problematic as well, since any tests that are in ext4.git tree
might change if people want to rename Kunit's publically exported
functions (for example).

> Hey Ted, do you know if that ext4 timestamp test can go in through
> linux-kselftest? It seemed fairly self-contained. Or is that what you
> were saying wouldn't work for you?

Well, I was hoping that we might start creating more tests beyond just
the ext4 timestamp tests....

						- Ted
Linus Torvalds Oct. 6, 2019, 5:18 p.m. UTC | #15
On Sun, Oct 6, 2019 at 9:55 AM Theodore Y. Ts'o <tytso@mit.edu> wrote:
>
> Well, one thing we *can* do is if (a) if we can create a kselftest
> branch which we know is stable and won't change, and (b) we can get
> assurances that Linus *will* accept that branch during the next merge
> window, those subsystems which want to use kself test can simply pull
> it into their tree.

Yes.

At the same time, I don't think it needs to be even that fancy. Even
if it's not a stable branch that gets shared between different
developers, it would be good to just have people do a "let's try this"
throw-away branch to use the kunit functionality and verify that
"yeah, this is fairly convenient for ext4".

It doesn't have to be merged in that form, but just confirmation that
the infrastructure is helpful before it gets merged would be good.

               Linus
Brendan Higgins Oct. 7, 2019, 8:20 a.m. UTC | #16
On Sun, Oct 6, 2019 at 9:55 AM Theodore Y. Ts'o <tytso@mit.edu> wrote:
>
> On Fri, Oct 04, 2019 at 06:18:04PM -0700, Brendan Higgins wrote:
> > > Let's talk about current state. Right now kunit is in linux-next and
> > > we want to add a few more tests. We will have to coordinate the effort.
> > > Once kunit get into mainline, then the need for this coordination goes
> > > down.
> >
> > Sure, I was just thinking that getting other people to write the tests
> > would be better. Since not only is it then useful to someone else, it
> > provides the best possible exercise of KUnit.
>
> Well, one thing we *can* do is if (a) if we can create a kselftest
> branch which we know is stable and won't change, and (b) we can get
> assurances that Linus *will* accept that branch during the next merge
> window, those subsystems which want to use kself test can simply pull
> it into their tree.

Yeah, I can't think of any reason that you haven't outlined already
why that might not work, but that seems kind of like circumventing
Linus.

> We've done this before in the file system world, when there has been
> some common set of changes needed to improve, say, Direct I/O, where
> the changes are put into a standalone branch, say, in the xfs tree,
> and those file systems which need it as a building block can pull it
> into their tree, and then add the changes needed to use those changes
> into their file system git tree.  These changes are generally not
> terribly controversial, and we've not had to worry about people want
> to bikeshed the changes.
>
> There is a risk with doing this of course, which is that if the branch
> *is* controversial, or gets bike-shedded for some reason, then Linus
> gets upset and any branches which depended on said branch will get
> rejected at the next merge window.  Which is the requirement for (a)
> and (b) above.  Presumably, the fact that people were unwilling to let
> Kunit land during this merge window might will *because* we think more
> changes might be pending?

My understanding, based on what I have been told, is that we were
simply unlucky with the timing when Linus pulled the branch in the
first week of the 5.4 merge window (Friday), such that once I fixed
the directory naming issue, the updated changes didn't spend enough
time in linux-next. And now with this issue fixed and KUnit back in
linux-next, if nothing interesting happens between now and 5.5, it
will be accepted in the 5.5 merge window. I do not think that anyone
is expecting anymore changes before merging.

Shuah, Linus, is my understanding correct?

> The other thing I suppose I can do is to let the ext4 kunit tests land
> in ext4 tree, but with the necessary #ifdef's around things like
> "#include <kunit/test.h>" so that the build won't blow up w/o kunit
> changes being in the tree that I'm building.  It means I won't be able
> to run the tests without creating a test integration branch and
> merging in kunit by hand, which will super-annoying, of course.  And
> if some of the bike-shedding is in Kunit's interfaces, then that
> becomes problematic as well, since any tests that are in ext4.git tree
> might change if people want to rename Kunit's publically exported
> functions (for example).

Yeah, that seems even worse. I'm sorry to have caused this frustration.

> > Hey Ted, do you know if that ext4 timestamp test can go in through
> > linux-kselftest? It seemed fairly self-contained. Or is that what you
> > were saying wouldn't work for you?
>
> Well, I was hoping that we might start creating more tests beyond just
> the ext4 timestamp tests....

Okay, that's what I thought (and what I hoped) you were saying :-)

I hope we can figure out something that will work for you. Or
otherwise that you won't mind waiting until 5.5.

Sorry
Brendan Higgins Oct. 7, 2019, 8:40 a.m. UTC | #17
On Sun, Oct 6, 2019 at 10:18 AM Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> On Sun, Oct 6, 2019 at 9:55 AM Theodore Y. Ts'o <tytso@mit.edu> wrote:
> >
> > Well, one thing we *can* do is if (a) if we can create a kselftest
> > branch which we know is stable and won't change, and (b) we can get
> > assurances that Linus *will* accept that branch during the next merge
> > window, those subsystems which want to use kself test can simply pull
> > it into their tree.
>
> Yes.
>
> At the same time, I don't think it needs to be even that fancy. Even
> if it's not a stable branch that gets shared between different
> developers, it would be good to just have people do a "let's try this"
> throw-away branch to use the kunit functionality and verify that
> "yeah, this is fairly convenient for ext4".
>
> It doesn't have to be merged in that form, but just confirmation that
> the infrastructure is helpful before it gets merged would be good.

I thought we already had done this satisfactorily.

We have one proof-of-concept test in the branch in the kselftest repo
(proc sysctl test) that went out in the pull request, and we also had
some other tests that were not in the pull request (there is the ext4
timestamp stuff mentioned above, and we also had one against the list
data structure), which we were planning on sending out for review once
Shuah's pull request was accepted. I know the apparmor people also
wrote some tests that they said were useful; however, I have not
coordinated with them on upstreaming their tests. I know of some other
people who are using it, but I don't think the tests are as far along
for upstreaming.

The point is: I thought we had plenty of signal that KUnit would be
useful to have merged into the mainline kernel. I thought the only
reason it was rejected for 5.4 was due to the directory name issue
combined with bad timing.

Please correct me if I missed anything.

Thanks!
Steven Rostedt Oct. 7, 2019, 2:40 p.m. UTC | #18
On Sun, 6 Oct 2019 10:18:11 -0700
Linus Torvalds <torvalds@linux-foundation.org> wrote:

> On Sun, Oct 6, 2019 at 9:55 AM Theodore Y. Ts'o <tytso@mit.edu> wrote:
> >
> > Well, one thing we *can* do is if (a) if we can create a kselftest
> > branch which we know is stable and won't change, and (b) we can get
> > assurances that Linus *will* accept that branch during the next merge
> > window, those subsystems which want to use kself test can simply pull
> > it into their tree.  
> 
> Yes.
> 
> At the same time, I don't think it needs to be even that fancy. Even
> if it's not a stable branch that gets shared between different
> developers, it would be good to just have people do a "let's try this"
> throw-away branch to use the kunit functionality and verify that
> "yeah, this is fairly convenient for ext4".
> 
> It doesn't have to be merged in that form, but just confirmation that
> the infrastructure is helpful before it gets merged would be good.

Can't you just create an ext4 branch that has the kselftest-next branch
in it, that you build upon. And push that after the kunit test is
merged?

In the past I've had to rely on other branches in next, and would just
hold two branches myself. One with everything not dependent on the other
developer's branch, and one with the work that was. At the merge
window, I would either merge the two or just send two pull requests
with the two branches.

-- Steve
Shuah Oct. 7, 2019, 2:42 p.m. UTC | #19
On 10/7/19 2:40 AM, Brendan Higgins wrote:
> On Sun, Oct 6, 2019 at 10:18 AM Linus Torvalds
> <torvalds@linux-foundation.org> wrote:
>>
>> On Sun, Oct 6, 2019 at 9:55 AM Theodore Y. Ts'o <tytso@mit.edu> wrote:
>>>
>>> Well, one thing we *can* do is if (a) if we can create a kselftest
>>> branch which we know is stable and won't change, and (b) we can get
>>> assurances that Linus *will* accept that branch during the next merge
>>> window, those subsystems which want to use kself test can simply pull
>>> it into their tree.
>>
>> Yes.
>>
>> At the same time, I don't think it needs to be even that fancy. Even
>> if it's not a stable branch that gets shared between different
>> developers, it would be good to just have people do a "let's try this"
>> throw-away branch to use the kunit functionality and verify that
>> "yeah, this is fairly convenient for ext4".
>>
>> It doesn't have to be merged in that form, but just confirmation that
>> the infrastructure is helpful before it gets merged would be good.
> 
> I thought we already had done this satisfactorily.
> 
Adding a couple more tests will only help in the long run. The idea is
to see can this help

> We have one proof-of-concept test in the branch in the kselftest repo
> (proc sysctl test) that went out in the pull request, and we also had
> some other tests that were not in the pull request (there is the ext4
> timestamp stuff mentioned above, and we also had one against the list
> data structure), which we were planning on sending out for review once
> Shuah's pull request was accepted. I know the apparmor people also
> wrote some tests that they said were useful; however, I have not
> coordinated with them on upstreaming their tests. I know of some other
> people who are using it, but I don't think the tests are as far along
> for upstreaming.
> 

Maybe that is a good start. To get the tests that are already in use
and get them in shape for upstream.

> The point is: I thought we had plenty of signal that KUnit would be
> useful to have merged into the mainline kernel. I thought the only
> reason it was rejected for 5.4 was due to the directory name issue
> combined with bad timing.
> 

That is probably the initial thought. However, it makes perfect sense
to add a couple of tests in. We have a few weeks anyway and it gives
us more confidence on kunit.

I already have a branch that is in linux-next and it just has kunit in
it and I will rebase it to 5.4-rc1.

https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git/log/?h=test

Let's use that for kunit work for 5.5. I won't add any kselftest patches
to it and keep it dedicated for kunit work. When tests are ready for
upstream, I can keep adding them to this branch.

thanks,
-- Shuah
Shuah Oct. 7, 2019, 2:57 p.m. UTC | #20
On 10/7/19 8:40 AM, Steven Rostedt wrote:
> On Sun, 6 Oct 2019 10:18:11 -0700
> Linus Torvalds <torvalds@linux-foundation.org> wrote:
> 
>> On Sun, Oct 6, 2019 at 9:55 AM Theodore Y. Ts'o <tytso@mit.edu> wrote:
>>>
>>> Well, one thing we *can* do is if (a) if we can create a kselftest
>>> branch which we know is stable and won't change, and (b) we can get
>>> assurances that Linus *will* accept that branch during the next merge
>>> window, those subsystems which want to use kself test can simply pull
>>> it into their tree.
>>
>> Yes.
>>
>> At the same time, I don't think it needs to be even that fancy. Even
>> if it's not a stable branch that gets shared between different
>> developers, it would be good to just have people do a "let's try this"
>> throw-away branch to use the kunit functionality and verify that
>> "yeah, this is fairly convenient for ext4".
>>
>> It doesn't have to be merged in that form, but just confirmation that
>> the infrastructure is helpful before it gets merged would be good.
> 
> Can't you just create an ext4 branch that has the kselftest-next branch
> in it, that you build upon. And push that after the kunit test is
> merged?
> 
> In the past I've had to rely on other branches in next, and would just
> hold two branches myself. One with everything not dependent on the other
> developer's branch, and one with the work that was. At the merge
> window, I would either merge the two or just send two pull requests
> with the two branches.
> 

I do something similar when I am working on top of a branch that isn't
already in the mainline. In any case, repeating myself

Let's work on top of - it is rebased to 5.4-rc1 and ready for use.

https://git.kernel.org/pub/scm/linux/kernel/git/shuah/linux-kselftest.git/log/?h=test

Let's use that for kunit work for 5.5. I won't add any kselftest patches
to it and keep it dedicated for kunit work. When tests are ready for
upstream, I can keep adding them to this branch.

thanks,
-- Shuah